Mystique Unicorn App backend is hosted on mongodb. Recenly one of their devs discovered that AWS released Amazon DocumentDB (with MongoDB compatibility) a fast, scalable, highly available, and fully managed document database service that supports MongoDB workloads.
Can you help them migrate from mongodb to documentdb?
We will follow an multi-stage process to accomplish our goal. We need the following components to get this right,
- Source Database - MongoDB
- If in AWS: EC2 instance in a VPC, Security Group, SSH Keypair(Optional)
- Some dummy data inside the database
- Destination Database - DocumentDB
- Subnet Groups
- VPC Security Groups
- Database Migration Service(DMS) - Replication Instance
- DMS IAM Roles
- Endpoints
- Database Migration Tasks
In this article, we will build an architecture, similar to the one shown above - A simple API using API Gateway which will trigger a Lambda function. We will have an stageVariable lambdaAlias
and lets assume it is going to be an prod
environment. The lambda will have multiple alias point at different stage of development. prod
pointing to the most stable version and dev
pointing to the bleeding edlge version.
In this Workshop you will practice how to migrate your MongoDB databases to Amazon DocumentDB using different strategies.
-
This demo, instructions, scripts and cloudformation template is designed to be run in
us-east-1
. With few modifications you can try it out in other regions as well(Not covered here).- ๐ AWS CLI Installed & Configured - Get help here
- ๐ AWS CDK Installed & Configured - Get help here
- ๐ Python Packages, Change the below commands to suit your OS, the following is written for amzn linux 2
- Python3 -
yum install -y python3
- Python Pip -
yum install -y python-pip
- Virtualenv -
pip3 install virtualenv
- Python3 -
As there are a number of components that need to be setup, we will use a combination of Cloudformation(generated from CDK), CLI & GUI.
-
-
Get the application code
git clone https://github.com/miztiik/dms-mongodb-to-documentdb cd dms-mongodb-to-documentdb
-
-
We will need cdk to be installed to make our deployments easier. Lets go ahead and install the necessary components.
# If you DONT have cdk installed npm install -g aws-cdk # Make sure you in root directory python3 -m venv .env source .env/bin/activate pip3 install -r requirements.txt
The very first time you deploy an AWS CDK app into an environment (account/region), youโll need to install a
bootstrap stack
, Otherwise just go ahead and deploy usingcdk deploy
.cdk bootstrap cdk ls # Follow on screen prompts
You should see an output of the available stacks,
vpc-stack dms-prerequisite-stack mongodb-on-ec2
-
Let us walk through each of the stacks,
-
Stack: vpc-stack This stack will do the following,
- Create an custom VPC
miztiikVpc
(We will use this VPC to host our source MongoDB, DocumentDB, DMS Replication Instance)
Initiate the deployment with the following command,
cdk deploy vpc-stack
- Create an custom VPC
-
Stack: dms-prerequisite-stack This stack will do the following,
- DocumentDB & DMS Security groups - (created during the prerequisite stack)
- Port -
27017
Accessible only from within the VPC
- Port -
- DMS IAM Roles - (This stack will FAIL, If these roles already exist in your account)
AmazonDMSVPCManagementRole
AmazonDMSCloudWatchLogsRole
- SSH KeyPair using a custom cfn resource
- This resource is currently not used. The intial idea was to use the SSH Keypair to administer the source mongodb on EC2. SSM Session Manager does the same job admirably.
Initiate the deployment with the following command,
cdk deploy dms-prerequisite-stack
After successful completion, take a look at all the resources and get yourself familiar with them. We will be using them in the future.
- DocumentDB & DMS Security groups - (created during the prerequisite stack)
-
Stack: Source Database - MongoDB This stack will do the following,
- Create an EC2 instance inside our custom VPC(created during the prerequisite stack)
- Attach security group with mongo port(
27017
) open to the world (For any use-case other than sandbox testing, you might want to restrict it) - Instance IAM Role is configured to allow SSM Session Manager connections(No more SSH key pairs)
- Instance is bootstrapped using
user_data
script to installMongodb 4.x
- Create user
mongodbadmin
& password (We will need this later for inserts and DMS) - Creates a table
miztiik_db
(_Later we will add a collectioncustomers
)
Initiate the deployment with the following command,
cdk deploy mongodb-on-ec2
As our database is a fresh installation, it does not have any data in it. We need some data to migrate. This git repo also includes a
insert_records_to_mongodb.py
that will help us to generate some dummy data and insert them to the database. After successful launch of the stack,-
Connect to the EC2 instance using SSM Session Manager - Get help here
-
Switch to privileged user using
sudo su
-
Navigate to
/var/log
-
Run the following commands
git clone https://github.com/miztiik/dms-mongodb-to-documentdb cd dms-mongodb-to-documentdb/dms_mongodb_to_documentdb/stacks/back_end/bootstrap_scripts python3 insert_records_to_mongodb
-
You should be able to see some
id
printed out and a summary at the end, Expected Output,{"no_of_records_inserted":114} {"total_coll_count":343}
If you want to interact with mongodb, you can try out the following commands,
# Open Mongo shell mongo # List all Database show dbs # Use one of the datbases use miztiik_db db.stats() # List all collections show collections # List some documents in the customer collection db.customers.find() # List indexes db.customers.getIndexes() # Quit quit()
Now we are all done with our source database.
-
-
We can automate the creation of DocumentDB & DMS using CDK, But since this will be the first time we use these services,let us use the Console/GUI to set them up. We can leverage the excellant documentation from AWS on how to setup our DocumentDB. You should only need to do Step 3 - Create an Amazon DocumentDB cluster(Use your own judgement, as docs tend to change over a period of time)
Couple of things to note,
- For VPC - Use our custom VPC
miztiikVpc
- For Security Group - Use
docsdb_sg_dms-prerequisite-stack
Download the public key for Amazon DocumentDB. We will need this to connect to DocumentDB Cluster from your machine and also from DMS Replication Instance
wget https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
- For VPC - Use our custom VPC
-
We can leverage the excellant documentation from AWS on how to setup our DMS Replication Instance.
Couple of things to note,
- For VPC - Use our custom VPC
miztiikVpc
- For Security Group - Use
dms_sg_dms-prerequisite-stack
After creating the replication instance, We need to create few more resources to begin our replication. We will use defaults mostly
- Endpoints for source MongoDB(custom values listed below)
- Source choose mongodb
- For server address se the private dns of the ec2 instance
- Auth Mode should be
password
- Update user as
mongodbadmin
, the passwordSom3thingSh0uldBe1nVault
- Authentication source as
admin
- Database name
miztiik_db
- For endpoing specific attributes, choose the DMS Replication instance we create in the previous step
- Endpoint for destination databases - DocumentDB(custom values listed below)
- Choose
docsdb
as target - For server name use the dnsname from docsdb, here is my example,
docsdb.cluster-konstone.us-weast-2.docdb.amazonaws.com
- Ensure you choose SSL verification
verify-full
and upload CA certificate for the Amazon DocumentDB public key we downloaded earlier - Database name
miztiik_db
- For endpoing specific attributes, choose the DMS Replication instance we create in the previous step
- Choose
- Database Migration Task
- Choose our replication instance, source & destination endpoints
- For Migration Type, choose
Migrate Existing Data
- For Table Mappings, Add new selection rule, you can create a custom schema name and leave
%
for the table name and ActionInclude
- Create Task
- For VPC - Use our custom VPC
-
Navigate to DMS task, under
Table Statistics
You should be able observe that the dms has copied the data from mongodb to documentdb. You can connect to documentdb and test the records using the same commands that we used with mongodb earlier.Additional Learnings: You can check the logs in cloudwatch for more information or increase the logging level of the database migration task.
-
Here we have demonstrated how to use Amazon Database Migration Service(DMS) to migrate data from MongoDB to DocumentDB.
-
We have shown how to migrate existing data using DMS. It is possible to use DMS to replicate changes(Change Data Capture - CDC). For this, you need to setup your mongodb(on EC2/OnPrem) as a replica set and set the migration type of database migration task to
Migrate existing data and replication ongoing changes
-
If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well
- Resources created during Deploying The Application
- Delete CloudWatch Lambda LogGroups
- Any other custom resources, you have created for this demo
# Delete from cdk cdk destroy # Follow any on-screen prompts # Delete the CF Stack, If you used cloudformation to deploy the stack. aws cloudformation delete-stack \ --stack-name "MiztiikAutomationStack" \ --region "${AWS_REGION}"
This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.
This repository aims to teach api best practices to new developers, Solution Architects & Ops Engineers in AWS. Based on that knowledge these Udemy course #1, course #2 helps you build complete architecture in AWS.
Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here
Buy me a coffee โ.
Level: 300