Prerequisites

• The MongoDB version is 4.0.7 to 8.0.

• Your MongoDB deployment runs as a replica set or sharded cluster. Standalone instances are not supported.

• A database user with the permissions to read your MongoDB databases is available.

• Hevo’s IP address(es) for your region are allowlisted to reach your MongoDB host through your firewall or security group.

• The hostname or IP address and port of your MongoDB server are available.

Set up MongoDB Replication for Change Streams

Note: Hevo supports data ingestion from MongoDB only through Change Streams. Change Streams are built on the MongoDB oplog, so your deployment must run as a replica set, and the oplog must be large enough to retain changes between syncs.

To set up replication for Change Streams, perform the following steps:

1. Modify the MongoDB server configuration: The MongoDB configuration file, mongod.conf, is generally found in the /etc/ directory in a Linux system. Set the following options:

   • replication.replSetName: The name of the replica set. Every member of the replica set must use the same name.

   • replication.oplogSizeMB: The maximum size of the oplog. We recommend setting it to retain at least 72 hours of changes.

   • net.bindIp: The IP addresses on which this MongoDB server listens, as a comma-separated list.

     Example configuration:

     net:
       bindIp: 127.0.0.1,10.0.0.5
     replication:
       replSetName: "repSet0"
       oplogSizeMB: 2048
     

     Note:

     • Do not set bindIp to 0.0.0.0 on a host that’s reachable from the internet. Bind only to the interfaces that Hevo or your SSH host connects through.

     • The oplogSizeMB configuration takes effect only when the oplog is first created. To resize the oplog of an existing replica set, read Change the Size of the Oplog.

2. Restart MongoDB: Restart mongod for the changes to take effect.

   sudo systemctl restart mongod
   

3. Initiate the replica set: Using mongosh, connect to the node you want to make primary, and run the following commands:

   • rs.initiate(): Initializes the replica set. Run this command only once on one member if the replica set is not already initialized. Running it again returns an error.

   • rs.conf(): Displays the replica set configuration. Verify that the _id matches your replSetName.

   • rs.status(): Displays the replica set status. Verify that one member is PRIMARY and the remaining members are SECONDARY.

Set up Permissions to Read your MongoDB Databases

After configuring replication for the Change Streams, grant the following roles to the database user so Hevo can read the databases you want to replicate:

• readAnyDatabase on the admin database, or read on each database you want to replicate: To read your collections and their changes.

• clusterMonitor on the admin database: To check your deployment’s configuration and health, such as its replica set status and shards.

To create the user or grant the roles, perform the following steps:

1. Using mongosh, connect to your replica set or to a mongos router in your sharded cluster as an admin user.

2. Do one of the following:

   Note: Replace the placeholder values in the commands below with your own. For example, replace <database_username> with hevouser, <password> with a strong password, and <database_name> with sales.

   Create a new user

   • Run the following commands to create a user with access to all databases:

     use admin
     db.createUser({
       user: "<database_username>",
       pwd: "<password>",
       roles: [ "readAnyDatabase", "clusterMonitor" ]
     })
     

   • Run the following commands to create a user with access to specific databases:

     use admin
     db.createUser({
       user: "<database_username>",
       pwd: "<password>",
       roles: [
         { role: "read", db: "<database_name_1>" },
         { role: "read", db: "<database_name_2>" },
         "clusterMonitor"
       ]
     })
     

   Grant roles to an existing user

   • Run the following commands to grant access to all databases to an existing user:

     use admin
     db.grantRolesToUser(
       "<database_username>",
       [ "readAnyDatabase", "clusterMonitor" ]
     )
     

   • Run the following commands to grant access to specific databases to an existing user:

     use admin
     db.grantRolesToUser(
       "<database_username>",
       [
         { role: "read", db: "<database_name_1>" },
         { role: "read", db: "<database_name_2>" },
         "clusterMonitor"
       ]
     )
     

Allowlist Hevo IP Addresses for your Region

You must add Hevo’s IP address for your region to the inbound rules of the firewall or security group that protects your MongoDB host, on the MongoDB port (default 27017). For more information, read Network and Configuration Hardening for Self-Managed Deployments.