Cloud Launcher for MongoDB in the Google Compute Engine

In this post you will learn how to use Google’s Cloud Launcher to set up instances for a MongoDB replica set in the Google Compute Engine.

Replication in MongoDB

A minimal MongoDB replica set consists of two data bearing nodes and one so-called arbiter that only takes part in the election of a new primary in case of failure.

Unlike other distributed databases, MongoDB does not offer auto-discovery of cluster nodes. If all nodes are up, you have to initialize the cluster via an administrative command inside the Mongo CLI that takes list of all replica set members. This fact makes it hard (but not impossible) to script this with the cloud provisioning tool of your choice.

Cloud Launcher for MongoDB

With the help of Google’s Cloud Launcher for MongoDB, the provisioning of a replica set is done in just a few steps.

First, we define a deployment name (that will prefix the names of the instances), a zone and the name of the replica set.

Then, we set up the instances that will run in the Compute Engine. We define a minimal replica set with two server instances and one arbiter. All other parameters will use the defaults for now (the server instances will use a n1-highmem-4 machine type).

This will finally lead to three running instances.

Compute Engine Instances

The instances will show up in the Compute Engine dashboard where they can be managed:

If you prefer your CLI, you can list the instances with the gcloud tool:

$ gcloud compute instances list
NAME                   ZONE            MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
mongo-0-arbiters-vm-0  europe-west3-b  n1-standard-1               10.156.0.2   35.198.139.198  RUNNING
mongo-0-servers-vm-0   europe-west3-b  n1-highmem-4                10.156.0.4   35.198.158.94   RUNNING
mongo-0-servers-vm-1   europe-west3-b  n1-highmem-4                10.156.0.3   35.198.189.164  RUNNING

Replica Set Status

In order to check if everything is up and running, we open a SSH window to one of the instances from the dasboard and start the mongo CLI:

After connecting to the MongoDB, we run the rs.status() to get the status of the replica set:

rs0:ARBITER>rs.status()
{
  "set" : "rs0",
  "date" : ISODate("2018-02-12T12:52:47.562Z"),
  "myState" : 7,
  ...
  },
  "members" : [
      {
      "_id" : 0,
      "name" : "mongo-0-servers-vm-0:27017",
      "health" : 1,
      "state" : 1,
      "stateStr" : "PRIMARY",
      "uptime" : 787,
      ...
      },
      {
      "_id" : 1,
      "name" : "mongo-0-servers-vm-1:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 787,
      ...
      },
      {
        "_id" : 2,
        "name" : "mongo-0-arbiters-vm-0:27017",
        "health" : 1,
        "state" : 7,
        "stateStr" : "ARBITER",
        "uptime" : 801,
        "configVersion" : 3,
        "self" : true
      }
  ],
  "ok" : 1
}

(I ommitted some of the output, so we can focus on the important things.)

We can see the name of the replica set rs0 and our three instances

• mongo-0-servers-vm-0
• mongo-0-servers-vm-1
• mongo-0-arbiters-vm-0

Restore Data Dump from Google Storage

We’ll restore a data dump from a file located in the Google Storage. Since only the the primary node can do write operations, we connect to the instance mongo-0-servers-vm-0 that has to role PRIMARY right now.

Inside that instance, we download a BSON dump from the storage by using the gsutil tool, assuming your storage bucket is called [BUCKET_ID]:

mongo-0-servers-vm-1:/tmp$ cd /tmp
mongo-0-servers-vm-1:/tmp$ gsutil cp gs://[BUCKET_ID].appspot.com/pois.* ./
Copying gs://[BUCKET_ID].appspot.com/pois.bson...
Copying gs://[BUCKET_ID].appspot.com/pois.metadata.json...             
/ [2 files][  5.6 KiB/  5.6 KiB]                                                
Operation completed over 2 objects/5.6 KiB.

Now the dump is on our disk and we restore its data to the MongoDB replica set:

mongo-0-servers-vm-1:/tmp$ mongorestore pois.bson
2018-02-12T13:12:33.402+0000    checking for collection data in pois.bson
2018-02-12T13:12:33.402+0000    reading metadata for test.pois from pois.metadata.json
2018-02-12T13:12:33.418+0000    restoring test.pois from pois.bson
2018-02-12T13:12:33.481+0000    restoring indexes for collection test.pois from metadata
2018-02-12T13:12:33.590+0000    finished restoring test.pois (7 documents)
2018-02-12T13:12:33.590+0000    done

Finally, we perfom a simple query from the OS command line:

mongo-0-servers-vm-0:/tmp$ mongo --quiet --eval "db.pois.count()" test
7

to check if our 7 imported documents are really there.

Summary

We created a MongoDB replica set and deployed it to the Compute Engine. Then we checked the replica set status and imported some data.

In one of my next posts I will show you how to build a REST-ful microservice in the AppEngine that will access our MongoDB backend in the Compute Engine.