Monitoring MongoDB 3 using StackDriver in GCE

Question

Has anyone successfully managed to monitor a MongoDB 3 cluster (or standalone db) with StackDriver in GCE ?

I've setup a MongoDB 3.0.6 cluster in GCE (replica set with 2 replica and 1 arbiter)

I'm trying to monitor it through StackDriver provided by Google.

I've followed all instructions to install monitoring agent and mongodb plugin found here : https://cloud.google.com/monitoring/agent/plugins/mongodb

When I start the agent on the replica where it is configured :

sudo /etc/init.d/stackdriver-agent restart

I get the following errors in /var/log/syslog :

collectd[6013]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.
collectd[6013]: mongo plugin: Authenticating to localhost:27017 failed:
collectd[6013]: mongo plugin: Connecting to localhost:27017 failed:
collectd[6013]: read-function of plugin `mongodb' failed. Will suspend it for 120.000 seconds.

I suspect the StackDriver agent is not compatible with MongoDB 3, because :

• In the past, I've used the Click-to-deploy feature of GCE to create a cluster and was able to monitor it with StackDriver. At that time, it was with MongoDB 2.6.x.

• I've quickly setup again a standalone installation of MongoDB 2.6.x, configured the StackDriver agent in the same way, and ... it works :-(

Any help will be really appreciated.

Details of configuration :

Mongodb :

• auth=true

• in database admin, user with roles : dbAdminAnyDatabase, clusterAdmin and readAnyDatabase

Stackdriver Mongodb plugin :

• this user and password used in /opt/stackdriver/collectd/etc/collectd.d/mongodb.conf

Additional information :

No Auth : the plugin initializes successfully

/etc/mongod.conf :

# Turn on/off security.  Off is currently the default
#noauth = true
#auth = true

/opt/stackdriver/collectd/etc/collectd.d/mongodb.conf :

LoadPlugin mongodb

<Plugin "mongodb">
    Host "localhost"
    Port "27017"

   #  If you restricted access to the database, you can
   #  set the username and password here
   #  User "user_name"
   #  Password "user_password"

   # For performance/eventually consistent trade-offs you may add this line
   # PreferSecondaryQuery true
</Plugin>

Auth mode : authentication errors when restarting the agent

/etc/mongod.conf :

# Turn on/off security.  Off is currently the default
#noauth = true
auth = true

3 users configured

use admin
db.createUser(
  {
    user: "siteUserAdmin",
    pwd: "xxx",
    roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
  }
)

db.auth("siteUserAdmin", "xxx");

db.createUser( {
    user: "siteRootAdmin",
    pwd: "xxx",
    roles: [ { role: "root", db: "admin" } ]
  });

db.createUser(
  {
    user: "monitoring",
    pwd: "xxx",
    roles: [ 
    { role: "dbAdminAnyDatabase", db: "admin" },
    { role: "clusterAdmin", db: "admin" },
    { role: "readAnyDatabase", db: "admin" } ]
  }
)

/opt/stackdriver/collectd/etc/collectd.d/mongodb.conf :

LoadPlugin mongodb

<Plugin "mongodb">
    Host "localhost"
    Port "27017"

   #  If you restricted access to the database, you can
   #  set the username and password here
   User "monitoring"
   Password "xxx"

   # For performance/eventually consistent trade-offs you may add this line
   # PreferSecondaryQuery true
</Plugin>

Same errors when I use siteRootAdmin in the plugin configuration.

Explanation and solution

The culprit was actually the auth schema used by StackDriver agent

I adapted the solution suggested by Adam C, because I already have users created with the SCRAM-SHA-1 schema.

In fact, I only need that the user for the monitoring uses MONGODB-CR.

To do so :

Restart MongoDB 3.0 with auth disabled

Connect to the instance

Change temporarily the auth schema to MONGODB-CR

use admin
var schema = db.system.version.findOne({"_id" : "authSchema"});
schema.currentVersion = 3;
db.system.version.save(schema);

Create the user for the StackDriver plugin

db.createUser(
      {
        user: "monitoring",
        pwd: "xxx",
        roles: [ 
        { role: "dbAdminAnyDatabase", db: "admin" },
        { role: "clusterAdmin", db: "admin" },
        { role: "readAnyDatabase", db: "admin" } ]
      }
    )

Check it has the correct auth schema : MONGODB-CR

> db.system.users.find({"user":"monitoring"})
{ "_id" : "admin.monitoring", "user" : "monitoring", "db" : "admin", "credentials" : { "MONGODB-CR" ...

Set back the auth schema to SCRAM-SHA-1

var schema = db.system.version.findOne({"_id" : "authSchema"});
schema.currentVersion = 5;
db.system.version.save(schema);

Restart MongoDB 3.0 with auth enabled

Restart the StackDriver agent

When StackDriver will support SCRAM-SHA-1, it will be useful to upgrade the auth schema for this user

db.adminCommand({authSchemaUpgrade: 1});

Answer 1

I work at Google on the Stackdriver agent. Adam C's answer is correct. I've been working on this lately, and now we have a version of this in beta that has been working well in our own testing. This new version addresses not only the SCRAM-SHA-1 issue but also has some performance improvements as well. Before releasing it more broadly, we would love the opportunity to test it in a couple of customer environments.

If anyone is willing to be a beta tester for the new version of our agent, I could arrange to deliver a .deb or .rpm file to you appropriate to your platform. As always there is a small risk that something can go wrong, so I think this only makes sense if you have a non-production environment you can try it in.

EDIT: As of July 12, 2016, this change is now available in production!

Answer 2

I suspect that the Stack Driver does not (yet) support the new SCRAM-SHA-1 Authentication mechanism. This new mechanism was added in 3.0 to replace MONGODB-CR and is the default in 3.0+, however it requires that the driver support the new mechanism.

To get MongoDB 3.0 to use the older mechanism, you can either start with 2.6 and create your users there, then upgrade or you can do the following (based on this comment):

• Start MongoDB 3.0 with auth disabled (make sure no users have been added yet)

• Connect to the instance and run the following:

  var schema = db.system.version.findOne({"_id" : "authSchema"}); schema.currentVersion = 3; db.system.version.save(schema);

• Start MongoDB with auth enabled and create your users

These uses should now be created with MONGODB-CR. The StackDriver is using libmongoc which supports SCRAM-SHA-1 as of version 1.1 but their version looks a lot older based on some Github browsing. Once they update their driver, this issue should go away - for now you will have to work around it.