Has anyone successfully managed to monitor a MongoDB 3 cluster (or standalone db) with StackDriver in GCE ?
I've setup a MongoDB 3.0.6 cluster in GCE (replica set with 2 replica and 1 arbiter)
I'm trying to monitor it through StackDriver provided by Google.
I've followed all instructions to install monitoring agent and mongodb plugin found here : https://cloud.google.com/monitoring/agent/plugins/mongodb
When I start the agent on the replica where it is configured :
sudo /etc/init.d/stackdriver-agent restart
I get the following errors in /var/log/syslog :
collectd[6013]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.
collectd[6013]: mongo plugin: Authenticating to localhost:27017 failed:
collectd[6013]: mongo plugin: Connecting to localhost:27017 failed:
collectd[6013]: read-function of plugin `mongodb' failed. Will suspend it for 120.000 seconds.
I suspect the StackDriver agent is not compatible with MongoDB 3, because :
In the past, I've used the Click-to-deploy feature of GCE to create a cluster and was able to monitor it with StackDriver. At that time, it was with MongoDB 2.6.x.
I've quickly setup again a standalone installation of MongoDB 2.6.x, configured the StackDriver agent in the same way, and ... it works :-(
Any help will be really appreciated.
Details of configuration :
Mongodb :
auth=true
in database admin, user with roles : dbAdminAnyDatabase, clusterAdmin and readAnyDatabase
Stackdriver Mongodb plugin :
- this user and password used in /opt/stackdriver/collectd/etc/collectd.d/mongodb.conf
Additional information :
No Auth : the plugin initializes successfully
/etc/mongod.conf :
# Turn on/off security. Off is currently the default
#noauth = true
#auth = true
/opt/stackdriver/collectd/etc/collectd.d/mongodb.conf :
LoadPlugin mongodb
<Plugin "mongodb">
Host "localhost"
Port "27017"
# If you restricted access to the database, you can
# set the username and password here
# User "user_name"
# Password "user_password"
# For performance/eventually consistent trade-offs you may add this line
# PreferSecondaryQuery true
</Plugin>
Auth mode : authentication errors when restarting the agent
/etc/mongod.conf :
# Turn on/off security. Off is currently the default
#noauth = true
auth = true
3 users configured
use admin
db.createUser(
{
user: "siteUserAdmin",
pwd: "xxx",
roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
}
)
db.auth("siteUserAdmin", "xxx");
db.createUser( {
user: "siteRootAdmin",
pwd: "xxx",
roles: [ { role: "root", db: "admin" } ]
});
db.createUser(
{
user: "monitoring",
pwd: "xxx",
roles: [
{ role: "dbAdminAnyDatabase", db: "admin" },
{ role: "clusterAdmin", db: "admin" },
{ role: "readAnyDatabase", db: "admin" } ]
}
)
/opt/stackdriver/collectd/etc/collectd.d/mongodb.conf :
LoadPlugin mongodb
<Plugin "mongodb">
Host "localhost"
Port "27017"
# If you restricted access to the database, you can
# set the username and password here
User "monitoring"
Password "xxx"
# For performance/eventually consistent trade-offs you may add this line
# PreferSecondaryQuery true
</Plugin>
Same errors when I use siteRootAdmin in the plugin configuration.
Explanation and solution
The culprit was actually the auth schema used by StackDriver agent
I adapted the solution suggested by Adam C, because I already have users created with the SCRAM-SHA-1 schema.
In fact, I only need that the user for the monitoring uses MONGODB-CR.
To do so :
Restart MongoDB 3.0 with auth disabled
Connect to the instance
Change temporarily the auth schema to MONGODB-CR
use admin
var schema = db.system.version.findOne({"_id" : "authSchema"});
schema.currentVersion = 3;
db.system.version.save(schema);
Create the user for the StackDriver plugin
db.createUser(
{
user: "monitoring",
pwd: "xxx",
roles: [
{ role: "dbAdminAnyDatabase", db: "admin" },
{ role: "clusterAdmin", db: "admin" },
{ role: "readAnyDatabase", db: "admin" } ]
}
)
Check it has the correct auth schema : MONGODB-CR
> db.system.users.find({"user":"monitoring"})
{ "_id" : "admin.monitoring", "user" : "monitoring", "db" : "admin", "credentials" : { "MONGODB-CR" ...
Set back the auth schema to SCRAM-SHA-1
var schema = db.system.version.findOne({"_id" : "authSchema"});
schema.currentVersion = 5;
db.system.version.save(schema);
Restart MongoDB 3.0 with auth enabled
Restart the StackDriver agent
When StackDriver will support SCRAM-SHA-1, it will be useful to upgrade the auth schema for this user
db.adminCommand({authSchemaUpgrade: 1});