I am wondering if anyone has seen an issue upgrading from 3.0.9 to 3.2.8 with the $external user database not getting replicated?
I have an existing 3.0.9 replica set environment running with no encrypted communication (no-TLS) and no auth; the environment works fine. But when I upgrade using 3.2.8 which has TLS and RBAC (using $external) the SECONDARY members in the cluster do not appear to have gotten the $external user data.
If I perform a pristine install of 3.2.8 with my configuration, it works fine, that is applications that connect to the replica set and read/write from primary or just read from secondary can connect and auth. But if I try to do the same from an upgraded 3.0.9-3.2.8 cluster, the connect, auth and read/write from primary is fine, but they can connect but not auth to either secondary.
Is this a known bug? I have read all the upgrade docs on mongodb but cannot find where anyone has attempted this.
The error I get when it does fail is:
test7-mongodb-rs:SECONDARY> db.getSiblingDB("$external").auth( { mechanism: "MONGODB-X509", user: "CN=*******,OU=******,O=**********,L=***,ST=**,C=US@$external"
Error: 11 { ok: 0.0, errmsg: "Could not find user CN=*******,OU=******,O=**********,L=***,ST=**,C=US@$external", code: 11 }
0
Some additional information which seems interesting. If I use my cluster member cert instead of the client cert I can connect and auth to the secondaries. Then when I run the getUsers() command on the $external DB it throws the following error:
test7-mongodb-rs:SECONDARY>
db.getSiblingDB("$external").getUsers() 2016-08-17T16:49:56.188-0400 E QUERY Error: not master and slaveOk=false at Error (<anonymous>) at DB.getUsers (src/mongo/shell/db.js:1342:15) at (shell):1:30 at src/mongo/shell/db.js:1342
I am also going to add, that once I hit the issue above, if I stop the cluster members and then remove the lib directory and restart my setup, it works. So there is something in 3.0.9 data left behind that is causing the issue. Must I perhaps perform a dropAllUsers() prior to creating the new ones for RBAC? That is, are there default or hidden data that are created with no auth in 3.0.9 that prevents this $external replication from occurring when upgrading?
Update, I see what the problem is but not sure why its a problem. The 'admin' DB on the primary is not being replicated. If I copy it to the secondaries, authentication there works. Question now it, why is it not syncing this DB
