I'm a MongoDB newbie. I have a replica set with 3 members (1 primary and 2 secondaries) on CentOS 6 and Mongo 2.6.8.
One of the secondaries crashed because of high consumption of memory and I couldn't restart it properly (due to some data corruption), so I deleted the whole content of the datadir to force a full re-sync.
After 4 hours, the secondary was synced and back to the replica set. However, it generated 25 "local" files (local.0 ... local.24), while it used to be only 2 (like the other members), occupying more than 60 GB on disk only because of those files.
Moreover, the size of the oplog changed (it used to be 990MB, and now it's 47 GB):
rs:SECONDARY> rs.printReplicationInfo(); configured oplog size: 47774.441162109375MB log length start to end: 579127secs (160.87hrs) oplog first event time: Tue Jun 23 2015 16:53:13 GMT+0100 (IST) oplog last event time: Tue Jun 30 2015 09:45:20 GMT+0100 (IST) now: Tue Jun 30 2015 09:45:20 GMT+0100 (IST)
Since that happened, the server has been consuming around 130 GB of virtual memory and it's been showing a really poor performance.
I had to do a full sync on the other secondary as well (because of similar issues) and nothing changed (it generated 2 "local" files and the size of the oplog was still 990MB).
I'm wondering:
What's the reason of this behaviour of the "local" database?
Why is this behaviour affecting the performance if the "local" database it's not replicated?
Is there a way to change this secondary node back to how it was (only 2 local files and a smaller oplog)? I know it is possible to change the size of the oplog, but I'm wondering if I could just stop the service, delete the "local" files so they sync again.
Any other suggestions are more than welcome (:
