1

tl;dr: the new 2.6 replica eats up a lot more disk space than the 2.4 members.

We're heavy users of MongoDB's GridFS. We're currently on 2.4, and are intending to upgrade to 2.6 by simply adding a new replica members and gradually deprecate the 2.4 nodes as recommended.

We have one primary, one secondary and an arbiter. Here are some data on the current system:

$ mongod --version
db version v2.4.10
Sun Jul 27 13:56:27.250 git version: e3d78955d181e475345ebd60053a4738a4c5268a

Here's some data on the database:

> db.stats()
{
    "db" : "SomeDatabase",
    "collections" : 4,
    "objects" : 2797931,
    "avgObjSize" : 284877.9334958582,
    "dataSize" : 797068801344,
    "storageSize" : 946689860768,
    "numExtents" : 469,
    "indexes" : 5,
    "indexSize" : 251248480,
    "fileSize" : 950804676608,
    "nsSizeMB" : 16,
    "dataFileVersion" : {
        "major" : 4,
        "minor" : 5
    },
    "ok" : 1
}

As you can see, the database is ~950GB and fits (but barely) on the Primary and Secondary (which is 1TB of storage).

Since we're running low on storage, I gave the new node that I wanted to add to the system another 200G (i.e. 1.2TB).

Here's the MongoDB version from the new node:

$ mongod --version
db version v2.6.3
2014-07-27T12:01:21.242+0000 git version: 255f67a66f9603c59380b2a389e386910bbb52cb

After making it a member of the replica set it starts to sync just fine (like every other time we've rotated ReplicaSet members). This is where it gets weird. After syncing for a few hours, it reaches the expected database size (~950GB). However, it just simply continues after that until it finally fills up the entire disk and crashes.

Now, I know that 2.6 is a new version, but how come that the database on disk exceeds the ones on the Primary and Secondary? Any pointers would be very helpful.

vpetersson
  • 721
  • 9
  • 21

1 Answers1

1

In MongoDB 2.6 PowerOf2sizes (http://docs.mongodb.org/master/reference/command/collMod/#usePowerOf2Sizes) is the default setting for collections. It will cause fewer disk relocations during updates but will consume more disk space.

Dharshan
  • 321
  • 3
  • 5
  • Thanks @Dharshan. Yeah I noticed that too, but wasn't sure how it would affect GridFS. I bumped up the size of the disk to see if that resolves the issue. Will spin up another one later and see if that solves the issue. – vpetersson Jul 27 '14 at 17:43
  • 1
    note that you do have an option to re-enable the old default. If you don't ever delete files in this collection, then you don't have to worry about efficient space re-use and it might be a better option for you (adding it here in case @Dharshan wants to add a note about that to his answer). – Asya Kamsky Jul 28 '14 at 04:29
  • Actually, even after disabling usePowerof2Sizes, I ran out of disk space on a 2.6 member with *more* disk space than the 2.4 members. Very odd. – vpetersson Jul 29 '14 at 08:19
  • I'm going to try to bring up the new node with 2.4 and then upgrade it to 2.6 to see if that solves the issue. In any case, this has been very painful. – vpetersson Jul 31 '14 at 20:19
  • I tried the 2.4 approach, still no go. Now trying to restore from a backup instead. – vpetersson Aug 16 '14 at 12:00
  • For now, I've settled for a temporary workaround. I took a file system snapshot from one of the existing 2.4 nodes, then I simply mounted that on my 2.6 node and allowed it to sync up. It is hardly a clean solution, but that's what I had to resort to. – vpetersson Aug 17 '14 at 12:17