7

I'm trying to use e2fsck on a large raid array that is 2TB large and uses GPT for partitioning (because of size).

There is only 1GB of ram installed on the system and I do not have the ability to add more at this time.

The problem is, shortly after starting the fsck on the device, I get an error that says:

Error storing directory block information (inode=115343515, block=0, num=108120142): Memory allocation failed
e2fsck: aborted

After a little bit of online searching and research I ran across this posting below:

Running out of memory running fsck on large filesystems

I followed the advice after looking up the MAN page for e2fsck.conf and basically created a /etc/e2fsck.conf file that looks like:

[scratch_files]
directory = /var/cache/e2fsck

and tried the fsck again after making sure to create the /var/cache/e2fsck directry. After starting the fsck again and watching the available memory, cpu usage, and teh directory size of /var/cashe/e2fsck... i can say it definetely helped a lot... but it still eventually failed giving the same error. Basically it helped slow down the memory consumption but did not negate it all together.

So i tried playing with the addition flags for the e2fsck.conf file using:

dirinfo = false

or

dirinfo = true

and 

icount = false

or

icount = true

neither of which seemed to have much of an effect as the same error happens after a short while.

Am i missing something? I'm ok with the fsck taking a long time... But i need it to actually complete instead of erroring out.

Community
  • 1
Jason
  • 71
  • 1
  • 2

3 Answers3

2

If you can, add some swap space to the system. The fsck will take an insanely long time, but it will eventually complete. Next time, chop your filesystem into smaller pieces.

womble
  • 95,029
  • 29
  • 173
  • 228
  • the filesystem is used for a backup service we run... so a large continuous drive/partition is desirable. Unfortunately it doens't seem to be using much swap space before it fails... the current swap space is 1GB as well... but it only fills up about 500MB before it errors out since the memory usage gets eaten up much faster. I'm going to try taking the server down today and running a e2fsck from a live cd and will report back if the memory consumption is different... i dont think it will be but we'll see. – Jason Jul 11 '11 at 16:54
  • tried it from a boot cd with no luck... it basically ran as if it was inside the OS and errored out with the Memory Allocation error after a few minutes... i'm kind of stumped on this one... does this mean the filesystems is so damaged that its impossible to fsck it? I would really like to avoid rebuilding the filesystem on that drive... – Jason Jul 11 '11 at 17:20
  • 1
    MOAH SWAPZ! Running from a live CD will, if anything, consume more memory than running locally (ramdisks use, well, *RAM*). When I say more swap I really mean *more* *swap* -- 10, 20GB. I don't see why "we run a backup service" translates to "we need one big partition"... make several small ones and mount them all next to each other in separate buckets. If you're not bucketing your data, you're going to make your dindexes sad pandas anyway. – womble Jul 11 '11 at 22:57
1
  1. Did you add those flags to the [scratch_files] section? Also, try to use "numdirs_threshold " to control when it starts to use the scratch files.
  2. Sometimes the e2fs utils are too old on the installed system (Like Centos4). You could boot a rescue image with the newer version of kernel and filesystem utils. Setup e2fsck.conf and let e2fsck use it.

HTH.

AndreasM
  • 1,083
  • 8
  • 13
0

Is the array internal or external storage? If it's external/attached storage, try plugging it into a different box with more RAM for the fsck. Alternately, repartition your setup and break it into smaller chunks. You can mount multiple partitions under a single "backup" filesystem hierarchy. Your clients and backup software don't need to know that they are touching multiple or separate filesystems.

If this storage doesn't get fsck'ed until later in the boot process, make sure you turn off any applications that might be eating up your precious memory until the fsck is completed.

Christopher Cashell
  • 8,999
  • 2
  • 31
  • 43