1

Running CentOS 5.6 with mdadm RAID1, 2TB, 3 partitions (boot, swap and root). I had a kernel panic in the server, and after rebooting, the system stays up for about 5-10 minutes, the it starts printing out fsck errors to the console and the root file system is put into readonly mode.

I rebooted into single-user mode, unmounted the / partition and tried to run 

e2fsck -f /dev/md2

but it hung at Stage 1 for over 8 hours without updates. I notice that the RAID is being resynched (cat /proc/mdstat) -- could this be impacting the e2fsck? I ran smartctl on both disks, they both came back healthy so I don't think this is a disk problem, though I don't have any clue what triggered the RAID rebuild.

Anyway, is there a problem running e2fsck on a raid device that is being resynched? Anything I can do to get fsck out of stage 1, or get more feedback on progress? Why is the error pointing to loop0 instead of one of the actual devices?

John P
  • 1,659
  • 6
  • 37
  • 56

1 Answers1

1

The (loop0) should have given me a clue. The problem was not with any hard drive filesystem -- it was with the RAM based tmpfs. My /tmp directory was mounted using a tmpfs (/dev/shm)

/usr/tmpDSK             /tmp   ext3

The ext3 errors were occurring in the RAM based tmpfs. Once I unmounted /tmp (had to force it using umount -l /tmp), the errors went away and everything started working. The question remains why an in-memory filesystem would get ext3 errors, but at least I'm up and running.

John P
  • 1,659
  • 6
  • 37
  • 56