Centos 6 unexpected inconsistency within 100% disk

Question

I have a runing centos 6, which suddenly fail due to hd full... which cause us sever problems since several days.

when i do:

df -h
S.ficheros            Size  Used Avail Use% Montado en
rootfs                9,7G  9,1G  158M  99% /
/dev/root             9,7G  9,1G  158M  99% /
none                  7,8G  404K  7,8G   1% /dev
/dev/md2              101G  188M   96G   1% /home
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/var/named
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.conf
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.rfc1912.zones
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/rndc.key
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/usr/lib64/bind
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.iscdlv.key
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.root.key

And when i go ahead, I get:

du -sh /var/named/*
du: ATENCIÓN: Estructura de directorios circular.
Esto quiere decir seguramente que el sistema de ficheros está corrupto.
COMUNÍQUELO AL ADMINISTRADOR DEL SISTEMA.
El siguiente directorio es parte del ciclo:
«/var/named/chroot/var/named/chroot»

268K /var/named/chroot
144K /var/named/data
12K /var/named/dynamic
4,0K /var/named/named.ca
4,0K /var/named/named.empty
4,0K /var/named/named.localhost
4,0K /var/named/named.loopback
4,0K /var/named/slave

So, i reboot into rescue mode, then i did:

fdisk -l

Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000bd74

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1275    10238976+  fd  Linux raid autodetect
/dev/sda2            1275       14528   106451968   fd  Linux raid autodetect
/dev/sda3           14528       14593      523264   82  Linux swap / Solaris

Disk /dev/sdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000fef6

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1275    10238976+  fd  Linux raid autodetect
/dev/sdb2            1275       14528   106451968   fd  Linux raid autodetect
/dev/sdb3           14528       14593      523264   82  Linux swap / Solaris

Disk /dev/md2: 109.0 GB, 109006749696 bytes
2 heads, 4 sectors/track, 26612976 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md2 doesn't contain a valid partition table

Disk /dev/md1: 10.5 GB, 10484645888 bytes
2 heads, 4 sectors/track, 2559728 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

So, i made the fsck as below:

root@rescue:~# fsck -yvf /dev/md2 
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

      12 inodes used (0.00%)
       0 non-contiguous files (0.0%)
       0 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 0/0/0
  257937 blocks used (0.97%)
       0 bad blocks
       1 large file

       0 regular files
       3 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       0 symbolic links (0 fast symbolic links)
       0 sockets
--------
       3 files
root@rescue:~# fsck -yvvvvf /dev/md2
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

      12 inodes used (0.00%)
       0 non-contiguous files (0.0%)
       0 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 0/0/0
  257937 blocks used (0.97%)
       0 bad blocks
       1 large file

       0 regular files
       3 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       0 symbolic links (0 fast symbolic links)
       0 sockets
--------
       3 files
root@rescue:~# fsck -yvvvvf /dev/md1
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

  293166 inodes used (45.66%)
    2318 non-contiguous files (0.8%)
      98 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 9776/113/0
 2391131 blocks used (93.41%)
       0 bad blocks
       2 large files

  236606 regular files
   53806 directories
      15 character device files
      77 block device files
       2 fifos
    2973 links
    2625 symbolic links (2625 fast symbolic links)
      26 sockets
--------
  296130 files
root@rescue:~# e2fsck -yvvvvf /dev/md1
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

  293166 inodes used (45.66%)
    2318 non-contiguous files (0.8%)
      98 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 9776/113/0
 2391131 blocks used (93.41%)
       0 bad blocks
       2 large files

  236606 regular files
   53806 directories
      15 character device files
      77 block device files
       2 fifos
    2973 links
    2625 symbolic links (2625 fast symbolic links)
      26 sockets
--------
  296130 files
root@rescue:~# e2fsck -yvvvvf /dev/md2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

      12 inodes used (0.00%)
       0 non-contiguous files (0.0%)
       0 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 0/0/0
  257937 blocks used (0.97%)
       0 bad blocks
       1 large file

       0 regular files
       3 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       0 symbolic links (0 fast symbolic links)
       0 sockets
--------
       3 files
root@rescue:~# reboot

But when i restart, I just login, and then the same:

 df -h
S.ficheros            Size  Used Avail Use% Montado en
rootfs                9,7G  9,1G  158M  99% /
/dev/root             9,7G  9,1G  158M  99% /
none                  7,8G  404K  7,8G   1% /dev
/dev/md2              101G  188M   96G   1% /home
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/var/named
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.conf
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.rfc1912.zones
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/rndc.key
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/usr/lib64/bind
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.iscdlv.key
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.root.key

Please, any idea what's going on there, I already delete the tmps from the root, and var, delete messages log... which are unsignificant, but it's unconsistency in the file system, any idea??

Well, you can run `du -sxh *` from `/` to see which of the top level directories are taking the most space, and then drill down to the next level, etc. — cjc, Jul 12 '12 at 11:14

score 1 · Answer 1 · answered Jul 12 '12 at 11:25

Find what's taking up the space with du -hx --max-depth=1 /, then repeat as necessary (du -hx --max-depth=1 /usr, for instance) until you find something big you can afford to lose or bind-mount into /home. And next time, use LVM, and don't pre-allocate all your space onto a partition you barely use.

score 0 · Answer 2 · answered Jul 12 '12 at 11:31

0

To find out what is filling your root filesystem

cd /
du -sx | sort -rn

will give you a list of files/directories sorted by usage. Then cd into the one with the highest usage and repeat till you find out what's filling the disk.

answered Jul 12 '12 at 11:31

user9517

114,104
20
206
289

Centos 6 unexpected inconsistency within 100% disk

2 Answers2