We are facing an issue in our production system, running on Cent OS 7. The / (root) partition reached 90% of disk space but we don’t have any huge files / hidden files on that machine. Kindly help us to solve this issue.
For your information:
Version : CentOS Linux release 7.2.1511 (Core).
Machine : Virtual Machine.
Output of df -h
[root@dbtestlab ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 45G 5.6G 89% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 96K 16G 1% /dev/shm
tmpfs 16G 49M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sdb 493G 146G 322G 32% /logdata
/dev/sdc 493G 181G 287G 39% /dbdata
/dev/mapper/centos-home 442G 212M 442G 1% /home
/dev/sda1 497M 195M 303M 40% /boot
tmpfs 3.2G 16K 3.2G 1% /run/user/1000
tmpfs 3.2G 0 3.2G 0% /run/user/0
Outpu of du -sch *
[root@dbtestlab /]# du -sch *
0 backupdata
0 bin
169M boot
181G dbdata
96K dev
0 disk1
0 disk2
30M etc
179M home
0 lib
0 lib64
146G logdata
0 media
0 mnt
0 opt
du: cannot access ‘proc/13270/task/13270/fd/4’: No such file or directory
du: cannot access ‘proc/13270/task/13270/fdinfo/4’: No such file or directory
du: cannot access ‘proc/13270/fd/4’: No such file or directory
du: cannot access ‘proc/13270/fdinfo/4’: No such file or directory
0 proc
40M root
du: cannot access ‘run/user/1000/gvfs’: Permission denied
49M run
0 sbin
0 srv
0 sys
8.0K tmp
3.8G usr
2.4G var
333G total
Partition Information:
[root@dbtestlab /]# cat /proc/partitions
major minor #blocks name
2 0 4 fd0
8 16 524288000 sdb
8 0 524288000 sda
8 1 512000 sda1
8 2 523774976 sda2
8 32 524288000 sdc
8 48 52428800 sdd
11 0 1048575 sr0
253 0 8273920 dm-0
253 1 52428800 dm-1
253 2 463069184 dm-2
Disk Information
[root@dbtestlab /]# fdisk -l
Disk /dev/sdb: 536.9 GB, 536870912000 bytes, 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 536.9 GB, 536870912000 bytes, 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000daca1
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 1048575999 523774976 8e Linux LVM
Disk /dev/sdc: 536.9 GB, 536870912000 bytes, 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdd: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 8472 MB, 8472494080 bytes, 16547840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-home: 474.2 GB, 474182844416 bytes, 926138368 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
EDIT 1
Inode Usage:
[root@dbtestlab /]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/centos-root 23556176 140689 23415487 1% /
devtmpfs 4093940 408 4093532 1% /dev
tmpfs 4097775 9 4097766 1% /dev/shm
tmpfs 4097775 596 4097179 1% /run
tmpfs 4097775 13 4097762 1% /sys/fs/cgroup
/dev/sdb 32768000 37 32767963 1% /logdata
/dev/sdc 32768000 6612 32761388 1% /dbdata
/dev/mapper/centos-home 463069184 207 463068977 1% /home
/dev/sda1 512000 337 511663 1% /boot
tmpfs 4097775 26 4097749 1% /run/user/1000
tmpfs 4097775 1 4097774 1% /run/user/0
EDIT 2
Files and folders greater than 1 GB
[root@dbtestlab /]# du -xh / 2> /dev/null | grep '[0-9].[0-9]G'
2.1G /var/log
2.4G /var
1.1G /usr/lib64
1.3G /usr/share
3.8G /usr
6.1G /
Kernel List:
[root@dbtestlab /]# rpm -q kernel
kernel-3.10.0-123.el7.x86_64
kernel-3.10.0-327.10.1.el7.x86_64
kernel-3.10.0-327.13.1.el7.x86_64
The kernel I am using:
[root@dbtestlab /]# uname -a
Linux dbtestlab 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I have already installed 50 GB hard disk to that machine but not mounted, since increasing size of the LVM may corrupt the system, since it is root partition. Sorry for the long question, I was searching solution for this issues and I found that this information needed for debugging, so I added all the output status. Feel free to ask me if any other further information required.
EDIT 3
Output of ls -la:
[root@dbtestlab /]# ls -la
total 52
dr-xr-xr-x. 22 root root 4096 May 30 15:28 .
dr-xr-xr-x. 22 root root 4096 May 30 15:28 ..
-rw-r--r-- 1 root root 0 Apr 7 12:36 .autorelabel
drwxr-xr-x 2 root root 6 Apr 26 12:44 backupdata
lrwxrwxrwx. 1 root root 7 Mar 22 16:56 bin -> usr/bin
dr-xr-xr-x. 4 root root 4096 May 30 15:35 boot
drwxrwxrwx 4 mysql mysql 4096 Apr 8 16:49 dbdata
drwxr-xr-x 19 root root 3320 May 26 23:56 dev
drwxr-xr-x. 2 root root 6 Apr 5 23:22 disk1
drwxr-xr-x. 2 root root 6 Apr 5 23:22 disk2
drwxr-xr-x. 141 root root 8192 May 30 15:33 etc
drwxr-xr-x. 6 root root 4096 May 30 15:59 home
lrwxrwxrwx. 1 root root 7 Mar 22 16:56 lib -> usr/lib
lrwxrwxrwx. 1 root root 9 Mar 22 16:56 lib64 -> usr/lib64
drwxrwxrwx 8 mysql mysql 4096 Apr 9 22:59 logdata
drwxr-xr-x. 2 root root 6 Aug 12 2015 media
drwxr-xr-x. 2 root root 6 Aug 12 2015 mnt
drwxr-xr-x. 3 root root 15 Aug 12 2015 opt
dr-xr-xr-x 250 root root 0 May 26 23:53 proc
dr-xr-x---. 14 root root 4096 May 30 14:37 root
drwxr-xr-x 38 root root 1180 May 30 17:23 run
lrwxrwxrwx. 1 root root 8 Mar 22 16:56 sbin -> usr/sbin
drwxr-xr-x. 2 root root 6 Aug 12 2015 srv
dr-xr-xr-x 13 root root 0 May 26 23:53 sys
drwxrwxrwt. 14 root root 4096 May 31 03:28 tmp
drwxr-xr-x. 13 root root 4096 Mar 22 16:56 usr
drwxr-xr-x. 23 root root 4096 May 26 23:53 var
Output of cat /proc/swaps
[root@dbtestlab /]# cat /proc/swaps
Filename Type Size Used Priority
/dev/dm-0 partition 8273916 899852 -1
Output of pvdisplay
[root@dbtestlab /]# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name centos
PV Size 499.51 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 127874
Free PE 0
Allocated PE 127874
PV UUID KP0RnD-kAZ3-sJ0o-B7SH-ePI3-8zig-2Vhmq3
Output of du -sch /* --exclude=/{dev,run,sys,logdata,dbdata,home,boot}
[root@dbtestlab /]# du -sch /* --exclude=/{dev,run,sys,logdata,dbdata,home,boot}
0 /backupdata
0 /bin
0 /disk1
0 /disk2
30M /etc
0 /lib
0 /lib64
0 /media
0 /mnt
0 /opt
du: cannot access ‘/proc/13991/task/13991/fd/4’: No such file or directory
du: cannot access ‘/proc/13991/task/13991/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/13991/fd/4’: No such file or directory
du: cannot access ‘/proc/13991/fdinfo/4’: No such file or directory
0 /proc
40M /root
0 /sbin
0 /srv
8.0K /tmp
3.8G /usr
2.4G /var
6.1G total
