I found several other questions regarding the theory behind my problem (e.g. this, this), but I don't know how to apply the answers to my machine.
# du -hsx /
11000283 /
# df -kT /
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/mapper/csisv13-root ext4 516032952 361387456 128432532 74% /
There is a big difference between 11G (du) and 345G (df). Where are the remaining 334G?
It's not in deleted files. There was only one, it was short, and I truncated it just in case. This is what remains:
# lsof -a +L1 /
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
zabbix_ag 4902 zabbix 1w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4902 zabbix 2w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4906 zabbix 1w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4906 zabbix 2w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4907 zabbix 1w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4907 zabbix 2w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4908 zabbix 1w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4908 zabbix 2w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4909 zabbix 1w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4909 zabbix 2w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4910 zabbix 1w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4910 zabbix 2w REG 252,0 0 0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
I rebooted to see if fsck does anything. But, from /var/log/boot.log, it seems there are no issues:
/dev/mapper/server-root: clean, 3936097/32768000 files, 125368568/131064832 blocks
Thinking maybe someone overzealously reserved root space, I checked the master record:
# tune2fs -l /dev/mapper/server-root
tune2fs 1.42 (29-Nov-2011)
Filesystem volume name: <none>
Last mounted on: /
Filesystem UUID: 86430ade-cea7-46ce-979c-41769a41ecbe
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 32768000
Block count: 131064832
Reserved block count: 6553241
Free blocks: 5696264
Free inodes: 28831903
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 992
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Fri Feb 1 13:44:04 2013
Last mount time: Tue Aug 19 16:56:13 2014
Last write time: Fri Feb 1 13:51:28 2013
Mount count: 9
Maximum mount count: -1
Last checked: Fri Feb 1 13:44:04 2013
Check interval: 0 (<none>)
Lifetime writes: 1215 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
First orphan inode: 28836028
Default directory hash: half_md4
Directory Hash Seed: bca55ff5-f530-48d1-8347-25c004f66d43
Journal backup: inode blocks
The system is:
# uname -a
Linux server 3.2.0-67-generic #101-Ubuntu SMP Tue Jul 15 17:46:11 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"
Does anyone have any tips on what exactly to do to find and hopefully reclaim the missing space?
