0

Recently I've faced situation, when "standard" disk space monitoring failed - Zabbix uses vfs.fs.size item for checking disk usage, which is documented and preconfigured in bundled templates.

The Problem:

When files are deleted, but file descriptor/handle is still open - disk might be filled, but Zabbix will report no usage in this case.

How can one monitor disk space in this case?

StephenKing
  • 932
  • 1
  • 8
  • 18
GioMac
  • 4,444
  • 3
  • 24
  • 41
  • Zabbix can use only data, which are reported by OS and OS report free space in this case. You can eventually calculate size of opened files per fs and then use calculated item vfs.fs.size + vfs.fs.openedfiles.size as real current disk usage -> But are you able to implement vfs.fs.openedfiles.size efficiently? My standard workaround: monitor also app logs - there should be some error "No space left" for this case - but it'll be too late to prevent incident from operation point of view. – Jan Garaj Sep 03 '15 at 10:42

1 Answers1

1

Here is a test i've done :

df /
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda1       15350768 14367976    809124  95% /

I create a 600Mb file and open it with less :

dd if=/dev/zero of=test bs=1024k count=600 & less test
[1] 1952
"test" may be a binary file.  See it anyway? 600+0 records in
600+0 records out
629145600 bytes (629 MB) copied, 11.895 s, 52.9 MB/s
[1]+  Done                    dd if=/dev/zero of=test bs=1024k count=600

Now df show that more space is used :

df /
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda1       15350768 14981664    195436  99% /

now if I remove my test file :

rm test

df says :

df /
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda1       15350768 14981712    195388  99% /

so the space is still in use, we can see it with lsof |grep deleted :

lsof |grep deleted|grep test
less       1953             mick    4r      REG                8,1  629145600       5407 /tmp/test (deleted)

now if I stop less :

pkill less

df show that the disk usage is free :

df /
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda1       15350768 14367584    809516  95% /

I think your problem comes more from a bad intepretation of df due to percent reserved for root.

From the last df, total minus used minus available equals :

15350768-14367584-809516=173668

So maybe your user has a disk full error, but root can still write to disk, and Zabbix will not see disk full due to percent reserved for root.

You can check and change this with tune2fs

mick
  • 21
  • 2