I`ve installed icinga2 on a virtual machine for monitoring 5 other ones with graphite support for the graphics and icingaweb2 as an interface. Recently on my web interface I got the follow:
Fatal error: Uncaught exception 'ErrorException' with message 'session_start(): open(/var/lib/php5/sess_3aabfsu76vcdj2hd2u3teofta5dvanpn, O_RDWR) failed: No space left on device (28)' in /usr/share/php/Icinga/Web/Session/PhpSession.php:105 Stack trace: #0 [internal function]: Icinga\Application\ApplicationBootstrap->Icinga\Application\{closure}(2, 'session_start()...', '/usr/share/php/...', 105, Array) #1 /usr/share/php/Icinga/Web/Session/PhpSession.php(105): session_start() #2 /usr/share/php/Icinga/Web/Session/PhpSession.php(215): Icinga\Web\Session\PhpSession->open() #3 /usr/share/php/Icinga/Authentication/Auth.php(344): Icinga\Web\Session\PhpSession->refreshId() #4 /usr/share/php/Icinga/Authentication/Auth.php(166): Icinga\Authentication\Auth->persistCurrentUser() #5 /usr/share/php/Icinga/Authentication/Auth.php(262): Icinga\Authentication\Auth->setAuthenticated(Object(Icinga\User)) #6 /usr/share/php/Icinga/Authentication/Auth.php(93): Icinga\Authentication\Auth->authExternal() #7 /usr/share/php/Icinga/Application/Web. in /usr/share/php/Icinga/Web/Session/PhpSession.php on line 105
Fatal error: Uncaught exception 'ErrorException' with message 'Unknown: open(/var/lib/php5/sess_3aabfsu76vcdj2hd2u3teofta5dvanpn, O_RDWR) failed: No space left on device (28)' in Unknown:0 Stack trace: #0 [internal function]: Icinga\Application\ApplicationBootstrap->Icinga\Application\{closure}(2, 'Unknown: open(/...', 'Unknown', 0, Array) #1 {main} thrown in Unknown on line 0
To check what`s going on I tried to find a problem in the disk free space:
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/hostname--vg-root 6.5G 4.8G 1.4G 78% /
tmpfs 201M 412K 200M 1% /run
...
...
/dev/sda1 236M 154M 70M 69% /boot
The problem occurred in the fulfilled inodes:
#df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/hostname--vg-root 441504 441504 0 100% /
...
...
/dev/sda1 62248 318 61930 1% /boot
To check where are the biggest part of the tiny-files I did:
#for i in /*; do echo $i; find $i |wc -l; done
And found that that the point is in /usr and /var/spool/icinga2/perfdata/ directory
/usr
177882
/var/spool/icinga2/perfdata
229326
The question is in the reasons why icinga2 doesn't clean it's performance data and in the way how to clean it manually without damaging. Thanks.