2

I`ve installed icinga2 on a virtual machine for monitoring 5 other ones with graphite support for the graphics and icingaweb2 as an interface. Recently on my web interface I got the follow:

Fatal error: Uncaught exception 'ErrorException' with message 'session_start(): open(/var/lib/php5/sess_3aabfsu76vcdj2hd2u3teofta5dvanpn, O_RDWR) failed: No space left on device (28)' in /usr/share/php/Icinga/Web/Session/PhpSession.php:105 Stack trace: #0 [internal function]: Icinga\Application\ApplicationBootstrap->Icinga\Application\{closure}(2, 'session_start()...', '/usr/share/php/...', 105, Array) #1 /usr/share/php/Icinga/Web/Session/PhpSession.php(105): session_start() #2 /usr/share/php/Icinga/Web/Session/PhpSession.php(215): Icinga\Web\Session\PhpSession->open() #3 /usr/share/php/Icinga/Authentication/Auth.php(344): Icinga\Web\Session\PhpSession->refreshId() #4 /usr/share/php/Icinga/Authentication/Auth.php(166): Icinga\Authentication\Auth->persistCurrentUser() #5 /usr/share/php/Icinga/Authentication/Auth.php(262): Icinga\Authentication\Auth->setAuthenticated(Object(Icinga\User)) #6 /usr/share/php/Icinga/Authentication/Auth.php(93): Icinga\Authentication\Auth->authExternal() #7 /usr/share/php/Icinga/Application/Web. in /usr/share/php/Icinga/Web/Session/PhpSession.php on line 105
Fatal error: Uncaught exception 'ErrorException' with message 'Unknown: open(/var/lib/php5/sess_3aabfsu76vcdj2hd2u3teofta5dvanpn, O_RDWR) failed: No space left on device (28)' in Unknown:0 Stack trace: #0 [internal function]: Icinga\Application\ApplicationBootstrap->Icinga\Application\{closure}(2, 'Unknown: open(/...', 'Unknown', 0, Array) #1 {main} thrown in Unknown on line 0

To check what`s going on I tried to find a problem in the disk free space:

# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/hostname--vg-root  6.5G  4.8G  1.4G  78% /
tmpfs                         201M  412K  200M   1% /run
...
...
/dev/sda1                     236M  154M   70M  69% /boot

The problem occurred in the fulfilled inodes:

#df -i
Filesystem                   Inodes  IUsed  IFree IUse% Mounted on
/dev/mapper/hostname--vg-root 441504 441504      0  100% /
...
...
/dev/sda1                     62248    318  61930    1% /boot

To check where are the biggest part of the tiny-files I did:

#for i in /*; do echo $i; find $i |wc -l; done

And found that that the point is in /usr and /var/spool/icinga2/perfdata/ directory

/usr
177882
/var/spool/icinga2/perfdata
229326

The question is in the reasons why icinga2 doesn't clean it's performance data and in the way how to clean it manually without damaging. Thanks.

Keith
  • 4,627
  • 14
  • 25
Ver
  • 33
  • 7

1 Answers1

3

Icinga 2 does not clean rotated performance data files. It expects external daemons/applications to remove these files after being processed. If you're using PNP for example, the NPCD daemon will collect the rotated files, process them and remove them afterwards.

You'd also want to add a "disk" check with inode threshold values to your monitoring in order to find out about such issues in advance.

dnsmichi
  • 845
  • 5
  • 12