Service just stops

Question

I have a serious production problem. A service running a java application on OpenJDK just stops at seemingly random times.

It starts correctly, when invoked manually:

sudo /etc/init.d/name start

But then after an unknown while the process is suddenly no longer there. About 3 hours after start in the last occurence.

Console output is redirected to a log file. There is no explanation to be found there. Likewise in the application log (through a Java logging framework).

Nothing of interest in the service journal:

journalctl -u name.service

No Java crash report files can be found on the entire system

sudo find / -name 'hs_err*'

Where else can I look for an explanation? This is quite urgent - any help would be greatly appreciated!

Answer 1

The most likely cause for applications disappearing without further logs is the system running out of memory.

You can confirm this using

$ grep -i -A20 "Out of memory" /var/log/syslog

This typically happens for one of the following reasons on your average server:

1. Your application has a memory leak and keeps using more and more memory.
2. Your application uses too much memory during regular operation, but you only notice that when your Virtualization framework limits the provisioned memory to the lower guarantee.
3. Some other application is using too much RAM. This could be a misconfigured SQL server that your application uses. Remember: The oom killer does not necessarily kill the application that caused the out of memory situation (rather one that contributed to it to some degree).

Answer 2

Bingo! Nov 7 02:00:51 web1 kernel: [2993761.744187] Out of memory: Kill process 23861 (java) score 461 or sacrifice child. How is this controlled?

To control it, you find out what is causing the system to run out of memory and you solve that issue. You will need to install monitoring an then use it. Watch what the system is doing, identify the process(es) using memory and then figure out what they are doing and why.