0

Ubuntu 18.04. 2Gb RAM + 512mb swap.

When running clamav, it consumes 800+mb of memory because it loads all the signatures into the memory. Because of that, I set it to run every day at 3am instead of ongoing.

So far, tomcat and clamav got along very well. Last night, at 3am, tomcat service was shut down when clamav started working.

[4643256.375812] OOM killed process 8145 (clamscan) total-vm:1149268kB, anon-rss:969476kB, file-rss:4kB
[7667218.452649] OOM killed process 8865 (java) total-vm:4568248kB, anon-rss:1067312kB, file-rss:0kB

Mar 26 03:00:31 user systemd[1]: tomcat.service: Main process exited, code=killed, status=9/KILL
Mar 26 03:00:31 user systemd[1]: tomcat.service: Failed with result 'signal'.
Mar 26 03:17:08 user systemd[1]: Reloading The Apache HTTP Server.
Mar 26 03:17:08 user systemd[1]: Reloaded The Apache HTTP Server.

I know upgrading is an immediate answer but until then, my questions are:

  1. Is there a way to run clamav without it consuming 800+mb?

  2. Is there a way to automatically restart tomcat if something like that happens again?

  3. Did Java really took 4,568,248kB = 4.5gb or am I missing something?

Edit inside tomcat.service file I have the following:

Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djava.awt.headless=true"
Amos
  • 237
  • 1
  • 4
  • 10

1 Answers1

1

Java did not use 4.5 GiB of physical memory, that figure refers to virtual memory. You have about 128 TiB of virtual memory address space (cf. this answer) and it is used to access physical memory, but also files as if they were in memory. Most of the allocated virtual space does not use any physical memory and contains only 0s or points to data present on hard drives.

What you are interested in is the anon-rss number: it is physical memory used by the process, mostly for private use, and is not just a cache/copy of a file (unlike file-rss). The kernel can not just delete this memory and you get an OOM.

The answer by Janne to your previous question points you to a solution of your problem:

  1. You should tune Tomcat's memory settings, which can be done by adding options to the variable JAVA_OPTS in /etc/default/tomcat9. The most commonly used are -Xms (minimum heap size) and -Xmx (maximum heap size). E.g. you can use:

    JAVA_OPTS="-Djava.awt.headless=true -Xms256m -Xmx1024m"

    The JVM will use between 256 MiB and 1024 MiB of heap space (plus other kinds of Java memory) and might give some of it back, when it is not used.

  2. Your amount of swap is quite small, especially if you don't have a lot of RAM. Extend it to around 4 GiB (cf. this article) so that the kernel can swap some memory instead of killing Tomcat.

  3. To configure an automatic restart of Tomcat after failure, edit the tomcat.service file:

    systemctl edit --full tomcat.service

    and add to the [Service] section, e.g.:

    RestartSec=5s
Restart=on-failure

    You can find the possible values for Restart in the SystemD documentation.

Piotr P. Karwasz
  • 5,292
  • 2
  • 9
  • 20
  • Thank you for your detailed reply. I can't change the swap size (vps). I have -Xms256m -Xmx1024m in catalina_opts. I prefer tomcat to have its resources (maybe even more than 1024m) and have clamav calm down :) Is it possible? I will try the restart option you wrote me. – Amos Mar 27 '20 at 10:50
  • Or maybe is there a way for set tomcat to take less RAM and more swap just when/if clamav is running and requires more than it has. – Amos Mar 27 '20 at 11:02
  • You can try digging deeper into Linux memory management subsystem, e.g. by adjusting the [swappiness](https://www.howtogeek.com/449691/what-is-swapiness-on-linux-and-how-to-change-it/) and [overcommit](https://serverfault.com/q/606185/530633) setting. You can also adjust Tomcat's [OOM killer score](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#OOMScoreAdjust=) to make it more resistant to killing. As for clamav, I don't use it, so can not tell you why it uses up so much memory. – Piotr P. Karwasz Mar 27 '20 at 18:17
  • Thanks for the tip. The OOM killer score is also defined under [service] section in systemctl edit --full tomcat.service? – Amos Mar 27 '20 at 19:49
  • Yes, in `*.service` files it should be defined under the `[Service]` section. – Piotr P. Karwasz Mar 27 '20 at 20:51
  • thank you very much – Amos Mar 27 '20 at 21:02