0

I've been running into this problem where every few days one of my java servers is just dying. To figure out why, I ran cat /var/log/syslog which produces:

Sep 21 06:36:24 ip-172-31-42-5 systemd-timesyncd[16235]: Network configuration changed, trying to establish connection.
Sep 21 06:36:24 ip-172-31-42-5 systemd-timesyncd[16235]: Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).
Sep 21 06:46:45 ip-172-31-42-5 systemd[1]: Starting Daily apt upgrade and clean activities...
Sep 21 06:47:35 ip-172-31-42-5 kernel: [5731409.370359] systemd-resolve invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
...(skipping a bunch) ...
Sep 21 06:47:35 ip-172-31-42-5 kernel: [5731409.370580] Out of memory: Kill process 28251 (java) score 368 or sacrifice child
Sep 21 06:47:35 ip-172-31-42-5 kernel: [5731409.374244] Killed process 28251 (java) total-vm:2613628kB, anon-rss:178048kB, file-rss:0kB, shmem-rss:0kB
Sep 21 06:47:35 ip-172-31-42-5 kernel: [5731409.403233] oom_reaper: reaped process 28251 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

After reading this post, I learned that the oom_reaper will kill the largest memory-using process if ANY process tries to allocate too much memory. So, I'm guessing that systemd-resolve is trying to request a ridiculous amount of memory, and then the reaper is killing my poor java process.

I could work around this by setting vm.oom_kill_allocating_task as suggested in the above link, but I'm trying to figure out the root cause of why systemd-resolve is trying to request such an absurd amount of memory. Anyone know what the root problem is here?

edit: when the java server is running, memory usage is only at 234M/460M

satnam
  • 51
  • 3
  • When the Java server is killed, it's at 2.5 GiB of memory. Sounds like a standard Java memory hog that eats up all resources and any allocation, whatever the size, will fail since I'm assuming you have between 2 and 4 GiB of memory and some other processes around. – Ginnungagap Sep 24 '19 at 20:47
  • I have only 460 MB of memory on this machine – satnam Sep 25 '19 at 00:39
  • `total-vm:2613628kB, anon-rss:178048kB` is Java actively using 178 MB of that memory, with a total allocated (virtually) of 2.5 GiB which is why it scores so high when OOM killer is involved. You should have a list of processes with their allocation information and how they score in the kernel log in the part you skipped in the question. Take a look at that if you want to try and better manage who consumes what to avoid running out of memory. – Ginnungagap Sep 25 '19 at 06:07

0 Answers0