4

There are quite number of software running on my server: httpd, varnish, mysql, memcache, java..

Each of them is using a part of the virtual memory and varnish was configured to be allocated 3GB of memory to run.

Due to high traffic load which is 100K, our server ran out of memory and oom-killer is invoked. We've to reboot the server.

We have 8GB of Virtual Memory and due to some reason we cannot extend to larger memory.

My question is - Is there any automated script, which will monitor how much virtual memory left and based upon certain criteria, lets say if 500MB left than restart the server automatically?

I do know this is not the proper solution but we have to do it, otherwise we don't know when server will get OOM and by the time we know and restart the server, we lost our visiting users.

Sukhjinder Singh
  • 393
  • 1
  • 4
  • 12
  • 2
    Wouldn't disabling the OOM killer make more sense? – voretaq7 Dec 11 '12 at 04:48
  • Are you sure you mean virtual memory, not physical memory? 8GB is an absurdly low virtual memory limit. Also, there's no way a server can have 500MB of virtual memory left, because virtual memory is a per-process resource. – David Schwartz Dec 11 '12 at 05:06
  • Are you talking about virtual memory as in a VPS? – Grumpy Dec 11 '12 at 06:14
  • @voretaq7: OOM killer is fired manually when server is OOM. – Sukhjinder Singh Dec 11 '12 at 07:54
  • @DavidSchwartz yes I mean Virtual memory. I have just given an example in case we have 500MB (this criteria can be changed). My actual question is - **is there any automated script which can restar server when centain defined limit of virtual memory reaches?** – Sukhjinder Singh Dec 11 '12 at 07:57
  • @Peter partially right yes, virtual memory as in a Dedicated Server. – Sukhjinder Singh Dec 11 '12 at 07:58
  • 1
    @SukhjinderSingh: That makes no sense, since each process has its own virtual memory limits on dedicated server. It wouldn't make sense the restart the server if a process was running low on virtual memory. (Do you mean RAM + swap, maybe?) – David Schwartz Dec 11 '12 at 08:19
  • 2
    Wouldn't it be MUCH better to just limit memory resources in order to have it not go beyond the limits of the server? Apache MPM tuning, MySQL buffer pool limit, etc. Better to refuse more connections than have processes killed in an unclean state. – gertvdijk Dec 11 '12 at 15:11
  • @SukhjinderSingh Um, the OOM killer *cannot* be "fired manually". It's "fired" by the kernel when it thinks you're going to run out of RAM (rather than the *correct, POSIX-conformant behavior* of returning `NULL` for a `malloc()` request it can't satisfy). This at least lets you know which process made the RAM-grab. Your permanent solution is to add sufficient RAM and/or swap to meet your resource requirements. – voretaq7 Dec 11 '12 at 16:12
  • @DavidSchwartz my server's lots memory is taken my Varnish due to high traffic load,but I cannot avoid varnish otherwise serving pages fastly will not happen. That is why I said please advise proper solution to this issue. – Sukhjinder Singh Dec 11 '12 at 17:41
  • @voretaq7 I understood – Sukhjinder Singh Dec 11 '12 at 17:42
  • @gertvdijk Thanks for the advice, can you please provide more details or reference links? – Sukhjinder Singh Dec 11 '12 at 17:44
  • Use common sense. Use monitoring. Find the cause. Fix the cause. Do not focus on the symptoms. – gertvdijk Dec 12 '12 at 11:55

2 Answers2

8

If I understand you correctly, you want something like the following:

  1. Check how much memory is left on the VPS.
  2. If 500M memory is left , reboot the VPS.

This could be done as follow

  1. Write a script that checks how much memory is left and reboot the VPS
  2. Add this script to crontab to automate the task.

e.g

#!/bin/bash

mem=$(free -m | awk '/Mem:/{print $4}')

(( mem <= 500 )) && reboot

Make the script executable

chmod +x scriptname // note don't add an extension

Add the script to the cron

crontab -e

* * * * * user_to_run_the_script /path/to/the/script

Hope you get the idea.

Valentin Bajrami
  • 3,870
  • 1
  • 17
  • 25
  • Right, I will check this out and let you know if this works for me. – Sukhjinder Singh Dec 11 '12 at 10:51
  • 4
    This is a direct answer to the question, which is good. It is also a HORRIBLE idea. "Just reboot the server" is almost ***NEVER*** the right solution -- find and fix the root cause (memory leak, inadequately provisioned system, etc), don't just bounce the box whenever a given parameter crosses a threshold. – voretaq7 Dec 11 '12 at 16:16
  • 1
    @voretaq7 I totally agree. I should have mentioned that this is not a desirable solution. There are of course better ways to this approach. As I tend to be concise on my answers I just pictured out what sukhijnder asked. – Valentin Bajrami Dec 11 '12 at 22:25
  • Does this really treat the `mem` variable as a number when comparing to 500 or is it still a string? I only ask because I see it printed out as less than 500 but yet not rebooting. – frakman1 Dec 26 '19 at 03:09
2

I had a similar problem, and while not wanting to question your question which is nice and specific, I have to say that you need a long term fix.

OOM killer kicks in because your server is running out of memory. Turning off OOM killer won't help with that - you'll still be out of memory and your server will eventually crash. Sure OOM killer doesn't always help, but turning it off won't either.

Rebooting your server will temporarily fix the problem, but it will happen again.

I had a similar issue with a server. Installing monit and configuring it to warn me when memory was running out allowed me to access the server when something was starting to happen, so I could properly diagnose it and solve it. I also added swap via a swapfile to increase the time I had to access the server while the problem was occurring.

In my case, my webserver was configured to start way too many spare servers for the load the server could handle. Once I figured out the root cause of the problem, I tackled it, and the server has not crashed since.

Community
  • 1
dunxd
  • 9,482
  • 21
  • 80
  • 117