1

I have a memory leak on one of my apps being served with nginx + passenger and a min/max_instances 2 setting. Memory keeps growing up as time passes by, and this can be a problem on a machine with limited memory (eg Amazon Web Services) that can end up crashing badly.

Is there a way / trick to restart some passenger processes of the same app, either periodically or when memory reaches a certain threshold ?

Suppose I have an application group A that can start up to 2 processes A1 and A2. I'd like to restart A1 and A2 one at a time to ensure service continuity (this is not a zero-downtime deployment since it relies on the same unmodified app, but some way to purge the memory leak and restart a fresh process until I figure out the problem).

Eg suppose I have the following state:

Requests in queue: 0

  * PID: 11124   Sessions: 0       Processed: 14638   Uptime: 1d 12h 31m 29s
    CPU: 0%      Memory  : 341M    Last used: 24s
  * PID: 11131   Sessions: 0       Processed: 9323    Uptime: 1d 12h 31m 29s
    CPU: 0%      Memory  : 389M    Last used: 24s

Process n°2 starts taking up a lot of memory (~400) so I'd like to restart it (and only this process) which should bring it back to 100MB ish, and I'd do the same with Process n°1 later

The memory leak takes some time to build and it would be best if I could restart only when detecting too high memory usage, but if I can schedule a process restart every night or so this could also work totally fine (eg restart process n°1 at 3am, then process n°2 at 4am)

Is there any signal I can send to this passenger group that would let me restart processes one by one ?

Cyril Duchon-Doris
  • 242
  • 2
  • 15

2 Answers2

1

If this is your production application, I'd suggest you go with passenger enterprise. There is a nice feature you can add to the nginx config:

passenger_memory_limit 1024;

Passenger will reload the process when the memory hits 1024 or whatever number you want to specify. You can also take advantage of multi-threading since passenger standalone is single threaded.

Andres Barcenas
  • 11
  • 1
1

This is may be a little bit to late for the OP, but we have the following cron-script running on our AWS hosts every minute:

#!/bin/bash
let "TARGET = 400 * 1024" # = 400 MB
ps aux --sort -rss | grep "Passenger RackApp" | while read -r LINE; do
        read PID MEMORY <<< $(echo $LINE | awk '{ print $2; print $6 }')
        if [ $MEMORY -gt $TARGET ]
        then
                `kill -15 $PID`
        fi
done
exit 0

Depending on the configuration, passenger will immediately re-spawn the stopped process.

pensan
  • 111
  • 2