37

Is there any way supervisord can automatically restart a failed/exited/terminated job and send me a notification email with a dump of the last x lines of log file?

masegaloeh
  • 17,978
  • 9
  • 56
  • 104
Sebastian Hoitz
  • 3,019
  • 3
  • 23
  • 19

2 Answers2

52

There is a plugin called superlance.

You install it with pip install superlance or download it at: http://pypi.python.org/pypi/superlance

The next thing you do is you go into your supervisord.conf and add the following lines:

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -a -m email1@example.com
events=PROCESS_STATE

This should be followed by a "supervisorctl update". When a process "exits" you will now get a notification sent to email1@example.com.

If you only want to listen to some selected apps you can exchange the -a for a -p program1 or if it is a group group1:program2 One example would be

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -p program1 -p group1:program2  -m email1@example.com
events=PROCESS_STATE

Regarding the automatic restart: you should make sure that autorestart is set to true (it is set to unexpected by default). This way the package will be restarted 3 times. If after that it still exits, it gives up, but you can change that with startretries.

Example program:

[program:cat]
command=/bin/cat
autorestart=true
startretries=10
Community
  • 1
Thomaschaaf
  • 3,012
  • 5
  • 29
  • 24
  • Despite being inactive for a few years this thread is still valid - I just tested superlance 1.0.0 (dated oct 2016) with supervisor 3.1.4 (CentOS7) and the crashmail worked just fine. – David Ramirez Oct 04 '18 at 19:48
  • How do we test it? – Curious Developer Jul 17 '20 at 14:43
  • If you have it configured correctly, you should see `crashmail` as process when running `supervisorctl`. You can test the mail by doing `ps aux | grep programthatshouldnotcrash`, then killing that process with `kill -9 123456789` where `123456789` is the pid of the program that is currently running via supervisor. Note: The documentation says to listen to `PROCESS_STATE_EXITED` instead of `PROCESS_STATE`. – Sumurai8 Sep 04 '20 at 13:09
  • 1
    And, where do i setup the SMTP for sending the email? – Jagesh Maharjan Aug 01 '21 at 14:23
0

I tried installing superlance and running crashmail like this:

sudo apt-get install python-pip
sudo pip install superlance

after i do:

sudo nano /etc/supervisor/supervisord.conf

and after i added:

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -a -m mymail@mail.fr
events=PROCESS_STATE

and I do not receive anything....

My crashmail file is:

#!/usr/bin/python

-- coding: utf-8 --
import re
import sys

from superlance.crashmail import main

if name == 'main':
sys.argv[0] = re.sub(r'(-script.pyw?|.exe)?$', '', sys.argv[0])
sys.exit(main())
RalfFriedl
  • 3,008
  • 4
  • 12
  • 17
Ben Pgm
  • 1