39

On Windows, you can set what should happen if/when a service fails. Is there a standard way of achieving the same thing on Linux (CentOS in particular)?

A bigger part of my question is: how do you handle sockets that have been left open - for example in TIME_WAIT, FIN_WAIT1, etc states.

At the moment if the service I am developing crashes, I have to wait for the sockets to clear or change the listen port before I can then manually restart it.

Thanks for your help.

Pryo
  • 615
  • 1
  • 5
  • 11

4 Answers4

34

Only answering the service restart part. I came across Monit as well, but on CentOS 7 systemd takes care of all that for you. You just need to add these two lines to the .service file (if they're not there already):

Restart=always
RestartSec=3

Run man systemd.service for reference.

If you want to create a custom systemd service, it's pretty straightforward to write your own service file. See the example below, for a custom http server.

Start the editor with a new service file:

vim /etc/systemd/system/httpd.service

And add the following content, which you can edit as required:

[Unit]
Description=My httpd Service
After=network.target

[Service]
Type=simple
User=root
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PERLLIB=/perl
ExecStart=/bin/httpd /etc/httpd.conf
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

I want it to start automatically on boot:

systemctl enable httpd

Tell systemd about the changes and start the service:

systemctl daemon-reload
systemctl start httpd

And now you can see the status:

systemctl status httpd

There are many options for Restart. The following is an excerpt from the man page:

Restart=

Configures whether the service shall be restarted when the service process exits, is killed, or a timeout is reached...

Takes one of no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, or always. If set to no (the default), the service will not be restarted. If set to on-success, it will be restarted only when the service process exits cleanly. In this context, a clean exit means any of the following:

• exit code of 0;

• for types other than Type=oneshot, one of the signals SIGHUP, SIGINT, SIGTERM, or SIGPIPE;

• exit statuses and signals specified in SuccessExitStatus=.

If set to on-failure, the service will be restarted when the process exits with a non-zero exit code, is terminated by a signal (including on core dump, but excluding the aforementioned four signals), when an operation (such as service reload) times out, and when the configured watchdog timeout is triggered.

The restart can be tested by sending an appropriate signal to the application. See the manual at man kill for details.

Nagev
  • 528
  • 1
  • 5
  • 9
  • Yes, exactly, `systemd`, the standard service manager on *most* popular distributions, can do that for you. – Rolf Apr 08 '19 at 12:53
  • Also note that you can override anything in a system-installed service file by placing your own .conf file containing the lines to add/replace into e.g. `/etc/systemd/system/httpd.service.d`. – hackel Jun 18 '20 at 21:12
  • How do I test that this setting is actually working? I have just found that my previous setting `Restart=on-failure` `RestartSec=5s` did not restart my service ... – Rho Phi Sep 15 '22 at 11:36
  • It could be that exit was not considered a failure by `systemd`, e.g. a clean exit code. To test, you could simulate a failure condition. Updated answer with additional details. – Nagev Sep 15 '22 at 14:53
29

monit is a great way to monitor and restart services when they fail--and you'll probably end up using this for other essential services (such as Apache). There's a nice article on nixCraft detailing how to use this for services specifically, although monit itself has many more functions beyond this.

As for the socket aspect, @galraen answered this spot on.

Andrew M.
  • 10,982
  • 2
  • 34
  • 29
  • It's a shame I have to decide which is _the_ answer. I should have asked two seperate questions. @gelraen your answer just ended my weeks of searching for a solution. Thanks you *so* much! @Redmumba thanks, Monit does look good! – Pryo Mar 26 '11 at 14:01
  • Whichever one you decide to mark correct, definitely *upvote* @gelraen's answer. Its spot on correct, and very informative. – Andrew M. Mar 26 '11 at 16:15
13

You can call setsockopt(2) for listening socket with SO_REUSEADDR, so you will be able to bind(2) it again without waiting for expiring all connections. Another possibility: drop connections from kernel. FreeBSD have tcpdrop command for this, don't know about Linux.

gelraen
  • 2,311
  • 20
  • 19
4

If your linux distro uses Upstart instead of SysV init, then the respawn keyword does this.

http://upstart.ubuntu.com/cookbook/#respawn

JeffG
  • 1,184
  • 6
  • 18
  • Thanks, this looks like just the thing, but unfortunately I'm stuck with SysV init. – Pryo Mar 26 '11 at 14:07