How to automatically restart a linux background process if it fails?

Question

I have a process which is executed by init.d script on the background. Eg:

case "$1" in 
    start)
       /bin/myprocess &
    stop)
       killall myprocess
    restart)
       killall myprocess
       /bin/myprocess &
esac

In certain conditions, myprocess can fail and return. Is there any (standard) way how to detect its fail and restart in automatically?

Answer 1

The easiest way would be to add it to /etc/inittab, which is designed to do this sort of thing:

respawn If the process does not exist, start the process. Do not wait for its termination (continue scanning the /etc/inittab file). Restart the process when it dies. If the process exists, do nothing and continue scanning the /etc/inittab file.

For example, you could do this:

# Run my stuff
myprocess:2345:respawn:/bin/myprocess

Answer 2

Buildroot has three possible init systems, so there are three ways to do this:

BusyBox init

With this, one adds an entry to /etc/inittab.

::respawn:/bin/myprocess

Note that BusyBox init has an idiosyncratic /etc/inittab format. The second field is meaningless, and the first field is not an ID but a device basename.

Linux "System V" init

Again, one adds an entry to /etc/inittab.

myprocess:2345:respawn:/bin/myprocess

systemd

One writes a unit file in, say, /etc/systemd/system/myprocess.service:

[Unit]
Description=My Process

[Service]
ExecStart=/bin/myprocess
Restart=always

[Install]
WantedBy=multi-user.target

Enable this to autostart at bootup with:

systemctl enable myprocess.service

Start it manually with:

systemctl start myprocess.service

Further reading

• "3.1.3 init system". The Buildroot user manual.

Answer 3

What about creating a subshell with a loop that calls constantly the same process?

If it ends, the next iteration of the loop goes on and starts it again.

(while true; do 
    /bin/myprocess
done) &

If the subshell dies, it's over though. The only possibility in that case would be to create another process (I'll call it necromancer) that checks whether yourprocess is alive, start it if it isn't and run this necromancer with cron, so that you can check that regularly.

Next step would be wondering what could happen if cron dies, but at some point you should feel safe and stop worrying.

Answer 4

You could make use of Monit . It's really easy to use and quite flexible. See for example this configuration for restarting the Tomcat process on failure.

check process tomcat with pidfile /var/run/tomcat.pid
   start program = "/etc/init.d/tomcat start"
   stop  program = "/etc/init.d/tomcat stop"
   if failed port 8080 type tcp then restart

It also has a lot of configuration examples for many use cases.

Answer 5

If you're not a super user or root, and if your Linux system has Docker installed, then you can create a docker image of your process, using docker to restart your process if the system is rebooted.

File: docker-compose.yml

version: "3"
services:
  lserver:
    image: your_docker_image:latest
    ports:
    - 8080:8080   # just use 8080 as an example
    restart: always  # this is where your process can be guaranteed to restart

To start your docker container,

docker-compose up -d

I find it's easy to handle my-own-process with auto-restart if I am not a super user of the system.

For a sample example of how to create a docker image, here is a quick example:

File: Dockerfile

FROM alpine:3.5

RUN apk update && apk upgrade && rm -rf /var/cache/apk/*
WORKDIR /app
COPY my-process-server /app
RUN ln -s /app/my-process-server /usr/local/bin/my-process-server

EXPOSE 8080

CMD ["my-process-server"]

Answer 6

You may use restarter

start)
   restarter -c /bin/myprocess &
stop)
   pkill -f myprocess

On newer systems use systemd which solves all those trivial issues

Answer 7

In my case, as a quick-fix, I modified and used the solution of @Trylks to wrap the program I was launching. I wanted it to end only on clean exit.

Should run in most shells:

#!/bin/sh

echo ""
echo "Use: $0 ./program"
echo ""

#eg="/usr/bin/apt update"

echo "Executing $1 ..."

EXIT_CODE=1
(while [ $EXIT_CODE -gt 0 ]; do
    $1
    # loops on error code: greater-than 0
    EXIT_CODE=$?
done) &