Start N processes with one systemd service file

Question

I found this systemd service file to start autossh to keep up a ssh tunnel: https://gist.github.com/thomasfr/9707568

[Unit]
Description=Keeps a tunnel to 'remote.example.com' open
After=network.target

[Service]
User=autossh
# -p [PORT]
# -l [user]
# -M 0 --> no monitoring
# -N Just open the connection and do nothing (not interactive)
# LOCALPORT:IP_ON_EXAMPLE_COM:PORT_ON_EXAMPLE_COM
ExecStart=/usr/bin/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -p 22 -l autossh remote.example.com -L 7474:127.0.0.1:7474 -i /home/autossh/.ssh/id_rsa

[Install]
WantedBy=multi-user.target

Is there a way to configure systemd to start several tunnels in one service.

I don't want to create N system service files, since I want to avoid copy+paste.

All service files would be identical except "remote.example.com" would be replace with other host names.

1.5 year later ...

I asked this question roughly 1.5 year ago.

My mind has changed. Yes, it's nice, that you can do this with systemd, but I will use configuration-management in the future.

Why should systemd implement a template language and substitute %h? .. I think it makes no sense.

Several months later I think this looping and templating should be solved on a different level. I would use Ansible or TerraForm for this now.

In other words, you're saying use a configuration management system to generate multiple almost identical service files to accomplish this task? Hmmm, maybe. As with most such matters, there's not a clear dividing line separating these. — pgoetz, Jan 12 '18 at 17:46
@pgoetz config management is still new to me, but it has a benefit if you look at the topic of this question: If you look at the result of the config managment everybody who knows systemd service files will understand it:plain and simple service files. I think it makes more sense to learn and use a config management system since the knowledge can be used for all config in /etc, not just systemd. — guettli, Jan 14 '18 at 10:03
I resorted to using Supervisor http://supervisord.org to define multiple processes in one file. Some repetition there stays, that's how I arrived here. — fmalina, May 08 '20 at 18:46

score 55 · Accepted Answer · edited Mar 18 '17 at 21:34

Well, assuming that the only thing changing per unit file is the remote.example.com part, you can use an Instantiated Service.

From the systemd.unit man page:

Optionally, units may be instantiated from a template file at runtime. This allows creation of multiple units from a single configuration file. If systemd looks for a unit configuration file, it will first search for the literal unit name in the file system. If that yields no success and the unit name contains an "@" character, systemd will look for a unit template that shares the same name but with the instance string (i.e. the part between the "@" character and the suffix) removed. Example: if a service getty@tty3.service is requested and no file by that name is found, systemd will look for getty@.service and instantiate a service from that configuration file if it is found.

Basically, you create a single unit file, which contains a variable (usually %i) where the differences occur and then they get linked when you "enable" that service.

For example, I have a unit file called /etc/systemd/system/autossh@.service that looks like this:

[Unit]
Description=AutoSSH service for ServiceABC on %i
After=network.target

[Service]
Environment=AUTOSSH_GATETIME=30 AUTOSSH_LOGFILE=/var/log/autossh/%i.log AUTOSSH_PIDFILE=/var/run/autossh.%i.pid
PIDFile=/var/run/autossh.%i.pid
#Type=forking
ExecStart=/usr/bin/autossh -M 40000 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC %i

[Install]
WantedBy=multi-user.target

Which I've then enabled

[user@anotherhost ~]$ sudo systemctl enable autossh@somehost.example.com
ln -s '/etc/systemd/system/autossh@.service' '/etc/systemd/system/multi-user.target.wants/autossh@somehost.example.com.service'

And can intereact with

[user@anotherhost ~]$ sudo systemctl start autossh@somehost.example.com
[user@anotherhost ~]$ sudo systemctl status autossh@somehost.example.com
autossh@somehost.example.service - AutoSSH service for ServiceABC on somehost.example
   Loaded: loaded (/etc/systemd/system/autossh@.service; enabled)
   Active: active (running) since Tue 2015-10-20 13:19:01 EDT; 17s ago
 Main PID: 32524 (autossh)
   CGroup: /system.slice/system-autossh.slice/autossh@somehost.example.com.service
           ├─32524 /usr/bin/autossh -M 40000 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC somehost.example.com
           └─32525 /usr/bin/ssh -L 40000:127.0.0.1:40000 -R 40000:127.0.0.1:40001 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC somehost.example.com

Oct 20 13:19:01 anotherhost.example.com systemd[1]: Started AutoSSH service for ServiceABC on somehost.example.com.
[user@anotherhost ~]$ sudo systemctl status autossh@somehost.example.com
[user@anotherhost ~]$ sudo systemctl status autossh@somehost.example.com
autossh@somehost.example.com.service - AutoSSH service for ServiceABC on somehost.example.com
   Loaded: loaded (/etc/systemd/system/autossh@.service; enabled)
   Active: inactive (dead) since Tue 2015-10-20 13:24:10 EDT; 2s ago
  Process: 32524 ExecStart=/usr/bin/autossh -M 40000 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC %i (code=exited, status=0/SUCCESS)
 Main PID: 32524 (code=exited, status=0/SUCCESS)

Oct 20 13:19:01 anotherhost.example.com systemd[1]: Started AutoSSH service for ServiceABC on somehost.example.com.
Oct 20 13:24:10 anotherhost.example.com systemd[1]: Stopping AutoSSH service for ServiceABC on somehost.example.com...
Oct 20 13:24:10 anotherhost.example.com systemd[1]: Stopped AutoSSH service for ServiceABC on somehost.example.com.

As you can see, all instances of %i in the unit file get replaced with somehost.example.com.

There's a bunch more specifiers that you can use in a unit file though, but I find %i to work best in cases like this.

You don't show how to start automatically at boot, including of which ones to start. — Craig Hicks, Jul 12 '18 at 09:56
With Systemd, the `enable` action is what makes a unit/service start at boot. — GregL, Jul 12 '18 at 12:08
Yeah, that's what you're doing when you enable/disable them. — GregL, Oct 01 '18 at 12:13
I can get this to work, but I'm required to have a log file. I tried with `StandardOutput=file:/path/to/log/file`, but it's only logging one instance at a time. Is this possible, or will the logging have to be in the script itself instead of the unit file? — DevOpsSauce, Dec 09 '21 at 14:43

radtek · Answer 2 · 2019-05-08T14:35:55.540

Here is a python example, which was what I was looking for. The @ in the service filename lets you start N processes:

$ cat /etc/systemd/system/my-worker@.service

[Unit]
Description=manages my worker service, instance %i
After=multi-user.target

[Service]
PermissionsStartOnly=true
Type=idle
User=root
ExecStart=/usr/local/virtualenvs/bin/python /path/to/my/script.py
Restart=always
TimeoutStartSec=10
RestartSec=10

Various methods to call it

Enabling various counts for example:

Enable 30 workers:

sudo systemctl enable my-worker\@{1..30}.service

Enable 2 workers:

sudo systemctl enable my-worker\@{1..2}.service

Then be sure to reload:

sudo systemctl daemon-reload

Now you can start/stop then in various ways:

Start 1:

sudo systemctl start my-worker@2.service

Start Multiple:
```
sudo systemctl start my-worker@{1..2}
```
Stop Multiple:
```
sudo systemctl stop my-worker@{1..2}
```
Check status:
```
sudo systemctl status my-worker@1
```

UPDATE: To manage instances as one service, you can do something like this:

/etc/systemd/system/some-worker@.service:

[Unit]
Description=manage worker instances as a service, instance %i
Requires=some-worker.service
Before=some-worker.service
BindsTo=some-worker.service

[Service]
PermissionsStartOnly=true
Type=idle
User=root
#EnvironmentFile=/etc/profile.d/optional_envvars.sh
ExecStart=/usr/local/virtualenvs/bin/python /path/to/my/script.py
TimeoutStartSec=10
RestartSec=10

[Install]
WantedBy=some-worker.service

/usr/bin/some-worker-start.sh:

#!/bin/bash
systemctl start some-worker@{1..10}

/etc/systemd/system/some-worker.service:

[Unit]
Description=manages some worker instances as a service, instance

[Service]
Type=oneshot
ExecStart=/usr/bin/sh /usr/bin/some-worker-start.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

And now you can manage all instances with sudo systemctl some-worker (start|restart|stop)

Here is some boilerplate for your script.py:

#!/usr/bin/env python

import logging


def worker_loop():
    shutdown = False
    while True:

        try:
            if shutdown:
                break

            # Your execution logic here.
            # Common logic - i.e. consume from a queue, perform some work, ack message
            print("hello world")

        except (IOError, KeyboardInterrupt):
            shutdown = True
            logging.info("shutdown received - processing will halt when jobs complete")
        except Exception as e:
            logging.exception("unhandled exception on shutdown. {}".format(e))


if __name__ == '__main__':
    worker_loop()

@radek: Two things I do not understand: First, %i is only used in the description of the unit file. How does the start command know what to start? Second, how does `systemctl some-worker (start|restart|stop)` know which instances to work on? — U. Windl, Mar 20 '19 at 13:31
%i is the output from @ in the name of the service file. Second part is already explained in the answer, see `Now you can start/stop then in various ways`. — radtek, Mar 20 '19 at 20:03
I think his answer is incomplete without the scripts being involved. Most "magic" is done inside the scripts which are missing. — U. Windl, May 07 '19 at 12:43
I have provided a full working solution here actually. Which "scripts" are you referring to? /path/to/my/script.py can be whatever you want, a "hello world" if you want. Something that will stay up running until it receives a kill signal. Please note the question is not specific to python. — radtek, May 08 '19 at 14:36
@radtek, how does this work say if one of the instance was masked or disabled? I want to restart all instances except the ones that were masked/disabled. Thanks. — D.prd, Jan 20 '21 at 09:23
@D.prd not sure but try it and let me know how it works. I assume the disabled service wouldn't start if its disabled. — radtek, Jan 20 '21 at 21:24
@radtek yes, I tested it out. The masked instance won't start (which is correct) but the parent service (some-worker in your example) shows a status failure even though all dependent units except the masked one are restarted successfully. Is it possible for the parent service (some-worker.service) to report a failure only and only if *all* of its dependent %i instances fail? Thanks. — D.prd, Jan 21 '21 at 05:40

score 2 · Answer 3 · answered Mar 23 '17 at 05:21

GregL's answer helped me a great deal. Here is an example of a unit template I used in my code using the example above for a gearman job server. I made a shell script that lets me create X amount of "workers" using this one template.

[Unit]
Description=az gearman worker
After=gearman-job-server.service

[Service]
PIDFile=/var/run/gearman_worker_az%i.pid
Type=simple
User=www-data
WorkingDirectory=/var/www/mysite.com/jobs/
ExecStart=/usr/bin/php -f gearman_worker_az.php > /dev/null 2>&1
Restart=on-success
KillMode=process

[Install]
WantedBy=multi-user.target

score -1 · Answer 4 · answered Mar 20 '20 at 16:31

I have searched for solution to a similar task and actually have found one, that i believe is easier to accomplish, but is supposed to be hacky. (and is not mentioned here)

I needed to create multiple ssh connections after vpn creates a tunnel, so i have created a service that is dependent on tun device and calls shell script with appropriate commands.

Service /etc/systemd/system/ssh_tunnel.service:

[Unit]
Description=Reverse SSH Service to access hidden services
ConditionPathExists=|/usr/bin
Wants=sys-devices-virtual-net-tun0.device
After=network.target sys-devices-virtual-net-tun0.device

[Service]
Type=forking
ExecStart=/bin/sh /etc/openvpn/ssh_tunnels.sh 
RemainAfterExit=yes
TimeoutSec=0
GuessMainPID=no

[Install]
WantedBy=multi-user.target

/etc/openvpn/ssh_tunnels.sh:

!/bin/bash
#sleep 15

echo 'Tunelling some ports'
killall -HUP ssh

su - user -c 'ssh -f admin@172.171.1.1 -p 9999 -L :3690:svn.newbox.ru:3690 -L :8888:10.1.20.55:80 -L :8181:10.1.10.10:80 -N -vvv'

ssh -i /home/user/.ssh/id_rsa -f admin@172.171.1.1 -p 9999 -L :587:mail.domain.ru:587 -L :995:mail.newbox.ru:995 -L :22:10.1.2.1:22 -N -vvv &

exit 0

Result:

# systemctl status ssh_tunnel.service
● ssh_smartex.service - Reverse SSH Service to access hidden services
     Loaded: loaded (/etc/systemd/system/ssh_tunnel.service; enabled; vendor preset: disabled)
     Active: active (running) since Fri 2020-03-20 16:01:07 UTC; 22min ago
    Process: 156 ExecStart=/bin/sh /etc/openvpn/ssh_tunnel.sh (code=exited, status=0/SUC>
      Tasks: 2 (limit: 4915)
     Memory: 3.8M
     CGroup: /system.slice/ssh_tunnel.service
             ├─166 ssh -f admin@172.171.1.1 -p 9999 -L :3690:svn.newbox.ru:3690 -L :8888:10.1.20.55:80 ->
             └─168 ssh -i /home/user/.ssh/id_rsa -f admin@172.171.1.1 -p 9999 -L :587:mail.newbox.ru:5>
...

However, i haven't checked yet, how it survives vpn restart, but it's another topic.

Start N processes with one systemd service file

1.5 year later ...

4 Answers4

Various methods to call it

Linked

Related