Systemd not starting an app from wrapper

Question

I've created a simple wrapper for Elasticsearch and a systemd service file, and for some reason I am not able to start Elasticsearch from systemd, however I'm able to do so from the wrapper itself.

Here's my (very simple) wrapper:

#!/bin/sh
SERVICE_NAME=elasticsearch
PATH_TO_APP="/opt/$SERVICE_NAME/bin/$SERVICE_NAME"
PID_PATH_NAME="/var/run/$SERVICE_NAME/$SERVICE_NAME.pid"
SCRIPTNAME=elasticsearch-wrapper.sh
ES_USER=$SERVICE_NAME
ES_GROUP=$SERVICE_NAME
SUDO="sudo -u $SERVICE_NAME"

case $1 in
    start)
        echo "Starting $SERVICE_NAME ..."
        if [ ! -f $PID_PATH_NAME ]; then
        mkdir $(dirname $PID_PATH_NAME) > /dev/null 2>&1 || true
            chown $ES_USER $(dirname $PID_PATH_NAME)
            $SUDO $PATH_TO_APP -d -p $PID_PATH_NAME
        echo "Return code: $?"
            echo "$SERVICE_NAME started ..."
        else
            echo "$SERVICE_NAME is already running ..."
        fi
    ;;
    stop)
        if [ -f $PID_PATH_NAME ]; then
            PID=$(cat $PID_PATH_NAME);
            echo "$SERVICE_NAME stopping ..."
            kill -15 $PID;
            echo "$SERVICE_NAME stopped ..."
            rm $PID_PATH_NAME
        else
            echo "$SERVICE_NAME is not running ..."
        fi
    ;;
    restart)
        if [ -f $PID_PATH_NAME ]; then
            PID=$(cat $PID_PATH_NAME);
            echo "$SERVICE_NAME stopping ...";
            kill -15 $PID;
        sleep 1;
            echo "$SERVICE_NAME stopped ...";
            rm -rf $PID_PATH_NAME
            echo "$SERVICE_NAME starting ..."
            mkdir $(dirname $PID_PATH_NAME) > /dev/null 2>&1 || true
            chown $ES_USER $(dirname $PID_PATH_NAME)
            $SUDO $PATH_TO_APP -d -p $PID_PATH_NAME
            echo "$SERVICE_NAME started ..."
        else
            echo "$SERVICE_NAME is not running ..."
        fi
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|restart}" >&2
    exit 3
    ;;
esac

And here's my elasticsearch.service file:

[Unit]
Description=ElasticSearch Server
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
#ExecStart=/opt/elasticsearch/bin/elasticsearch -d
ExecStart=/opt/elasticsearch/bin/elasticsearch-wrapper.sh start
ExecStop=/opt/elasticsearch/bin/elasticsearch-wrapper.sh stop
ExecReload=/opt/elasticsearch/bin/elasticsearch-wrapper.sh restart
#ExecStop=/opt/elasticsearch/bin/elasticsearch-stop.sh
LimitNOFILE=65536
LimitMEMLOCK=infinity
User=root
StandardOutput=journal+console

This is the result of systemd status elasticsearch after attempting to run it:

● elasticsearch.service - ElasticSearch Server
   Loaded: loaded (/etc/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

Mar 23 17:38:30 ip-10-96-13-64 systemd[1]: Started ElasticSearch Server.
Mar 23 17:38:30 ip-10-96-13-64 elasticsearch-wrapper.sh[2866]: Starting elasticsearch ...
Mar 23 17:38:30 ip-10-96-13-64 sudo[2872]:     root : unable to resolve host ip-10-96-13-64
Mar 23 17:38:30 ip-10-96-13-64 elasticsearch-wrapper.sh[2866]: sudo: unable to resolve host ip-10-96-13-64
Mar 23 17:38:30 ip-10-96-13-64 sudo[2872]:     root : TTY=unknown ; PWD=/ ; USER=elasticsearch ; COMMAND=/opt/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch.pid
Mar 23 17:38:30 ip-10-96-13-64 sudo[2872]: pam_unix(sudo:session): session opened for user elasticsearch by (uid=0)
Mar 23 17:38:30 ip-10-96-13-64 sudo[2872]: pam_unix(sudo:session): session closed for user elasticsearch
Mar 23 17:38:30 ip-10-96-13-64 elasticsearch-wrapper.sh[2866]: Return code: 0
Mar 23 17:38:30 ip-10-96-13-64 elasticsearch-wrapper.sh[2866]: elasticsearch started ...
Mar 23 17:38:30 ip-10-96-13-64 elasticsearch-wrapper.sh[2895]: elasticsearch is not running ...

I've put a debug statement that returns the "return code" when starting elasticsearch and as you can see in the systemctl status it is returning 0. I've even tailed elasticsearch's logs immediately after executing systemctl start elasticsearch and nothing as in literally nothing in the logs; like it didn't even start elasticsearch.

I'm not sure why this is happening, so as a last resort I'm posting it here to see if anyone can help me out and point out what's going on here. To reiterate, starting/stopping/restarting elasticsearch from the wrapper directly works just fine with no problems, but with systemd, it doesn't work.

EDIT: The reason why I'm using the wrapper because, according to their documentation, you have to do kill the process. Also, I'd rather to have also the the ExecReload option.

EDIT2: To confirm that that there's no stop option in elasticsearch, here's the output of elasticsearch --help

USER# ./elasticsearch --help
starts elasticsearch

Option                Description                                              
------                -----------                                              
-E <KeyValuePair>     Configure a setting                                      
-V, --version         Prints elasticsearch version information and exits       
-d, --daemonize       Starts Elasticsearch in the background                   
-h, --help            show help                                                
-p, --pidfile <Path>  Creates a pid file in the specified path on start        
-q, --quiet           Turns off standard ouput/error streams logging in console
-s, --silent          show minimal output                                      
-v, --verbose         show verbose output

Why do you have a wrapper script at all? Everything necessary can be placed directly in the systemd unit. — Michael Hampton, Mar 23 '17 at 21:31

score 0 · Accepted Answer · answered Mar 23 '17 at 18:24

Okay, so building on the information that iwaseatenbyagrue provided, Adding PIDFile worked only when I changed the Type from simple to forking, so now it works as expected!

Here's how my elasticsearch.service looks like now:

[Unit]
Description=ElasticSearch Server
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target

[Service]
Type=forking
#ExecStart=/opt/elasticsearch/bin/elasticsearch -d
ExecStart=/opt/elasticsearch/bin/elasticsearch-wrapper.sh start
ExecStop=/opt/elasticsearch/bin/elasticsearch-wrapper.sh stop
ExecReload=/opt/elasticsearch/bin/elasticsearch-wrapper.sh restart
#ExecStop=/opt/elasticsearch/bin/elasticsearch-stop.sh
PIDFile=/var/run/elasticsearch/elasticsearch.pid
LimitNOFILE=65536
LimitMEMLOCK=infinity
User=root
StandardOutput=journal+console

iwaseatenbyagrue · Answer 2 · 2017-03-23T18:01:03.603

In most cases (and in the case of your unit file), systemd expects to be tracking a programme's PID directly.

Adding a wrapper means systemd sees the wrapper's PID, but cannot track ES's PID.

You can see this in your output:

Mar 23 17:38:30 ip-10-96-13-64 elasticsearch-wrapper.sh[2866]: Return code: 0

Your wrapper is exiting cleanly, and so systemd concludes nothing is running.

I would strongly recommend you move any necessary set-up functions to use systemd's ExecStartPre, and run ES directly from systemd.

If you absolutely must use a wrapper, then you should ensure ES writes a PID file, and provide systemd the path to it, using PIDFile.

However, I think you are better off having something like:

[Unit]
Description=ElasticSearch Server
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=elasticsearch
Group=elasticsearch
ExecStartPre=-mkdir /opt/elasticsearch/data
ExecStart=/opt/elasticsearch/bin/elasticsearch OPTIONS
ExecStop=/opt/elasticsearch/bin/elasticsearch stop
LimitNOFILE=65536
LimitMEMLOCK=infinity
StandardOutput=journal

Ah, that makes sense, I'll try adding `PIDFile` to the systemd service file and report back.. And about the `chown`, the wrapper isn't finished and that's going to change for sure.. — Fadi, Mar 23 '17 at 18:00
`PIDFile` doesn't seem to make it work.. Also the service file from your edited answer doesn't work because there's no `stop` option in elasticsearch's executable file, and also I'd like to have a `ExecReload` option too, hence why I created the wrapper. — Fadi, Mar 23 '17 at 18:07

Systemd not starting an app from wrapper

2 Answers2