Amazon Linux Init.d Script Not Daemonizing Properly

Question

I'm working on Amazon Linux and am running into an issue with my init.d script. I'm trying to run Kibana 4, which is just a script that calls nodejs to run as a service, but it's not working.

Typing service kibana status works fine, but whenever I type service kibana start the command just hangs (i.e. doesn't run in the background). When I Ctrl-C it, I get a [FAILED] response, but the process has been created and is running.

When I look in my /var/run/ directory for kibana.pid I don't see anything, even though the process is actually running. Can anyone help me understand what I'm doing wrong?

I have a feeling I'm not using daemon correctly...

#!/bin/bash
#
#       /etc/init.d/kibana

# Set defaults.
exec="/kibana/kibana-{{ kibana.version }}-linux-x64/bin/kibana > /var/log/kibana/kibana.log"
prog="kibana"
pid_file="/var/run/${prog}.pid"
lock_file="/var/lock/subsys/$prog"
kibana_user="kibana"

# Execute init.d functions
. /etc/rc.d/init.d/functions

case "$1" in
    start)
        echo -n "Start Kibana: "
        daemon --user $kibana_user --pidfile $pid_file $exec
        retval=$?
        echo
        [ $retval -eq 0 ] && touch $lock_file
        ;;
    stop)
        echo -n "Stopping Kibana: "
        su $kibana_user -c "kill `cat $pid_file`"
        retval=$?
        [ $retval -eq 0 ] && rm -f $lock_file
        echo "OK"
        ;;
    status)
        status -p $pid_file kibana
        exit $?
        ;;
    *)
        echo "Usage: `basename $0` start|stop|status"
        exit 1
esac

exit 0

By the way, here's the kibana file I'm trying to execute: https://github.com/elasticsearch/kibana/blob/master/src/server/bin/kibana.sh — Josh Padnick, Feb 20 '15 at 01:29
Amazon Linux is generally not compatible with CentOS, and so I've removed all the CentOS references to avoid confusion. — Michael Hampton, Feb 20 '15 at 01:49
Thanks, Michael. Unfortunately, doesn't look like theres much interest in answering a CentOS init.d question. — Josh Padnick, Feb 20 '15 at 19:19
I would just advise you to not use Amazon Linux for anything important...or anything at all. It's too unstable. — Michael Hampton, Feb 20 '15 at 19:22
I hear you, but this is for a project where that decision can't be reversed. Any thoughts in general on how I would debug this? — Josh Padnick, Feb 20 '15 at 21:28
Well, I can't help you with office politics, which is the only thing that could cause "that decision can't be reversed"... — Michael Hampton, Feb 20 '15 at 21:31

Josh Padnick · Accepted Answer · 2015-02-23T01:31:42.363

I figured out that the issue was that daemon was not playing nicely with exec. I updated my code to call kibana directly and added & at the end to get it to daemonize.

Update:

Unfortunately, I can't copy the full code I used, but I can tell you that I based it on https://github.com/chovy/node-startup. I basically forked this code and made the customizations that made sense for my setup.

Here's the key function in my init.d script:

KIBANA_EXEC="/path/to/my/kibana/bin"
LOG_FILE="/path/to/kibana/log/file"
PID_FILE="/path/to/kibana/pid/file"

...

start_it() {
    mkdir -p "$PID_DIR"
    mkdir -p "$LOG_DIR"

    # This is an artifact of the way I set things up and is a hacky solution to
    # fix a race condition, so copy with caution
    sleep 10

    echo "Starting Kibana..."
    $KIBANA_EXEC 1>"$LOG_FILE" 2>&1 &
    echo $! > "$PID_FILE"
    echo "Kibana started with pid $!"
}

Yes, doesn't look like there's a command-line option to make kibana fork and return, ie run as a daemon so it has to be backgrounded in bash with '&'. Possibly `nohup` or `disown` would help as well. Could you maybe show your code changes for the benefit of anyone with a similar problem? — Cedric Knight, Feb 22 '15 at 23:09

score 0 · Answer 2 · edited Jun 19 '15 at 10:24

I've just been going through this issue and created a solution based on Josh's above. It get's around the 'sleep 10' issue by using a chkconfig header that ensures that the script waits for suitable services to be available before attempting to start. Heres my init.d script:

#!/bin/bash
#
# Kibana    Init script for Kibana
#
# chkconfig: 345 99 76
# processname: kibana
#
KIBANA_EXEC="/opt/kibana/bin/kibana"
now=$(date +"%Y-%m-%d-%S")
LOG_FILE="/opt/kibana/bin/log/kibana.$now.log"
PID_FILE="/opt/kibana/bin/log/kibana.$now.pid"
RETVAL=0

start() {
    echo "Starting Kibana..."
    $KIBANA_EXEC 1>"$LOG_FILE" 2>&1 &
    echo $! > "$PID_FILE"
    echo "Kibana started with pid $!"
}

case "$1" in
    start)
    start
    ;;
    *)
        echo "Usage: $0 {start}"
        exit 0
    ;;
esac
exit $RETVAL

You'll then need to update the script permissions and register the service (don't forget to create a suitable logging directory and set the correct paths for kibana, log and pid location etc:

sudo chmod 0755 /etc/init.d/kibana
sudo chkconfig kibana on

Amazon Linux Init.d Script Not Daemonizing Properly

2 Answers2