Nginx hangs using "service nginx start"

Question

I've compiled nginx with custom paths for my production server and when I try to start/restart the service using:

service nginx start

or

service nginx restart

It enters a new line without returning the shell:

So the problem is that I can't control nginx using service command. The service actually runs but it won't return a shell to me so I always have to press ctrl+c to get it back.

I also have to mention that nginx runs just fine when calling it by its own nginx command and stops or reloads easily using nginx -s stop/reload.

This problem persists using systemctl start nginx as well, but systemctl stop nginx works just fine.

Info:

$ lsb_release -a
    Distributor ID: Ubuntu
    Description:    Ubuntu 15.10
    Release:    15.10
    Codename:   wily

$ uname -r
    4.2.0-27-generic

$ nginx -V
    nginx version: nginx/1.9.11
    built by gcc 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2) 
    built with OpenSSL 1.0.2d 9 Jul 2015
    TLS SNI support enabled
    configure arguments: --sbin-path=/usr/bin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-debug --with-pcre --with-http_ssl_module

$ cat /etc/default/nginx
    NGINX_CONF_FILE=/etc/nginx/nginx.conf
    DAEMON=/usr/bin/nginx

$ cat /etc/init.d/nginx
    NGINX_BIN=/usr/bin/nginx
    test -x $NGINX_BIN || { echo "$NGINX_BIN not installed"; 
        if [ "$1" = "stop" ]; then exit 0;
        else exit 5; fi; }
    NGINX_PID=/var/run/nginx.pid

    # Check for existence of needed config file and read it
    #NGINX_CONFIG=/etc/sysconfig/nginx
    #test -r $NGINX_CONFIG || { echo "$NGINX_CONFIG not existing";
    #   if [ "$1" = "stop" ]; then exit 0;
    #   else exit 6; fi; }
    #
    # Read config   
    #. $NGINX_CONFIG

    # Source LSB init functions
    # providing start_daemon, killproc, pidofproc, 
    # log_success_msg, log_failure_msg and log_warning_msg.
    # This is currently not used by UnitedLinux based distributions and
    # not needed for init scripts for UnitedLinux only. If it is used,
    # the functions from rc.status should not be sourced or used.
    #. /lib/lsb/init-functions

    # Shell functions sourced from /etc/rc.status:
    #      rc_check         check and set local and overall rc status
    #      rc_status        check and set local and overall rc status
    #      rc_status -v     be verbose in local rc status and clear it afterwards
    #      rc_status -v -r  ditto and clear both the local and overall rc status
    #      rc_status -s     display "skipped" and exit with status 3
    #      rc_status -u     display "unused" and exit with status 3
    #      rc_failed        set local and overall rc status to failed
    #      rc_failed <num>  set local and overall rc status to <num>
    #      rc_reset         clear both the local and overall rc status
    #      rc_exit          exit appropriate to overall rc status
    #      rc_active        checks whether a service is activated by symlinks
    . /etc/rc.status

    # Reset status of this service
    rc_reset

    # Return values acc. to LSB for all commands but status:
    # 0   - success
    # 1       - generic or unspecified error
    # 2       - invalid or excess argument(s)
    # 3       - unimplemented feature (e.g. "reload")
    # 4       - user had insufficient privileges
    # 5       - program is not installed
    # 6       - program is not configured
    # 7       - program is not running
    # 8--199  - reserved (8--99 LSB, 100--149 distrib, 150--199 appl)
    # 
    # Note that starting an already running service, stopping
    # or restarting a not-running service as well as the restart
    # with force-reload (in case signaling is not supported) are
    # considered a success.

    case "$1" in
        start)
        echo -n "Starting nginx "
        ## Start daemon with startproc(8). If this fails
        ## the return value is set appropriately by startproc.
        /sbin/startproc -p $NGINX_PID $NGINX_BIN

        # Remember status and be verbose
        rc_status -v
        ;;
        stop)
        echo -n "Shutting down nginx "
        ## Stop daemon with killproc(8) and if this fails
        ## killproc sets the return value according to LSB.

        /sbin/killproc -p $NGINX_PID -TERM $NGINX_BIN

        # Remember status and be verbose
        rc_status -v
        ;;
        try-restart|condrestart)
        ## Do a restart only if the service was active before.
        ## Note: try-restart is now part of LSB (as of 1.9).
        ## RH has a similar command named condrestart.
        if test "$1" = "condrestart"; then
            echo "${attn} Use try-restart ${done}(LSB)${attn} rather than condrestart ${warn}(RH)${norm}"
        fi
        $0 status
        if test $? = 0; then
            $0 restart
        else
            rc_reset    # Not running is not a failure.
        fi
        # Remember status and be quiet
        rc_status
        ;;
        restart)
        ## Stop the service and regardless of whether it was
        ## running or not, start it again.
        $0 stop
        $0 start

        # Remember status and be quiet
        rc_status
        ;;
        force-reload)
        ## Signal the daemon to reload its config. Most daemons
        ## do this on signal 1 (SIGHUP).
        ## If it does not support it, restart the service if it
        ## is running.

        echo -n "Reload service nginx "
        ## if it supports it:
        /sbin/killproc -p $NGINX_PID -HUP $NGINX_BIN
        #touch /run/nginx.pid
        rc_status -v

        ## Otherwise:
        #$0 try-restart
        #rc_status
        ;;
        reload)
        ## Like force-reload, but if daemon does not support
        ## signaling, do nothing (!)

        # If it supports signaling:
        echo -n "Reload service nginx "
        /sbin/killproc -p $NGINX_PID -HUP $NGINX_BIN
        #touch /run/nginx.pid
        rc_status -v

        ## Otherwise if it does not support reload:
        #rc_failed 3
        #rc_status -v
        ;;
        reopen)
            echo -n "Reopen the logfiles "
            /sbin/killproc -p $NGINX_PID -USR1 $NGINX_BIN
            rc_status -v
            ;;

        status)
        echo -n "Checking for service nginx "
        ## Check status with checkproc(8), if process is running
        ## checkproc will return with exit status 0.

        # Return value is slightly different for the status command:
        # 0 - service up and running
        # 1 - service dead, but /run/  pid  file exists
        # 2 - service dead, but /var/lock/ lock file exists
        # 3 - service not running (unused)
        # 4 - service status unknown :-(
        # 5--199 reserved (5--99 LSB, 100--149 distro, 150--199 appl.)

        # NOTE: checkproc returns LSB compliant status values.
        /sbin/checkproc -p $NGINX_PID $NGINX_BIN
        # NOTE: rc_status knows that we called this init script with
        # "status" option and adapts its messages accordingly.
        rc_status -v
        ;;
        probe)
        ## Optional: Probe for the necessity of a reload, print out the
        ## argument to this init script which is required for a reload.
        ## Note: probe is not (yet) part of LSB (as of 1.9)

        test /etc/nginx/nginx.conf -nt /run/nginx.pid && echo reload
        ;;
        *)
        echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload|probe}"
        exit 1
        ;;
    esac
    rc_exit

UPDATE: The problem also persists while using Docker container on CoreOS alpha.

UPDATE 2: Here are the outputs for strace -o log -f service nginx start and journalctl -xe:

strace -o log -f service nginx start Log output [too long to post here]

---

    journalctl -xe
    Feb 26 07:25:38 lucifer polkitd(authority=local)[870]: Registered Authentication Agent for unix-process:8181:8813595 (system bus name :1.77 [/usr/bin/pkttyagent --notify-fd 5 --fallback], o
Feb 26 07:25:38 lucifer systemd[1]: Starting The NGINX HTTP and reverse proxy server...
-- Subject: Unit nginx.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit nginx.service has begun starting up.
Feb 26 07:25:38 lucifer nginx[8211]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Feb 26 07:25:38 lucifer nginx[8211]: nginx: configuration file /etc/nginx/nginx.conf test is successful
Feb 26 07:25:38 lucifer systemd[1]: nginx.service: PID file /var/run/nginx.pid not readable (yet?) after start: No such file or directory
Feb 26 07:25:43 lucifer polkitd(authority=local)[870]: Unregistered Authentication Agent for unix-process:8181:8813595 (system bus name :1.77, object path /org/freedesktop/PolicyKit1/Authen

Answer 1

I'm having the same issue with Ubuntu 16.04, systemd and NginX 1.10.1 from source.

I was using the default nginx.service file: https://www.nginx.com/resources/wiki/start/topics/examples/systemd/

The issue was the nginx.pid location. To fix it, I:

1. Fired up Nginx without the service

   sudo nginx start
   

2. Updated the locate db

   sudo updatedb
   

3. Found the location of the pid file

   locate "nginx.pid"
   

4. Updated the nginx.service file to the location where I found it

   PIDFile=/usr/local/nginx/logs/nginx.pid
   

   (No idea why it was stored in my logs dir...)

5. Then run daemon-reload to reload the nginx.service file

   systemctl daemon-reload
   

Afterwards, systemctl start nginx works like a charm. Hope this helps.

Answer 2

It hangs because of this error:

PID file /var/run/nginx.pid not readable (yet?) after start

Newer Linux distros comes with systemd. If you use a service bundled with your distro you'll get it already configured for systemd.

Since you are compiling nginx from the sources and you are using a SysV init file (/etc/init.d/nginx), systemd will use a generator to parse it (systemd-sysv-generator).

In your SysV script, you define the pid file and starts the process with:

NGINX_PID=/var/run/nginx.pid
...
/sbin/startproc -p $NGINX_PID $NGINX_BIN

If I'm not wrong you are using a SUSE Linux init script on Ubuntu (because of the startproc command), that startproc command only reads the pid file (specified by the -p parameter), it doesn't create it, thus systemd can't find a pid file and it hangs.

In your case, the solution is either to create the pid file in your SysV init script (on the /var/run/nginx.pid location), use an Ubuntu SysV init script, or a systemd one.

This may also happen (not what's happening to you) when you have a correct SysV init script which creates a pid file, but it's different from the one commented at the top of the file. The systemd generator reads the comments, for example this one:

# pidfile: /var/run/nginxd.pid

And uses the pidfile defined there.

More information:

• nginx init scripts for SysV init, systemd, etc.
• SysV generator processing the pidfile comment.
• How systemd reads SysV init scripts.
• SysV generator man page.
• startproc source code (SUSE FTP site).

Answer 3

It sounds like /usr/lib/systemd/system/nginx.service is malformed. Make sure this line is set:

PIDFile=/var/run/nginx.pid

Alternatively you can copy the [Service] section to a new file called /etc/systemd/system/nginx.service. Unit files placed in /etc/systemd/system allow you to override sections without duplicating the entire file installed by the package manager.

Answer 4

Inspiration https://serverfault.com/a/788576 and https://askubuntu.com/a/733604

Ubuntu 20.04

I changed two things

First: I runned service NetworkManager-wait

sudo systemctl enable NetworkManager-wait-online.service
sudo systemctl start NetworkManager-wait-online.service

Second: I added to path PIDFile (and pidfile) prefix var

[Service]
...
PIDFile=/var/run/nginx.pid
...
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /var/run/nginx.pid
..