0

IMPORTANT NOTE: I no longer need to restart the servers(my 3 VPS!) to have the pid file back in the dir /run/nginx.pid

I just typed the command: sudo fuser -k 443 / tcp && service nginx start

solves the problem for 24h, as the pid returns to /run/nginx/ without having to restart the service, and is always uptime, but every day the PID disappears! even though a nginx process is in progress and the server always uptime!

so I can use service nginx start and the pid returns to appear the monit returns the monitor the nginx process and I came back to have control over the commands: service nginx status, service nginx stop etc...

Attention, I do not say that nginx stops working, the sites continue oline, I say that the file: /run/nginx.pid disappears every day, so I can no longer use the nginx related commands, even me enter the command

sudo fuser -k 443/tcp

the absence of the file /run/nginx.pid brings several problems, the worst of them is I can not use the commands, service nginx status, start, stop etc..

========================= I need the nginx pid to work with the monit, the pid is simply disappearing unexpectedly, I have to restart the vps to get it back! every night this is happening! but even if he disappears, he's still running! the online site! but obviously I'm unable to monitor status, restart, stop ... etc ... I need pid! when I try to see the status via terminal I have

Error output on the terminal commandline:

service nginx status
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: en
   Active: failed (Result: exit-code) since Sun 2017-11-05 06:25:07 WET; 1h 50mi
  Process: 5062 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 -
  Process: 1527 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code
  Process: 1522 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process 
 Main PID: 1486 (code=exited, status=0/SUCCESS)

Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 faile
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 faile
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] still could not bind()
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Control process exited, code=exi
Nov 05 06:25:07 vps1 systemd[1]: Failed to start A high performance web server a
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Unit entered failed state.
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Failed with result 'exit-code'.
lines 1-18/18 (END)...skipping...
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: en
   Active: failed (Result: exit-code) since Sun 2017-11-05 06:25:07 WET; 1h 50mi
  Process: 5062 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 -
  Process: 1527 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code
  Process: 1522 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process 
 Main PID: 1486 (code=exited, status=0/SUCCESS)

Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 faile
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 faile
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] still could not bind()
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Control process exited, code=exi
Nov 05 06:25:07 vps1 systemd[1]: Failed to start A high performance web server a
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Unit entered failed state.
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Failed with result 'exit-code'.
~
~
~
~
~
~
lines 1-18/18 (END)...skipping...
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2017-11-05 06:25:07 WET; 1h 50min ago
  Process: 5062 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=1/FAILURE)
  Process: 1527 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
  Process: 1522 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 1486 (code=exited, status=0/SUCCESS)

Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 failed (98: Address already in use)
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 failed (98: Address already in use)
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] still could not bind()
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Control process exited, code=exited status=1
Nov 05 06:25:07 vps1 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Unit entered failed state.
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Failed with result 'exit-code'.

my /etc/nginx/nginx.conf:

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    worker_connections 1024;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
    server_names_hash_bucket_size 128;
client_max_body_size 100M;
client_body_buffer_size 16k;

fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
fastcgi_connect_timeout 180;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
}


#mail {
#   # See sample authentication script at:
#   # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
# 
#   # auth_http localhost/auth.php;
#   # pop3_capabilities "TOP" "USER";
#   # imap_capabilities "IMAP4rev1" "UIDPLUS";
# 
#   server {
#       listen     localhost:110;
#       protocol   pop3;
#       proxy      on;
#   }
# 
#   server {
#       listen     localhost:143;
#       protocol   imap;
#       proxy      on;
#   }
#}

Nginx Log: 

2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: still could not bind()
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: still could not bind()
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: still could not bind()
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:80 failed (98: Address already in use)

Ubuntu 16.04-64b / nginx 1.10.3 /php7.0-fpm

I trying just see status: service nginx status:

nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2017-11-05 09:00:51 WET; 3h 5min ago
  Process: 5062 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=1/FAILURE)
  Process: 8636 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
  Process: 8633 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 1486 (code=exited, status=0/SUCCESS)

Nov 05 09:00:50 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 09:00:50 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:80 failed (98: Address already in use)
Nov 05 09:00:50 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 09:00:51 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:80 failed (98: Address already in use)
Nov 05 09:00:51 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 09:00:51 vps1 nginx[8636]: nginx: [emerg] still could not bind()
Nov 05 09:00:51 vps1 systemd[1]: nginx.service: Control process exited, code=exited status=1
Nov 05 09:00:51 vps1 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Nov 05 09:00:51 vps1 systemd[1]: nginx.service: Unit entered failed state.
Nov 05 09:00:51 vps1 systemd[1]: nginx.service: Failed with result 'exit-code'.

If I restart the machine in force, reboot now (I just do this again), the nginx pid reappears(/run/nginx.pid), and monit returns to monitor the process:

https://imgur.com/a/IWfch

but of course no one may have to restart vps just to do a nginx status or nginx restart because there is no more nginx pid on linux server!

I really appreciate your help.

Ariane Martins Gomes Do Rego
  • 1
  • 1
  • 3
  • Another process is blocking ports 80 and 443. When the problem happens, enter `netstat -tulpan | egrep ':80|:443'` to find the processes that are blocking the ports. Depending on the outcome, dig further to troubleshoot. – Thomas Nov 05 '17 at 11:40
  • Thanks @Thomas.. my output: https://pastebin.com/1VhyQxfT – Ariane Martins Gomes Do Rego Nov 05 '17 at 11:46
  • There is running a `nginx` already. Did you try to restart `nginx`. Is the problem rather related to `monit`? – Thomas Nov 05 '17 at 11:54
  • No, the monit is ok, it just can not work with nginx, because the problem is the nginx pid disappearing from linux server !!, I have no control over nginx, I can not see status, restart etc .. I have to reboot the machine as I say in question :( `sudo find /run -iname nginx*.pid` without pid, I can't nothing .. I add in question output about service nginx status :( – Ariane Martins Gomes Do Rego Nov 05 '17 at 12:05
  • Hm....but the `nginx` server is running as you can see in the output of `netstat -tulpan ...` with the two PIDs 5080 and 5081. What do you plan to do with the PID? Normally on Ubuntu 16.04 */run/nginx.pid` does hold the process of the master process. You can also check the PIDs with `ps aux | grep nginx`. – Thomas Nov 05 '17 at 12:20
  • I updated again, see IMAGE, the path of the pid is correct(/run/nginx.pid), the same as you informed, is what I have always used, and I find this path with the find as I said above, the problem is that out of nowhere, this link to pid disappears in `/run/nginx.pid` and without that, I have to restart vps when I want to do something like nginx. – Ariane Martins Gomes Do Rego Nov 05 '17 at 12:29
  • @Thomas This path we are talking about `/run/nginx.pid` must exist, with `nano/run /nginx.pid` we can see the link to the process, with `sudo find/run -iname nginx * .pid` output : `/run/nginx.pid` happens, that every day this disappears and I can not do anything with nginx, so every day I have to restart the machine so he can come back! so that the service nginx restart, status ... return to work and I can find the file nginx.pid in linux with find again, neither I nor monit can find nginx.pid in linux server again! my uptime :-( https://imgur.com/a/vedTr – Ariane Martins Gomes Do Rego Nov 05 '17 at 12:44
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/68240/discussion-between-ariane-martins-gomes-do-rego-and-thomas). – Ariane Martins Gomes Do Rego Nov 05 '17 at 17:17
  • Why do you want a pid file? Why are you bothering with monit? – Michael Hampton Dec 17 '17 at 19:09
  • Thanks for your participation @Michael, yes, as I said in the post, the pid link is mandatory not only for monit, but for anything else I want to do, like service status, restart etc... without the pid nothing can be done regarding nginx. so I have to use the command I mention for the link(PID) to appear again. – Ariane Martins Gomes Do Rego Dec 17 '17 at 19:42
  • Eh? systemd doesn't require a PID file and can perfectly well monitor nginx itself, as well as restart nginx without ever using a PID file. It keeps track of the PIDs internally. – Michael Hampton Dec 17 '17 at 19:43
  • Please, tell me, how to I can doing this process? Because the file disappears between 24h and 72h is a concern, but if I have how to manage and monitor nginx automatically as monit does for me, without needing the PID, it would be a miracle for my life. – Ariane Martins Gomes Do Rego Dec 17 '17 at 19:56

1 Answers1

0

First try: pkill nginx && service nginx restart

If the file still gets deleted, then you can use ftrace to find which process is deleting your file:

trace-cmd record -e \*unlink\*

Wait for the file to be deleted, press CTRL+C to stop trace-cmd record ..., then run:

trace-cmd report

See also: https://serverfault.com/a/310477/30946

Also make sure that pid in /usr/lib/systemd/system/nginx.service or /etc/systemd/system/nginx.service files is pointing to the same file as in /etc/nginx/nginx.conf

Mircea Vutcovici
  • 16,706
  • 4
  • 52
  • 80
  • sorry for long delay. I always use: `pkill nginx OR fuser -k 443/tcp && service nginx START` to retrieve the PID nginx link back to /run/nginx/, do not even need to restart nginx, just start! The PID file link back to system, but disappears again from the system between 24h and 72h! I've already followed all its guidelines except the `trace-cmd` part, I'll do it and I'll come back to say, this will re-trace the process that are you deleting my link every day? or will it solve the problem so it will not be deleted? I've been even thinking about making it immutable, `chattr + i` Thankyou s.m. – Ariane Martins Gomes Do Rego Nov 27 '17 at 22:12
  • trace-cmd is to find what is deleting your file – Mircea Vutcovici Nov 28 '17 at 03:13
  • @Vutcovici Thanks a lot for your help, my main objective is to find out, even to be useful for future experience, two servers stopped with the disappearance of the PID file, a VPS still has the problem, so I was able to do what you taught:https://pastebin.com/ZDj1AV9M I'm taking a long time to come here to answer, because I keep monitoring the VPS to see the changes and reactions, waiting for reactions to the attempts that I make. I really want to figure this out and why the problem occurs! This is important for other future cases. – Ariane Martins Gomes Do Rego Dec 04 '17 at 11:39
  • It seems that only `fail2ban-server` deleted files. You can try to disable it for a few hours to see if this is indeed the root cause. If `fail2ban-server` is the problem, I would trace it using `strace -e file fail2ban......`. – Mircea Vutcovici Dec 05 '17 at 17:48
  • I think I already found the problem, I created two server blocks in my vhosts, with http and another one for https. and I had not commented the line list https in http block and the line list http in https block, I will be sure and to create a good explanation and to play in pastebin for you to see, I will take a week to do this, because I am doing tests. Thanks @Miscea – Ariane Martins Gomes Do Rego Dec 14 '17 at 19:06
  • Hi @Mircea Vutcovici I waited the necessary days to see if the problem disappeared from the link to the file would happen again, and it happened this morning, so I already had fal2ban stopped, so I noticed that the only difference between my VPS that this does not occur, for VPS this occurs, it is the use of two server block server {} please see my update above (do not need to analyze everything, I put a separator line in update, where there is new content). – Ariane Martins Gomes Do Rego Dec 17 '17 at 15:10
  • Thank you for the update. I was not expecting Nginx to do this. – Mircea Vutcovici Dec 18 '17 at 13:18