My server setup is nginx directly connects to a node.js server (nginx and node.js are in the same node and nginx is forwarding request to node.js: 127.0.0.1:8000). The symptom is sometimes there are some 504 logs in nginx log. And node.js log doesn't show any sign of ever receiving the request. I then enabled tcp log using iptables, which logs all tcp packets to/from port 8000. After checking the tcp log, it seems that nginx was trying to establish a tcp connection with node.js server, but it never succeeds. It just kept retrying sending SYN packets and then got timed out by nginx. Here's an example (tcp + nginx log):
May 14 15:27:44 iZ255cnkjm1Z kernel: [5320615.762381] IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63441 DF PROTO=TCP SPT=33533 DPT=8000 WINDOW=43690 RES=0x00 SYN URGP=0
May 14 15:27:45 iZ255cnkjm1Z kernel: [5320616.760064] IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63442 DF PROTO=TCP SPT=33533 DPT=8000 WINDOW=43690 RES=0x00 SYN URGP=0
May 14 15:27:47 iZ255cnkjm1Z kernel: [5320618.764069] IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63443 DF PROTO=TCP SPT=33533 DPT=8000 WINDOW=43690 RES=0x00 SYN URGP=0
May 14 15:27:51 iZ255cnkjm1Z kernel: [5320622.768069] IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63444 DF PROTO=TCP SPT=33533 DPT=8000 WINDOW=43690 RES=0x00 SYN URGP=0
May 14 15:27:59 iZ255cnkjm1Z kernel: [5320630.784094] IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63445 DF PROTO=TCP SPT=33533 DPT=8000 WINDOW=43690 RES=0x00 SYN URGP=0
May 14 15:28:15 iZ255cnkjm1Z kernel: [5320646.816070] IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63446 DF PROTO=TCP SPT=33533 DPT=8000 WINDOW=43690 RES=0x00 SYN URGP=0
Then followed by a nginx log:
x.x.x.x - - [14/May/2017:15:28:44 +0800] "GET /javascripts/version.json HTTP/1.1" 504 ...
During the period, the CPU load is pretty light, memory < 50%, incoming request is less than 50 per minute, and other requests were processed normally.
Server is Ubuntu 14.04.2 LTS
Any idea what's going on? Seems like an OS level issue? Thank you in advance.
Updated:
Firewall doesn't have any special settings. iptables -nvL shows the following:
Chain INPUT (policy ACCEPT 426K packets, 552M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 350K packets, 218M bytes)
pkts bytes target prot opt in out source destination
