0

In an Ubuntu 16.04 machine hosting an internet-facing web application, my /var/log/syslog is being flooded by messages of the sort:

Jan  9 17:41:50 ip-172-31-11-100 kernel: [483324.699896] [UFW BLOCK] IN=ens5 OUT= MAC=0a:16:21:97:4e:74:0a:af:bd:31:30:da:08:00 SRC=88.201.58.59 DST=172.31.11.100 LEN=52 TOS=0x10 PREC=0x20 TTL=40 ID=63099 DF PROTO=TCP SPT=6450 DPT=443 WINDOW=343 RES=0x00 ACK URGP=0

Jan  9 17:41:50 ip-172-31-11-100 kernel: [483324.719775] [UFW BLOCK] IN=ens5 OUT= MAC=0a:16:21:97:4e:74:0a:af:bd:31:30:da:08:00 SRC=88.201.58.59 DST=172.31.11.100 LEN=569 TOS=0x10 PREC=0x20 TTL=40 ID=63098 DF PROTO=TCP SPT=6450 DPT=443 WINDOW=343 RES=0x00 ACK PSH URGP=0

Jan  9 17:43:13 ip-172-31-11-100 kernel: [483408.133979] [UFW BLOCK] IN=ens5 OUT= MAC=0a:16:21:97:4e:74:0a:af:bd:31:30:da:08:00 SRC=103.255.6.65 DST=172.31.11.100 LEN=40 TOS=0x00 PREC=0x00 TTL=38 ID=0 DF PROTO=TCP SPT=3277 DPT=443 WINDOW=0 RES=0x00 RST URGP=0

It seems user traffic is being blocked from accessing 443 for ttls of up to 40 seconds. I am unable to decipher this issue. Can an expert point out what is going on, and how I can reverse this situation?

Do note that myself and numerous users are able to access this web applcation - i.e. the blocking is not universal. But there are legitimate users complaining about it. Maybe it is a rate-limit issue of some sort?

sudo ufw status verbose yields:

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22                         ALLOW IN    Anywhere                  
80                         ALLOW IN    Anywhere                  
443                        ALLOW IN    Anywhere                  
22 (v6)                    ALLOW IN    Anywhere (v6)             
80 (v6)                    ALLOW IN    Anywhere (v6)             
443 (v6)                   ALLOW IN    Anywhere (v6)

sudo iptables -S | grep ACCEPT yields:

-P OUTPUT ACCEPT
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-input -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A ufw-before-input -d 224.0.0.251/32 -p udp -m udp --dport 5353 -j ACCEPT
-A ufw-before-input -d 239.255.255.250/32 -p udp -m udp --dport 1900 -j ACCEPT
-A ufw-before-output -o lo -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-skip-to-policy-output -j ACCEPT
-A ufw-track-output -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-output -p udp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 22 -j ACCEPT
-A ufw-user-input -p udp -m udp --dport 22 -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 80 -j ACCEPT
-A ufw-user-input -p udp -m udp --dport 80 -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 443 -j ACCEPT
-A ufw-user-input -p udp -m udp --dport 443 -j ACCEPT
-A ufw-user-limit-accept -j ACCEPT

sudo iptables -S | grep ufw-user-limit yields the following (in case rate-limits are in play):

-N ufw-user-limit
-N ufw-user-limit-accept
-A ufw-user-limit -m limit --limit 3/min -j LOG --log-prefix "[UFW LIMIT BLOCK] "
-A ufw-user-limit -j REJECT --reject-with icmp-port-unreachable
-A ufw-user-limit-accept -j ACCEPT

Lastly, sudo iptables -S | grep "UFW BLOCK" yields:

-A ufw-after-logging-forward -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-after-logging-input -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
Hassan Baig
  • 2,033
  • 11
  • 27
  • 47
  • Do you have more log entries? The three shown above look like an attack, not legitimate traffic. – Michael Hampton Jan 09 '21 at 21:57
  • Might need more than just rules filtered by `grep ACCEPT`. I'd try `grep "UFW BLOCK"` and work backwards from there. – tater Jan 10 '21 at 08:18
  • @tater - added that output at the end of the question. Is that what you wanted – Hassan Baig Jan 10 '21 at 09:16
  • @MichaelHampton Can you explain how they are attacks? Yes there are tons more log entries. Unfortunately, I do not understand these log entries, and they are thinly documented online. I have talked to at least one user who is beset by this issue - and they are not an attacker. Their ip does show up in the logs. My guess right now is they are being rate-limited. – Hassan Baig Jan 10 '21 at 09:20

2 Answers2

0

I have seen this before and read a few topics around this on line.

UFW is blocking the RST packets.

https://frankfu.click/linux/basic/ufw-blocking-fin-rst-and-ack-packets-when-rules-should-allow-it/

Gary Mclean
  • 1
0

Very likely these are BLOCKs for invalid packets. Do a tcpdump to capture the packets with something like this and watch for the blocks to happen e.g.:

sudo tcpdump port 443 -w /tmp/ufw.log -i eno1 -G 200

Break that to stop writing to the output file. Next look at the source port on one of the BLOCKS (SPT=6450 for example) and do a filter to get just the packets for that session into another file:

tcpdump -r /tmp/ufw.log -w /tmp/ufw.port6450 port 6450

Then look at that output file with wireshark. Very likely those ACK packets will show up in red as spurious retransmissions, and those RST packets are sent after the FIN-ACKs, which make them invalid.

brianc
  • 1