iptables blocking some of the traffic on ports 80 and 443 when it shouldn't?

Question

A web server I'm administering shows weird iptables denials from IPv4 addresses on destination port 443, despite HTTPS traffic being explicitly allowed. Port 80 is also allowed in the same rule, but the site is HTTPS-only and 80 is immediately redirected to 443 by nginx.

Thing is: browsing the site works. You can load all the pages, all the resources come in just fine etc. But something is clearly amiss here, and this could be hurting page load performance.

Here are some examples of errors that are logged in /var/log/iptables_deny.log for port 443 and 80, respectively. These can come individually or in bursts, judging from my tail -f on the log file. The vast majority are for port 443:

iptables denied: IN=eth0 OUT= MAC=f2:3c:91:26:1e:1f:84:78:ac:0d:8f:41:08:00 SRC=(redacted IP) DST=(redacted IP) LEN=40 TOS=0x08 PREC=0x00 TTL=53 ID=61266 DF PROTO=TCP SPT=49264 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
iptables denied: IN=eth0 OUT= MAC=f2:3c:91:26:1e:1f:84:78:ac:0d:8f:41:08:00 SRC=(redacted IP) DST=(redacted IP) LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=11186 DF PROTO=TCP SPT=58445 DPT=443 WINDOW=254 RES=0x00 ACK FIN URGP=0
iptables denied: IN=eth0 OUT= MAC=f2:3c:91:26:1e:1f:84:78:ac:0d:8f:41:08:00 SRC=(redacted IP) DST=(redacted IP) LEN=40 TOS=0x00 PREC=0x00 TTL=116 ID=16941 DF PROTO=TCP SPT=16278 DPT=80 WINDOW=255 RES=0x00 ACK FIN URGP=0

A quick grepping says that the packet types in the denials can be at least ACK, ACK FIN, ACK RST, ACK PSH and RST. Perhaps others, but they didn't catch my eye.

Below, the output from iptables -nvL. The basic pattern (allow all related and established first, later allow all new traffic on 80 and 443) ought to be solid based on everything I've read. The LOG & DROP rules are the last ones in the INPUT chain, as they should be.

Chain INPUT (policy ACCEPT 1 packets, 391 bytes)
 pkts bytes target     prot opt in     out     source          destination   
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0       0.0.0.0/0     
 8893  770K ACCEPT     all  --  *      *       0.0.0.0/0       0.0.0.0/0      ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       (redacted IP)   0.0.0.0/0      ctstate NEW tcp dpt:22
    0     0 ACCEPT     tcp  --  *      *       (redacted IP)   0.0.0.0/0      ctstate NEW tcp dpt:22
    0     0 ACCEPT     tcp  --  *      *       (redacted IP)   0.0.0.0/0      ctstate NEW tcp dpt:22
    0     0 ACCEPT     tcp  --  *      *       (redacted IP)   0.0.0.0/0      ctstate NEW tcp dpt:22
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0       0.0.0.0/0      icmptype 8
   63  3840 ACCEPT     tcp  --  *      *       0.0.0.0/0       0.0.0.0/0      ctstate NEW multiport dports 80,443
    0     0 DROP       udp  --  *      *       0.0.0.0/0       0.0.0.0/0      udp dpt:137
    0     0 DROP       udp  --  *      *       0.0.0.0/0       0.0.0.0/0      udp dpt:138
    0     0 DROP       udp  --  *      *       0.0.0.0/0       0.0.0.0/0      udp dpt:139
    0     0 DROP       udp  --  *      *       0.0.0.0/0       0.0.0.0/0      udp dpt:445
    1    40 LOG        all  --  *      *       0.0.0.0/0       0.0.0.0/0      limit: avg 15/min burst 5 LOG flags 0 level 7 prefix "iptables denied: "
    1    40 DROP       all  --  *      *       0.0.0.0/0       0.0.0.0/0     

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source          destination   
    0     0 DROP       all  --  *      *       0.0.0.0/0       0.0.0.0/0     

Chain OUTPUT (policy ACCEPT 1 packets, 65 bytes)
 pkts bytes target     prot opt in     out     source          destination   
 7311   19M ACCEPT     all  --  *      *       0.0.0.0/0       0.0.0.0/0

The actual relevant rules, if they're of any use after that output:

-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p tcp -m conntrack --ctstate NEW -m multiport --dports 80,443 -j ACCEPT

What's clear is that this is not malicious traffic, by and large. I browsed the site from a couple of different IP addresses, and while the site loaded seemingly fine, my IP also appeared in the deny log without any pattern I could discern, and without any user-visible error conditions in the browser.

Any ideas what could be behind this?

Answer 1

RST and ACK,FIN packets are part of the very tail end of a TCP connection.

My understanding is that iptables's connection tracking engine takes a pretty robust approach to deleting state table entries, for security reasons: as soon as it sees one end of a connection trying to close it down, then (knowing that such a request signals the end of a conection, because one end of it now thinks the connection's down) it removes the state table entries that have been permitting that connection's traffic.

It would arguably be unsafe for it to wait longer to do this, in case some other similar firewall were also in the path, blocking the rest of the tidy-up packets that the RFCs specify; delaying reaping the state table entry until those were seen might risk leaving invalid entries in the table for some period of time, and that would present a potential vulnerability.

But the RFCs specify some more tidying-up to be done, and it's those packets that you see being refused. Yes, this means that the endpoints in the connection will use more memory, waiting for the connections to age out instead of seeing the full closedown and being able to free connection memory that much faster; but increased security often requires resources, and this is one of those trade-offs.

There's no harm in those packets never getting through, so you can safely ignore the log entries.

Edit: if you don't want to see them in the logs, deal with them before your logging line, eg

iptables -I INPUT 13 -p tcp -m conntrack --ctstate INVALID --tcp-flags ACK,FIN ACK,FIN -j DROP