This is a tricky one.
TL;DR:
1) Client establishes TCP handshake with the firewall on (closed/unavailable) port 21, even though firewall does not respond to client's SYN packet.
2) Client sends 1 SYN packet (no retransmit), firewall sees 3 SYN packets from client.
3) Same thing happens for a Linux server on another continent with block rule on its PREROUTING chain.
Recently we noticed our edge enterprise firewall (top 3 in Gartner ranking) is showing port 21 as an open port on its WAN port.
Even though the firewall itself does not use that port for any of its services nor there is a policy/NAT/LB redirecting it to another host/server, it still shows open in NMAP or telnet/NC output.
Explicitly denying it under firewall's interface and through security policies, did not make a difference. My search in that vendor's documents did not yield any result whatsoever.
At that time, to make sure that the firewall is the one responding to port 21, I ran tcpdump on my system and that firewall's "packet flow" tool to see what was happening and lo and behold, I see my connection to port 21 flowing into the firewall, proving the packets from client reach the firewall itself.
Oddly enough, my client sends 1 SYN packet and the firewall receives 3 SYN packets, does not respond to my client (no SYN/ACKs sent) and yet somehow the connection succeeds.
The thing is only the TCP handshake is successful on the client side and if you press enter in your client terminal, your connection immediately closes.
Since these firewalls are UNIX based at their core, out of curiosity, I set out to try and see if I can replicate this behavior on a VPS (running Ubuntu Server 19) I have on another continent.
Using IPTABLES, I created a DROP policy at PREROUTING chain to block traffic destined to port 21. I repeated my test and the result is exactly the same! Client sends 1 SYN packet, Ubuntu Server sees 3 SYN packets, does not respond with an ACK, and yet somehow, the connection establishes for the client! Needles to say upon pressing enter on client terminal the connection closes immediately. The exact same behavior on the firewall.
Below is the tests I have run and that are identical on both the server and the firewall:
Client telnets to server on port 21:
Client~# telnet Server-IP 21
Trying Server-IP...
Connected to Server-IP.
Escape character is '^]'.
Connection closed by foreign host.
TCPDUMP run on client when running telnet:
Client~# tcpdump -ni any port 21
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
19:24:05.456477 IP Client-priv-IP.36284 > Server-IP.21: Flags [S], seq 3093982956, win 64240, options [mss 1460,sackOK,TS val 3370721563 ecr 0,nop,wscale 7], length 0
19:24:05.557556 IP Server-IP.21 > Client-priv-IP.36284: Flags [S.], seq 3130560876, ack 3093982957, win 4080, options [mss 1360,sackOK,TS val 1773913297 ecr 3370721563], length 0
19:24:05.557679 IP Client-priv-IP.36284 > Server-IP.21: Flags [.], ack 1, win 64240, options [nop,nop,TS val 3370721664 ecr 1773913297], length 0
19:24:17.740636 IP Server-IP.21 > Client-priv-IP.36284: Flags [R.], seq 1, ack 1, win 0, length 0
TCPDUMP run on server when running telnet:
Server~# tcpdump -ni any -Q in port 21
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
19:25:05.001238 IP Client-pub-IP.36288 > Server-IP.21: Flags [S], seq 2605288404, win 4080, options [mss 1360,sackOK,TS val 1773972678 ecr 0], length 0
19:25:08.000566 IP Client-pub-IP.36288 > Server-IP.21: Flags [S], seq 2605288404, win 4080, options [mss 1360,sackOK,TS val 1773975678 ecr 0], length 0
19:25:10.999775 IP Client-pub-IP.36288 > Server-IP.21: Flags [S], seq 2605288404, win 4080, options [mss 1360,sackOK,TS val 1773978678 ecr 0], length 0
19:25:16.999702 IP Client-pub-IP.36288 > Server-IP.21: Flags [R.], seq 2605288405, ack 0, win 0, length 0
IPTABLES drops 3 SYN packets:
Server~# iptables -t raw -L -n -v
Chain PREROUTING (policy ACCEPT 36M packets, 26G bytes)
pkts bytes target prot opt in out source destination
3 2468 DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:21
We have tested this from different PCs and ISPs to exclude any weird client or internet connection issues as well.
For the sake of keeping secrecy, I could not share the IP addresses and vendor name.
Can anyone explain this behavior? The most puzzling thing is the fact that client sends 1 SYN and server/firewall receives 3.
Thanks in advance