1

This is a tricky one.

TL;DR:

1) Client establishes TCP handshake with the firewall on (closed/unavailable) port 21, even though firewall does not respond to client's SYN packet.

2) Client sends 1 SYN packet (no retransmit), firewall sees 3 SYN packets from client.

3) Same thing happens for a Linux server on another continent with block rule on its PREROUTING chain.

Recently we noticed our edge enterprise firewall (top 3 in Gartner ranking) is showing port 21 as an open port on its WAN port.

Even though the firewall itself does not use that port for any of its services nor there is a policy/NAT/LB redirecting it to another host/server, it still shows open in NMAP or telnet/NC output.

Explicitly denying it under firewall's interface and through security policies, did not make a difference. My search in that vendor's documents did not yield any result whatsoever.

At that time, to make sure that the firewall is the one responding to port 21, I ran tcpdump on my system and that firewall's "packet flow" tool to see what was happening and lo and behold, I see my connection to port 21 flowing into the firewall, proving the packets from client reach the firewall itself.

Oddly enough, my client sends 1 SYN packet and the firewall receives 3 SYN packets, does not respond to my client (no SYN/ACKs sent) and yet somehow the connection succeeds.

The thing is only the TCP handshake is successful on the client side and if you press enter in your client terminal, your connection immediately closes.

Since these firewalls are UNIX based at their core, out of curiosity, I set out to try and see if I can replicate this behavior on a VPS (running Ubuntu Server 19) I have on another continent.

Using IPTABLES, I created a DROP policy at PREROUTING chain to block traffic destined to port 21. I repeated my test and the result is exactly the same! Client sends 1 SYN packet, Ubuntu Server sees 3 SYN packets, does not respond with an ACK, and yet somehow, the connection establishes for the client! Needles to say upon pressing enter on client terminal the connection closes immediately. The exact same behavior on the firewall.

Below is the tests I have run and that are identical on both the server and the firewall:

Client telnets to server on port 21:

Client~# telnet Server-IP 21
Trying Server-IP...
Connected to Server-IP.
Escape character is '^]'.

Connection closed by foreign host.

TCPDUMP run on client when running telnet:

Client~# tcpdump -ni any port 21

listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
19:24:05.456477 IP Client-priv-IP.36284 > Server-IP.21: Flags [S], seq 3093982956, win 64240, options [mss 1460,sackOK,TS val 3370721563 ecr 0,nop,wscale 7], length 0
19:24:05.557556 IP Server-IP.21 > Client-priv-IP.36284: Flags [S.], seq 3130560876, ack 3093982957, win 4080, options [mss 1360,sackOK,TS val 1773913297 ecr 3370721563], length 0
19:24:05.557679 IP Client-priv-IP.36284 > Server-IP.21: Flags [.], ack 1, win 64240, options [nop,nop,TS val 3370721664 ecr 1773913297], length 0
19:24:17.740636 IP Server-IP.21 > Client-priv-IP.36284: Flags [R.], seq 1, ack 1, win 0, length 0

TCPDUMP run on server when running telnet:

Server~# tcpdump -ni any -Q in port 21

listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
19:25:05.001238 IP Client-pub-IP.36288 > Server-IP.21: Flags [S], seq 2605288404, win 4080, options [mss 1360,sackOK,TS val 1773972678 ecr 0], length 0
19:25:08.000566 IP Client-pub-IP.36288 > Server-IP.21: Flags [S], seq 2605288404, win 4080, options [mss 1360,sackOK,TS val 1773975678 ecr 0], length 0
19:25:10.999775 IP Client-pub-IP.36288 > Server-IP.21: Flags [S], seq 2605288404, win 4080, options [mss 1360,sackOK,TS val 1773978678 ecr 0], length 0
19:25:16.999702 IP Client-pub-IP.36288 > Server-IP.21: Flags [R.], seq 2605288405, ack 0, win 0, length 0

IPTABLES drops 3 SYN packets:

Server~# iptables -t raw -L -n -v
Chain PREROUTING (policy ACCEPT 36M packets, 26G bytes)
 pkts bytes target     prot opt in     out     source               destination         
   3  2468 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:21

We have tested this from different PCs and ISPs to exclude any weird client or internet connection issues as well.

For the sake of keeping secrecy, I could not share the IP addresses and vendor name.

Can anyone explain this behavior? The most puzzling thing is the fact that client sends 1 SYN and server/firewall receives 3.

Thanks in advance

PoJam
  • 111
  • 5
  • Are your client and server located on the same subnet? It may happen of some misconfigured router or switch between them answering. Try to shut server down and check if replies are still there. I had something like that and found that very hard to diagnose. If it is the same subnet - you can use -e in tcpdump to check what mac address is answering. If not - you need to debug network devices between them. – kab00m Apr 11 '20 at 17:50
  • It is very common to have some router trying to proxy ftp requests, to comply to FTP protocol and open neccesary ports on demand. – kab00m Apr 11 '20 at 17:57
  • Try a "traceroute -T -p 21 server_ip", compare that to a normal traceroute, see if it actually gets as far as your server. Probably being intercepted between. (Assuming your test PC and server are not on the same IP subnet and ethernet segment.) You said you tried different PCs and ISPs, but you didn't mention changing out your local router? – Stingray Apr 11 '20 at 18:03
  • Guys, This is happening for a server acting as a firewall and an actual enterprise firewall on the internet on two different continents from different ISPs. The TCPDUMP output I provided is proving the receive of packets on both the server and the firewall. Please note this server is not behind a firewall. – PoJam Apr 11 '20 at 18:30

3 Answers3

0

Is this not expected behaviour? "If the initial TCP handshake is failing because of packet drops then you would see that the TCP SYN packet is retransmitted only 3 times." I don't know if Microsoft is an authority on this subject, but I found this: docs Microsoft

Gerard H. Pille
  • 2,469
  • 1
  • 12
  • 10
  • But the thing is those "retransmitted" SYNs are not evident in my TCPDUMP output. because the connection establishes. Please refer to the outputs I have provided. – PoJam Apr 11 '20 at 17:18
0

Port 21 is the well-known port for FTP, which is not exactly a NAT-friendly protocol. Your redacted tcpdump from the client says "Client-priv-IP" whereas the one from the server says "Client-pub-IP", so I take it that the client sits behind a NAT router. I'll bet that router is intercepting FTP in order to make it work over NAT, completing the three-way handshake on its own and in parallel doing three attempts to reach the destination server. When the destination server doesn't respond it terminates the connection from the client on the next opportunity.

Tilman Schmidt
  • 3,778
  • 10
  • 23
  • You've got a very valid point. Must be a middle-box. To provide more info, yes, in order to reach the firewall or the firewall, clients pass through many boxes and their ADSL modem is one of the very first ones so the NAT happens. – PoJam Apr 12 '20 at 15:26
0

Thanks to Tilman Schmidt I was able to search and find RFC 3234, it is in fact a middle-box acting up with FTP traffic.

In the TCPDUMP output you can see that factors like MSS value and some TCP options like SACK differ from client to server which is an indication of a middle-box presence.

A middle-box tends to manipulate:

  • some TCP options (SACK, NOP, etc.)
  • TTL value
  • MSS value
  • in some cases source port
  • sequence numbers
  • packet ID

I suspect (optimistically) these middle-boxes act in this way because protocols like FTP work in two different TCP channels. This behavior of proxying my FTP connection is to be aware and not disrupt the data channel flowing from server to client.

PoJam
  • 111
  • 5