FTP server cannot complete SSL handshake (VSFTPD)

Question

I'm trying to set up an internet-accessible FTP server with encryption, using VSFTPD as the server program, on Fedora 25. Despite seemingly setting everything up correctly, I can never connect to the server from outside my LAN when encryption is enabled. However, connecting is possible if I disable encryption or if I connect from within my LAN.

The issue I'm having is that the VSFTPD server can't complete the SSL handshake after receiving the AUTH command from a client. Using Wireshark, I can see that the server is trying to send what looks like the handshake response several times.

If it helps, here is Wireshark's report of a client attempting to connect to the server:

From    Info
------  ----
Client  64423 → 21 [ACK] Seq=1 Ack=1 Win=13952 Len=0 TSval=996262 TSecr=3062736173
Server  Response: 220 (vsFTPd 3.0.3)
Client  64423 → 21 [ACK] Seq=1 Ack=21 Win=13952 Len=0 TSval=996281 TSecr=3062736371
Client  Request: AUTH TLS
Server  21 → 64423 [ACK] Seq=21 Ack=11 Win=29056 Len=0 TSval=3062736436 TSecr=996282
Server  Response: 234 Proceed with negotiation.
Server  [TCP Retransmission] 21 → 64423 [PSH, ACK] Seq=21 Ack=11 Win=29056 Len=31 TSval=3062736822 TSecr=996282
Server  [TCP Retransmission] 21 → 64423 [PSH, ACK] Seq=21 Ack=11 Win=29056 Len=31 TSval=3062737214 TSecr=996282
Server  [TCP Retransmission] 21 → 64423 [PSH, ACK] Seq=21 Ack=11 Win=29056 Len=31 TSval=3062737214 TSecr=996282
Server  [TCP Retransmission] 21 → 64423 [PSH, ACK] Seq=21 Ack=11 Win=29056 Len=31 TSval=3062737214 TSecr=996282
...

Other info: I have VSFTPD configured to use TLSv1 or higher, to work in passive mode & with explicit FTPS, and with a self-signed RSA certificate. I don't think there's a problem with my certificate, since I am able to use the same certificate to host an https server with httpd, which I can access from outside my LAN just fine. So the problem must be related to VSFTPD somehow.

I have also set my router & firewall to forward & accept port 21 for ftp control port connections. I also set VSFTPD to have port 2048 as the only PASV data port, but for some reason I didn't need to forward that port on my router to let unencrypted FTP connections work...and besides, the failure I'm having is happening before the data port gets involved, anyway.

Does anyone have any idea for how to fix this? Is there something obvious that I'm missing here?

Answer 1

Server  Response: 234 Proceed with negotiation.
Server  [TCP Retransmission] 21 → 64423 [PSH, ACK] Seq=21 Ack=11 Win=29056 Len=31 TSval=3062736822 TSecr=996282

What you see is not the TLS handshake. The TLS handshake would be started by the client which is not the case here. What you instead see is a retransmission of the last response of the server, i.e. 234 Proceed with negotiation.\r\n which is exactly 31 bytes.
This means that the server does not receive any ACK from the client to this response and thus it is retransmitting it, i.e. common behavior of TCP connections on missing ACK.

The question is why the server is not receiving the ACK. It is not clear from your question if the packet capture was done on the client or server side but I assume it was done on the server side. In this case my guess would be that there is some firewall between client and server which tampers with the connection:
Since FTP is a protocol featuring dynamic ports for data transfer and these dynamic ports are exchanged inside the control connection a firewall needs to have clear text access to the control connection in order to find out which dynamic ports gets used and open these ports. If the control connection is encrypted (AUTH TLS) this is no longer possible so some firewalls try to explicitly or inadvertently block the use of TLS. And what you see might the result of this.