4

Since at least a few months, issuing a netstat -t command on our web server, which has TCP ports 22, 80 and 443 exposed to the Internet, often reveals dozens of connections in SYN_RECV status:

$ netstat -nt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 128.1XX.XX.X:22         142.93.230.186:50603    SYN_RECV
tcp        0      0 128.1XX.XX.X:80         104.248.89.196:36332    SYN_RECV
tcp        0      0 128.1XX.XX.X:80         167.71.79.217:40430     SYN_RECV
tcp        0      0 128.1XX.XX.X:22         142.93.235.90:36742     SYN_RECV
tcp        0      0 128.1XX.XX.X:80         178.128.243.146:35451   SYN_RECV
tcp        0      0 128.1XX.XX.X:443        178.62.253.100:43994    SYN_RECV
tcp        0      0 128.1XX.XX.X:80         188.166.91.16:42461     SYN_RECV
tcp        0      0 128.1XX.XX.X:22         178.62.242.138:33381    SYN_RECV
tcp        0      0 128.1XX.XX.X:80         142.93.235.90:57784     SYN_RECV
tcp        0      0 128.1XX.XX.X:443        165.22.198.207:36181    SYN_RECV
tcp        0      0 128.1XX.XX.X:22         68.183.11.83:63899      SYN_RECV
tcp        0      0 128.1XX.XX.X:22         104.248.93.253:41816    SYN_RECV
tcp        0      0 128.1XX.XX.X:80         104.248.85.129:48833    SYN_RECV
tcp        0      0 128.1XX.XX.X:80         178.62.207.43:59050     SYN_RECV
tcp        0      0 128.1XX.XX.X:80         128.199.55.152:55891    SYN_RECV
tcp        0      0 128.1XX.XX.X:443        178.62.205.23:52978     SYN_RECV
tcp        0      0 128.1XX.XX.X:80         165.22.198.135:36668    SYN_RECV
tcp        0      0 128.1XX.XX.X:80         165.22.203.65:65273     SYN_RECV
tcp        0      0 128.1XX.XX.X:80         68.183.15.91:46722      SYN_RECV
tcp        0      0 128.1XX.XX.X:443        167.71.4.136:55194      SYN_RECV
tcp        0      0 128.1XX.XX.X:80         142.93.228.103:60458    SYN_RECV
tcp        0      0 128.1XX.XX.X:443        178.62.253.100:52599    SYN_RECV
tcp        0      0 128.1XX.XX.X:22         104.248.89.56:46283     SYN_RECV
tcp        0      0 128.1XX.XX.X:443        178.62.250.235:43637    SYN_RECV
tcp        0      0 128.1XX.XX.X:443        167.71.7.181:37556      SYN_RECV
tcp        0      0 128.1XX.XX.X:80         104.248.89.200:64128    SYN_RECV
tcp        0      0 128.1XX.XX.X:22         142.93.139.213:38821    SYN_RECV
tcp        0      0 128.1XX.XX.X:443        104.248.92.161:63004    SYN_RECV
tcp        0      0 128.1XX.XX.X:22         68.183.15.91:46210      SYN_RECV
tcp        0      0 128.1XX.XX.X:443        104.248.92.138:39651    SYN_RECV
tcp        0      0 128.1XX.XX.X:22         178.62.241.74:43953     SYN_RECV
tcp        0      0 128.1XX.XX.X:22         142.93.224.74:42996     SYN_RECV
tcp        0      0 128.1XX.XX.X:443        165.22.196.167:38597    SYN_RECV
tcp        0      0 128.1XX.XX.X:80         178.128.254.3:36751     SYN_RECV
tcp        0      0 128.1XX.XX.X:22         104.248.93.27:32904     SYN_RECV
tcp        0      0 128.1XX.XX.X:443        167.71.78.255:60529     SYN_RECV
tcp        0      0 128.1XX.XX.X:22         165.22.201.209:45397    SYN_RECV
tcp        0      0 128.1XX.XX.X:80         178.62.207.205:39291    SYN_RECV
tcp        0      0 128.1XX.XX.X:22         165.22.198.207:61967    SYN_RECV
tcp        0      0 128.1XX.XX.X:443        178.62.234.81:39309     SYN_RECV
[---cut---]

The first though I had was that it was a SYN flooding attack, however (a) the server doesn't seem to suffer at all from the "attack", (b) the SYN rate is pretty low, at just slightly over 2 packets/sec., and (c) I generally observe those SYN packets coming from the same netblock both on that server (University lab web server hosted in our University network) and on a home server which is on a completely different netblock (connected via a regular residential fiber connection with a public dynamic IP), and (d) the Linux kernel never complies about a possible SYN flood (TCP SYN cookies are enabled).

What I'm wondering, is what the actual purpose of those SYN packets can be. They don't seem to come from network problems (badly configured firewalls or similar issues). They almost always come from IP spaces of datacenters, which change from one day to another (it could be Amazon EC2, Serverius, DigitalOcean, etc.). Sometimes it's just one to a few IPs, sometimes there are dozens. Sometimes the IP answers if I ping it, sometimes not. Although I only tested sporadically, I've almost never found a reachable web server at those IPs on port 80 or 443 (never tried to scan more ports than this), but see below for the first one I found, today. They come by bursts for perhaps one hour, then disappear; I didn't try yet to automatically detect them to have a hourly plot to see if there is some "schedule".

I was also thinking that this might be some kind of amplification attack using forged IP source addresses (each received SYN packet will eventually produce 5 SYN ACK packets before the server times out), but it doesn't make much sense as (a) a normal server would quickly reply with a RST packet thus stopping further SYN ACKs, and (b) as mentioned, those servers don't appear to host public websites that might worth of a DDoS amplification attack.

The (possibly spoofed?) source addresses mostly have no PTR record, or a generic one, but in some cases they give actual hostnames (e.g. in the list mentioned above, 142.93.230.186 gives parimatchklub.com, apparently a Russian sports betting site). The site was very slow to reach, so perhaps it is indeed some form of reflection attack against the apparent source addresses?

Did anybody else already observe this? Any explanation about the actual purpose of those packets?

Ale
  • 1,613
  • 17
  • 25
  • Do you use any IPtables rules or external firewall? Have you ruled out that you're actually blocked by a sender rather than the receiver just spoofing addresses? – Matthew Ife Nov 07 '19 at 10:22

3 Answers3

2

It looks like the footprint left behind by something like nmap host discovery. Nmap is a host discovery tool and port scanner.

nmap’s host discovery phase can send syn packets to webserver ports in order to detect servers which block icmp ping. Someone may be using nmap to discover live hosts in your ip range (or even 0.0.0.0/0) and may have no interest whatsoever in your host.

You could probably detect this by packet sniffing SYN and ICMP packets and comparing them with the scans described in: https://nmap.org/book/man-host-discovery.html

You can also see if they’re following up with a wider port scan. They may be looking for hosts with a particular port with a vulnerability known to them.

Fenster34
  • 173
  • 6
  • This sounds like an interesting explanation -- however I didn't observe any ICMP packets coming from the same IP addresses. Moreover, why would nmap scan the same port over and over? – Ale Nov 11 '19 at 10:04
  • It may not be nmap specifically but a group of hosts performing a similar task. I know that nmap will retry if it gets no response which is likely if they’re spoofing the source address. Just a thought – Fenster34 Nov 11 '19 at 19:39
  • I dont ever see the same source address. Maybe they’re being really thorough incase their target hosts are blocking certain source ip ranges. Or they’re trying to avoid detection as a port scanner which would normally hit multiple ports from the same source host. I’ve got to admit, if I’m right, they’re moderately sophisticated. – Fenster34 Nov 11 '19 at 20:06
1

It could be HTTP Evasion to bypass your server security. The IP addresses are possible malicious hosts, because they are budget hosting providers, which are known to host Malware and participate in Botnet activity.

Olaf V
  • 11
  • 1
  • But how would could it work in this situation? Connections in SYN_RECV state can't transfer any data. – Ale Aug 23 '19 at 05:18
  • Botnets can create such behavior if you are participating in a DDoS attack. It would help if you post some configurations of your kernel hardening and iptables – Olaf V Aug 23 '19 at 20:12
  • Participating how, you mean by reflecting or amplifying packets from spoofed IP addresses? A dump of the network traffic shows these SYN packets and the SYN ACK answers going back. Who is the target of the DDoS, the apparent source addresses? Re the kernel config and iptables rules, I'll extract them next week. – Ale Aug 23 '19 at 20:33
1

Any explanation about the actual purpose of those packets?

The output of netstat shows that your server was unable to ESTABLISH connection, that could be due interim connectivity issues or spoofed (forged) IP source endpoints — replies it sent simply didn't reach no initiator(s).

If we choose "spoofing version", I'd suggest trying to track those connections at least to the channel the server is getting them from. It might be not upstream but your LAN (be it real LAN or VMs network, etc.).

Then if it's upstream you might forward your concerns to theirs NOC (starting from support, more likely of course). If it's your side, well — clear it up.

poige
  • 9,171
  • 2
  • 24
  • 50
  • It's clearly not a local network problem, I'm sporadically observing this on completely unrelated networks (e.g., home server on FTTH connection and university server). It's definitely coming from upstream, I already verified that the packets actually come from the router, and the fact that I observe them at several locations also excludes local injection. Looks like they come from the wide Interne. The SYN packets seem to come from hosts having no intention at all of opening a connection (they never ACK the SYN ACK packet), but it's clearly not a SYN flood either (rate way too low). – Ale Nov 08 '19 at 11:44
  • I'd say it's better to verify than simply assume than it's external traffic. Sometimes it can be LAN traffic which crosses gateway first and then comes back. As to if it's SYN flood or not, it's a matter of terminology and it's rather question if it's a flood or not. Obviously, if the rate isn't alarming, it's not flooding. But in essence it's SYN packets from spoofed address anyways. – poige Nov 08 '19 at 12:35
  • Right, I see your point about LAN traffic coming back, but this would not explain why I observe the SYNs from the same sources at home. We have ingress and egress filtering here, so if the source was internal, the packets with spoofed IPs wont reach my FTTH connection... Concerning the flood, I don't find a rate of 2 SYNs/sec. alarming, I don't observe adverse effects on servers' performance. – Ale Nov 08 '19 at 12:39
  • That's what I'm saying. It's too low rate to consider it alarming but in nutshell if you see something that walks like duck … © – poige Nov 08 '19 at 13:05