24

For the past week I've been getting a huge stream of traffic from a wide range of Chinese IP addresses. This traffic appears to be from normal people and their HTTP requests indicate that they think I'm:

  • Facebook
  • The Pirate Bay
  • various BitTorrent trackers,
  • porn sites

All of which sounds like things people would use a VPN for. Or things that would make Great Wall of China angry.

User-agents include web browsers, Android, iOS, FBiOSSDK, Bittorrent. The IP addresses are normal commercial Chinese providers.

I have Nginx returning 444 if the host is incorrect or the user agent is obviously wrong:

## Deny illegal Host headers
if ($host !~* ^({{ www_domain }})$ ) {
   return 444;
}
## block bad agents
if ($http_user_agent ~* FBiOSSDK|ExchangeWebServices|Bittorrent) {
    return 444;
}

I can handle the load now, but there were some bursts of up to 2k/minute. I want to find out why they are coming to me and stop it. We also have legitimate CN traffic, so banning 1/6th of planet earth is not an option.

It is possible that its malicious and even personal, but it may just be a misconfigured DNS over there.

My theory is that its a misconfigured DNS server or possibly some VPN services that people are using to get around Great Fire Wall.

Given a client IP address:

183.36.131.137 - - [05/Jan/2015:04:44:12 -0500] "GET /announce?info_hash=%3E%F3%0B%907%7F%9D%E1%C1%CB%BAiF%D8C%DE%27vG%A9&peer_id=%2DSD0100%2D%96%8B%C0%3B%86n%8El%C5L%11%13&ip=183.36.131.137&port=11794&uploaded=4689970239&downloaded=4689970239&left=0&numwant=200&key=9085&compact=1 HTTP/1.0" 444 0 "-" "Bittorrent"

I can know:

descr:          CHINANET Guangdong province network
descr:          Data Communication Division
descr:          China Telecom
  • How can I find out what DNS server those customers are using ?
  • Is there anyway to determine if an HTTP request is coming from a VPN ?
  • What is really going on here ?
Chris Sattinger
  • 343
  • 2
  • 7
  • 5
    I've seen this very problem before, both being the target of the traffic, and having traffic intended for my server sent somewhere else. I don't have any answers though. I reduced the impact of the first problem with a firewall, and the second with a software solution that was only possible in our specific situation (our software was making the requests). In the process of instigating, we found that some DNS servers refuse to respect very low TTLs, instead caching the result for months, which might explain the list of sites you're getting traffic for. – xofer Jan 09 '15 at 15:27
  • 1
    Also check this question. I had the same problem http://serverfault.com/questions/656093/mod-security-block-requests-by-http-host-header I am just curious as to why an ISP would do something like that. I cannot see the value in in. – Cha0s Jan 09 '15 at 15:31
  • 4
    In my experience, these are attempts to find open web proxies. Some web servers will allow you to request **any** URL; I once got called in to deal with one such, which had exceeded it's (generous) monthly bandwidth allocation before it was even in service. A bunch of students at the Nanjing Institute of Technology had discovered they could make HTTPS connections to it and request **any** webpage, and were thus eagerly sneaking all their porn past the Great Firewall. If you're not actually serving the requested content, you should be fine. – MadHatter Jan 09 '15 at 16:26
  • @MadHatter: if they are looking for a proxy, shouldn't the GET header of the request also include a protocol and domain, eg. `GET http://myexampleporn.com/video/87461234` ? – r_3 Jan 09 '15 at 16:32
  • 1
    Usually, yes. Only one log entry was quoted above, so I thought it worth mentioning; it's not intended to be a complete, thought-out answer, or I'd've submitted it as one! – MadHatter Jan 09 '15 at 16:46
  • @Cha0s I think it may be a new way to block forbidden content. – Chris Sattinger Jan 09 '15 at 16:56
  • It could be the case, but it still does not make sense to do it by resolving to random IPs (or not so random maybe?). I can see other countries' ISPs blocking *.thepiratebay.org by resolving to 127.0.0.1. That makes more sense in order to block content than resolving to completely unrelated IPs. – Cha0s Jan 09 '15 at 18:08
  • 1
    1. Have you ever tried to hunt down the sysadmins for a registered domain here in US? If so you know how difficult that can be. I imagine its darn near impossible to not only find the right person to talk to at "Chinanet" but also the right person who actually cares enough to help you. – Michael Martinez Jan 10 '15 at 01:00

3 Answers3

31

There is one theoretical way of determining the DNS resolver of your clients, but it's quite advanced and I don't know any off-the-shelf software that will do that for you. You'll for sure have to run a authoritative DNS server for that in addition to your nginx.

In case the HTTP Host header is incorrect, serve an error-document and include a request to a dynamically created, unique FQDN for each and every request which you log to a database. eg.

http://e2665feebe35bc97aff1b329c87b87e7.example.com/img.png

As long as Chinas great firewall doesn't fiddle with that request and the client requests the document from that unique FQDN+URI, each request will result a new DNS lookup to your authoritative DNS for example.com where you can log the IP of the DNS resolver and later correlate this with your dynamically generated URIs.

r_3
  • 886
  • 5
  • 9
  • 6
    This is the same approach I would suggest, though I think another level of domains would be needed to make it work. If the primary domain is `example.com`, you would create an NS record for one subdomain such as `ns-detect.example.com`. Then you would create a unique name beneath that domain name, such the full domain would be `e2665feebe35bc97aff1b329c87b87e7.ns-detect.example.com`. – kasperd Jan 09 '15 at 20:04
  • 1
    Its an interesting approach. I'm suspecting now that the redirection is deliberate (because I'm not the only one seeing this). So I would assume that the various Chinese DNS servers would not bother to do a lookup to the authoritative one to get the subdomain. It would not make sense for them to do it. – Chris Sattinger Jan 10 '15 at 16:25
  • If they use a whitelisting aproach, you are probably right. If they are blacklisting there is actually no reason why the shouldn't lookup an innocent looking FQDN. Of course this could also a result of much more advanced filtering techniques than just playing around with DNS responses. – r_3 Jan 10 '15 at 18:24
  • 1
    +1 for kasperd's idea of making a subdomain with its own NS, to keep logs for that separate from your normal DNS. And to do it with less chance of screwing up your normal stuff. If you see http requests for hostnames that weren't looked up in your DNS by anyone, then you know the DNS server your bad clients are using is forging DNS responses (and getting it wrong, since they probably meant to send that traffic somewhere else. Maybe a Chinese admin somewhere typoed an IP address in a config?). – Peter Cordes Jan 11 '15 at 07:19
  • Accepting because its answers the first question most exactly. Doesn't really solve why we get the traffic, but stackexchange doesn't allow vague questions. – Chris Sattinger Jan 13 '15 at 11:30
5

I've heard the great firewall used to redirect "blocked" traffic to a handful of phony IPs, but this was causing their blocks to be easily spotted (I'm not sure if it allowed easy subversion). In any case the administrators have started redirecting to random IPs. This has led to some Chinese users getting porn, instead of facebook or vpns, apparently.

I suspect one of your IPs has turned out to be a recipient of blocked chinese traffic - hence you seeing Facebook IPI user agents.

This means the host-header check should be a good one. Most user agents support SNI these days, so you should be able to drop no-host-header traffic with relative impunity.

Edit: http://www.infosecurity-magazine.com/news/great-firewall-upgrade-redirects/

Tom Newton
  • 4,021
  • 2
  • 23
  • 28
4

How can I find out what DNS server those customers are using ?

Contact Chinanet and ask? Seriously, DNS is configurable on the client side. Most people get DNS settings via DHCP, but OpenDNS and Google's DNS offering wouldn't have a business model if you couldn't change them.

Is there anyway to determine if an HTTP request is coming from a VPN ?

Not really, except that the IP would be of the VPN, not the end user in China.

What is really going on here ?

That I can't tell you, but perhaps there's some kind of misconfiguration in the Great Firewall of China?

unor
  • 246
  • 2
  • 19
Katherine Villyard
  • 18,510
  • 4
  • 36
  • 59