12

So, my apache server was slow, and I looked in the log files. Turned out they had grown to 12GB of accesses from tons and tons of different hosts trying to access /wpad.dat on one of my Vhosts.

Now, the virtual host in question is the "catch-all" vhost that is invoked when a browser doesn't supply a known hostname.

I am currently getting thousands of requests per minute to "/wpad.dat" and as far as Google can tell me, this is something that has something to do with proxy servers? But I don't use proxy servers, so why am I being literally bombarded by these requests.

I am getting more requests per minute for this non-existant file than I am getting normal requests. So my assumption is that I am under some form of attack. Funny thing is that it generally only occurs at night (here in Sweden) and not during the day.

A sample size of the latest 500 requests (i.e. half a minute) shows that it consists of 200 different hosts, and a small sample of those shows that they are all valid hosts (not TOR proxies) so is this some DNS servers being incorrectly configured? I do run a DNS server on the machine.

Please help! :)

EDIT The host they are accessing is "cluster.atlascms.se" so what they do is access http://cluster.atlascms.se/wpad.dat thousands of times per minute.

Now, cluster.atlascms.se is my DNS failover host. So all my clients point their subdomains to cluster.atlascms.se, which in turn points them to the current IP (master server of failover server).

As it seems - this means I am getting tons and tons of requests to cluster.arlascms.se - could that mean that my DNS is misconfigured?

Sandman
  • 349
  • 1
  • 4
  • 15
  • Is there anything in common among the hosts? – Michael Hampton May 23 '13 at 21:04
  • They all seems to be from Sweden, but that's just to be expected... No, nothing in common. Spread over the country and over different ISP... – Sandman May 23 '13 at 21:08
  • 3
    You know, you could put up your own WPAD.DAT file and really have some fun with these people. >smile< Seriously, though, somebody has b0rked up a configuration somewhere horribly if they're pulling a WPAD.DAT from an untrusted source. You redirect all their browsers to a proxy you control and MiTM their traffic. – Evan Anderson May 23 '13 at 21:21
  • But the reqauests comes from hundreds and hundreds of valid IP's from swedish ISP's, it's not one company or one source. – Sandman May 23 '13 at 21:23
  • 3
    I would strongly be tempted to put up a `wpad.dat` that simply points the the local host. That should break things enough whoever is causing the problem might take the time to fix it. – Zoredache May 23 '13 at 21:23
  • @Zoredache - You're no fun! – Evan Anderson May 23 '13 at 21:24
  • I now have put up a wpad.dat file on the server: http://atlas.eklundh.com/wpad.dat - This changed nothing. Still thousands and thousands of request - only, now they actually find something there... :) – Sandman May 23 '13 at 21:26
  • 1
    @Sandman - The WPAD.DAT file needs to be Javascript to work. Have a look here: http://en.wikipedia.org/wiki/Web_Proxy_Autodiscovery_Protocol – Evan Anderson May 23 '13 at 21:28
  • Ok, I've tried to use a example file I found now... But still, it's the *requests* that are bogging down my server, and they remain the same... :/ – Sandman May 23 '13 at 21:31
  • Maybe a wpad like this instead? `function FindProxyForURL(url, host) { return "PROXY 127.0.0.1:445"; }` See if they can DoS themselves. – Zoredache May 23 '13 at 21:33
  • 1
    BTW, it is extremely rude for you to discover a problem, and then try to dump your trash on someone else's lawn. So please don't set your wpad to point at someone elses systems. – Zoredache May 23 '13 at 21:36
  • @Zoredache You are correct, I was merely testing to see if it made any difference. – Sandman May 23 '13 at 21:39
  • 2
    The problem with your DNS setup is that any of your clients that use a wildcard dns entry to point all of their sub domains to cluster.atlascms.se will by default be pointing wpad.theirdomain.whatever there as well. Which means if they set their desktop hostname to something.theirdomain.whatever then it will lookup wpad.theirdomain.whatever, get your IP and repeatedly request wpad.dat thousands of times a day. – Justin Buser Aug 02 '14 at 06:35
  • Where are the clients coming from? Should only be from internal clients of one of your wildcard domains. – eckes Nov 12 '17 at 07:54

5 Answers5

11

Machines will look for a WPAD.dat file hierarchically based on their own FQDN, if they are configured for proxy autodiscovery. So, if a windows PC is a member of a domain c.d.e.com, it will look for WPAD.dat in:

http://wpad.c.d.e.com/wpad.dat
http://wpad.d.e.com/wpad.dat
http://wpad.e.com/wpad.dat
http://wpad/wpad.dat

Chances are that somewhere, someone has a domain which is a subdomain of one of the ones you are hosting HTTP on, and hasn't properly configured or disabled proxy auto-discovery. In consequence, they are likely searching hierarchically.

It's possible that a virus may have caused them to do this; likely, if the machines making the query are extremely numerous and in diverse subnets, this is what is up.

If possible, avoid defining a DNS record for the wpad subdomain of anything you don't intend to use for proxy auto-discovery.

If this is not an option, you could consider using layer 7 filtering to find queries for wpad.dat and reject the packets with an ICMP message. This might actually be the most effective way to stop the traffic, unless the IPs are all from the same network and their technical contact in whois is responsive.

Things that will point a host at a particular location for wpad.dat include domain settings, the domain name option in DHCP replies, and an explicit setting in the web browser to load proxy information from some URL.

Falcon Momot
  • 24,975
  • 13
  • 61
  • 92
  • I do not have a wpad subdomain, but I do have a wildcard subdomain. I think the culprit may be my atlascms.se domain (for which I do not need a wildcard subdomain) which is the domain I use for my clients for their CNAME records. I have updated my DNS and I'll have to wait and see if this fixes things. – Sandman May 24 '13 at 05:15
  • The problem is that all these hundreds of thousands of requests I get from hundreds and hundreds of different hosts couldn't all possibly be thinking that atlascms.se is in their own subnet, surely? – Sandman May 24 '13 at 05:16
  • It has nothing at all do with subnets; it's all domains. – Falcon Momot May 24 '13 at 05:56
  • Yeah, sorry for the terminology confusion. – Sandman May 24 '13 at 07:03
  • It's entirely possible someone is trying to use your domain to host a malicious wpad.dat (and failing). There might be a virus out there setting your URL as the place to get wpad.dat from explicitly, or otherwise pointing hosts there, or there might be a misconfigured network out there. – Falcon Momot May 24 '13 at 07:09
  • I suppose it's possible. I set the wpad.* subdomain to point to localhost and have seen no change as of yet. I also use iptables to string-block the incoming requests. The server is still bogged down (so maybe these requests weren't the entire reason for it being slow) but at least my logs aren't filling up... – Sandman May 24 '13 at 08:44
  • You might have to wait your TTL (or 48 hours) for DNS convergence. Also, this is a relatively harmless way to handle it - the machines will generally still get their WPAD failure from your domain, and less spurious traffic will be generated. – Falcon Momot May 24 '13 at 08:53
9

It appears that your DNS zone eklundh.com has a wildcard record defined pointing to cluster.atlascms.se. This includes wpad.eklundh.com. I suggest you add a DNS record explicitly defining wpad.eklundh.com. to 127.0.0.1 or something.

Zoredache
  • 128,755
  • 40
  • 271
  • 413
  • 7
    I hate wildcard DNS records. They've never been anything but trouble. – Evan Anderson May 23 '13 at 22:03
  • Ok, I have now added a wpad subdomain to all my domains in my DNS that all point to 127.0.0.1 - I have to wait to see it propagate and see if this fixes the problem. – Sandman May 24 '13 at 05:13
  • Looking in my log files, the vast bulk of the requests aren't directed at a wpad subdomain, but to the IP or to cluster.atlascms.se... – Sandman May 24 '13 at 13:09
4

The first thing I would do is to try to find out where these requests are going to, i.e. their destination. Apache doesn't log the hostname by default, so you can either use tcpdump to get a brief capture and inspect it for the Host: request header, or change your Apache log format to log it. I prefer to log it in the otherwise useless second field, for example:

LogFormat "%h %{Host}i %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

Once you know who these mistaken requests are being addressed to, what to do next may become clear. For instance, it might turn out to be some big company example.se in which case you can go find their network admins and yell at them.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • using server-status, I see the host they are "attacking", and it's not even a configured vhost (which is why it is logged by the catch-all vhost) - the host they ALL are connecting to is "atlas.eklundh.com" – Sandman May 23 '13 at 21:24
  • And also, all these requests comes from hundreds and hundreds of different hosts from all over the country and all over the ISP spectra, this is not coming from one source or one misconfiguredc company. I am wondering if I am doing something wrong with my eklundh.com domain here, whose DNS server I also run on the machine – Sandman May 23 '13 at 21:25
  • "atlas.eklundh.com" resolves to the same IP as "sandman.net". – Evan Anderson May 23 '13 at 21:26
  • Yes, they are both vhosts on this machine. What do you mean by that? – Sandman May 23 '13 at 21:27
  • I can't tell you *why* all those machines are looking for a WPAD.DAT / Proxy Autoconfig file at "atlas.eklundh.com", but they are. Getting to the bottom of that is going to be interesting but, for now, you could change the "A" record on that name, assuming you control it, to something like 127.0.0.1 to stem the tide of traffic. – Evan Anderson May 23 '13 at 21:29
  • OK, so now you need to hunt down Hans Eklundh and ask him what he's up to. – Michael Hampton May 23 '13 at 21:33
  • No - I see now that it is the host "cluster.atlascms.se" they are accessing - which is the CNAME my clients use for their domains... So "sandman.net" points to "cluster.atlascms.se" which in turn points to my IP... So maybe there is something wrong with my DNS? – Sandman May 23 '13 at 21:37
  • There's no way for us to know whether that's correct or not. If you weren't expecting this wpad.dat file to be served, you should contact the person who is apparently responsible for it (and I already named him out of the whois record). – Michael Hampton May 23 '13 at 21:43
  • Hans Eklundh is the owner of the eklundh.com domain (and also my father). He is not the culprit here. I am the owner of the atlascms.se domain, and I am wondering what I could have done wrong here... – Sandman May 23 '13 at 21:45
  • So he didn't configure a bunch of company machines to look for a wpad.dat file? – Michael Hampton May 23 '13 at 21:45
  • No he did not. I manage the DNS for eklundh.com (and atlascms.se) and it is done on this server (i.e. I run both the DNS and the web server on this machine). Am I seeing HTTP traffic due to a misconfigured DNS server? – Sandman May 23 '13 at 21:47
  • No, you're seeing traffic because a lot of machines are configured to look for a proxy server configuration file (wpad.dat) at your hostname. _Somebody_ set up those machines to do that. – Michael Hampton May 23 '13 at 21:48
  • 1
    You don't really need to configure systems to look for a wpad.dat. You just to have a valid DNS record like `wpad.eklundh.com` (you do have that record) and computers that have a FQDN that is set to something like *.eklundh.com` will automatically try to do the WPAD lookup. – Zoredache May 23 '13 at 21:49
  • @Zoredache Oh, you're right. I bet that's a wildcard CNAME. And there you have it. – Michael Hampton May 23 '13 at 21:51
  • @Zoredache Yes, I have it set up as *.atlascms.se and *.eklundh.com (which catches the wpad.eklundh.com as well). Shouldn't I have it set up that way? – Sandman May 23 '13 at 21:51
  • 1
    The problem then becomes that any computer that thinks it's in the eklundh.com domain will see that wpad.eklundh.com is valid, and attempt to download a proxy server configuration from it. You can remove the DNS record, or reconfigure all of the computers. – Michael Hampton May 23 '13 at 21:52
  • But that doesn't make any sense - noe of the hundreds of hundreds of IP's that is currently spamming me should be thinking they are in any of these domains. Right? – Sandman May 23 '13 at 21:54
  • Again, we have no way of knowing that. – Michael Hampton May 23 '13 at 21:56
0

Just FYI, ModSecurity will catch this and block it. There is a ruleset provided by Comodo. Here is a log entry. I stripped out account relevant data so it has in it just so as to use it as an example.

Apache-Error: [file ""] [] [] [client xxx.xxx.xxx.xxx] ModSecurity: Access denied with code 403 (phase 2). Matched phrase ".dat/" at TX:extension. [file ""] ["] [id "210730"] [rev "2"] [msg "COMODO WAF: URL file extension is restricted by policy"] [data ".dat"] [severity "CRITICAL"] [hostname "removed"] [uri "/wpad.dat"] [unique_id "WjFa06qDOW3DDPRieFmICgAAAEg"]

islandnet.com Web Hosting
  • 1
  • 2
-1

Had this problem and fixed it by creating a wpad.dat file putting "this page left blank" page in it.

CPU went to nearly zero. Problem seems solved.

Brian Tolman
  • 1
  • But it solved the symptom. In this instance, it reduced the server load, go the site running again, and gave me time to redo the A-Records where the real problem lived. – Brian Tolman Nov 15 '17 at 21:58