5

I have a list of about 20000 hostnames and I need to check which ones are up. I would say 98% of them are down. I have already tried powershell and python scripts without much success (couldn't set a decent timeout in each). I'm trying to use NMap with the following command:

nmap.exe -iL hostnames.txt -sn --host-timeout=200ms -oN results.txt

Even then, each down host takes about 8 seconds because NMap gets stuck trying to resolve the hostname into an IP address. Also, I see a warning at the start that says NMap couldn't import all necessary NPCap functions but i don't know if that is responsible.

Am I doing something wrong? Any suggestion?

Anders
  • 64,406
  • 24
  • 178
  • 215
Lucas Cioffi
  • 177
  • 1
  • 6
  • 3
    divide and conquer: do dns lookups first then pipe the IPs to a scanner – schroeder Jan 09 '18 at 17:42
  • 1
    When privileged, nmap by default will send tcp syn to port 443 and port 80, icmp echo request and icmp timestamp request, if any of these result in success, host will be marked as live. Often, the icmp is being blocked by firewall. – Tryna Learn Somethin Jan 09 '18 at 17:47
  • 1
    Strange that your scan is taking a long time. I was able to scan an entire /16 network in about 37 seconds. Is the windows version significantly slower? What hardware/network speeds are you running? – Mrdeep Jan 09 '18 at 17:50
  • Mrdeep - pretty good speeds, that shouldn't be the problem. Tryna - i'm almost sure ICMP isn't being blocked by a firewall. Schroeder - I think i'll do that *thoughtful emoji* – Lucas Cioffi Jan 09 '18 at 21:14
  • @schroeder has a good idea. Nmap's forward lookups of hostnames are one of the slowest parts of it because they are not parallelized/asynchronous. Something to fix, I guess. – bonsaiviking Jan 10 '18 at 03:32

2 Answers2

2

There are tools made for large-scale scanning that have implemented optimizations exactly for solving this problem. One of the best known and most developed options is masscan. It can reach speeds of 10 MPPS (Million Packets per Second) and can scan the entire IPv4 internet in ~6 minutes. It even has flags that let you benchmark on your local network without spamming the whole internet. Masscan has optimizations that don't require your OS to maintain full TCP/IP connections just to see which hosts are responding.

David
  • 15,814
  • 3
  • 48
  • 73
1

You could do this backwards.

  1. Nmap scan all the numeric IP addresses in the block, save live ones into a file
  2. Resolve those IP's into hostnames. (I suspect the DNS server is slowing things down deliberately, while simultaneously alerting the security watch officer.)
  3. Do a diff to identify which ones match your list.

I'd do it this way, if my goal were to identify "at a point in time" what hosts were up... resolving their name could come later, and take all the time it wants. Just a thought.

Russ
  • 161
  • 2
  • So, problem is... I don't know which blocks these hostnames came from. And i don't think scanning every IP in the park is feasible... What i wanted was some way for nmap to timeout on the DNS request and maybe scan a bunch of hostnames in parallel. – Lucas Cioffi Jan 09 '18 at 21:15
  • I'm not an nmap user, but I recall there is a rich set of command line options for doing things in parallel, specifying timeouts, and the like. If the parallelization idea doesn't work in the tool itself, maybe you can split your hostnames file into N units. I found two ways of doing this, if you are on UNIX. – Russ Jan 10 '18 at 13:33
  • First is using split: `split -l 5000 hostnames.txt host5k` which will put the first 5000 lines into "host5kaa," the next 5000 into "host5kab," and so on. You can then invoke 4 nmaps each with their own file in parallel for all 20,000. (Split as fine as you want). – Russ Jan 10 '18 at 13:39
  • Second (my new fav) is sed: `sed -n '0~3p' hostnames.txt > host0.txt` then `sed -n '1~3p' hostnames.txt > host1.txt` and `sed -n '2~3p' hostnames.txt > host2.txt`, the semantic being 'N~STEPp' starts at line N and steps by STEP, so first file gets 3, 6, 9, ... second gets 1, 4, 7, 10, and third gets 2, 5, 8, 11, ... and of course run separate nmap processes. Credit to [this answer at superuser](http://superuser.com/a/396557). – Russ Jan 10 '18 at 13:42