2

It looks like there are companies that sell the lists of domain names, yet their lists are incomplete, and each company has a different amount of entries for a particular TLD.

Why so much conspiracy about the complete list of existing/registered/active domain names? Why isn't it just available online for everyone (of course, without any additional data, just domain names)?

And the main question is - is there a right technical way to build such a list on my own?

If I understand correctly, registrars are accredited for each TLD, and then they are responsible to host zone files.

So, if one is to build the complete list, they would need to get a complete list from all registrars and join them together.

Sounds simple, yet there are no such lists available. So why is it so and how to do it properly on my own from the technical point of view?

meglio
  • 149
  • 2
  • 8

3 Answers3

5

In some sense one might argue that the "right" technical approach is to jump through whatever hoops the relevant registries require to get access to their TLD zone data or, if they do not have such a program, just not do it.

Regarding gTLDs, ICANN does provide a central repository:

The Centralized Zone Data Service provides a centralized point for interested parties to request access to Zone Files provided by participating Top Level Domain Registries. The service is the solution for scaling zone data transfer as hundreds of new gTLDs are added to the Internet.

Every new gTLD Registry is required to provide zone data to approved requesters (e.g. law enforcement agents, IP attorneys, researchers) upon technical delegation of its gTLD.

New Registries will be provided with instructions for utilizing the Centralized Zone Data Service once a gTLD Registry Agreement has been executed.

For a much longer time individual zones have been made available separately. For instance the com and net zones have been made available by Verisign (operating these zones on behalf of ICANN).


For ccTLDs there is no such convenience as a central repository and each registry may or may not have a program that allows access to the full zone data.

Registrars do not host zone files, the nameservers for the TLD are operated on behalf of the registry. The registrars feed data to the registry.

Håkan Lindqvist
  • 33,741
  • 5
  • 65
  • 90
  • So there is no any rules that would make it compulsory to make the full list of ccTLDs available by request? Is there any such a regulation - no, no luck? – meglio Jun 01 '15 at 12:12
  • @meglio Am I right to guess that you actually mean the contents of a ccTLD zone, not the list of ccTLDs? The list of all TLDs (ccTLDs included) is [readily available](https://www.iana.org/domains/root/db). – Håkan Lindqvist Jun 01 '15 at 16:24
  • Yes you are right. I need the contents of a ccTLD zone, no the list of ccTLDs. – meglio Jun 02 '15 at 22:42
  • A caveat to CZDS (or any zone file) data is that it won't contain domain names that are not delegated (e.g. have no name servers) and also names which have been explicitly excluded (see https://www.icann.org/resources/pages/epp-status-codes-2014-06-16-en) – Neil C. Obremski Apr 24 '19 at 20:46
  • @NeilC.Obremski Indeed, it is the set of "active" (in the sense of existing in dns) domain names rather than the set of registered domain names. – Håkan Lindqvist Apr 25 '19 at 06:36
3

There's no conspiracy, it's just that the list is HUGE (133,182,309 domains handled just by VeriSign (.net and .com) as of this writing, supposing an average of 13 byte-width characters per domain, that's almost 2 GB of just domain names, not to mention the associated information attached to a domain entry) and bandwidth costs money.

You can request the zone file from or sign up as a registrar (to get better access to the zone file) for the individual zone sponsor. Continuing with the VeriSign example, you can look for .net and .com here: http://www.verisigninc.com/en_US/channel-resources/domain-registry-products/zone-file/index.xhtml and http://www.verisigninc.com/en_US/channel-resources/become-a-registrar/index.xhtml

You can also sign up to access the API of a whois database (usually will cost money because bandwidth isn't cheap), or download the zone file from a site that provides access to their cached copies, like http://www.premiumdrops.com/zones.html.

Again, accessing said list for each TLD will require its own process.

Lastly, this information becomes stale the moment that you download it, since domains are being continuously registered/renewed/expired. This issue decreases the value of maintaining such a list available for download.

Addendum: This doesn't begin to cover the case where the DNS servers you (or your intended clients) are set up NOT to serve all DNS records as they are registered with their registrars, or many other cases that would make such a list useless.

austinian
  • 1,699
  • 2
  • 15
  • 29
2

No supported (anymore) by any TLD operator but the technical mechanism provided by the DNS protocol established in RFC 833 (and refined in subsequent RFC's) is a zone transfer for the TLD root zone, with the AXFR request.

Community
  • 1
HBruijn
  • 72,524
  • 21
  • 127
  • 192
  • That's very intriguing, but I'm afraid I can't understand what you mean. Could you explain in a bit more detail? – meglio May 28 '15 at 11:35