Where or how can I get list of domain dependencies of top 100 popular sites?

Question

For example google.com depends on ssl.gstatic.com, lh3.googleusercontent.com, fonts.gstatic.com.

I need this list for Proxy settings

Could you please explain how did you determine that *`google.com* depend on `googleusercontent.com` and `gstatic.com`*? — F. Hauri - Give Up GitHub, Dec 11 '17 at 09:09
@f-hauri If these 2 domains will be blocked, current version of google.com won't work properly — Rustam, Dec 11 '17 at 15:41

F. Hauri - Give Up GitHub · Answer 1 · 2017-12-11T09:47:47.197

About dns domain dependencies

Domain dependencies are only registrars. There is nothing else.

But you could ask for:

whois gstatistic.com | grep -A1 Holder

Hmm. nothing? Ok looking for names servers:

whois gstatic.com | grep Name.Server
   Name Server: NS1.GOOGLE.COM
   Name Server: NS2.GOOGLE.COM
   Name Server: NS3.GOOGLE.COM
   Name Server: NS4.GOOGLE.COM

and

whois googleusercontent.com | sed '/Name Server/q;d'
   Name Server: NS1.GOOGLE.COM

Also, building a proxy (meaning web proxy), is a HTTP concern. What you call dependencies are only web links to other repositories where to find some objects.

Web domain dependencies

Web pages depend on a lot of things:

scripts: Libraries could be stored on development site
fonts: Fonts are generally located on fonts servers.
styles: Styles could be linked to scripts, stored on development servers.
videos: Mostly located on stream servers.
map: Geographic maps, stellar maps, oceanographic, human... And other specialised big data systems..
photos, images, others...: Some copyrighted files could only be shown from original site (artist, photograf...),

But web domains strongly depends on

advertiser where links to advertiser host is required for log count and further analisys.
captcha I hate captcha but...
online banks required for web shops.

Where to get such a list

On your web history, cache and mostly on the log of your proxy server...

Unfortunely such a list could not exist, because every web site could be build and modified at any time.

So if you plan to build a strongly restrictive proxy server, you have to keep some hotline up and stay ready to change your rules, adding some new kind of third party site, at any time.

And finally, about Top 100 popular sites

Answer could only be subjectives...

You could try:

... Then have a look at your proxy logs ;-)

Tip or trick:

Have a look at my answer about How to display the Subject Alternative Name of a certificate?,

Displaying Subject Alternative Name of SSL certificates give some exhaustive list of directly related domains.