I have Lighttpd on an Ubuntu server. I just checked the access logs from lighttpd for a particular domain. This domain only has a very simple index.html
file that basically says "coming soon". Below are 10 of the most recent ones. I don't completely understand this. Why are search engine bots trying these weird subdomains and URLs? I've found the following bots doing weird things: mail.ru, bing, baidu. Google and Yahoo are not found in the logs. I've changed the domain to example.com
to protect it of course.
217.69.133.239 power-steering-pump-ford.example.com - [31/Dec/2014:05:17:37 -0500] "GET /robots.txt HTTP/1.1" 404 345 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
217.69.133.240 power-steering-pump-ford.example.com - [31/Dec/2014:05:17:39 -0500] "GET /bedroom-boy-furniture-quality.html/ HTTP/1.1" 404 345 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
217.69.133.238 power-steering-pump-ford.example.com - [31/Dec/2014:05:17:44 -0500] "GET /10-car-hottest-top.html/ HTTP/1.1" 404 345 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
157.55.39.173 best-mixed-drink-recipes.example.com - [31/Dec/2014:05:26:43 -0500] "GET / HTTP/1.1" 200 187 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
217.69.133.234 cannon-printer-model-mp450.example.com - [31/Dec/2014:05:31:49 -0500] "GET /robots.txt HTTP/1.1" 404 345 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
217.69.133.240 cannon-printer-model-mp450.example.com - [31/Dec/2014:05:31:50 -0500] "GET / HTTP/1.1" 200 187 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
217.69.133.240 smart-car-bike-rack.example.com - [31/Dec/2014:05:31:52 -0500] "GET /robots.txt HTTP/1.1" 404 345 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
217.69.133.238 smart-car-bike-rack.example.com - [31/Dec/2014:05:31:54 -0500] "GET / HTTP/1.1" 200 187 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
202.46.53.179 winter-clothing-for-kids.example.com - [31/Dec/2014:05:52:05 -0500] "GET / HTTP/1.1" 200 230 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
180.76.4.195 winter-clothing-for-kids.example.com - [31/Dec/2014:05:52:47 -0500] "GET / HTTP/1.1" 200 230 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"