0

I noticed that my website isn't archived anymore by web.archive.org. When I look at http://web.archive.org/web/*/http://www.cnn.com it is clearly visible that it stopped working in july 2008. web.archive.org has a 6-month-delay policy. This means that 6 month after the crawl run it should be visible to the enduser. So it should have a crawl run from end of 2009 available and not only from mid 2008.

What can I do to get it working again for me and is there a official statement somewhere?

1 Answers1

1

The last time their FAQ was updated, the visibility delay was "6-14 months" so it's possible that delay has extended, and you are seeing normal behaviour.

Check that your web site doesn't have a robots.txt file that is telling crawlers not to index it.

That FAQ also lists two other ways to get your site indexed: install the Alexa toolbar in your browser and visit your site; or go to http://www.dmoz.org/ and use the "Suggest URL" form.

William
  • 1,158
  • 8
  • 9
  • Yes, it is already submitted to dmoz - http://www.dmoz.org/News/ And the robots.txt only tells the crawler to ignore some specific folders. But it seems http://web.archive.org/web/*/http://www.foxnews.com/ also has that problem. So nothing I can do on my server –  Mar 05 '10 at 12:13