1

I work on Sharepoint 2007 intranet with 500+Gb of documents. Running a full crawl takes over 48 hours. When we first set Sharepoint up, and it was much smaller, we ran Full crawl jobs on a weekly basis, and incrementals each night.

Is there any benefit to running full crawls? Or should I reduce the frequency to monthly - or even do it less frequently than that?

dunxd
  • 9,482
  • 21
  • 80
  • 117

1 Answers1

1

A weekly or even regularly scheduled full crawl, normally, isn't necessary.

Based on Microsoft's recommendations ([Article][1]), you would only need to manually do a full crawl after your initial full crawl in the following cases (FTA):

  • One or more hotfix or service pack was installed on servers in the farm. See the instructions for the hotfix or service pack for more information.

  • An SSP administrator added a new managed property.

  • To re-index ASPX pages on Windows SharePoint Services 3.0 or Office SharePoint Server 2007 sites.

  • To resolve consecutive incremental crawl failures. In rare cases, if an incremental crawl fails one hundred consecutive times at any level in a repository, the index server removes the affected content from the index.

  • Crawl rules have been added, deleted, or modified.

  • To repair a corrupted index.

  • The search services administrator has created one or more server name mappings.

  • The account assigned to the default content access account or crawl rule has changed.

Also full crawls are performed automatically (when an Incremental crawl is requested) in the following scenarios (FTA):

  • A SSP administrator stopped the previous crawl.

  • A content database was restored from backup.

  • A farm administrator has detached and reattached a content database.

  • A full crawl of the site has never been done.

  • The change log does not contain entries for the addresses that are being crawled. Without entries in the change log for the items being crawled, incremental crawls cannot occur.

  • The account assigned to the default content access account or crawl rule has changed.

  • To repair a corrupted index.

HTH, Dan

SQL3D
  • 670
  • 1
  • 6
  • 11
  • Great answer! I guess the current set up is due to some superstition about full crawls being somehow better than incremental, or some idea there is a risk of index corruption over time. I'll switch off the full crawls and let Sharepoint decide what to do. – dunxd Sep 14 '10 at 21:06
  • By the way - the link didn't get added to your answer. Please could you add it, in case I need to back up this decision with something "official" ;-) – dunxd Sep 14 '10 at 21:07
  • 1
    Sorry about that. Here is the link: http://technet.microsoft.com/en-us/library/cc262926(office.12).aspx – SQL3D Sep 14 '10 at 21:22
  • BTW, when I first started managing SharePoint, I made the same assumptions. Like you, my full crawls started to take too long. So I went with the regularly scheduled incremental crawls with the occasion full crawl should we change something, etc. – SQL3D Sep 14 '10 at 21:25
  • Switched off the full crawls, and now seeing much less pressure on the index server. Thanks for your help Dan! – dunxd Sep 20 '10 at 08:33