0

I am about to use OWASP CRS rules with mod_security on my WHM/cPanel enabled CentOS server (with apache). But I fear that accidentally Googlebot may be blocked by one rule or the other. After enabling OWASP CRS, If I add the below custom rule separately, will my rule prevent Googlebot from being blocked by any of the CRS rules. Will the below rule override any CRS rule that may suspect Googlebot.

The proposed custom rule: SecRule REMOTE_HOST googlebot.com$ allow,pass

Update: My website has thousands of URLs and google's crawl rate is between 10K to 50K urls per day. Hence, there is every chance that some brute force rule in OWASP CRS may be triggered for googlebot.

Kannan
  • 101
  • 2

1 Answers1

1

In the document about veryfying gooblebot, crawlers might come both from googlebot.com or google.com.

Assuming that you don’t check user-agent (easily spoofed), you need to do something like:

SecRule REMOTE_HOST “@rx google(bot|)\.com$” “id:50000,nolog,allow”

Also, this will only work if you enabled the Apache directive HostnameLookups On. Otherwise, you’ll only get IPs.

Felipe
  • 51
  • 4
  • But can any OWASP CRS rules override this rule? Is there a way to tell mod_security to ignore all other rules when this rule is executed? – Kannan Jun 12 '21 at 02:34
  • You are explicitly allowing it here. If you want to remove all CRS rules, you can add to the actions: `ctl:ruleRemoveByTag=OWASP_CRS`, which is the recommended way of "disabling" OWASP CRS for anything related this particular transaction. – Felipe Jul 14 '21 at 13:08