mod_security rule 960015 keeps catching Google and other good bots. I have the following in the vhost to prevent good bots from being caught:
SecRule REQUEST_HEADERS:User-Agent "Mail.ru" log,allow
SecRule HTTP_USER_AGENT "Mail.RU_Bot" log,allow
Same for Google and Yandex.
It works 99% of the times, however fails at other times for some really bizare reason, here are the logs example for Mail.ru bot:
Successfull:
217.69.134.79 - - [07/Mar/2014:10:17:13 +0400] "GET / HTTP/1.1" 200 189934 "-"
"Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0;
+http://go.mail.ru/help/robots)"
[Fri Mar 07 10:17:13 2014] [error] [client 217.69.134.79] ModSecurity: Access
allowed (phase 2). Pattern match "Mail" at REQUEST_HEADERS:User-Agent.
[file "/etc/apache2/sites-enabled/xxx"] [line "28"] [hostname "xxx"]
[uri "/"] [unique_id "UxlkaQp-d4EAABU9BSIAAAAV"]
And next minute it fails:
217.69.134.79 - - [08/Mar/2014:02:14:19 +0400] "GET / HTTP/1.1" 403 389 "-" "
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/
help/robots)"
[Sat Mar 08 02:14:19 2014] [error] [client 217.69.134.79] ModSecurity: Access
denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS.
[file "/usr/share/modsecurity-crs/activated_rules/
modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"]
[rev "2.2.5"] [msg "Request Missing an Accept Header"] [severity "CRITICAL"]
[tag "PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"]
[tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "xxx"] [uri "/"]
[unique_id "UxpEuwp-d4EAAEMnBFQAAAAE"]
I know the proper way is to do reverse lookups, however they slow down the website, and I want to have at least some security but as it is at the moment cant use the 960015 because it blocks Google and others. In the same time it is a very usefull rule that caught 100s of bad bots.
If someone knows how to set it up with reverse lookup that will actually work and allow Google and other good bots to index - you are welcome to post here. However I am also looking for a quick and dirty solution to make it work right now, since some security is better then no security.