Block Crawlers on Apache

Question

I am trying to block crawlers on Apache

    <Directory /to/my/site/>
        Options Indexes FollowSymLinks
        AllowOverride All
        Order deny,allow
        deny from 18.*.*.*
        deny from 35.*.*.*
        deny from *.us-west-2.compute.amazonaws.com
        deny from *.*.compute.amazonaws.com
        Allow from all
    </Directory>

However this does not seem to block traffic coming from any of these IPs. What am I missing?

score 1 · Accepted Answer · answered Jan 28 '20 at 06:55

First you should try to add a robots.txt to your web page to stop polite crawlers.

Your Deny statements do not work, since you use a not supported syntax for wildcard (cf. Apache's documentation). The directives should be:

Deny from 18 35
Deny from .compute.amazonaws.com

Block Crawlers on Apache

1 Answers1