I moved a web service to a new server. I figured out that, in the old server, the following gives approximately the same number of hits as awstats (e.g., for a given day the following gives 5537, whereas awstats indicates 5557 hits):
grep -v bot myaccess.log| # file contains given vhost for given date range \
grep -v rss2email|\
grep -v Slurp|\
grep -v pider| # Ignore spiders \
egrep 'HTTP/.... (200|304) '| # Catch only 200 and 304 responses \
grep -v Wget|\
grep -v Bot|\
grep -v rawler| # Ignore crawlers \
grep -v favicon.ico|\
grep -v robots.txt|\
grep -v HTTrack|\
grep -v simplepie|\
grep -v BingPreview|\
wc -l
Adding the following to the end of the grep chain gives approximately the same number of pages (e.g. 2916 for a given day) as awstats (3042):
egrep -v '(css)|(js)|(class)|(gif)|(jpg)|(jpeg)|(png)|(bmp)|(ico)|(swf) HTTP'
Now, I moved server. Lots of things changed: apache became nginx; the log format changed; the awstats configuration has been rewritten; Debian squeeze became wheezy, and awstats 6.9.5 became 7.0.
The large pipeline above still approximates awstats's hits well (e.g. 5521 vs. 5541), but adding the egrep that excludes the NotPageList does not: I get, for a given day, 2948, whereas awstats gives 1580. (Whether the exclusion list contains rss and xml makes no significant difference.) Indeed, since the day the service was moved, the hits remain approximately the same, whereas the pages and the visits have approximately halved. I can't figure out why.
