We have trouble on the AWStats background job reading the access.log file: the awstats.pl scripts takes more than 24 hours and it seems never ends.
We have a web site which has more than 8 million page views per day, generating a 2 GB Apache access.log file every day.
Here is the command log when we tried to run the awstats.pl scripts manually:
root@hostname:~# /usr/lib/cgi-bin/awstats.pl -config=org.mysite -update
Create/Update database for config "/etc/awstats/awstats.org.mysite" by AWStats version 7.4 (build 20150714)
From data in log file "/var/log/apache2/org.mysite-access.log"...
Phase 1 : First bypass old records, searching new record...
Direct access to last remembered record is out of file.
So searching it from beginning of log file...
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
Flush history file on disk (unique url reach flush limit of 5000)
^C
We Use ctrl+C to stop here, since it has been taken more than 1 hour.
We have tried to Disable the DNS lookup (set as 0), well it did not help:
Edit the file /etc/awstats/awstats.org.mysite.conf
# 0 - No DNS Lookup
# 1 - DNS Lookup is fully enabled
# 2 - DNS Lookup is made only from static DNS cache file (if it exists)
# Default: 2
DNSLookup=0
The hardware should not the bottleneck, it is the ovh.com HOST-128L dedicated server
- Intel Xeon D-1520 - 4 core / 8 thread
- 128GB DDR4 ECC 2133 MHz
- 2 x 480 GB / SSD
The operation system is Ubuntu 16.04.3 LTS
root@hostname:~# cat /etc/*-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
VERSION_CODENAME=xenial
So,
- Do we have anything to do to improve the AWStats performance?
- Or are we hitting the performance threshold of AWStats?
