0

Hoping someone can point out what is wrong with the LogFormat I'm attempting to use with the access logs generated by Amazon ELB.

Multi-line for ServerFault readability:

%time5 %elb %host %host_port %host_r %host_r_port %request_processing_time 
%backend_processing_time %response_processing_time %code 
%backend_status_code %received_bytes %bytesd %methodurl

Example log entry (after cleanup):

2014-08-28T17:59:14 awseb-e-2-AWSEBLoa-AAAAAAAA 123.123.123.123 44153 10.123.123.123 80 0.000046 0.536613 0.000045 200 200 0 13129 "GET /path/to/web/app HTTP/1.1"

Logs are cleaned up before getting sent to AWStats. I use the following replacements:

cat ${s3_logs_dir}/* \
| sed -e "s/\(\.[0-9]\{1,3\}\):\([0-9]\{2,5\}\)/\1 \2/g" \
| sed -e "s/\(:[0-9]\{2\}\)\(\.[0-9]\{6\}Z\)/\1/g" \
| sed -e "s/http:\/\/www\.example\.com:80//g" \
> ${combined_log} 2>>${log_file}

First I detach host from port, secondly I remove microseconds from the %time5 ISO date, and finally I tried removing the absolute URL from the request.

Matt Beckman
  • 1,512
  • 17
  • 33
  • Currently, I believe the issue is related to an AWStats bug with the %time5 ISO date. I'm confirming if an actual bug or issue with the EC2 instance. – Matt Beckman Sep 02 '14 at 18:28

3 Answers3

1

Turns out it appears to be a bug with %time5 LogFormat option in AWStats.

I converted YYYY-MM-DDTHH:MM:SS to YYYY-MM-DD HH:MM:SS and used %time2, and the logs were parsed successfully.

Matt Beckman
  • 1,512
  • 17
  • 33
1

Morning, now this is outdated, but with awstats Advanced Web Statistics 7.7 (build 20180105) and your parsing rules with this criteria it also match agent.

In current elb logs is also logs type of request even not documented, so new format is

LogFormat="%other %time5 %other %host %host_port %host_r %host_r_port %request_processing_time %backend_processing_time %response_processing_time %code %backend_status_code %received_bytes %bytesd %methodurl %uaquot"

%time5 is working now.

thx for the posting.

Kind regards David

David Roman
  • 11
  • 1
0

  • there is no dot followed by 1 to 3 digit followed by 2 dot then 2 to 5 digit in sample but a space is already in place for the port separator

    sed -e "s/(.[0-9]{1,3}):([0-9]{2,5})/\1 \2/g"

  • there is no double dot followed by 2 digit fowwloed by a dot, 6 digit and 'Z'

    sed -e "s/(:[0-9]{2})(.[0-9]{6}Z)/\1/g"

...

Also, you dont need to pipe the sed action, just separe action by a semi column s/pat11/pat12/opt1;s/pat21/pat22/opt2;other action

NeronLeVelu
  • 128
  • 4
  • I clarified in the post, but those replacements are performed before being sent to AWStats. The example log entry was what AWStats is seeing post-cleanup. – Matt Beckman Sep 02 '14 at 16:32