Get list of user-agents from nginx log

Question

I have nginx log file, and I want to find out market share for each major version of browsers. I am not interested in minor versions and operating systems. I would like to get something like this:

100 IE6
 99 IE7
 20 IE8
200 FF2
300 FF3

I know how to get the list of user agents from the file, but I want to aggregate the list to see only the major versions of the browsers. Is there a tool that does it?

SaveTheRbtz · Accepted Answer · 2013-05-22T16:33:06.820

23

awk -F'"' '/GET/ {print $6}' /var/log/nginx-access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn

awk(1) - selecting full User-Agent string of GET requests
cut(1) - using first word from it
sort(1) - sorting
uniq(1) - count
sort(1) - sorting by count, reversed

PS. Of course it can be replaced by one awk/sed/perl/python/etc script. I just wanted to show how rich unix-way is.

edited May 22 '13 at 16:33

answered Dec 01 '09 at 19:10

SaveTheRbtz

5,621
4
29
45

Thanks, this looks interesting. I will try it out immediately. :) – Željko Filipin Dec 02 '09 at 11:14
I am trying to get your code to work. Could you please provide more information about each command? (I will do some research too.) – Željko Filipin Dec 03 '09 at 10:54
you can use `man grep`, `man awk` etc. – SaveTheRbtz Dec 03 '09 at 15:55
Thanks, will do that. I was not explicit enough. The only thing confusing me so far is `{print $6}`, but I guess it represents the piece of data in a line. – Željko Filipin Dec 04 '09 at 09:16
it's generic awk script. It prints 6th column using `"` as separator – SaveTheRbtz Dec 04 '09 at 11:48
1

Thanks a lot. I guess my log is not standard, but I just had to change $6 to $8 to get it working. – Željko Filipin Dec 31 '09 at 14:35

score 7 · Answer 2 · edited Apr 13 '17 at 12:14

While the one liner by SaveTheRbtz does the job, it took several hours to parse my nginx access log.

Here is a faster version based on his, which takes less than 1 minute per 100MB of log file (corresponding to about 1 million lines):

sed -n 's!.* "GET.* "\([[:alnum:].]\+/*[[:digit:].]*\)[^"]*"$!\1!p' /var/log/nginx/access.log | sort | uniq -c | sort -rfg

It works with the default access log format of nginx, which is the same as the combined format of Apache's httpd and has the User-Agent as the last field, delimited by ".

score 3 · Answer 3 · answered Feb 17 '20 at 07:15

3

To get user agent

sudo awk -F"\"" '{print $6}' /var/log/nginx/access.log | sort | uniq -dc

answered Feb 17 '20 at 07:15

Arun

131
2

score 3 · Answer 4 · edited Sep 25 '17 at 07:44

3

This is a slight variation of the accepted answer, using fgrep and cut.

cat your_file.log | fgrep '"GET ' | cut -d'"' -f6 | cut -d' ' -f1 | sort | uniq -c | sort -rn

There is something appealing about using "weaker" commands when it is possible.

edited Sep 25 '17 at 07:44

Aakash Jain

103
3

answered Jan 19 '14 at 02:16

solidsnack

131
3

score 2 · Answer 5 · answered Dec 01 '09 at 12:57

2

Awstats should do the trick, but will supply far more information. I hope this helps...

answered Dec 01 '09 at 12:57

Kevin Worthington

327
2
6
19

Thanks, Awstats looks similar to Webalizer. I will try it out. – Željko Filipin Dec 01 '09 at 14:23
Awstats was to complicated to install. – Željko Filipin Dec 03 '09 at 10:52

Željko Filipin · Answer 6 · 2009-12-01T15:07:53.943

2

Webalizer can do it.

Example:

webalizer -o reports_folder -M 5 log_file

-o reports_folder specifies folder where report is generated
-M 5 displays only the browser name and the major version number
log_file specifies log file name
source: ftp://ftp.mrunix.net/pub/webalizer/README

edited Dec 01 '09 at 15:07

answered Dec 01 '09 at 14:19

Željko Filipin

223
1
3
12

score 0 · Answer 7 · answered Dec 01 '09 at 16:17

0

I'd use shell script for that: cat, awk pipe, sort and uniq will do the job

answered Dec 01 '09 at 16:17

alexus

12,342
27
115
173

Thanks. I know how to parse logs. I was looking for a tool that already knows how to do it, so I do not have to write a script. – Željko Filipin Dec 02 '09 at 11:07

Get list of user-agents from nginx log

7 Answers7

Linked