How to print statistics from lines in a file

Question

I often find myself computing percentages manually after filtering through a log file to find the proportion of X or Y in it. Can this easily be achieved via common CLI tools?

Answer 1

Typically I will identify a few traits in a log file that I want to distinguish on, and get percentages from. This can be done easily with sed, replacing anything you don't need in every line, and then counting the occurrence of each. For example to distinguish between Linux and Windows hits in a log file, you could do:

$ cat some.log | sed -r 's/.*(Windows|Linux).*/\1/' | sort | uniq -c | sort -rn
23940 Windows
12390 Linux

This gets you the absolute count for each trait you are looking for, but not percentage so it's not ideal yet.

It seems awk can not easily loop twice over the lines to first compute the total and then output percentages, but with a small hack we can first add a line on top that shows the sum of all matched traits:

$ ... | awk '{s+=$1;lines=lines"\n"$0} END {printf "%d Total",s;print lines}' 
Total 36330
Windows 23940
Linux 12390

Finally now that we have the total, we can easily compute and print percentages using this:

$ ... | awk '!max{max=$1}{s=$1/max*100;c=$1;$1="";printf "%30s %10d %7.2f%%\n",$0,c,s;}'
Total     36330   100.00%
Windows   23940    65.90%
Linux     12390    34.10%

The combined one-liner would then be:

cat some.log | sed -r 's/.*(Windows|Linux).*/\1/' | sort | uniq -c | sort -rn | awk '{s+=$1;lines=lines"\n"$0} END {printf "%d Total",s;print lines}' | awk '!max{max=$1}{s=$1/max*100;c=$1;$1="";printf "%30s %10d %7.2f%%\n",$0,c,s;}'

Where some.log is the file you want to inspect, and Windows|Linux is a pipe-delimited list of terms to match on/distinguish.

Should you want to remove the Total line at the end since it becomes slightly irrelevant, you can append | tail -n +2 to it.