How can I count commands in a log file?

Question

I have hundreds of log files in a single directory, and I want an automated way to browse through the log files to count the number of times each command is used.

• Log files are text files
• There are 100+ different commands
• they all start as "Command: " and then the name of the command.
• It's one line per command
• other lines should be ignored

So I'd like to know if there's a script or software that would do that and sort the results by most used commands, and display the frequency of each command. Something like statistics.

(edit: I'm on Win XP and I have no problems installing programs for this purpose)

Answer 1

With Windows PowerShell you could do something along the lines of:

Get-Content *.log |
    Where-Object { $_.StartsWith('Command:') } |
    Group-Object {
        $null = $_ -match '^Command: (\w+)';
        $Matches[1]
    } |
    Select-Object Name,Count

For my test file this yields an output like

Name Count
---- -----
foo      2
bar      2
baz      1

Above code simply reads the log files line by line, pushing each line through the pipeline, it then filters the lines to only use those that start with “Command:”, indicating a command to follow. Then those lines are grouped into the individual commands. This is done by the regular expression

 ^Command: (\w+)

which matches the string “Command:” at the start of the line, followed by one or more word characters. This assumes the command name follows the colon and space immediately; adjust the regex accordingly if this is not the case. The command name is captured in a capture group which is used for grouping. After that only the name and frequency of the commands are selected.

The $null = part for the match is to suppress the output of the -match operator which would return always True here. We don't want to group by True bar but only by bar.

ETA: Depending on how exactly your input looks, you might want to tweak things a bit.

• Allow empty commands:

   ^Command: (\w*)
  

• Allow arbitrary non-space characters in command names (and empty commands):

   ^Command: ([^ ]*)
  

Answer 2

What OS?

This isn't a perfect match to your requirement, but...

grep -i -c Command *.log

will get you the number of occurrences broken down by file. You could then parse that output to get your totals. Dump it to a text file and pull it into Excel or something. Not hugely difficult, but admittedly not completely automated either. I often find that I spend more time searching for a completely automated solution than I do using something like this... You can get grep for Windows using cygwin.