How can I combine two commands to tally data from compressed and uncompressed log files?

Question

I am running this command to get a tally of failed login attempts from a server’s auth.log and it works well:

sudo cat /var/log/auth.{log,log.1} | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

But the issue is—like every server in the world—I have some GZipped archives created by log rotation that won’t be parsed by this command:

-rw-r----- 1 syslog adm  7822722 Oct 31 13:44 /var/log/auth.log
-rw-r----- 1 syslog adm 12532511 Oct 25 06:59 /var/log/auth.log.1
-rw-r----- 1 syslog adm  2250939 Oct 18 06:55 /var/log/auth.log.2.gz
-rw-r----- 1 syslog adm  2139669 Oct 11 07:06 /var/log/auth.log.3.gz
-rw-r----- 1 syslog adm  2769919 Oct  4 06:54 /var/log/auth.log.4.gz

To deal with those I have a variant of the command that uses zcat instead of cat but it’s a bit of a nuisance to have to run two commands to get this data. Is there any way to combine the cat and zcat commands into one command so I can get combined results? I know I could write some Bash script to filter through compressed versus uncompressed files, but I use this current one-liner on various servers and I want to have a one-liner I can quickly refer to and use when needed.

More details.

I’ve been doing some security auditing and firewall tweaking and have this command I run that gives me a nice tally of “Failed password” attempts from the auth.log files on an Ubuntu Linux machine:

sudo cat /var/log/auth.{log,log.1} | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

It works great! The output looks something like this:

 5909 Oct 18
13444 Oct 19
  351 Oct 20
  162 Oct 21
  499 Oct 22
  377 Oct 23
  145 Oct 24
10897 Oct 25
   76 Oct 26
   54 Oct 27
  310 Oct 28
 1024 Oct 29
  208 Oct 30
   30 Oct 31

And while this works well for uncompressed logs, since the logs rotate and get compressed there are always some GZip compressed files that would be nice to tally up as well. So I run this variant of the above command that uses zcat:

sudo zcat -q /var/log/auth.log* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

And the output is something like this:

gzip: /var/log/auth.log: not in gzip format

gzip: /var/log/auth.log.1: not in gzip format
  10413 Sep 27
  15977 Sep 28
  12297 Sep 29
  14438 Sep 30
  23394 Oct 1
  12912 Oct 2
  15844 Oct 3
  19697 Oct 4
  15350 Oct 5
  12358 Oct 6
  12692 Oct 7
   8377 Oct 8
  10875 Oct 9
    565 Oct 10
  16027 Oct 11
  10422 Oct 12
   6808 Oct 13
  26891 Oct 14
   9493 Oct 15
   5138 Oct 16
   9415 Oct 17
   2226 Oct 18

As you can see, the output works when it works, but it would be nice if the cat and zcat variants of this command could simply be combined into one command. How can that be done?

Bonus Points:

These are not critical issues, but it would be nice if somehow they could be addressed in a solution:

• Note that at the top of the zcat output there are two errors from zcat attempting to process two uncompressed files. Suppressing that would be nice.
• Also note how the cat and zcat commands show data for October 18th; a day when log rotation happened. Would there be any way to add up those two values in the command? If not, I’m fine with having two lines of October 18th data with different values I would have to add up manually.

Answer 1

Can use zgrep which will decompress if necessary and therefore works with both straight text and compressed input. As well grep/zgrep can process multiple files directly which is needed in this case since mixed compressed and text to standard input doesn't always work as intended. Supress filenames in the grep output with -h or --no-filename.

sudo zgrep -h 'Failed password' /var/log/auth.* | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

Man page:

ZGREP(1)                                                              ZGREP(1)

NAME
       zgrep - search possibly compressed files for a regular expression

SYNOPSIS
       zgrep [ grep_options ] [ -e ] pattern filename...

DESCRIPTION
       Zgrep  invokes grep on compressed or gzipped files.  All options specified
       are passed directly to grep.  If no file is specified, then  the  standard
       input  is  decompressed if necessary and fed to grep.  Otherwise the given
       files are uncompressed if necessary and fed to grep.

       If the GREP environment variable is set, zgrep uses it as the grep program
       to be invoked.

AUTHOR
       Charles Levert (charles@comm.polymtl.ca)

SEE ALSO
       grep(1), gzexe(1), gzip(1), zdiff(1), zforce(1), zmore(1), znew(1)

Answer 2

While ugly to type each time, you could do something like:

for log in /var/log/auth.log*; do if ! sudo zcat "$log" 2>/dev/null; then sudo cat "$log"; fi; done | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

Probably it would be better to create a function for at least the cat part, something like:

getLogs() {
    for log in /var/log/auth.log*; do
        if ! sudo zcat "$log" 2>&/dev/null; then
            sudo cat "$log"
        fi
    done
}

Then your command can be

getLogs | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

or you can make the whole thing be a function:

getSSHFailures() {
    for log in /var/log/auth.log*; do
        if ! sudo zcat "$log" 2>/dev/null; then
            sudo cat "$log"
        fi
    done | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c
}

Then you just call

getSSHFailures

You could then define that function in your .bashrc and have that function available in each shell you start

Also, this should also fix both your observation about the error trying to zcat non compressed files (because we redirect zcat errors to /dev/null) and get rid of the duplicate entry, because we're printing all the results together as a single stream before processing them.

Finally, awk can actually do the pattern matching for us, so we can also get rid of the 2 greps like:

getSSHFailures() {
    for log in /var/log/auth.log*; do
        if ! sudo zcat "$log" 2>/dev/null; then
            sudo cat "$log"
        fi
    done | awk '/sshd.*Failed password/ {print $1,$2}' | sort -k 1,1M -k 2n | uniq -c
}