Count amount of new lines with interval in log

Question

Is there a way/tool to tail or watch a logfile and report the amount of lines it grows?

E.g. I want to see the amount of new entries in mysql.log every 3 seconds.

watch wc -l /path/to/log.log1 shows only acumulated amount. I'd prefer not to truncate the log inbetween.

score 1 · Accepted Answer · answered Dec 10 '10 at 02:04

1

All you have to do is redirect the file into wc and it will only show what was added rather than the total.

watch wc -l < /path/to/log.log1

The < makes all the difference.

answered Dec 10 '10 at 02:04

Dennis Williamson

60,515
14
113
148

score 0 · Answer 2 · answered Dec 09 '10 at 19:22

Here's some quick-and-dirty perl to do what you want:

#!/usr/bin/perl

open(I, "<$ARGV[0]") || die "Can't open $ARGV[0]: $!\n";

while (1) {
  my $count = 0;
  while (<I>) { $count++; }
  print scalar(localtime()), ": $count lines\n";
  sleep(3);
  seek(I, 0, 1); # clear the EOF on I
}

close(I);

Here's the output of it running in one window:

[root@g3 tmp]# perl tt.pl FILE
Thu Dec  9 13:18:38 2010: 0 lines
Thu Dec  9 13:18:41 2010: 0 lines
Thu Dec  9 13:18:44 2010: 0 lines
Thu Dec  9 13:18:47 2010: 0 lines
Thu Dec  9 13:18:50 2010: 0 lines
Thu Dec  9 13:18:53 2010: 1 lines
Thu Dec  9 13:18:56 2010: 0 lines
Thu Dec  9 13:18:59 2010: 1 lines
Thu Dec  9 13:19:02 2010: 0 lines
Thu Dec  9 13:19:05 2010: 0 lines
Thu Dec  9 13:19:08 2010: 0 lines
Thu Dec  9 13:19:11 2010: 0 lines
Thu Dec  9 13:19:14 2010: 0 lines
Thu Dec  9 13:19:17 2010: 0 lines
Thu Dec  9 13:19:20 2010: 100 lines

And here's the commands I ran in another window to generate the output above:

[root@g3 ~]# cd /tmp
[root@g3 tmp]# echo foo > FILE
[root@g3 tmp]# echo foo >> FILE
[root@g3 tmp]# for i in `seq 1 100`
> do
> echo $i >> FILE
> done
[root@g3 tmp]#

Hope that helps...

score 0 · Answer 3 · answered Dec 09 '10 at 19:23

I don't know any standard tool but you could use the following Python script:

import sys, time

if not len(sys.argv) >= 2:
    print 'usage: %s FILENAME FREQUENCY' % (sys.argv[0])
    sys.exit(1)

fd=open(sys.argv[1])
freq=int(sys.argv[2])
fd.seek(0, 2)
while True:
    current = fd.tell()
    nlines = len(fd.readlines())
    if nlines:
        print nlines
    time.sleep(freq)

To test:

$ (while true; do echo test; sleep 0.2; done) > log &
$ python watchandcount.py log 3

It is not perfect because it cannot "monitor" STDIN or detect when the file has been replaced (the --follow=name tail's option.

Chose this one bacause the server had python (and not perl) and my Bash is poor. :) — berkes, Dec 09 '10 at 20:07

score 0 · Answer 4 · answered Dec 09 '10 at 19:30

And here is a bash script:

TOTAL=0
while true; do
  NEW_TOTAL=$(wc -l /path/to/log.log1 | cut -f1 -d' ')
  echo $(($NEW_TOTAL - $TOTAL))
  TOTAL=$NEW_TOTAL
  sleep 3
done

It has the advantage of being usable directly in the shell, provided you are not alergic to ";" ;)

TOTAL=0; while true; do NEW_TOTAL=$(wc -l TOTAL=0; while true; do NEW_TOTAL=$(wc -l big | cut -f1 -d' '); echo $(($NEW_TOTAL - $TOTAL)); TOTAL=$NEW_TOTAL; sleep 3; done | cut -f1 -d' '); echo $(($NEW_TOTAL - $TOTAL)); TOTAL=$NEW_TOTAL; sleep 3; done

Count amount of new lines with interval in log

4 Answers4