Live (e)grepping on a file

Question

Every week or so I need to load a file that has millions of lines and start running greps on it. Some greps are positive, some are negative ("-v" flag). Some are grep and some are egrep.

I do it manually today - run a grep, save the results to a file, then run another grep on this file and save to file2, then file3, etc.

Is there some tool that can make my life easier where I can easily manipulate a list of greps, egreps, that the tool will calculate and show the results live? The tool needs to cache the results of previous greps to avoid re-running them all each time.

Answer 1

I'm not quite sure what you mean by "show the results live" (but see the end of my answer for an idea) but generally I'd use make for this sort of task. For example, given a file srcfile.txt on which I'm going to run a number of greps, I'd put the greps in a makefile something like this:

all: e.out b.out zy.out

single: e.out b.out

e.out: srcfile.txt
    grep e srcfile.txt >e.out

b.out: srcfile.txt
    grep b srcfile.txt >b.out

zy.out: srcfile.txt
    grep zy srcfile.txt >zy.out

... I can then run all the greps that are required to bring the output files up-to-date with

> make

... I can run all the greps that search for single letters with

> make single

... etc. Over a few weeks in your situation, I think I'd quickly accumulate a set of make rules that covered all (or almost all) of the cases in which I was interested.

You might be able to save some more time and effort by putting the commands to load your million+ line file in the makefile as well.

If choosing one or more targets using the make command doesn't provide enough flexibility, you could use conditionals in the make file, based on variables that you set when running make, e.g. see the GNU makefile tutorial on conditionals and this article on passing variables from the make command line.

To show the results of the grepping as it happens, I would be inclined to use tail or less as recommended here. A difficulty is that either method only shows one file at a time. I'd need to know more about your need for "live" viewing to know if that might be satisfactory or if another solution might be required.