Is there a more efficient way to use grep?

2

I have been using grep to search databases. I've been using the following command when searching multiple folders with multiple text files.

grep 'text_search' */*

is there a better command that will speed up the process of searching through large amounts of data? I am open to using other tools if needed.

dsec35

Posted 2019-07-25T16:35:33.307

Reputation: 23

Answers

4

I'm a fan of ripgrep

If you prefer to stick with grep, grep -F matches strings not patterns (which may or may not be faster; I'm not sure if modern greps simplify a simple pattern to a string search).

Running grep in parallel is also an option. I use GNU parallel for this.

find . -type f | parallel --jobs {#jobs} -n 500 -k -m grep -H {search-pattern} {}

(jobs and search-pattern aren't surrounded in braces; they indicate a variable you need to enter)

meangrape

Posted 2019-07-25T16:35:33.307

Reputation: 41

Theres also ack and ag, off the top of my head, and fzf depending on what youre doing. – D. Ben Knoble – 2019-07-26T01:08:58.010

I'd avoid parallelism if you have a rotational hard disk, however fast it may be. Or at least benchmark it with a cold cache (echo 3 > /proc/sys/vm/drop_caches) before settling on it as a real benefit. The only time I have benefited from any parallelism is when the data can fit in RAM and I have to make multiple greps, one after another. In that case, the first one is slow (it reads from disk), but subsequent ones are very fast. – sitaram – 2019-07-29T01:10:30.953