cat and grep with a huge file - Broken pipe

1

I am tryting to cat and grep a huge 40 GB file. After some output, my process get killed.

Any idea what can be the issue? Using OSX

[1]    37392 broken pipe  cat output.txt |
       37393 killed       grep --color=auto -C 20 201804090004054066628236

I can confirm its because of grep memory usage. Its memory usage shoots up to almost in GB's

Madhur Ahuja

Posted 2018-04-13T08:46:41.990

Reputation: 1 667

Answers

2

Looks like cat reported the broken pipe because grep was killed. This would typically happen if the processes consumed too much memory and the OOM (out of memory) killer was triggered. You could try without the useless cat:

grep --color=auto -C 20 201804090004054066628236 output.txt

l0b0

Posted 2018-04-13T08:46:41.990

Reputation: 6 306

1grep doesn't store very much information in memory so the OOM seems somewhat unlikely, unless -C 20 collects lines which are on the order of gigabytes each. – tripleee – 2018-04-13T09:46:22.313

Yeah, but I can't see another reason why grep would be killed by something other than the user. – l0b0 – 2018-04-13T21:09:04.040

Asked: 2018-04-13T08:46:41.990

Viewed: 467 times

Active: 2018-04-16T05:57:03.433