Increase efficiency of bash loop with awk

I have to write a fairly large text file(>300,000 lines) using text extracted from the second column of one file, then prepending and appending text to each line and finally writing it out to a new file.

I have the following while loop and it is working fine. But it is slow, taking many minutes per run. I suspect that there is a better awk recipe/method that would be much faster. Can anyone suggest a faster method.

SOURCEFILE example

useless9   important1   more useless stuff
useless8   important2   more useless stuff
useless7   important3   more useless stuff
useless6   important4   more useless stuff

Extract text from source file and output final result file.

while read line; do

  mytext=`echo $line | awk -v RS='\r\n' '{print $2}'`

  echo "$PrePattern $mytext $PostPattern" >> $OUTFILE

done < $SOURCEFILE

OUTFILE

PrePattern text important1 PostPattern text
PrePattern text important2 PostPattern text
PrePattern text important3 PostPattern text
...

Slow Poke

Posted 2016-01-21T18:57:11.517

Reputation: 11

1show some sample lines from the source file. That whole loop can probably be expressed as awk -v pre="$PrePattern" -v post="$PostPattern" '{print pre, $2, post}' "$SOURCEFILE" – glenn jackman – 2016-01-21T19:15:37.840

Thanks very much, glen jackman! As I suspected there was a performance gain to be had. Your solution just turned a ~15 minute process into ~3 seconds – Slow Poke – 2016-01-21T19:50:31.333

much of that gain is not having to launch 300,000 awk processes, when just 1 will do – glenn jackman – 2016-01-21T22:15:10.433

2Can you start that education process by explaining what this does? Thanks. – fixer1234 – 2016-02-27T18:57:51.647

@fixer1234 Well instead of that, go read the awk man page and then look at the example I posted. After that if you have specific questions, I'll help you understand those points. – Bing Bang – 2016-02-28T04:58:17.283

Increase efficiency of bash loop with awk

Answers