1
I have to write a fairly large text file(>300,000 lines) using text extracted from the second column of one file, then prepending and appending text to each line and finally writing it out to a new file.
I have the following while loop and it is working fine. But it is slow, taking many minutes per run. I suspect that there is a better awk recipe/method that would be much faster. Can anyone suggest a faster method.
SOURCEFILE example
useless9 important1 more useless stuff
useless8 important2 more useless stuff
useless7 important3 more useless stuff
useless6 important4 more useless stuff
Extract text from source file and output final result file.
while read line; do
mytext=`echo $line | awk -v RS='\r\n' '{print $2}'`
echo "$PrePattern $mytext $PostPattern" >> $OUTFILE
done < $SOURCEFILE
OUTFILE
PrePattern text important1 PostPattern text
PrePattern text important2 PostPattern text
PrePattern text important3 PostPattern text
...
1show some sample lines from the source file. That whole loop can probably be expressed as
awk -v pre="$PrePattern" -v post="$PostPattern" '{print pre, $2, post}' "$SOURCEFILE"
– glenn jackman – 2016-01-21T19:15:37.840Thanks very much, glen jackman! As I suspected there was a performance gain to be had. Your solution just turned a ~15 minute process into ~3 seconds – Slow Poke – 2016-01-21T19:50:31.333
much of that gain is not having to launch 300,000 awk processes, when just 1 will do – glenn jackman – 2016-01-21T22:15:10.433