tee
forwards its stdin to every single file specified, while pee
does the same, but for pipes. These programs send every single line of their stdin to each and every file/pipe specified.
However, I was looking for a way to "load balance" the stdin to different pipes, so one line is sent to the first pipe, another line to the second, etc. It would also be nice if the stdout of the pipes are collected into one stream as well.
The use case is simple parallelization of CPU intensive processes that work on a line-by-line basis. I was doing a sed
on a 14GB file, and it could have run much faster if I could use multiple sed
processes. The command was like this:
pv infile | sed 's/something//' > outfile
To parallelize, the best would be if GNU parallel would support this functionality like so (made up the --demux-stdin
option):
pv infile | parallel -u -j4 --demux-stdin "sed 's/something//'" > outfile
However, there's no option like this and parallel
always uses its stdin as arguments for the command it invokes, like xargs
. So I tried this, but it's hopelessly slow, and it's clear why:
pv infile | parallel -u -j4 "echo {} | sed 's/something//'" > outfile
I just wanted to know if there's any other way to do this (short of coding it up myself). If there was a "load-balancing" tee
(let's call it lee
), I could do this:
pv infile | lee >(sed 's/something//' >> outfile) >(sed 's/something//' >> outfile) >(sed 's/something//' >> outfile) >(sed 's/something//' >> outfile)
Not pretty, so I'd definitely prefer something like the made up parallel
version, but this would work too.