I don't fully understand the command but your description looks like it may be a job for named pipes. To make this concept clear my example uses four of them. With proper substitutions you can reduce this number to two, I think, maybe even to one; but for now let's keep it simple.
mkfifo pre-audio-pipe pre-video-pipe audio-pipe video-pipe # creating pipes
(reading commands) | mbuffer -p 1 -m 5G | tee pre-audio-pipe > pre-video-pipe # splitting
This process will fill whatever buffers are created for the two named pipes, then it will wait for this data to be read elsewhere.
The "elsewhere" is in another console:
<pre-audio-pipe (isolate audio) | (process audio) > audio-pipe
and in yet another console:
<pre-video-pipe (isolate video) | (process video) > video-pipe
Again these two commands will wait until we read some data from the pipes. In the final console:
ffmpeg -i video-pipe -i audio-pipe (doing final conversion)
You may experience a lockdown in case the final command wants to read one stream ahead of the other. I don't know how likely this is. Additional buffers may be useful to avoid this. My first try would be to remove the mbuffer
(before tee
) and insert two independent buffers between respective (isolate)
and (process)
.
After it's all done:
rm pre-audio-pipe pre-video-pipe audio-pipe video-pipe # cleaning
Edit
From the OP's comment:
do you see any chance to implement a solution without using separate named pipes?
I've been thinking about coprocesses (coproc
builtin) but I don't know them much. There is this comprehensive answer about them. Search for the phrase "why they are not so popular". From therein:
The only benefit of using coproc
is that you don't have to clean up of those named pipes after use.
I totally agree. Look at the example there – it's basically your case with data stream forked three-way instead of two-way. The example uses shells other than bash
but from my experience it would be similarly awful in bash
.
Ideally, there would be a one-line-command working with unnamed pipes only, since the job should be started with an "economic effort" from the command prompt.
Seriously? With all those (doing some encoding here)
expanded? In my opinion, no matter whether you use named or unnamed pipes, an "economic effort" here will be to write a script, even if it's a one-time job. Comparing long one-liner and equivalent well written script, I find the latter easier to debug.
But since you asked for one-liner, you'll get it, still with named pipes though. My idea for maintaining named pipes is to create a temporary directory for them. The general concept:
my_temp=`mktemp -d` ; pre_audio_pipe="${my_temp}/pre-audio-pipe" ; pre_video_pipe="${my_temp}/pre-video-pipe" ; audio_pipe="${my_temp}/audio-pipe" ; video_pipe="${my_temp}/video-pipe" ; mkfifo "$pre_audio_pipe" "$pre_video_pipe" "$audio_pipe" "$video_pipe" ; (reading commands) | tee "$pre_audio_pipe" > "$pre_video_pipe" & <"$pre_audio_pipe" (isolate audio) | mbuffer -p 1 -m 1G | (process audio) > "$audio_pipe" & <"$pre_video_pipe" (isolate video) | mbuffer -p 1 -m 4G | (process video) > "$video_pipe" & ffmpeg -i "$video_pipe" -i "$audio_pipe" (doing final conversion) ; rm -rf "$my_temp"
According to this answer you probably can fit it into one command line, even after you dig into the command and expand all those (do something)
placeholders.
OK, the one-liner form was to show you how inconvenient it could be. The same concept as a script:
#!/bin/bash
my_temp=`mktemp -d`
pre_audio_pipe="${my_temp}/pre-audio-pipe"
pre_video_pipe="${my_temp}/pre-video-pipe"
audio_pipe="${my_temp}/audio-pipe"
video_pipe="${my_temp}/video-pipe"
mkfifo "$pre_audio_pipe" "$pre_video_pipe" "$audio_pipe" "$video_pipe" #creating actual pipes
# Main code here.
# Notice we put few commands into the background.
# In this example there are two separate mbuffers.
(reading commands) | tee "$pre_audio_pipe" > "$pre_video_pipe" & # splitting
<"$pre_audio_pipe" (isolate audio) | mbuffer -p 1 -m 1G | (process audio) > "$audio_pipe" &
<"$pre_video_pipe" (isolate video) | mbuffer -p 1 -m 4G | (process video) > "$video_pipe" &
ffmpeg -i "$video_pipe" -i "$audio_pipe" (doing final conversion)
# Then cleaning:
rm -rf "$my_temp"
Interesting, is that similar (or identical) to naming file descriptors? – Xen2050 – 2017-03-31T04:18:01.023
@Xen2050 Frankly I don't know. My explicit experience with file descriptors (in
bash
or elsewhere) is very limited. Of course I use them implicitly all the time. :) Unnamed pipes (|
) are ephemeral FIFOs; named pipes are more persistent FIFOs with names (paths). For now it seems to me there may be a similarity at most, not an identity. – Kamil Maciorowski – 2017-03-31T05:13:22.123Thank you for replying to my issue. I understand your approach; do you see any chance to implement a solution without using separate named pipes? Ideally, there would be a one-line-command working with unnamed pipes only, since the job should be started with an "economic effort" from the command prompt. – user415275 – 2017-03-31T06:58:16.403
@user415275 I have expanded my answer. – Kamil Maciorowski – 2017-03-31T10:19:34.093
Thank you very much for replying. Your big "one-liner" still makes use of fifos/named pipes anyway -- is there a chance to achieve the following:
I absolutely appreciate your work and your dedication, and of course it does work, however, it is not the pragmatic way I would like to choose, provided it would be possible to do so. – user415275 – 2017-04-02T16:20:50.060
@user415275 As I said: coprocesses; but I'm not going to include them in my code here because I'm not familiar enough with them and I think you already have all means to do it yourself, since you insist on abandoning named pipes. You may post your own answer and let others learn from you. In case there's a specific problem with coprocesses you cannot solve despite your reasonable research effort, I encourage you to ask a separate question explicitly about them (on Super User or Unix & Linux).
– Kamil Maciorowski – 2017-04-02T18:23:19.633