Why is this tee losing stdout?

Question

Simple script:

#!/bin/bash
remote_ssh_account="depesz@localhost"
directory_to_tar=pgdata
exec nice tar cf - "$directory_to_tar" | \
    tee >(
        md5sum - | \
        ssh "$remote_ssh_account" 'cat - > /tmp/h3po4-MD5-2012-03-13.tar'
    ) | \
    ssh "$remote_ssh_account" 'cat - > /tmp/h3po4-data-2012-03-13.tar'

Theoretically it should deliver the data and checksum, to remote machine.

But somehow the tee fails with:

tee: standard output: Resource temporarily unavailable

Did strace, but nothing came out of it. I see both ssh started, and tee writing to both of them, but only the pipe to ( md5sum | ssh ) gets data - strace of the ssh "data" doesn't get any data, and after 5 seconds tee shows the error.

Aside from this all works. 2 connections are established, tar works, md5sum and its delivery works.

On the `strace` front, try `strace -fF`. Works like a champ for me. — BMDan, Mar 14 '12 at 20:14
-fF is not in manual, but there is -f, -F and -ff. I used `strace -ff -o strace.log -s 512 ./z.sh` — , Mar 14 '12 at 20:25
`-fF` is the same as `-f -F`. The `-F` is actually unneeded on most of the systems I work on, but it (tries to) follow vforks on older versions of strace, and has no effect on newer ones, but it doesn't hurt. `-ff` ends up writing a bunch of files (one per PID) that you then have to collate back into a sensible timeline, so I avoid it in the vast majority of situations. That said, `-ff` should work, but the output files will be oddly-named. — BMDan, Mar 15 '12 at 12:55

BMDan · Answer 1 · 2012-03-14T20:10:46.490

5

Issue found. Here's the relevant portion of the strace:

[pid 10243] write(1, "pFl\r\347\345]\244Hi\336\253,-\231\247\344\234\241\332\302\252\315\243G\234\225+\241\323\316s"..., 4096 <unfinished ...>
[pid 10247] select(7, [3 4], [3], NULL, {10, 0} <unfinished ...>
[pid 10243] <... write resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid 10247] <... select resumed> )      = 1 (out [3], left {10, 0})
[pid 10243] write(2, "tee: ", 5tee:  <unfinished ...>
(...)
[pid 10243] write(2, "standard output", 15standard output <unfinished ...>
(...)
[pid 10243] write(2, ": Resource temporarily unavailab"..., 34: Resource temporarily unavailable) = 34

So, what's happening is that the remote ssh isn't yet ready for the write to continue. Most programs handle this correctly, but tee decides to die in a pile. See http://lists.freebsd.org/pipermail/freebsd-bugs/2012-February/047528.html for one reference to this sort of behavior. There are a couple others that I found too in a brief search for "EAGAIN tee".

The solution that lhunath found works because it effectively forces bash to handle the EAGAIN. Elegant.

edited Mar 14 '12 at 20:10

answered Mar 14 '12 at 19:26

BMDan

7,129
2
22
34

Same error - you can also easily test it yourself - just change directory to tar (it shouldn't be too small), and remote_ssh_account - to something that you have passwordless access to. – Mar 14 '12 at 19:32
Changed; please try again. Yes, I know that doesn't do exactly what you want, but I just want to try it. Please be sure you're running it the same way (i.e., as a script, not directly on the command line). – BMDan Mar 14 '12 at 19:43
This works, and delivers the file. While I do appreciate the help - tests like this can be ran everywhere - did my original script *not* fail for you? – Mar 14 '12 at 19:49
1

I can get the problem to reliably recur with `dd if=/dev/urandom bs=1M count=20 | tee >(md5sum - | ssh user@host 'cat - > /tmp/foo.md5') | ssh user@host 'cat > /tmp/foo.tar'`. Working on debugging why now. – BMDan Mar 14 '12 at 19:54
It also happens with the order reversed (that is, with the `cat>tar` inside the `>()`). Neat. Adding the `-i` flag to `tee` (on a blind hunch) didn't help. – BMDan Mar 14 '12 at 19:57
Yeah - it looks that the pipe before final ssh breaks. removing it to use > >( ), as in lhunath example - helps, but also helps if I'll do: `tee >( md5sum | ssh ) >( ssh ) > /dev/null` Weird stuff. – Mar 14 '12 at 20:00
Since the evolution of my answer can be confusing to read: I've updated my answer with the final analysis of why this failed in the first place. – BMDan May 07 '12 at 17:39

score 3 · Accepted Answer · answered Mar 14 '12 at 19:50

3

Try this, an alternative way of doing the pipe that breaks:

#!/bin/bash
remote_ssh_account="depesz@localhost"
directory_to_tar=pgdata
nice tar cf - "$directory_to_tar" | \
    tee >(
        md5sum | \
        ssh "$remote_ssh_account" 'cat > /tmp/h3po4-MD5-2012-03-13.sum'
    ) > >(
        ssh "$remote_ssh_account" 'cat > /tmp/h3po4-data-2012-03-13.tar'
    )

answered Mar 14 '12 at 19:50

lhunath

145
4

1

Seems to be working. I still would love to know why the | ssh thing breaks. – Mar 14 '12 at 19:51

Why is this tee losing stdout?

2 Answers2

Linked