How do I prepend a string to program output without waiting for the whole line?

Question

I have a script that runs a command on a remote server using SSH. I want to prepend the string Remote: to every line of the output, but I don't want each line to be delayed until the whole line is available. Here is the output from my command:

$ myproject-db-push my_database_name
Exporting from database... Done
Archiving data... Done
Uploading archive to remote... Done
Running install script on remote
Remote: Decompressing archive into temporary directory... Done
Remote: Using database: my_database_name
Remote: Dropping collections:
Remote:  - my_collection_foo
Remote:  - my_collection_bar
Remote: Importing new data... Done

In this case, I'm using sed like this:

echo "$INSTALLCMD" | ssh -T "deploy@$SERVER" | sed -u "s/^/Remote: /"

The problem is, as I explained, that no partial lines are printed to the screen. If I remove the | sed part, it works as expected. First this is written:

Importing new data... 

And a few seconds later, the line is completed:

Importing new data... Done

I'm assuming sed is only able to work on a line-by-line basis. I tried setting it to unbuffered, but it still waits for whole lines. Is there another way to accomplish this?

Answer 1

It's a bit tricky, because all those utillities (sed, awk, grep) are line buffered. That means they print the output only when the line has finished (a newline has appreaed). They cannot read the input character by character.

So for testing I made a small sequence, that simulates your behaviour:

{ 
  echo -n "first task: "
  sleep 2
  echo "done"
  echo -n "second task: "
  sleep 2
  echo "done"
}

As in your question, it prints first task: and after 2 seconds done. Try it yourself by copying it in your terminal.

Solution:

Add the following behind your command:

IFS=
command | { x=1; while IFS= read -d'' -s -N 1 char; do
  [ $x ] && printf "Remote: "
  printf "$char"
  unset x
  [ "$char" == "
" ] && x=1
done; }

Explanation:

The read builtin of bash can read input character by character. The part read -d'' -s -N 1 char disables the delimiter -d'', activates the silent mode -s and reads only 1 character a time -N 1 into the variable $char. Then the command checks if the variable $x exists. If yes, we are in a new line and we print the "prefix". Then print the character. Unset $x. Then last statement checks if the character is a newline. If it's a newline set $x to 1 and in the next loop, the "prefix" will be printed.

The whole thing can be tested, when you concatente the two sequences:

{ 
  echo -n "first task: "
  sleep 2
  echo "done"
  echo -n "second task: "
  sleep 2
  echo "done"
} | { x=1; while IFS= read -d'' -s -N 1 char; do
  [ $x ] && printf "Remote: "
  printf "$char"
  unset x
  [ "$char" == "
" ] && x=1
done; }