64

I have a long running batch process that outputs some debug and process information to stdout. If I just run from a terminal I can keep track of 'where it is' but then the data gets too much and scrolls off the screen.

If I redirect to output to a file '> out.txt' I get the whole output eventually but it is buffered so I can no longer see what it is doing right now.

Is there a way to redirect the output but make it not buffer its writes?

James Dean
  • 761
  • 1
  • 6
  • 5
  • 1
    Could you please have a look at my (and @cnst 's) "debate" below, I'm guessing the only thing you want is to see the output in the same time as logging it to a file. If you found a solution, let us know about it ;) ! – Benj Dec 02 '13 at 16:22
  • 2
    more upvoted question http://unix.stackexchange.com/questions/25372 – Trevor Boyd Smith Jun 10 '16 at 20:20

8 Answers8

67

You can explicitly set the buffering options of the standard streams using a setvbuf call in C (see this link), but if you're trying to modify the behaviour of an existing program try stdbuf (part of coreutils starting with version 7.5 apparently).

This buffers stdout up to a line:

stdbuf -oL command > output

This disables stdout buffering altogether:

stdbuf -o0 command > output
Eduardo Ivanec
  • 14,531
  • 1
  • 35
  • 42
10

You may achieve line buffered output to a file by using the script command like so:

stty -echo -onlcr   # avoid added \r in output
script -q /dev/null batch_process | tee output.log        # Mac OS X, FreeBSD
script -q -c "batch_process" /dev/null | tee output.log   # Linux
stty echo onlcr
Community
  • 1
melder
  • 109
  • 1
  • 2
  • -1 because: a) unlike the accepted answer this doesn't work if you want to run the command in the background (the command immediately terminates without finishing `batch_process` if you append `&` to the command above, at least on my Linux box), which seems like an extremely common use case, and b) there's no explanation here of how this incantation works. – Mark Amery Oct 10 '16 at 10:16
9

On Ubuntu, the unbuffer program (from the expect-dev) package did the trick for me. Just run:

unbuffer your_command

and it won't buffer it.

Calmarius
  • 191
  • 1
  • 3
6

The easiest solution that I found (didn't need any third-party packages installed) was mentioned in a similar thread over on at Unix & Linux site: use the script command. It's old, and likely already installed.

$ script -q /dev/null long_running_command | print_progress       # FreeBSD, Mac OS X
$ script -q -c "long_running_command" /dev/null | print_progress  # Linux

Note that the first filename parameter for the script command is the log file to be written. If you simply run script -q your_command, you'll overwrite the command you indented to run with the log file. Check man script, to be safe, before trying it.

Community
  • 1
Steve HHH
  • 321
  • 3
  • 5
4

try the script command; if your system has it, it takes a file name as argument, all text dumped to stdout gets copied to the file. It's very useful when a setup program requires interaction.

Chris S
  • 77,337
  • 11
  • 120
  • 212
  • I know of the 'script -a out.txt' trick. I was wondering if there is any other way to make the writing process not buffer. – James Dean Jul 27 '11 at 14:56
3

Personally I prefer piping output of a command I want to examine through tee.

script records too much information, including timing of key presses, and a lot of non-printable characters. What tee saves is much more human readable for me.

Paweł Brodacki
  • 6,451
  • 19
  • 23
  • I would also add " | less" to the command line. – HUB Jul 26 '11 at 15:43
  • 5
    I'm pretty sure that `tee` is affected by buffering also. I frequently still get partial lines displayed by tee when splitting output from `find` commands. – Magellan Jul 26 '11 at 16:19
3

You can use the tee command, just magic !

someCommand | tee logFile.log will both display in the console and write into the log file.

Benj
  • 399
  • 3
  • 8
  • 1
    Doesn't work. `tee` will not stop any buffering from taking place. – cnst Nov 24 '13 at 09:15
  • 1
    @cnst Effectively, `tee` will not avoid some buffering but will only allow you to have a look on what is the output. This is what @JamesDean wanted (as I undestood his question), but I think buffering is not really the problem here. If you have more details, let me know. – Benj Nov 30 '13 at 10:14
  • He wanted to see output, and he's not getting any output (in an unbuffered manner), and yet you suggest to use `tee` to get a better view at the output that one isn't getting? – cnst Nov 30 '13 at 17:56
  • 1
    How I red @JamesDean question : "I redirect to output to a file '> out.txt'" means for me no console output, wait for process to complete while the entire output is redirected. When you are using `>` you don't see anything on the console. I guess @JamesDean is using the word "buffering" to describe that. I'll post a comment on his question for him to say more about what he wants. – Benj Dec 02 '13 at 16:20
2

Redirect the output into a file and follow the file with the tail -f command.

Edit

If this still suffers from buffering, then use the syslog facility (which is generally unbuffered). If the batch process runs as a shell script, you can use the logger command to do this. If the batch job runs in a scripting language, there should be a logging facility anyway.

wolfgangsz
  • 8,767
  • 3
  • 29
  • 34