6
2
I have tested on CygWin doing "ls -R" with a large directory and sending the output to a Test.txt
file, so the process takes about 30 seconds to complete; the file is created, but it is empty; it will only be filled when the "ls -R" command finishes, and I need to see the file content meanwhile new data is entering.
The problem is supposed to come from the buffering that the operating system makes before writing into the file.
This is what I have tested in order to write into the file without buffering:
- The unbuffer command from the
expect
package: where is it? Modernexpect
version has no suchunbuffer
command (see below):
./unbuffer -p ls /cygdrive/y/Repositorio/ -R > Test.txt ./unbuffer ls /cygdrive/y/Repositorio/ -R | tee Test.txt ./unbuffer -p ls /cygdrive/y/Repositorio/ -R | tee Test.txt ./unbuffer ls /cygdrive/y/Repositorio/ -R | ./unbuffer -p egrep "" | tee Test.txt ./unbuffer -p ls /cygdrive/y/Repositorio/ -R | ./unbuffer -p egrep "" | tee Test.txt
- Installing the Expect package and naming "unbuffer" this script:
#!/usr/bin/expect -- # Description: unbuffer stdout of a program # Author: Don Libes, NIST eval spawn -noecho $argv set timeout -1 expect
- Installing the Expect package and naming "unbuffer" this other script:
#!/usr/bin/expect -- # Description: unbuffer stdout of a program # Author: Don Libes, NIST if {[string compare [lindex $argv 0] "-p"] == 0} { # pipeline set stty_init "-echo" eval spawn -noecho [lrange $argv 1 end] close_on_eof -i $user_spawn_id 0 interact { eof { # flush remaining output from child expect -timeout 1 -re .+ return } } } else { set stty_init "-opost" set timeout -1 eval spawn -noecho $argv expect }
- The script command:
script -c "ls /cygdrive/y/Repositorio/ -R" | tee Test.txt script -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | tee Test.txt script -q -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | tee Test.txt script -q -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | grep "" --line-buffered | tee Test.txt script -q -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | egrep "" --line-buffered | tee Test.txt
- The egrep command with unbuffering option:
cat BigFile.txt | egrep "" --line-buffered | tee Test.txt
The stdbuf: it does not exist on modern CygWin, or so I think.
Another methods:
ls /cygdrive/y/Repositorio/ -R 1>&2 | tee Test.txt ls /cygdrive/y/Repositorio/ -R 1>&2 |& tee Test.txt
Tests done with
bash
andmintty
.Checked and tested everything in these threads:
Writing "tail -f" output to another file
https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe
https://serverfault.com/questions/294218/is-there-a-way-to-redirect-output-to-a-file-without-buffering-on-unix-linux
Results are always the same: the output file Test.txt
is only filled at the end of the "ls -R" command.
Any more ideas, please?
1I'm not sure this is solely due to buffering. Buffers nowadays are usually 64 kb and 30 seconds is a long time; I'd expect
ls
to fill 64 kb faster than that. If this is still an issue for you, could you tell us the size of the resultingTest.txt
? – André Chalella – 2015-06-15T10:02:48.740