Understanding the terminal and shell interaction

Question

For a long while my understanding about a terminal in unix-like systems has been that it launches the shell process and provides a user interface to it by communicating with it over it's stdin, stdout & stderr.

However recently while looking into a problem with launching a windows console application via a cygwin terminal, I realize it may not be as simple as that.

At http://cygwin.com/1.5/cygwin-ug-net/using-effectively.html I see,

Another issue is receiving output from or giving input to console-based Windows programs. Unfortunately, interacting with Windows console applications is not a simple matter of using a translation utility. Windows console applications are designed to run under command.com or cmd.exe, and some do not deal gracefully with other situations. Cygwin can receive console input only if it is also running in a console (DOS box) since Windows does not provide any way to attach to the backend of the console device. Another traditional Unix input/output method, ptys (pseudo-terminals), is supported by Cygwin but not entirely by Windows. The basic problem is that a Cygwin pty is a pipe and some Windows applications do not like having their input or output redirected to pipes.

I wrote a little C program which I compiled under windows using VC++'s cl.exe -

#include <stdio.h>

int main(int argc, char *argv[]) {
    #define BUFFER_LEN 1024
    char buffer[BUFFER_LEN];

    printf("echo server started\n"); 
    while (fgets(buffer, BUFFER_LEN, stdin) != NULL) {
        printf("%s", buffer);
    }

    return 0;
}

When I run the Cygwin terminal (mintty.exe) and launch this program, I cannot interact with it -

[puneet@freestyle ~]$ /cygdrive/c/echo1.exe
Hello

^-- no response

But when I put it in a pipe, it works -

[puneet@freestyle ~]$ echo  -e "1\n2\n3" | /cygdrive/c/echo1.exe | while read line; do echo $line; done
1
2
3
[puneet@freestyle ~]$

Basically it fails to interact with the mintty.exe terminal. However when running bash.exe directly from a windows console, it can be correctly interacted with -

[puneet@freestyle ~]$ /cygdrive/c/echo1.exe
Hello
Hello
^Z
[puneet@freestyle ~]$

I then thought if I ssh'd into my machine and ran this program as a command, it would work as then the terminal wouldn't directly interact with it, but the SSH server will. However that doesn't work too -

[puneet@freestyle ~]$ ssh freestyle /cygdrive/c/echo1.exe
Hello

^-- no response

But putting this in a pipe again works ! -

[puneet@freestyle ~]$ echo  -e "1\n2\n3" | ssh freestyle /cygdrive/c/echo1.exe | while read line; do echo $line; done
1
2
3
[puneet@freestyle ~]$

Could someone explain the theory behind all these observations?

Is the interaction between the terminal and the shell more than just using the shell's stdin, stdout and stderr?

How is the windows console different? Why do the windows console programs seem to work fine when in a pipe with the cygwin programs?

Answer 1

If you add fflush(stdout) to your while loop (after printf), then your program will work as you expect, even inside mintty. You should also be able to call setbuf(stdout, NULL) as the first operation you do on stdout, and have that work.

You can also run c:\cygwin\bin\bash inside your windows console window and your original program will work as you except. I haven't tried it but you should also be able to run bash inside a conemu or console2 window and have your original program work as you expect.

That is to say: this is about mintty.

Here's (part of) what's going on:

The windows console is very special. Windows has a service called Client/Server Runtime Subsystem. When you start a cmd prompt on Windows, what you are actually doing is connecting to the Client/Server Runtime Subsystem, and it is creating the window for you.

Mintty, on the other hand is a "regular" window (that happens to be running a terminal emulator).

Microsoft's C library i/o routines actually test whether they are running under a window created by the Client/Server Runtime Subsystem and change their behavior if they are. One change they make is to turn off full buffering for stdout.

When you run under mintty the Window's gets/puts routines think they are not running in a command prompt, so they are doing full buffering.

Here's one more trick: under bash running under mintty you can run cat | echo1.exe. Then type some stuff, and when you hit ^D all of that buffered stdout output will show up all at once.

The reason that the cat | echo1.exe trick works is that cat is a cygwin program. Cygwin is really, at heart, a posix emulation library. So the stdin of cat is emulated by the Cygwin library, and the Cygwin library treats ^D differently than a "real" Windows stdin would.

But if you just run echo1.exe under bash under mintty then your keystrokes are going into a non-emulated file stream (i.e., your stdin is a real Windows stdin, not one emulated by the Cygwin library), so ^D does not send eof. Instead ^Z would send eof, but ^Z means something special to bash, so it doesn't get sent. Rather you need to hit ^C which terminates echo1.exe immediately (and doesn't flush the buffers, which is correct.)