In general, programs are not compatible due to the differences in their application binary interface (ABI).
Doesn't the program run directly on the CPU?
NO! That's the job of the operating system, to prevent applications from running "directly" on the CPU. Typically, at the lowest level (i.e. the one the OS API is built on), an application interfaces with the operating system's kernel.
Is it because the compiled program itself needs to reference OS specific libraries?
Yes. Many OS libraries are written to facilitate interfacing with the operating system itself, but there's just as many that are written to be cross-platform. These hide the low-level OS interfacing from the developer, and assumes the compiled version for that OS will be available at runtime (see below).
Although libraries can be written in a cross-platform manner, when compiled they can't be run cross-platform. They still need to be recompiled for the specific target operating system, again to utilize the particular underlying components of the operating system (kernel).
What is the difference between a compiled program for one OS vs another?
Finally, executable files themselves often contain very specific binary loading headers and so-forth (e.g. the PE Executable file format [.exe, .dll, etc...] for Windows, or ELF for Linux [none, .o, .so, etc...]). These can also include code to load the compiled OS-specific binaries for a particular software library.
Lastly, from a programmer's perspective: calling convention. Compiled code passes variables to functions in a given manner (i.e. through registers, or on the stack) in a very particular order. Even then, it also needs to be agreed upon who's responsible to "clean up" function calls (the caller or the callee?). Although there are several standard and widely used x86 calling conventions, some may not be supported by certain operating systems (this is part of the ABI).
In addition, different OSes have different standards for stack layout, memory alignment practices, etc, so even purely numeric/computational code may need to differ from one OS to the next. – Daniel R Hicks – 2013-01-30T01:49:59.673
Would these differences prevent statically compiled code from executing? Didn't know that.. – alexis – 2013-01-30T19:23:20.497
It's probably just a theoretical problem, since some other incompatibility would get you first. But OSes often have expectations for how stack frames are aligned, where registers are stored, etc. These may vary from one OS to the next for the same hardware architecture. You might in theory be able to RUN "foreign" code that is purely computational, but you'd never be able to get it started and never be able to end it cleanly. – Daniel R Hicks – 2013-01-30T20:55:39.420