Typically the "insecurity" of files comes into play when they are accessed, not just by having them downloaded. Downloading a file unopened and having it pwn your computer is the stuff of Hollywood and science fiction (as far as I know, though there may be research into this too).
Thus, to understand the problem, the real question is not about the file types, but rather about the typical programs commonly used to access those files.
Thus, a .txt file should be relatively harmless, since it's accessed in raw form by simple text readers and not recognized as code in any way. (Though of course a 0day vuln found in e.g. Windows Notepad could change that....)
PDFs and Office docs are typically opened by Adobe Reader and MS Office programs, respectively - and these all recognize numerous types of dynamic content, e.g. Office Macros. Furthermore, there is a whole lot of data handling, parsing, translating, activating, and rendering going on, and there can be (and are, and were) many different vulns at each step of the way, in each of those programs (and of course others).
Same thing goes for media files - JPEGs and GIFs can have hidden code (see GIFAR), movie files can have embedded codecs, etc.
Of course, executable files and code (compiled or not) are the most dangerous, since they are trivially exploitable - the code is already there, and the user explicitly asks to excute it. This includes, e.g. on Windows exe, dll, com, bat, htm/html, js, vbs, jar, etc.
Of course source code (.java, .cs, .vb, .c, .cpp, etc) can also be an issue, since these are typically passed directly to a compiler - arguably one of the most complex pieces of common software - and gives execution instructions, not to mention the later output... It is not too complicated to obfuscate malicious code.
Choice of OS is not necessarily the most important factor here, though of course different files will execute differently on each OS - and each OS has its own share of unsafe filetypes.
More importantly, is if/when a malicious file is executed, what kind of damage can it do? An OS that defaults to minimal user privileges (compare Win7 to WinXP), will of course be that much safer.
