I have an exe file that I don't trust (maybe its infected with malware). I know that running it in a VM will isolate the exe file from my system, but what about using mechanisms such as System Call Interposition or Software Fault Isolation?
Is either of them suitable for sandboxing the exe file? Are there any advantages or disadvantages between them and using the VM?
This may depend on what you are trying to do. Analyse the binary to determine if it truly is malware or just run the code without letting it affect your system.
I think with SFI you might run into a problem where the suspected malware uses an undocumented opcode and the SFI therefore refuses to run the code as it cannot analyse it.
With System Call Interposition you can rewrite operating system calls but unless you use interpositon to create a complete container you need to be able to identify which system calls are dangerous before allowing them through. This may not be obvious. Even using Interposition to create a container may fail if the code exploits a subtle kernel bug.
I think VMs would present a smaller attack surface than OS kernels and since modern cpus provide hardware support should protect even against undocumented opcodes.
In the days of Spectre, Meltdown and Rowhammer you probably still need to be wary though.
