How is system call executed

Question

From the point of view of application programmer the system call is just invocation of a procedure from wrapper library. As I don't have good knowledge of assembly I was wondering how it is done on lowest level. How else can an application invoke system call without using library?

Answer 1

System calls are architecture dependent, but they all share similar traits.

In x86, system calls are done either via software interrupts or via a special "systenter" instruction. Linux can use either, but a software interrupt is most common. On boot time, the kernel sets up handlers for interrupts, both interrupts generated by hardware and those generated by software. In x86 there is a special assembly instruction 'int' which generates a software interrupt. The kernel has set up a handler for software interrupt 0x80 to be the system call interrupt.

This table shows each possible system call and its parameters and how they can be passed. As you can see, a number in the eax register indicates which system call. The other registers point to some parameter to the system call.

So the userland software simply sets up the correct parameters in the registers, then issues interrupt 0x80. The kernel has the handler set up to interpret the system call, perform the function, and return having modified the registers with any possible return value.

While wrapper functions do all this and usually in a portable way, there is no reason why you couldn't write some inline assembly in any C/C++ code to issue a system call directly. You would only need a basic knowledge of assembly: Move the correct data in the correct registers and then issue 'int 0x80'