Why can't just any program use his own file-system-access-system?

Question

This may sound stupid for everyone who knows more than me about the UNIX-System and security in software:

Imagine you have a program trying to cause harm by deleting files. You just do something like "rm -rf /home/" and get an error message, because rm checks whether you are allowed to delete this directory or not and stops if you are not allowed to.

So far, so good.

Now, what exactly is stopping the virus-programmer to use his own "delete"-method? Why must he rely on system calls and cannot simply use his own methods to achieve his goals? He could write a delete-method that just doesn't check for rights and deletes anyway, couldn't he?

Another example would be file-editing. Why can't an attacker just write his own read-write-mechanisms that do not check for rights and use them to edit files?

So: What exactly is it that forces the evil-program to stay in the sandbox the OS is providing?

Answer 1

During system boot, the OS takes exclusive control of a lot of privileged operations and becomes the gatekeeper. The CPU and other hardware itself will only listen to the trusted process (the kernel) that has been given the keys to the kingdom. This means that in order to do things like get access to most system resources, the application has to actually request this access through the OS. The OS actually does the requested behavior, not the application. The application could try to make direct requests to hardware (which would have to be hardware specific), but since it isn't the kernel, the requests would be ignored if they are trying to access anything that is privileged to the kernel.

Answer 2

No, file system permissions are enforced by the operating system.

When a program wants to delete a file, it calls the unlink() function, which immediately turns control over to the kernel. The kernel first checks to see if the file exists, and then checks the permissions associated with it. If the user has permission to delete (which typically means write permission on the directory containing the file), then the kernel will, using the filesystem driver, perform all the necessary actions to remove the file from the disk. Once this is done, control is then returned back to the user's program.

This distinction between actions performed directly by your program (typically called "user mode" or "user land") and actions performed by the kernel on behalf of the user (typically called "kernel mode") are a key feature of all operating systems, not just Linux, and is the basis of how kernel rules are enforced, including process separation, privilege separation, memory segmentation, and everything that the kernel is responsible for providing.

Any flaw or loophole that would allow a program to circumvent this protection would be considered a privilege escalation vulnerability, and would have to be fixed for the OS to be safe to use. These are surprisingly common, but typically get fixed quickly once discovered.

With respect to filesystem privileges, the thing that prevents you from writing your own filesystem driver to circumvent the kernel's filesystem driver is the fact that a user doesn't have direct read or write access to the raw disk. These rights are controlled (once again) by filesystem permissions.

On Linux and Unix systems, there are a set of special files which represent physical devices, typically kept in the /dev/ directory. For example, /dev/sda represents the first SCSI or SATA disk on Linux. On BSD (including OSX) it's /dev/disk0.

Reading or writing these files will actually read or write directly to the disk, skipping the filesystem driver, but still going through the kernel. Since these files have their own permissions set to them, the very same checks are performed by the kernel before allowing access to the device, so the security remains intact.

If you were to change the permissions or ownership of the entry in /dev/ to allow ordinary users direct disk access, by doing so you would create (you guessed it) a privilege escalation vulnerability.

Answer 3

On any non-trivial computer (including PCs, Macs, smartphones, tablets, even most low-end home routers, etc.), the processor can run in two modes or more: system mode and user mode. Only the kernel of the operating system runs in system mode; all programs run in user modes (even programs executing under a system administrator account such as root on Unix).

When the system boots, the kernel is loaded into memory. It configures the MMU to ensure that only the kernel is allowed to access hardware peripherals, that only kernel code can affect kernel data, and that kernel code can only be executed through controlled interfaces (system calls). This way, only kernel code can access the hardware.

The kernel either does not provide any way for user code to access storage devices, or provides a way that is restricted by file permissions. For example, on Linux and most other Unix systems, there are special files in the directory /dev that allow direct access to the disk, but these entries have permissions that only allow access by root and perhaps other system users, not by applications run by an actual user or non-core service. The only way for unprivileged processes to access the disk is via the filesystem interface, which includes permission management.

A program cannot bypass the filesystem because it has no way to convince the processor to let it talk to the disk drive. The kernel won't let it.

Answer 4

To make it a bit clearer than @AJ Henderson's answer, you have two options to read/write data: one it to go through the kernel/system functions, which will obviously stick to the "sandbox" security. The other would be to directly access the hardware resource. That's fairly easy on *nix - just do an ls on /dev/sda1 (or whatever your disk is):

# ls -l /dev/disk0s2
brw-r-----  1 root  operator    1,   2  8 Feb 16:31 /dev/disk0s2

(sorry, on a mac here) As you can see, the disk is readable by anybody with admin privileges, but only writeable by root - so you're back to the System permissions. And as for direct hardware access, again that's restricted to kernel-level permissions.

Answer 5

What you're describing sounds like the pt_chown attack described at NullCon last week. The attack can be used to take control of psuedo terminals (ptys) including a root (uid=0) owned pty, which would lead to privilege escalation on a Unix platform. The platform tested and verified was Linux, but we speculated that it could happen to OS X or another BSD operating system.

The attack relied on a userspace filesystem, FUSE.