When is memory allocation/layout deterministic?

Question

I'm trying to learn about buffer overflow attacks, both on the stack and on the heap. However, I'm confused about when it's possible to determine the address of the buffer.

The classic "Smashing the Stack for Fun and Profit" mentions the need to use a sizable NOP sled in a stack-based buffer overflow attack in order to reliably jump into the shellcode placed into the buffer. This seems to be due to the fact that the buffer ends up at a different memory location in each run of the program. However, it's not completely random, since it is possible to guess the address of the buffer by increasing the chances using a NOP sled.

In contrast, the environment variables passed to the program are placed in the same addresses every time, enabling a much easier local attack in which the attacker places shellcode into an environment variable, and then can jump to that environment variable's location exactly.

I'm still in the dark about how this plays out for heap-based buffers (e.g. those allocated by malloc) and in other situations. Therefore:

1. In which circumstances is memory layout deterministic?
2. How predictable is it even when not entirely deterministic? Finally, how do exploit mitigations like ASLR and PIE affect this?

Answer 1

Part of the problem has to do with what the memory layout looks like at program initialisation and what the memory layout looks like after it has been running for a while and you manage to get your buffer to overflow. You also have to contend with the 'works on my machine' issue.

So at initialisation, all the environment variables will be loaded at the same place because this is a reasonable fixed and early process. When it comes to overflowing the buffer you cannot always ensure that you are triggering it at the same time each and every time. for example if the application receives network packets, it would be very difficult for you to send exactly the same packets all the time. So as a result by the time you come to trigger the overflow the memory layout may not be exactly the same. Hence the need for the NOP sled as somewhere to 'land'.

The 'works on my machine' problems is that, really, you want your exploit to work on other peoples machines, and they are unlikely to be configured exactly like yours is. There may be different versions of libraries, different hardware configurations, all of which mean that the target overflow address is unlikely to be the same as yours.

These problems can make it hard to ensure that the memory layout will be the same all the time, so instead you work on a range of likely values and hence the need for a NOP sled.

Answer 2

Determinism From Within Programs

Although all deterministic operations on known data produce deterministic results, memory layout is not intended to be thought of as deterministic for anyone other than kernel developers. Although layout is deterministic in the mathematical sense ¹, the fast changing and diversified nature of the software industry renders an application's view of system structures effectively stochastic.

Portable Security Features

Neither portable nor reliable code can rest on assumptions about memory outside of the standardized programming models provided. Stack, dynamically allocated, or static memory headers can change, and the compiler can change in sync so that a standards compliant program will run without a hitch, but one that exploits special OS knowledge can break and create more security risk than it purported to eliminate.

A program cannot depend on the consistency of kernel structures over time. CPU architectures, their instruction sets, execution models, stack handling, and the under-the-hood mechanics of malloc and free can change at any time. This is the reason for POSIX and other standards. Only the facades provided through standardized systems calls can be (usually) trusted to exhibit predictable behavior.

Avoiding Environment Variable Exploits

One can, if one wishes, iterate through the environment variables, copy the values needed, and then write zeros over the exact range occupied by each. But that is not the best strategy.

Generally Useful System and Source Practice

The low hanging fruit that has been used by many security teams is to block the execution of all executable software from all users except those user-software combinations that make sense. Those that are permitted are analyzed for potential security holes and appropriate patches are applied to minimize them.

The proper exploit-specific guard against buffer overflow and underflow is either careful buffer index validation in languages like C or proper use of objects that are designed to prevent these exploitable memory model manipulations.

Entropy Injection Methods

Address space layout randomization (ASLR) is an operating system feature that provides additional resistance to attack. You don't add ASLR to your program. Each OS that provides the model has a mechanism in which specific programs can be executed without providing much of a hint through process inspection from where executable instruction bytes are loaded into the CPU.

Position-independent executable (PIE) is a mechanism that supports randomization by ensuring that all addresses in the executable are relative or behave (in groups of instructions) as if they were.

You will need to look into how adding entropy to memory model positioning can be applied in your particular target operating system. There may be associated constraints on your code, compilation or link flags to use, and container execution that performs the needed randomization. There may be speed and resource impact, so you may wish to identify your executable risk levels and be selective.

How do they work? Introducing entropy into the memory location determination makes it difficult for attackers to guess where CPU instructions reside in memory so the jump source and desired destination are more difficult to determine in real time from within another time slice in a multitasking scenario.

Layering Strategies for Cumulative Security Strength

Remember that these are guards not security guarantees. Few cost effective strategies are as effective as the traditional combination of world class system administration, careful memory access within programs, and excellent sanitation of incoming data. More recent additions to this list are to add layers to these effective strategies to increase system resilience to attack.

---

[1] Memory layout is deterministic unless real entropy is captured by the OS from mouse movement, audio input, or some other transducer and then employed to position memory in a random (as opposed to pseudo-random) manner.