I would like to know how I can know which is the safest compilation line, that is:
Having several compilation lines in, for example, GCC, how do I know which one is more secure? Hardening would be a good solution? What do you recommend?
Does the compilation optimizations flags affect security?
Compiler options can influence the security of the resulting program in several different ways. Generally speaking, optimization can hurt security. However, this is not a reason to always turn off optimizations! For example, if your encryption code is so slow that users disable the encryption feature and send data around in cleartext, that makes security worse.
Low-level languages such as C and C++ have a lot of areas with undefined behavior. "Undefined behavior" means that anything can happen, and compilers take advantage of this to optimize code.
There a strong disconnect between the way typical developers see code and the way compilers see code. Often, what looks like an obvious optimization to a developer is hard for a compiler to detect, while compilers may perform optimizations that seem to the developer like it's exploiting a loophole.
A classic example is that optimizing compilers will eliminate redundant checks. Sounds good, right?
if (config == NULL) return ERROR_BAD_CONFIGURATION;
… // code that doesn't modify config
if (config != NULL) x = config->value;
An optimizing compiler is likely to compile that last line as if it had read x = config->value since it knows that config cannot be null at this point. Now consider this:
int *p = &config->value;
if (config == NULL) return ERROR_BAD_CONFIGURATION;
…
x = *p;
config->value promises that config is non-null at this point. Therefore the check config == NULL below is redundant and an optimizing compiler is likely to remove it. So, if config is null at runtime, x gets set to a value that's read from an invalid address, leading to a crash or worse. This is an example where an optimizing compiler makes incorrect code insecure, whereas a non-optimizing compiler would let the program get away with being incorrect.
Turning off optimization usually leads to fewer such surprising effects, but it is not a guarantee. The fix for such "optimization-induced bugs" is to write correct code in the first place, rather than hope that the compiler will figure out the intent of the programmer.
Undefined behavior is not limited to low-level languages. In particular, regardless of the language, concurrency inherently has some undefined behavior, because the order in which things happen is not determined. Race conditions can happen in any language (even without built-in support for concurrency, it can happen via communication between programs). Compiler options usually have little influence there.
Some vulnerabilities exploit direct ways of accessing information, for example a missing permission check or a buffer overread. Other vulnerabilities are due to indirect ways of accessing information: side channels. A common side channel is timing, including the timing of indirect events.
For example, on a typical high-end system such as a personal computer or smartphone, the operating system protects programs from accessing each others' memory. However a program may still be able to deduce information from the way other programs behave. In particular, by measuring the timing of memory accesses, an attacker program can deduce which cache lines a victim program is accessing, and this provides information about which addresses the victim accesses in memory. For example, the easy and fast way to implement the popular cryptographic algorithm AES uses look-up tables; if the attacker knows which row of the table the victim is accessing, that can allow the attacker to reconstruct the key.
Programs can be written in a defensive way to avoid leaking information in such ways. This is sometimes difficult when optimizing compilers turn code without obvious leaks into executable code that does have leaks, because the compiler has determined that the program would be faster that way. In this case, an optimizing compiler can make even correct code insecure.
Compiler bugs are uncommon, much less common than bugs in some random application. Any decent developer knows that if the program doesn't work, a bug in their own code is much, much, much more likely than a compiler bug. Nonetheless, compiler bugs can happen. The higher the optimization level, the higher the risk of bugs. Most code doesn't benefit from the highest optimization levels anyway: usually -O or -O2 is a good compromise between performance and risk.
Some safety features are partly under the control of the compiler. There, the obvious thing to look for is whether these safety features are enabled. These safety features generally don't help to defend against vulnerabilities, but they can make exploits harder. They often don't make exploits completely impossible, but they can make a difference between taking 5 minutes and 5 weeks to craft an exploit, and that can give the developer the time to develop and deploy a patch.
For example, many compilers can generate stack canaries. Stack canaries detect when a buffer overflow happens on the stack and halt the program. To exploit a stack buffer overflow, the attacker needs to limit the size of the overflow so that the canary isn't overwritten or arrange for the correct value to be written in the canary's place, neither of which are always possible or easy.
