I'm new to exploit development and while watching a tutorial I came across the topic of "Bad character identification". I'm referring to the process of sending all possible characters to the vulnerable process to see if there are characters which fail to pass to the receiver.
The existence and identification of those characters has been discussed many times before but I couldn't find the root cause of their existence.
Why are there bad characters a target process mis-handles?
There is no common root cause, although null byte is a common problematic character when dealing with C strings. Each vulnerability has its own buffer space and bad bytes due to the underlying code (char array size declaration f.ex), but also through the function calls that leads to the vulnerable function being called.
A payload that is transmitted over HTTP typically needs to avoid characters that influence how the HTTP request is parsed. Characters such as \r, \n, / and ? can cause the line that's being parsed to truncate prematurely and fail to overflow the buffer, or lead to a 404 error instead of calling the vulnerable function.
Characters being converted between upper and lower case is another example that will mess with shell code. There are also cases where a single byte is fine, but multiple sequential bytes are not, such as a function trimming whitespace in the payload before calling the vulnerable function.
So you can see that the overall combination of the bad bytes is a combination of the code, the call stack and the transport layer (and possibly more for second order bugs) and the bad bytes can be bad due to diverting the call stack as well as corrupting the payload.
