The ret2libc (and return oriented programming (ROP)) technique relies on overwriting the stack to create a new stack frame that calls the system function. This wikipedia article explains stack frames in great detail: https://en.wikipedia.org/wiki/Call_stack#Stack_and_frame_pointers
The stack frame dictates the order the function call and parameters are written:
function address
return address
parameters
So in your example you want to call system()
with cmdstring
and return to the exit()
function when the system()
call returns. So you write the following stack frame:
system_addr
exit_addr
cmdstring_addr
If you remove the exit address you change the stack frame to the following:
system_addr
cmdstring_addr
existingdataonstack_aka_junk
So now you're calling system()
with a junk address argument and trying to return into your command string once the function completes. Which is why it fails. You can replace the exit()
address with other data such as 0x41414141
which will cause a segfault
once the system()
call completes. Or you could replace it with the address of a pop pop ret
location and then write more stack frames underneath. This is now a ROP
payload. Here is a basic example of how that would look on the stack:
system_addr
poppopret_addr
cmdstring_addr
printf_addr
exit_addr
addr_of_string_to_printf
This would allow you to print something like You got hacked
before exiting.