When you say "Hack" I'm personally wondering what sort of hacking you mean - it's a fairly varied skill with many different interpretations.
Firstly, by far and away the biggest domain going forward in security will be web. That means SQL injection, javascript bugs, browser bugs, studies of authentication schemes etc. So as others have mentioned, OWASP and the like are fantastic resources.
Not my particular favourite area though, so here's my guide to "things to know" if you want to start looking for vulnerabilities in compiled code on operating systems. One of the first things you'll begin to realise is that there is a lot to know - you are not going to become an uber cool h@x0r!11!! overnight, or actually if you do nothing else for the next 6 months but read up on all of this, top to bottom.
Programming knowledge
- You need to know assembly. Contrary to popular belief it is not that hard - most assembly takes the form
instruction register, register and translates directly to machine code. You need to know an assembler, such as NASM, YASM or GNU AS. There are two different syntaxes in assembler - AT&T and Intel. They're not that far apart.
- You need to know your processor's instruction set. The Intel Software Manuals for IA-32 and Intel 64 are a great resource and explain every instruction you could ever need. AMD publish equivalents; since both AMD processors and Intel processors use the same x86 instruction sets, there are, mostly, a lot of commonalities.
- Knowing a debugger and a disassembler will help you. GDB is the canonical debugger used on Linux platforms, WinDBG is one such debugger from windows. Other people like OllyDbg. In terms of disassemblers, many Linux ones are powered by objdump. I personally like objconv from this author. The tool in the field of disassemblers is IDA Pro, which provides much more than just disassembly.
- Have an understanding of how to use hex notation and the various word sizes.
Program Internals knowledge
- Know C. Know C++. Know the difference between C/C++; for example what effect a
template keyword has. Know what name mangling is and why it exists. Know what C cannot do which assembly can.
- Understand how your CPU represents data and which C types most closely represent a register.
- Binary file formats: knowing how executables work is very important, especially what formats they exist in and how to manipulate them. You don't need to know the internals of the ELF, COFF or PE formats unless there's a really hairy exploit going on, but knowing how to extract symbols from these files and how data is laid out in them generally will give you an advantage.
- Shared object loading. Understand how shared objects or DLLs work and load.
- Understand the basic run time constructions of your program. Where is the "heap"? Where is the stack, what is on it and how does it work? Where is your vtable and how does that work?
Program Internals for other languages
Nobody said you'd be working with C/C++ programs, although if you're looking at traditional, OS software it probably is C/C++. That said, you may be interested in python or jvm bytecode and how each of these runtimes work. Or perhaps the CLR.
Program environment - Operating Systems I
- Basics. You should know that there are permissions and what they mean. You should know who the administrator account is, what the default is and how you acquire administrator access normally (e.g. root vs Administrator group).
- Program environment. You should know where and how config for programs and the OS is laid out. You should know how services run and as what user. How do programs run automatically/on boot? How is media loading handled?
System subversion - Operating Systems II
- Know the difference between the various CPU Rings.
- Know how drivers are loaded into the operating system and what they can do.
- Have an accurate idea of how the OS handles permissions, resources, filesystems etc. Better if you know exactly how.
- Know about mechanisms for intercepting system actions.
- Understand how to perform shared object injection on your system (1, 2).
- Know how various subsystems in both user and kernel space work and how to attack programs over these subsystems.
- See here for OS internals.
Networking & Network Services
- Knowing something about networking is important. What does a packet
look like? What is the OSI stack? How do you inspect packets and
network traffic; how do you connect to a network from your system?
- Know common network services for your target platform, most commonly HTTP, SSH. Understand how they can be exploited.
Common exploits and their defences
Analysis tools
This list is probably incomplete and reflects a set of knowledge that should be helpful for understanding how reverse engineering works and how you go from there to finding vulnerabilities and exploiting them. I do not, myself, know absolutely everything in the above list inside out and I've been researching this stuff for a while. As I said, becoming good at this takes a lot of time, dedication and patience.
Once here, you can start to study applications and how they process data and begin to work out how exploits against them work, such as PDF/Flash/Java vulnerabilities.
