The whole enchilada, huh?
OK, first reverse DNS (for ipV4) goes something like this.
Ya start with an ip address, say 1.2.3.4. You want to find out what A records (hosts) are associated with this ip address. The DNS guys are totally Clever Dudes, so they made the the lookups for IP addresses work pretty much the same as for regular domains. In order to do this, they have to make them go from more specific on the left, to more specific on the right.
With a forward host name:
www3.north.sales.example.com
The broadest is on the far right (.com) and the left (www3.) most likely specifies a single host - very specific. IP addresses are opposite
1.2.3.4
4 is the most specific part, indicating a single host. To fix this, the reverse lookup is done by checking
4.3.2.1.in-addr.arpa
Or the reversed, dotted quad notation plus in-addr.arpa whacked on to the end. Something critical to note here, is that with a reverse DNS lookup for 1.2.3.4 is completely independent of the forward lookup for that host name. That is why the reverse lookup is important for mail servers, ssl servers etc.
For a mail server, when an SMTP transaction is negotiated, the sender gives a HELO (or EHLO) command with the host name. Since this is all done over a TCP connection, you also get the sending mail servers IP address. If everything is on the up-and-up, the reverse DNS of the IP address should resolve to whatever the HELO specified. Of course as stated above by @Zordeache, the only header you can trust at all is the one issued by your server. Within this header, the sending hostname from the HELO (forgeable) is usually shown in the mail header along with the IP address from the TCP connection (not easily forgeable).
For email, all a spammer has to do is make sure their reverse and forward lookups for their scummy spam spewing barf box match, which is not hard. Fortunately though, many spammers are careless with their setups, and being strict about this check stops a lot of it.