The source of your confusion is an historical "explanation" of RSA signatures as being "encryption of the hash with the private key". This explanation is confusing, and flawed in many ways. It works only for RSA. Actually it does not work for RSA either.
Let's define things properly:
Since these four functions take different types of inputs and produce different kind of outputs, they cannot be swapped around.
In general, asymmetric encryption algorithms and signature algorithms live in separate worlds. It so happens that there is an asymmetric encryption algorithm known as "RSA", and there is a signature algorithm that is also known as "RSA". Moreover, they both use the same kind of public/private key pairs. However, still generally speaking, encryption and signature keys should be distinct.
If we look at the internals of RSA (both algorithms), as defined in PKCS#1, then we may find some common elements, and others which are distinct.
For asymmetric encryption, the message must be smaller (by at least 11 bytes) than the modulus. The message to encrypt is "padded" as follows:
0x00 0x02 rr rr ... rr 0x00 mm...
where mm...
is the message, and the rr
are random non-zero bytes. The number of rr
bytes is adjusted so that the total length is equal to that of the modulus, and there must be at least eight such non-zero bytes (hence the restriction on the message length). The resulting sequence of bytes is interpreted as an integer modulo N (the "modulus" part of the public/private key pair), and raised to the public exponent modulo N. The result is the encrypted message: it is an integer modulo N (the convention is to encode that message in big-endian, yielding a sequence of bytes of the same length as the modulus).
To give some numbers: if you use a 1024-bit RSA key, the modulus has size 128 bytes. Messages that can be encrypted must be no longer than 117 bytes. The encryption result is a sequence of 128 bytes.
For signatures, the message that is to be signed is first hashed with a suitable hash function like SHA-256. Thus, arbitrarily long messages can be processed (SHA-256 has no practical limit on input length). The hash value is then padded, with a padding which looks like this:
0x00 0x01 0xFF 0xFF ... 0xFF 0x00 ii... hh...
where ii...
is a conventional header that designates the hash function, and hh...
is the hash value. The number of 0xFF bytes is again adjusted so that the total length matches that of the modulus. The resulting sequence is again interpreted as in integer, and exponentiated modulo N, but this time the private exponent is used. Moreover, most RSA implementations actually optimize that operation by using the Chinese Remainder Theorem through their knowledge of the factors of N (these factors are part of the private key).
So, while both encryption and signature generation rely on some sort of padding followed by an exponentiation, they differ in significant ways:
For encryption, the message is taken "as is" and is limited in length. For signature, the message is first hashed, and has no length limit.
The paddings differ. It is important for security that the padding used in encryption includes sufficiently many random bytes. It is equally important for security that the padding used in signatures includes sufficiently many non-random bytes. Less obviously, it is very important that the padding bytes occur "on the left" in the big-endian convention.
The exponentiations differ. For encryption, you use the public exponent, which is usually short (possibly down to e = 3, although the value e = 65537 is traditional)(as it happens, 65537 is used for no good reason at all, but hey, tradition is tradition). For signature generation, the private exponent is used, but the operation rarely use that exponent "as is"; in fact, two sub-exponentiations are done modulo p and q (the prime factors of N), for a total speed-up of about 4x.
In fact, the only really common point is that both RSA-encrypted messages and RSA signatures are integers modulo N, and are encoded with big-endian convention. Everything else differs. And, as I said earlier, you should not do signatures and encryption with the same key pairs anyway; both kinds of key pairs call for distinct management and lifecycles.
Some implementations of signature generation/verification accept as input a pre-hashed value. The hash function is still an important part of the overall process. Even if a given piece of code accepts the hash value as input, the hash must still occur at some point.
Some implementations of the RSA signature verification algorithm actually on "just" the signature and the public key; instead of the simple boolean result (match/mismatch), they produce the hash value itself; the caller is then responsible for matching that hash value with the message that is purported to be signed. The ability to recover the hashed message from the signature+public key is a specificity of RSA (RSA is said to be a signature algorithm with recovery). Not all other signature algorithms can do that. And, again, it is important that the hash occurs; don't let it make you believe that you could do "RSA signatures without hashing".
Traditionally, RSA signatures used in SSL have relied on a slightly different procedure: the padding lacks the identifying header (the ii...
in my notations), and the hash value is actually the concatenation of MD5 and SHA-1, both computed over the message that is to be signed. This is a remnant from a past where the PKCS#1 standard was not as firmly entrenched as it is today.