I've been trying to find out exactly what a pepper is. The only things I could accurately determine are that a Pepper:
But I haven't been able to figure out it's purpose in adding security to a password hash.
Can someone please explain to me what purpose a pepper has (or had) in the password hashing process?
A "pepper" is a secret application-wide random value appended to a password before hashing it. This ensures that an attacker has no way of actually cracking the password hashes without compromising the pepper value one way or another. This is unlike a salt which usually isn't a secret and appended together with the password hash.
If done right, this can be helpful in ensuring that an attacker with limited access to your systems will not be able to crack the passwords you store. This is quite a common scenario. Imagine an attacker that found a SQL injection attack that allows him to dump your password database. He will be unable to crack those passwords without the pepper which is stored in a configuration file he cannot get at.
That being said, I don't think peppers are commonly implemented. I don't know of any password hashing libraries that provide this out of the box, and i have never used it in my own applications.
The "pepper" is a secret value which you assume to be out of reach of the attacker. You hash passwords because you suppose that the attacker could get a glimpse at your server entrails (see this for a discussion). The "pepper" adds additional security if you believe that the glimpse is partial: the attacker could read, for instance, the contents of the database (a common outcome of SQL injection attacks), but not configuration files on the Web server itself.
Without a pepper, an attacker obtaining the hashed password can "try passwords at home" because he can recompute the same hash on any potential password, and see whether the result matches what he purloined from your server. But if the hash computation is "peppered", then the attacker cannot compute hashes at home, unless he knows the pepper value as well. Cryptographically speaking, the "pepper" is a secret key and inserting it into the hashing process turns that hash function into a MAC. The pepper is exactly as valuable as it is secret, i.e. not guessable by the attacker.
See this answer for a primer on password hashing. There is a section on peppering (near the end). Remember that password hashing makes sense only when the attacker's breach is partial (he got a read-only peek only), and peppering makes sense only when the breach is doubly partial: not only did the attacker got only a read-only peek, but that peek was not for the full server contents. When piling assumptions over other assumptions, at some point it ceases to be realistic enough to be worth the effort; this is why peppering tends to remain rare in practice.
"Peppering" enters into the realm of what is feasible with cheap HSM-like devices, for which it makes a lot of sense. If you aim at a software-only solution, then the usefulness of the pepper is, at best, arguable; if you do use a pepper, at least do it right.
Namely, for a user password p and the pepper k, compute HMAC(p, k) (compute the HMAC of p with the pepper k as key for HMAC; use HMAC/SHA-256 if possible, although HMAC/SHA-1 would be fine in practice). Then, use the HMAC output in a "normal" password hashing function, complete with unique/random salt and many iterations, like bcrypt or PBKDF2 (no, the peppering does not remove the need for a good salt). HMAC/SHA-256 produces a 32-byte binary output, which you might need to convert into characters if your bcrypt or PBKDF2 library wants characters; Base64 will turn 32 bytes into 44 characters. The "normal hashing" is meant to ensure some level of robustness if the attacker succeeds at obtaining both the hashed passwords and the pepper value. If you combine HMAC and bcrypt/PBKDF2 the way I explain above, then, at least, the "pepper" won't make your server weaker, which is always a risk when doing homemade cryptography.
