If I hash passwords before storing them in my database, is that sufficient to prevent them being recovered by anyone?
I should point out that this relates only to retrieval directly from the database, and not any other type of attack, such as bruteforcing the login page of the application, keylogger on the client, and of course rubberhose cryptanalysis (or nowadays we should call it "Chocolate Cryptanalysis").
Of course any form of hash will not prevent those attacks.
Note: This answer was written in 2013. Many things have changed in the following years, which means that this answer should primarily be seen as how best practices used to be in 2013.
We need to hash passwords as a second line of defence. A server which can authenticate users necessarily contains, somewhere in its entrails, some data which can be used to validate a password. A very simple system would just store the passwords themselves, and validation would be a simple comparison. But if a hostile outsider were to gain a simple glimpse at the contents of the file or database table which contains the passwords, then that attacker would learn a lot. Unfortunately, such partial, read-only breaches do occur in practice (a mislaid backup tape, a decommissioned but not wiped-out hard disk, an aftermath of a SQL injection attack -- the possibilities are numerous). See this blog post for a detailed discussion.
Since the overall contents of a server that can validate passwords are necessarily sufficient to indeed validate passwords, an attacker who obtained a read-only snapshot of the server is in a position to make an offline dictionary attack: he tries potential passwords until a match is found. This is unavoidable. So we want to make that kind of attack as hard as possible. Our tools are the following:
Cryptographic hash functions: these are fascinating mathematical objects which everybody can compute efficiently, and yet nobody knows how to invert them. This looks good for our problem - the server could store a hash of a password; when presented with a putative password, the server just has to hash it to see if it gets the same value; and yet, knowing the hash does not reveal the password itself.
Salts: among the advantages of the attacker over the defender is parallelism. The attacker usually grabs a whole list of hashed passwords, and is interested in breaking as many of them as possible. He may try to attack several in parallel. For instance, the attacker may consider one potential password, hash it, and then compare the value with 100 hashed passwords; this means that the attacker shares the cost of hashing over several attacked passwords. A similar optimisation is precomputed tables, including rainbow tables; this is still parallelism, with a space-time change of coordinates.
The common characteristic of all attacks which use parallelism is that they work over several passwords which were processed with the exact same hash function. Salting is about using not one hash function, but a lot of distinct hash functions; ideally, each instance of password hashing should use its own hash function. A salt is a way to select a specific hash function among a big family of hash functions. Properly applied salts will completely thwart parallel attacks (including rainbow tables).
Slowness: computers become faster over time (Gordon Moore, co-founder of Intel, theorized it in his famous law). Human brains do not. This means that attackers can "try" more and more potential passwords as years pass, while users cannot remember more and more complex passwords (or flatly refuse to). To counter that trend, we can make hashing inherently slow by defining the hash function to use a lot of internal iterations (thousands, possibly millions).
We have a few standard cryptographic hash functions; the most famous are MD5 and the SHA family. Building a secure hash function out of elementary operations is far from easy. When cryptographers want to do that, they think hard, then harder, and organize a tournament where the functions fight each other fiercely. When hundreds of cryptographers gnawed and scraped and punched at a function for several years and found nothing bad to say about it, then they begin to admit that maybe that specific function could be considered as more or less secure. This is just what happened in the SHA-3 competition. We have to use this way of designing hash function because we know no better way. Mathematically, we do not know if secure hash functions actually exist; we just have "candidates" (that's the difference between "it cannot be broken" and "nobody in the world knows how to break it").
A basic hash function, even if secure as a hash function, is not appropriate for password hashing, because:
So we need something better. It so happens that slapping together a hash function and a salt, and iterating it, is not easier to do than designing a hash function -- at least, if you want the result to be secure. There again, you have to rely on standard constructions which have survived the continuous onslaught of vindicative cryptographers.
PBKDF2 comes from PKCS#5. It is parameterized with an iteration count (an integer, at least 1, no upper limit), a salt (an arbitrary sequence of bytes, no constraint on length), a required output length (PBKDF2 can generate an output of configurable length), and an "underlying PRF". In practice, PBKDF2 is always used with HMAC, which is itself a construction built over an underlying hash function. So when we say "PBKDF2 with SHA-1", we actually mean "PBKDF2 with HMAC with SHA-1".
Advantages of PBKDF2:
Drawbacks of PBKDF2:
bcrypt was designed by reusing and expanding elements of a block cipher called Blowfish. The iteration count is a power of two, which is a tad less configurable than PBKDF2, but sufficiently so nevertheless. This is the core password hashing mechanism in the OpenBSD operating system.
Advantages of bcrypt:
Drawbacks of bcrypt:
scrypt is a much newer construction (designed in 2009) which builds over PBKDF2 and a stream cipher called Salsa20/8, but these are just tools around the core strength of scrypt, which is RAM. scrypt has been designed to inherently use a lot of RAM (it generates some pseudo-random bytes, then repeatedly reads them in a pseudo-random sequence). "Lots of RAM" is something which is hard to make parallel. A basic PC is good at RAM access, and will not try to read dozens of unrelated RAM bytes simultaneously. An attacker with a GPU or a FPGA will want to do that, and will find it difficult.
Advantages of scrypt:
Drawbacks of scrypt:
I cite this one because you will use it if you do password-based file encryption with GnuPG. That tool follows the OpenPGP format which defines its own password hashing functions, called "Simple S2K", "Salted S2K" and "Iterated and Salted S2K". Only the third one can be deemed "good" in the context of this answer. It is defined as the hash of a very long string (configurable, up to about 65 megabytes) consisting of the repetition of an 8-byte salt and the password.
As far as these things go, OpenPGP's Iterated And Salted S2K is decent; it can be considered as similar to PBKDF2, with less configurability. You will very rarely encounter it outside of OpenPGP, as a stand-alone function.
Recent Unix-like systems (e.g. Linux), for validating user passwords, use iterated and salted variants of the crypt() function based on good hash functions, with thousands of iterations. This is reasonably good. Some systems can also use bcrypt, which is better.
The old crypt() function, based on the DES block cipher, is not good enough:
But the more recent variants, which are active by default, will be fine.
About everything else, in particular virtually every homemade method that people relentlessly invent.
For some reason, many developers insist on designing function themselves, and seem to assume that "secure cryptographic design" means "throw together every kind of cryptographic or non-cryptographic operation that can be thought of". See this question for an example. The underlying principle seems to be that the sheer complexity of the resulting utterly tangled mess of instruction will befuddle attackers. In practice, though, the developer himself will be more confused by his own creation than the attacker.
Complexity is bad. Homemade is bad. New is bad. If you remember that, you'll avoid 99% of problems related to password hashing, or cryptography, or even security in general.
Password hashing in Windows operating systems used to be mindbogglingly awful and now is just terrible (unsalted, non-iterated MD4).
Up to now, we considered the question of hashing passwords. A closely related problem is transforming a password into a symmetric key which can be used for encryption; this is called key derivation and is the first thing you do when you "encrypt a file with a password".
It is possible to make contrived examples of password hashing functions which are secure for the purpose of storing a password validation token, but terrible when it comes to generating symmetric keys; and the converse is equally possible. But these examples are very "artificial". For practical functions like the one described above:
Indeed, PBKDF2 and scrypt are KDF, not password hashing function -- and NIST "approves" of PBKDF2 as a KDF, not explicitly as a password hasher (but it is possible, with only a very minute amount of hypocrisy, to read NIST's prose in such a way that it seems to say that PBKDF2 is good for hashing passwords).
Conversely, bcrypt is really a block cipher (the bulk of the password processing is the "key schedule") which is then used in CTR mode to produce three blocks (i.e. 192 bits) of pseudo-random output, making it a kind of hash function. bcrypt can be turned into a KDF with a little surgery, by using the block cipher in CTR mode for more blocks. But, as usual, we cannot recommend such homemade transforms. Fortunately, 192 bits are already more than enough for most purposes (e.g. symmetric encryption with GCM or EAX only needs a 128-bit key).
As much as possible! This salted-and-slow hashing is an arms race between the attacker and the defender. You use many iterations to make the hashing of a password harder for everybody. To improve security, you should set that number as high as you can tolerate on your server, given the tasks that your server must otherwise fulfill. Higher is better.
MD5 is broken: it is computationally easy to find a lot of pairs of distinct inputs which hash to the same value. These are called collisions.
However, collisions are not an issue for password hashing. Password hashing requires the hash function to be resistant to preimages, not to collisions. Collisions are about finding pairs of messages which give the same output without restriction, whereas in password hashing the attacker must find a message which yields a given output that the attacker does not get to choose. This is quite different. As far as we known, MD5 is still (almost) as strong as it has ever been with regards to preimages (there is a theoretical attack which is still very far in the ludicrously impossible to run in practice).
The real problem with MD5 as it is commonly used in password hashing is that it is very fast, and unsalted. However, PBKDF2 used with MD5 would be robust. You should still use SHA-1 or SHA-256 with PBKDF2, but for Public Relations. People get nervous when they hear "MD5".
The main and only point of the salt is to be as unique as possible. Whenever a salt value is reused anywhere, this has the potential to help the attacker.
For instance, if you use the user name as salt, then an attacker (or several colluding attackers) could find it worthwhile to build rainbow tables which attack the password hashing function when the salt is "admin" (or "root" or "joe") because there will be several, possibly many sites around the world which will have a user named "admin". Similarly, when a user changes his password, he usually keeps his name, leading to salt reuse. Old passwords are valuable targets, because users have the habit of reusing passwords in several places (that's known to be a bad idea, and advertised as such, but they will do it nonetheless because it makes their life easier), and also because people tend to generate their passwords "in sequence": if you learn that Bob's old password is "SuperSecretPassword37", then Bob's current password is probably "SuperSecretPassword38" or "SuperSecretPassword39".
The cheap way to obtain uniqueness is to use randomness. If you generate your salt as a sequence of random bytes from the cryptographically secure PRNG that your operating system offers (/dev/urandom, CryptGenRandom()...) then you will get salt values which will be "unique with a sufficiently high probability". 16 bytes are enough so that you will never see a salt collision in your life, which is overkill but simple enough.
UUID are a standard way of generating "unique" values. Note that "version 4" UUID just use randomness (122 random bits), like explained above. A lot of programming frameworks offer simple to use functions to generate UUID on demand, and they can be used as salts.
Salts are not meant to be secret; otherwise we would call them keys. You do not need to make salts public, but if you have to make them public (e.g. to support client-side hashing), then don't worry too much about it. Salts are there for uniqueness. Strictly speaking, the salt is nothing more than the selection of a specific hash function within a big family of functions.
Cryptographers can never let a metaphor alone; they must extend it with further analogies and bad puns. "Peppering" is about using a secret salt, i.e. a key. If you use a "pepper" in your password hashing function, then you are switching to a quite different kind of cryptographic algorithm; namely, you are computing a Message Authentication Code over the password. The MAC key is your "pepper".
Peppering makes sense if you can have a secret key which the attacker will not be able to read. Remember that we use password hashing because we consider that an attacker could grab a copy of the server database, or possible of the whole disk of the server. A typical scenario would be a server with two disks in RAID 1. One disk fails (electronic board fries - this happens a lot). The sysadmin replaces the disk, the mirror is rebuilt, no data is lost due to the magic of RAID 1. Since the old disk is dysfunctional, the sysadmin cannot easily wipe its contents. He just discards the disk. The attacker searches through the garbage bags, retrieves the disk, replaces the board, and lo! he has a complete image of the whole server system, including database, configuration files, binaries, operating system... the full monty, as the British say. For peppering to be really applicable, you need to be in a special setup where there is something more than a PC with disks; you need an HSM. HSM are very expensive, both in hardware and in operational procedure. But with an HSM, you can just use a secret "pepper" and process passwords with a simple HMAC (e.g. with SHA-1 or SHA-256). This will be vastly more efficient than bcrypt/PBKDF2/scrypt and their cumbersome iterations. Also, usage of an HSM will look extremely professional when doing a WebTrust audit.
Since hashing is (deliberately) expensive, it could make sense, in a client-server situation, to harness the CPU of the connecting clients. After all, when 100 clients connect to a single server, the clients collectively have a lot more muscle than the server.
To perform client-side hashing, the communication protocol must be enhanced to support sending the salt back to the client. This implies an extra round-trip, when compared to the simple client-sends-password-to-server protocol. This may or may not be easy to add to your specific case.
Client-side hashing is difficult in a Web context because the client uses JavaScript, which is quite anemic for CPU-intensive tasks.
In the context of SRP, password hashing necessarily occurs on the client side.
Use bcrypt. PBKDF2 is not bad either. If you use scrypt you will be a "slightly early adopter" with the risks that are implied by this expression; but it would be a good move for scientific progress ("crash dummy" is a very honourable profession).
For storing passwords hashes, you need an algorithm slow enough that brute-force attacks are not feasible. Salting the password will help against rainbow attacks, but not against brute-force attacks. For storing password hashes, you need to use an algorithm specifically designed for this purpose; such as:
scrypt is new but interesting because it not only uses a variable work factor but also memory-hard functions. This dramatically increases the cost of brute-force attacks, because both running-time and memory requirements are increased.
Passwords stored in a database as hashed values can be recovered either through brute-force calculation of the hashes or through use of rainbow tables (which are specific to the algorithm used).
A rainbow table is created as a series of pre-computed values for either a dictionary file or more commonly every combination of a given character set [a-z, A-Z, 0-9] being a common example.
Essentially they can speed up the cracking of a password by allowing the hash value to be looked up in the table rather than requiring the attacker to create the hash for each password. Rainbow tables for common password algorithms (eg, NTLM, MD5, etc) can be found online, making it fairly straightforward to get access to large volumes of them.
There are a number of ways of improving the security of the hashes stored in the database.
First up is to use a per user salt value, this value is stored in the database along with the hashed password. It isn't intended to be secret but is used to slow down the brute force process and to make rainbow tables impractical to use.
Another add-on I've seen to this is to also add in what was called a pepper value. This was just another random string but was the same for all users and stored with the application code as opposed to in the database. the theory here is that in some circumstances the database may be compromised but the application code is not, and in those cases this could improve the security. It does, however, introduce problems if there are multiple applications using the same password database.
A third means of helping improve the security of the passwords is to use a slow password function, this won't have a huge impact on individual users, but will massively slow down an attacker in cracking passwords retrieved from the database. Some more information on this approach is available here
Update 4: By 2016, the hardware improvements and other factors caused the bitcoin hash rate to rise by a factor of more than 100,000 (!) in the 5 years since when this post was first written in 2011. Password cracking techniques have also improved on the software end. So users should to add a few more characters to the minimum length of their passwords, and iteration counts need to be raised, and we all really need to be preparing to move to better algorithms like Argon2.
Update 3: In 2015, the Password Hashing Competition selected a winner: Argon2. It was designed to be "memory hard" to make GPU implementations by crackers hard; simple; highly configurable; resistant to side-channel leaks, etc. If it passes the test of time, it may be a significant step forward, but as pointed out by Thomas at Are there more modern password hashing methods than bcrypt and scrypt?, you should be wary of shiny new algorithms, and probably give the pros more time to look for weaknesses.
Update 2: In 2013, several experts initiated a Password Hashing Competition which should result in improved and more usable methods, with winners selected by 2015. For excellent background on the need for that, and good advice in the interim, see Password security: past, present, future from Passwords^12. Note that the advent of faster and faster hardware (as discussed below) implies the need for memory-intensive algorithms like scrypt, and that bcrypt is also still resistant to GPU attacks unlike PBKDF2 or crypt.
Others here have pointed out that brute force attacks need to be defended against via salts, even though MYSQL still hasn't figured that out. The importance of iterations has also been noted, and has been known since the seminal paper on Unix crypt in 1978 by Robert Morris and Ken Thompson. But many people (and developers too, like Django!) evidently still think brute force must take a pretty long time or be pretty expensive, and thus think a single iteration of SHA-1 is OK for password hashing.
Not true! Moore's law and cloud computing has caught up with us. To crack a SHA-1 hash of an alphanumeric password of length 8 ((26+26+10)^8 = 62^8 = 218,340,105,584,896 = 218 trillion combinations) on a modern desktop machine takes 5 days, or 1 hour if you rent a bunch of Amazon compute nodes (How long does it take to actually generate rainbow tables? - IT Security)
Update: bitcoin hashing capacity
The most powerful organized hashing capability on the planet (excluding possible classified systems) is the bitcoin mining network. [As of May, 2011 it was] performing SHA-256 hashes at an aggregate rate of over 11 Thash/s, i.e. 11 * 10^12 hash/s (by 2016 that was 1700000 Thash/s - see update 4 above), and the rate has been rising quickly recently (graphs). Miners are working to earn the estimated $700,000 per week that mining yields at the current price of $14 per bitcoin (BTC) (graph), and rate of 50 BTC produced every 10 minutes. Popular hardware these days includes a Radeon HD 5970 GPU, each of which has a total of 3200 stream processors and can do about 800 Mhash/s. It is also thrifty in power consumption at about 2.3 Mhash/Joule. See Bitcoin mining hardware comparison for lots more options. It turns out that GPU nodes on Amazon's EC2 use Nvidia Tesla GPUs which are less efficient at hashing, and their nodes aren't cost effective for mining at today's prices.
This is about twice the capacity of one 5.5 Thash/s estimate for the hashing power of the world's top 500 supercomputers combined, though of course the supercomputers were typically designed for floating point performance, not hashing.
As an extreme current case, if this hashing capacity were redirected to trying to crack passwords, e.g. following a crash in bitcoin prices, it would be fearsome against non-iterated password algorithms. 8 character passwords using a completely random assortment of all 94 printing characters would fall in less than 10 minutes (94^8 / (11 * 10^12 * 60) = 9.2). 10 character passwords would take less than 57 days (94^10 / (11 * 10^12 * 3600 * 24) = 56.7). 10-character upper-lower case alphanumeric passwords (26+26+10 = 62 possible characters) would take less than a day (62^10 / (11 * 10^12 * 3600 * 24) = 0.88) even if well randomized.
But if programmers simply used e.g. an iteration count of 2000 as Thomas suggests, good 10-character passwords would last years. Though 8-character passwords would be easily cracked, within 13 days (2000 * 94^8 / 11 10^12 / 3600 / 24 = 12.8 days).
See also:
Passwords should always be salted and stretched before storing them. Basically this involves appending or prepending some text to the password and hashing the result several times. As for hash algos anything over and above MD5 and SHA-1 is currently advisable - go for SHA 256 or 512 (see http://www.schneier.com/blog/archives/2009/06/ever_better_cry.html)
I'd second the recommendations for PBKDF2. It's not the most computationally expensive, but it does have a precise standard for reference during implementation, and it's well-accepted.
https://www.rfc-editor.org/rfc/rfc2898
I'd really recommend reading Colin Percival's paper on scrypt, though. He does a good job describing the issues in play here. My guess is scrypt will look better and better with time.
http://www.tarsnap.com/scrypt.html
Having an implementable standard is not nothing, by the way - there have been differences between the algorithms described in papers and the reference implementations in both bcrypt and scrypt, if memory serves.
A good password hashing algorithm must have salt and something to make the calculation of the password expensive (usually iteration count).
The best, and most common, method for this is PBKDF2. Although not perfect, it should be a baseline for everybody:
Depending on the algorithm you use the answer is probably no.
First off you should Salt them, this basically means appending or prepending some text to the password.
Then you should use a strong algorithm (md5 doesn't cut it)
It is interesting to note that although bcrypt and scrypt are all good solutions to passwords, with a favor for the latter, scrypt seems to be prone to cache-timing attacks. As suggested here: http://eprint.iacr.org/2013/525 Catena will be safe against this, along with provable safety and a few other nice features.
bcrypt is said to be slower on GPUs, which makes brute-forcing slower. However, with ever evolving computing hardware, security should not only rely on the difficulty of implementing a particular hashing algorithm on a particular hardware.
Rather, you can arbitrarily increase the cost of brute-forcing a hash by using the "variable work/cost factor" (sometimes also called "rounds") that some hash functions libraries using hash functions support. Amongst them are bcrypt and SHA-512glibc's crypt function and the pam_unix library.
For example, a cost factor of 100000 for SHA-512 makes generation (and therefore brute-forcing) of the hash about 4 times slower than the cost factor 08 for bcrypt. This can be confirmed by using a hash solving program like hashcat.
If you assume that at some point your password hashes and salts will be stolen and the attackers will use ASIC hardware to brute-force them, you could simply increase this work/cost factor to still make it too costly for them, while hopefully not overloading your servers' CPU with regular user authentication.
The importance of long and random passwords applies nevertheless.
Note that POSIX user authentication security has been relying on this technique since 2007 (since glibc 2.7), with Debian 11 only recently switching to yescrypt from SHA-512 with 5000 rounds.
I wrote a blog post about this technique including details and reproducible measurements of brute-forcing hashes with hashcat.
Quote from man crypt:
Since glibc 2.7, the SHA-256 and SHA-512 implementations support a user-supplied number of hashing rounds, defaulting to 5000.
Quote from man pam_unix:
It uses standard calls from the system's libraries to retrieve and set account information as well as authentication.
sha512: ... The SHA512 algorithm must be supported by the crypt(3) function.
rounds=n: Set the optional number of rounds of the SHA256, SHA512 and blowfish password hashing algorithms to n.
Quote from man login.defs:
When ENCRYPT_METHOD is set to SHA256 or SHA512, this defines the number of SHA rounds used by the encryption algorithm by default. With a lot of rounds, it is more difficult to brute forcing the password. But note also that more CPU resources will be needed to authenticate users. If not specified, the libc will choose the default number of rounds (5000), which is orders of magnitude too low for modern hardware.
See also:
/etc/pam.d/common-password/etc/login.defs
I use SHA1 here
def __encrypt(self, plaintext, salt=""):
"""returns the SHA1 hexdigest of a plaintext and salt"""
phrase = hashlib.sha1()
phrase.update("%s--%s" % (plaintext, salt))
return phrase.hexdigest()
def set_password(self, new_password):
"""sets the user's crypted_password"""
#from datetime import datetime, timedelta
import datetime
if not self.salt:
self.salt = self.__encrypt(str(datetime.datetime.now()))
self.crypted_password = self.__encrypt(new_password, self.salt)
def check_password(self, plaintext):
return self.__encrypt(plaintext, self.salt) == self.crypted_password
Rather than salting passwords like this I would like that the algorithm is changed so that it doesn't need salt and still won't hash the same input twice to the same value.
