Normally you know the hash function from context. If you are trying to crack a password then this password is valuable to you, in that it would grant you access to some data or service somewhere, which you specifically target. So one has to assume that you already know the kind of system (in particular if you succeeded in grabbing a list of hashed passwords from a database, then you saw the database), and this often includes knowledge of the password hashing function. Moreover, in systems which support several types of hash function (like the /etc/shadow
file on Unix systems), the kind of hash is encoded along the hash value, because, by nature, the system itself must know it.
Remember that hashed passwords are intended to be used by some software, somewhere, which exists as executable files and source code. On a general basis, software cannot be considered to be secret, and reverse engineering is quite effective at uncovering the kind of processing that is done on data.
For Windows, passwords tend to be processed with two hash functions, the old LM (LanMan), and the newer NT Hash (which is MD4). There is also a "password verifier" used in some part, which is "salted" with the user name (not a good salt, really: every "Administrator" is called "Administrator"). See this page for details. LM hash is extremely weak and was used as the demonstration testbed to show the effectiveness of rainbow tables; fortunately, it is disabled by default in newer systems (since Vista). However, the NT hash is still quite weak (not as abysmal as LM, but still) since it is very fast (a single MD4 invocation...) and unsalted (thus making it worthwhile to build precomputed tables, such as rainbow tables).