It will be more difficult to brute-fore a hash without knowing the hash function used -- you'll have to guess both the hash function and the data to hash. However, there's no reason to keep the hash function secret.
First, it's usually quite apparent from the length of the hash (or from setting up a dummy account and trying many obvious possibilities) what algorithm was used to generate it. It's generally a bad idea to modify existing hash functions to "roll your own" hash function, as (1) its unnecessary, (2) will be a nightmare to maintain across various systems, (3) if done wrong can drastically weaken the security, and (4) how the hash function is calculated is going to be in the source code/executable and a determined attacker who compromises the list of hashes, may also be able to get their hands at the source code/executable and be able to reverse engineer the hash function from there.
If you really want to go down the route that the hash should not be brute-forceable (until the compromise the application source code or executable), you shouldn't use a hash but should use a MAC (Message Authentication Code). You can build a MAC out of a hash function using the HMAC construction: HMAC(password) = H(K1 ++ H(K2 ++ password))
where H
is a hash function, K
is a secret key (stored in the application computing the HMAC), K1 = K XOR 0x5c5c...5c
and K2 = K XOR 0x3636...36
are two different secret keys derived from K (where 0x5c5c...5c
and 0x3636...36
is a block of the byte 0x5c
or 0x36
repeated with the same length as the secret key). So if you have a 256-bit secret key and use SHA-256 as your hash function, there are about 2256 potential HMAC keys (~1077 -- for comparison there's only been about 1026 nanoseconds since the big bang) an attacker would have to iterate through before they find the K used (if they in fact find the correct K and not an unlikely collision that works for one specific password but not others).
MACs are usually used to verify message integrity (in that only someone with the secret key can generate a valid MAC) and not for storing passwords/authenticating users, but there's no reason they couldn't be used for this purpose. The problem is the same -- you do not want someone who obtained a message and the associated hash (MAC), be able to create new hashes (MACs) for other messages (as part of a brute-force attack). MACs are believed to have security against this attack; while choosing some random complicated scheme like SHA1(SHA512(pw)++SHA256(MD5(pw)++MD4(pw)))
is in principle much easier for an attacker to guess than a random 256-bit key (and more likely to be leaked from some sort of side-channel attack).
Granted using a MAC is probably overkill. If you use strong passwords and a modern key-strengthened salted hash (bcrypt, PBKDF, SHA512crypt) then brute-forcing is simply not feasible.