A password that is human-memorable, or even one that can be typed on a normal keyboard, is going to be way lower entropy than its bit length, because many of the individual bits aren't going to be random at all even if the characters in the password are random. For example, the high bit of every byte will almost certainly be 0, and there's an eighth of your entropy gone right off the bat. A 32-character password that a human can memorize and type quickly will almost certainly be way weaker then one made of truly random characters; it'll probably incorporate actual words and so on, which have much lower entropy per character than 8 (or even 7) bits.
To make up for the (relatively) low entropy of the password, you use a key derivation function to make brute-forcing the key as computationally difficult as possible. That's what the "cost" or "work factor" parameter in KDFs and password-hashing algorithms is for (and the "salt" - which does have to be totally random - is there to prevent pre-computing the keys/hashes). The attacker might still only have to try a few trillion passwords (this might sound like a lot but to a computer it really isn't; that's only 50-odd bits of entropy) but if they can only try tens (or even thousands) of them per second, that's pretty much infeasible. If they can instead try the passwords directly as keys, it'll take... minutes. Maybe a few hours if the attacker doesn't care to throw money at making it faster. Modern hardware is really fast, but more to the point it's extremely parallel, and this problem is trivially parallelizable (such that you can rent a bunch of EC2 instances or whatever to distribute it across, if you need the answer really fast).
Now, to be clear, this only applies to passwords, which are human-memorable and human-input secrets. If you instead generate 32 cryptographically random bytes, that's just a key, and can be used as such directly. You probably don't want to store it in plain text on your hard disk alongside your encrypted file, though, and you probably don't want to carefully type it in (with all its hex values or escape codes for the non-printing characters, never mind that it's probably not valid UTF-8 to begin with) every time you want to use it, so that means... probably encrypting it somewhere. Which brings us back to getting the key to decrypt it, which is probably from a password, and thus we're right back to password-derived keys.
