The needle-in-a-haystack method you're employing is precisely what I do with TrueCrypt. There are about 600 key files in a directory on a flash stick, and ~10 of them are used.
The flash stick has a hardware read-only switch, which prevents the OS from updating the "Last Accessed" metadata. Keep that in mind if you're using a writeable medium for storage - the key files you access will (likely) be given away by the access date. I've also added some extra "chaff" to this by randomising the created / modified / accessed dates on the files, to make all dates completely untrustworthy.
This makes it very difficult to find the right key files - 1,545,269,050,000,000,000,000 potential combinations, assuming 10 chosen from 600 files, if order doesn't matter and any one key file can only be used once. Each combination must be tried for every potential password, making such a brute-force completely infeasible. As long as you're using random key files of sufficient size (I usually use 4kB or more) there's no feasible way to brute-force the content, either. The key is (usually) generated by a hash of the file content, or a composite of those hashes if more than one key file is used, combined with the hash of the password. Note that "hash" could mean a simple hash like SHA256, or the output of a key derivation algorithm like PBKDF2 or bcrypt. As such, computing the key without the keyfiles is impossible. Conversely, it's impossible to identify the keyfile from the volume, since you'd need to know the key to deduce such a thing.
To address the potential cache timing attack mentioned in the comments by fatfredyy - this is highly unlikely. The proposed attack involves reading various files and comparing their access times. A recently read file should read faster than the others, since it'd be in the cache. However, read times from disk are noisy on hard disks (seek times from unknown current sectors make things interesting) and accessing the cache has a Heisenberg effect - reading a file will likely load it into the cache, and potentially corrupt the integrity of the cache in terms of potential for timing attacks. A similar problem occurs on SSDs, which employ relatively little read caching and a lot of write caching. In some systems, you may find that sequentially reading files in a directory will result in later files being prefetched, further complicating the timing attack. Short of physically extracting the hardware cache data from the drive, I don't believe this is feasible.