According to the Truecrypt documentation, only the first 1MB of contents in the keyfile is actually examined. So anything beyond that is totally superfluous.
But more interestingly, keyfiles are simply hashed (along with other keyfiles and any password) to produce the real key. As such, the real key length is limited -- as of course it would have to be.
The length of the hash output depends on the hashing algorithm, but the important point is that no matter how much entropy you put into the hash, the amount you get out is capped at the hash size. The result is far larger than could possibly ever by brute-force cracked, so that's not a concern. But if you're using truly random keyfiles, then just a few bytes (say 64 characters) is typically enough to hit that entropy cap. Anything beyond that is just further permutation of the same bits.
Again, you choose the hash type, and SHA-256 and SHA-512 are common candidates. As for how easy it is to brute-force crack a 256-bit key; it's impossible beyond the laws of physics.