For starters, yes, key bruteforce is unrealistic under normal circumstances. Any properly configured server will not allow you to make billions of guesses at the SSH key. If it does, there are some major problems with the security protocols of the server. All sorts of red lights should be going off from the logs of a production server if there are that many key failures against a single account.
http://blog.codinghorror.com/brute-force-key-attacks-are-for-dummies/
https://askubuntu.com/questions/2271/how-to-harden-an-ssh-server
For nearly all intents and purposes, it is in practice impossible.
Keep in mind that if someone has gained access to a client file system, you are already is very bad shape. However, that does not mean all further security should be abandoned.
For example, in the case of a remote exploit it is very common to gain shell access to a system without gaining full rights. For instance, a remote exploit of the Apache web server will land with a shell... as the server itself (whoami = apache), which isn't always as privileged as you might have hoped.
So in the case of a compromised client machine, you want to make sure that before compromise that machine has every sort of security protection it can to make post compromise life hellish.
There are many ways to compromise a client machine, so it isn't really possible for me to outline all the possible scenarios here but:
- Make sure to do everything you can to lock down the machine from physical access. Good hardware management, full disk encryption, good passwords.
- Ensure that once the attacker is on the machine the password to everything is neigh impossible to guess
- Use a program to generate large, securely generated passwords. Store the passwords in a keystore that is locked when the user is away, and has a very large passphrase itself.
- Enforce a policy that your users always full-disk-encrypt their laptops while on the move. This way a stolen laptop is useless to an attacker.
- Use client side log auditing software (like ossec) to raise alarms when fishy activity happens on a client machine.
- Configure a lockout on SSH key guesses. This will only slow an attacker down, but with a sufficiently good password it is almost a game over for the attack.