A passphrase-encrypted key provides two-factor authentication, but only if used correctly. It is easy for the user to misuse the key, providing only a single factor, and the server cannot detect incorrect usage. Hence a passphrase-encrypted key cannot be considered two-factor without additional assumptions. From the point of view of the system as a whole, the passphrase-encrypted key provides two factors, but from the point of view of the server, there is only one factor, which is the private key.
The password is what-you-know. However, the password is not visible to the server. The server does not know if you used a weak password or no password at all. In any case, typing a password on a machine which may be running a keylogger is not valid use of a password for authentication.
The key file is what-you-have, but only if you do not copy it willy-nilly. Strictly speaking, it's the USB stick where the key file is stored that is a something-you-have authentication factor. The key file itself stops being an authentication factor once you allow for it to be copied off the stick.
In the scenario that you describe, where you copy the key on a machine that you do not control, is not valid usage. It transforms what you have into what the attacker also has. If the attacker can install a keylogger on that machine, he can also install a program that makes an copy of the content of every removable media that's inserted into it. What you have must be tied to an actual physical object that is not accessible to the attacker. A key stored on your own laptop or smartphone is fine. A smartcard inserted into a smartcard slot is fine (for normal smartcard usage, where the secrets do not leave the card). A USB stick inserted into a public machine does not provide an authentication factor.
And yes, there is off-the-shelf malware that grabs the content of removable media. Depending on where you plug in, it may be more or less common than keyloggers (though I'd expect the two to often go together). The attacker who installs a removable disk imager may be after authentication data, or possibly after other confidential documents. There is a resale market for corporate secrets (confidential documents, contact lists, etc.) which fosters this kind of malware, and grabbing authentication data is an easy side benefit.
With a user who may insert his USB stick into a public machine and type his password there, the passphrase-encrypted key provides zero authentication factor.