I configured ssh to use GPG as my ssh-agent and if I remove the ~/.ssh folder, I can ssh into my server fine using my gpg key. However, my ~/.ssh folder has over a dozen different ssh keys in it, and if I try to ssh when it is there, I get a permission denied error because my ssh client is offering every single private key in the directory before trying the keys in the gpg ssh-agent.
With regular ssh-keys, I just use the IdentityFile config in my ~/.ssh/config file, but I can't do that because my identity is a gpg cardno. I am confused by why ssh is preferring the key files over the agent. Is there any way to force ssh to use the agent instead of the files? Or even better, is there any way to specify in the ~/.ssh/config file that the gpg key must be used for a given server?
I have confirmed that ssh-agent is not running and that gpg-agent is running and ssh-add -L shows my gpg key to be present, along with a single other ssh-style private key.