SSH Public Key Auth on VM when connecting to host remotely

Question

On my server I have two VMs that I normally run, and I typically access this server (the host) via ssh. When connecting to the host via SSH, it connects fine with the public key. I can connect to the VM via ssh, but the public key is only used if I am on the host computer itself.

If I am connecting to the host via my laptop or phone via ssh, and then try to connect to the VM via ssh the public key is ignored and I have to enter my password for the VM.

I have added the public keys from my laptop and my phone to both the host and the VM, but that hasn't made any difference. My sshd_config is the same on both host and VM as well.

I have several control/automation scripts that I run on the host to control the VM system and it is very troublesome to have to enter my password every time it sends an SSH command to the VM.

I have tried every search I can think of to see what config changes I need to make to facilitate this but everything I find is just how to setup public key authentication in the first place.

Is it possible to setup the VM's sshd to accept the public key of my phone/laptop when connecting to it though the host computer? Or to configure the host server to pass-on it's own key the same as when I am logged into it directly?