What you are asking about is typically called pre-shared key (PSK) scheme, where a previously shared secret between the parties is used to secure communication. Conceptually, it is secure from quantum attacks. However, PSK is typically implemented with symmetric keys, where you skip key exchange and shared key derivation and go directly from handshake to encrypted channel. This is done with symmetric keys because such algorithms are more computationally efficient. There is no technical reason why PSK could not operate with asymmetric keys, but there is also no reason to use asymmetric keys when you already established a shared secret. In addition, this is not how any of the existing protocols utilize public keys - where public key is transmitted in the clear. For example, as part of TLS protocol server will transmit its certificate that contains public key in the clear during handshake. At that point, public key can be trivially extracted by a passive eavesdropper.
Symmetric cryptography is resistant to quantum computing while asymmetric cryptography is not. The part of asymmetric cryptography that fails to quantum is key derivation, specifically establishing shared secret - it is theoretically possible for an attacker to derive symmetric keys based on information exchanged during handshake and negotiation. All quantum attacks on asymmetric schemes assume knowledge of public key. Assuming custom protocol that never transmits public keys in the clear is used, you would be safe from quantum attacks. Unfortunately, such protocol would not be at all useful in any way.
For now, the best you could do to improve your resistance to quantum is to utilize very, very long public keys. This is based on the assumption that quantum computer with larger number of quantum gates is harder to construct and operate.