Backstory
We've just set up our GCP project. We've got several Compute Engine VMs (centOS) running. We've set up a VPN between our production co-lo and GCP. All appears to be working fine.
We do have the networking (VPC) set up in one project and it is shared with the project that has the VMs.
Problem
I've set up Private DNS using the Cloud DNS and added in a test A record that has one of our co-lo IP addresses. The NS IP in the DNS matches the nameserver in the resolv.conf file.
When I try to ping the DNS record, I get "Name or service not known".
I believe that we've got everything set correctly, but we are not getting the name resolution.
Any help would be appreciated. Thanks!
Update 12/4/2018
Added the DNS to both projects and DNS resolves. If we remove it from either, then it doesn't work. Still doesn't sound right, but whatever.