We recently had a Google App Engine application fail to properly fail over during scheduled maintenance of our database server (hosted in Aiven).
During scheduled maintenance, the DB server will fail over to a replacement server by updating the DNS record. This is supposed to be instant, but we found that our node app running in GAE was crashing with connection failures for several minutes.
The connection error is taken as a hard error and so the node app exits, and is then immediately replaced with a new process by running npm start again. However, this process also failed as it also could not connect and so on until GAE decided the server was a lameduck and replaced it.
By the time the instances were replaced it looks like the connection issue was resolved, but i'm unclear on the reason for it taking so long to resolve.
My suspicion is that the old database hostname may have been cached and so it was stuck trying to connect to the old IP.
As a work around I'm wondering if it's possible to flush the DNS cache on a google app engine instance (from within the instance)?
I've looked for documentation on how App Engine resolves DNS, but end up at pages about setting up custom or internal DNS.
To summarise: Our node app running on GAE is connecting to an externally hosted database and is identifying the host by DNS lookup.
So the cache I need to flush is the one that is caching lookups of a public DNS record not hosted on google, but being requested by a GAE app.
ie:
Node GAE App -> { Public Internet } -> Database