1

We had a major issue with Google Cloud Platform some time between 2019-04-16T11:00:00Z and 2019-04-17T02:30:00Z, when all of our service accounts lost their access keys for an unknown reason, so our Google Compute Engine and Google Cloud Storage API requests failed, disrupting our production services.

ie. When we visited https://console.cloud.google.com/iam-admin/serviceaccounts, the entire column for "Key ID" displayed "No keys" and we had to create new keys for each service account to restore access.

Why or how could this happen and what can we do to prevent it happening again?

Sim Kennedy
  • 31
  • 3
  • 2
    Have you contacted GCP support? – ceejayoz Apr 24 '19 at 20:46
  • 4
    Check your [audit logs](https://cloud.google.com/logging/docs/audit/). – Michael Hampton Apr 25 '19 at 00:47
  • @ceejayoz We don't have a premium support package so I don't think we can't contact them directly. Instead I've reported an issue, submitted feedback and posted here and on Google Groups as they recommend here: https://cloud.google.com/support/docs/community – Sim Kennedy Apr 25 '19 at 13:49
  • @MichaelHampton Aha! I didn't know about these. I just checked and can now see the entries for DeleteServiceAccountKey from an old user with admin access. Exactly what we needed! Thank you so much! – Sim Kennedy Apr 25 '19 at 13:54

1 Answers1

2

Thanks to Michael Hampton's comment, we got to the bottom of this issue after looking at the Audit logs (https://console.cloud.google.com/logs/viewer). The Service Account key deletions were executed via API by Terraform (https://www.terraform.io/), which was being used by a dev with Editor access testing Terraform on what they thought were their own projects. Automation can be dangerous..

Sim Kennedy
  • 31
  • 3