2 days ago I have received a very frustrating email from Google Cloud Platform Compliance team and after some investigations I was recommended to post my question to the community and ask for their opinion and help!
So here is the first email I received!
Action required: Critical problem with your Google Cloud Platform / API project.
Dear Developer,
We have recently detected that your Google Cloud Project XXXXXXX (id: cuZZZZZZZZZ-XXX-12345) has been performing intrusion attempts against a third-party and appears to be violating our Terms of Service. You can fix the problem by ensuring that your project traffic directed at third-parties is expected, and that your project has not been compromised.
We will suspend your project in 3 days unless you correct the problem and respond to this email by submitting an appeal.
I checked with my team for any kind of strange behavior on our server and after digging deeper into our server we were not able to find anything wrong, so I hit the button in the email to "Request and appeal".
In the form I described that there is nothing that we found to be Violating Google guidelines!
After some time I received the second email with a little info about what had happened on our server. The Google team member claimed the following:
Based on our records, we detected SSH brute forcing attacks originating from IP 10x.1xx.xx.xx (our server IP) targeting more than 942 IP addresses on December 26, 2016.
As the project owner, you are responsible for securing the software installed on your machine. Please provide us with more information regarding the behavior of your project and specifically how it could relate to what we have detected.
So this means that on Dec 26, our server has made 942 different SSH requests to different IP addresses, which is impossible! We don't have such a script running on our end that is constantly establishing connections with port 22 to 942 different IP addresses... If there has been such a strange behavior on our server it would mean that our server was hacked and someone is running this strange script! But this is not the case as well, as we have checked the latest added and modified files one by one: all of them are added by us - through Git...
So I contacted back to the Google Cloud Console team asking them for some more info, but they got back to me that they can't give more info and that they recommend me to connect with the community and ask them for help!
Posting the latest email I received from them:
Dear Developer,
Thank you for your response. Unfortunately, we do not have visibility into your instance or how it can be fixed. However, you can reach out to the Google Cloud Platform community support at https://cloud.google.com/support/#community for recommendations. Also, you can look at some of the recommendations in the original external escalation as a starting point.
We strongly recommend adopting additional security measures and then completely re-installing your project.
We don't really want to reinstall out project as it is a huge environment with 15k users....
Anybody can help me with any idea or suggestion about how can I track this strange behavior or what would be the best way to with this. Please note that they initially told that they are going to suspend the instance if the behavior is not fixed and appealed.
Dear community members, I am looking forward to hearing your thoughts! This is very important for our startup!
