8

We are using a Continuous Integration service to automatically run our product test suite. Every time we push code to our central Git repository production branch, the CI services is notified and fetches the code to run the test suite.

The CI service allows us to write a post build script which can automatically push the code to our Heroku production server when the tests pass.

However we fear that if an attacker breaks into the CI service, he can then push any code change he wants to our production server.

We plan to change this architecture to have the CI service ping a specific 'deploy' server of ours when the tests pass. This 'deploy' server would fetch the latest production branch code from our central Git repository and the push to our Heroku production server.

This way, if the CI service is compromised, an attacker could not push from the CI service any code he wants to our production server. He would have to break into our 'deploy' server.

The goal of this new architecture is to move the risk from the CI SaaS to a server of ours. The Heroku credentials necessary to deploy are no more hosted on the CI SaaS but on a server of ours.

Does it make sense? Or is there any other simpler alternative when using a CI service (apart of setting up and securing one's own CI server)?

Bradley Kreider
  • 6,152
  • 2
  • 23
  • 36
Florent2
  • 183
  • 5
  • "a post build script which can automatically push the code to our [...] production server when the tests pass." does not seem like a wise plan to me. You want to have a responsible and trustworthy human making decisions about what goes to the production instance, don't you? – adric Sep 13 '12 at 22:06
  • 1
    Yes we want :) That happens when one of us merges code in the production branch and push it to our central git repository. From this point we want the deploy to be automated. I've edited my question to clarify this point. – Florent2 Sep 14 '12 at 00:04
  • 2
    I'd worry most about one of your developer machines getting breached. – CodesInChaos Sep 14 '12 at 07:01
  • Having one of our developer machine getting breached is always a risk, we want to mitigate the additional risk of having a breach on a service (the CI service) over which we have no control. – Florent2 Sep 14 '12 at 08:49
  • 1
    Don't you do any manual testing before you deploy? – tdammers Sep 14 '12 at 10:38
  • 1
    We do, our CI setup is a way to make sure the test suite is ran before the code is pushed to production even when we forget to do it manually. – Florent2 Sep 14 '12 at 14:00

2 Answers2

4

It sounds like you are mostly moving the problem around. The CI server used to have the credentials to push new installs to the Heroku server, now it's the deploy server.

So the big question from me is - what is the difference in security between the CI service and the deploy server? Is one hosted externally? Is there a reason you trust one more than the other. It's the credentials that give access to the Heroku server that give the most risk.

In the NEW framework, I'd ask these questions:

  • What is the likelihood that code could be tampered with on the Git server?

  • Is there a possibility for an illegitimate "ping" to the deploy server to cause problems? The race condition @roxOr mentions is one case. There's also the thought that if a CI server is tampered with, it could be triggered to send a ping that induces your deployment server to put non-working code on the Heroku server. Or... what is the level of authentication required on this ping? Can it be sent from anywhere by anyone? Then the attacker does not need to attack the CI server to trigger a bad deployment.

  • is there a chance that the deploy server can be hacked? that is now your big asset - it can get your intellectual property (the code) and deploy whatever malware it likes. So it needs the most stringent protection.

  • Is there a way for an outside agent to get the credentials used to access the Heroku and access it directly? In which case it doesn't matter how you deploy code, the attacker can push code however he likes.

  • Is there a chance that networking can be tampered with - a very good hack would be to force the deploy server to be redirected from the legitmate git hub site to a source of the attacker's choosing. Then when a legitimate ping is recieved from the CI server, the deployment server deploys attacker-made code instead of the tested code.

No system is ever "perfect", so the big question usually comes down to what kind of an attacker do you expect, and what do you think their motive might be. Disgruntled employees, corporate espionage, random punks and cybercriminals looking to use your site as a malware host will each have their own goals and each one will have different capabilities. It helps to ponder what you want to protect against before you update your architecture.

bethlakshmi
  • 11,606
  • 1
  • 27
  • 58
  • 1
    The main difference between our deploy server and the CI service is that we control the deploy server while the CI service is as SaaS over which we have no control. The Heroku credentials are indeed critical and we prefer to have them hosted in a server of ours rather than on a SaaS. – Florent2 Sep 14 '12 at 14:03
  • When pinging the deploy server only an existing commit of the production branch can mentioned, so hopefully an attacker should not be able to force the deploy server to deploy any code he wants. – Florent2 Sep 14 '12 at 14:09
  • Thanks for your detailed answer bethlakshmi, I have edited my question to clarify that the main goal of the new architecture is too move the risk from the CI SaaS to a server of ours. – Florent2 Sep 14 '12 at 14:10
3

You have 2 points that can get compromised: git repository access and the CI service. Are you using CI SaaS, because what is the difference if your deploy server or your CI machine get compromised? They can both push code.

If you have a CI service that is remote, you might as well build on your deploy server and push that actual build to the CI service. And when that build passes, push it to production. That way you know that you have tested the exact version that is getting deployed.

Race condition: CI passes the code. Someone checks new code into git. Deploy machine is alerted by CI and builds with the untested code and pushes it to production.

Bradley Kreider
  • 6,152
  • 2
  • 23
  • 36
  • We are using a CI SaaS, the solution I explain in the question is made to not have the CI machine able to push code. Thanks for the race condition, had not thought about that, to prevent it will have the CI Saas ping the deploy server by specifying the exact commit(s) to deploy. – Florent2 Sep 14 '12 at 08:50
  • 1
    The race condition is easily averted by keeping track of *which version* passed CI. – tdammers Sep 14 '12 at 10:38
  • @tdammers Agreed, but it didn't seem like it was considered. One more piece of complexity to manage if doing 2 builds and testing one. Besides building on 2 different machines. – Bradley Kreider Sep 14 '12 at 14:42