5

I have a VPC with public and private subnets.

  • Public subnet contains my Nating and Bastion instances
  • Private subnet contains my application servers (3 ec2 instances running tomcat7 with my project war file). These 3 instance are behined the ELB

Whenever I want to update war file, I can do it in 2 ways.

Manual

  1. I upload new war file to S3 bucket
  2. SSH to Bastion server (only this can be SSHed)
  3. from here SSH to private instance
  4. Download war file from S3, stop tomcat7 replace old war(and delete directory)
  5. Start the tomcat

I repeat above steps for each instance (currently 3 instances)

Semi Manual Here are the steps

  1. Upload new war to S3 bucket
  2. Kill an instance (from control panel of AWS web site)
  3. Scaling and Launch configuration automatically kicks in to maintain minimum number of instance. This launch a new instance and execute a script which actually download war from S3 and some other environment settings and folder etc.
  4. When I see new instance is created,up and running and added to ELB, I kill another old instance.
  5. That way I kill all the old instance one by one, and they got replaced with new instance with latest upgrades.

First method is quick but lots of manual steps. SSH in-and-out of the instances (SSH on instances is only allowed from BASTION server). download, stop, replace, start.

Second method is slow and not actually automatic. Old instance termination time + New instance launch time + Deploying the new war

I want to achieve following things

  1. Automate the upgrade process (Click of a button)
  2. Non-dev like QA and Support people can upgrade without having access to AWS website or SSH keys
  3. Quick upgrade
  4. Rollback in case of failure

One may think why don't I use Beanstalk, its simple and just upload new war file, automatically upgrade all the instances with roll back feature if anything goes wrong?

Here are my reasons (some things might be just my assumption or lack of knowledge).

  1. Overall architecture of the application is complex, VPC contains 9 subnets, and lots of ec2 instances in private and public subnets. and complex security configuration using security groups
  2. My own naming convention for the subnets (Public_Subnet_A, Public_Subnet_B, Private_WEB_A,Private_WEB_B, Private_API_A, SG_GLOBAL, SG_BASTION, SG_BASTION and many more). I don't see a way doing this in ElasticBeanstalk.
  3. Beanstalk only support limited predefined templates for pre-defined environments. Like I want my instances based on Ubuntu, having Tomcat7, running Java6 or 7 and few others.
  4. Beanstalk for Java application runs tomcat8, amazon-instance, it also keep tomcat behind Apache (and I think bridging is done through mode_jk) and then this Apache is behind ELB. And I think ELB->Apache-mode_jk->tomcat thing is little overkill(my be wrong).
  5. I am using ELB->tomcat approch
  6. My outward traffic is through NATing server, There is no way (theoretically) for the outer world to know instances.
  7. In beanstalk instances can be accessed directly. but in my environment its not possible you can only ssh BASTION instance from specific IP with a ssh-key and some sudo-random username (I think it is hard to guess), and private instances can be SSHed through BASTION only (Custom security Group configuration).
  8. Similarly my RDS instance is also in one of my appointed private sub-net and only API instances of application can connect to database (told you lots of custom configuration).
  9. And there are some other as well.

Now you know the background. Lets stick to the simple version of example I explained on the top.

I want the upgrade to be quick and automatic, also not to use third part tools to share my credentials with those tools(I have heard about some Cloud based tools that can provide you transparency and perform all the task on your choice of cloud environment).

Only thing I can think of is writing an ANT-script or Some Java code which may perform all those steps explained in first method. though not sure they would be allowed to perform these operations on aws.

PHP Avenger
  • 259
  • 2
  • 8
  • Have you tried [Jenkins](https://jenkins.io/)? – Dusan Bajic Sep 03 '16 at 12:05
  • Nope, but thinking on similar lines like, create ANT commands in Jenkins, to perform the steps explained in first method. – PHP Avenger Sep 03 '16 at 12:12
  • Jenkins is for good reason almost an (open source) industry standard for what you are trying to achieve. I personally use it to orchestrate shell scripts (even for non-continuous-deployment stuff), but try what ever language you are most comfortable with (and there is a huge number of plugins for more pointy clicky oriented folks). – Dusan Bajic Sep 03 '16 at 12:21
  • I think you can turn on automated deployment. On tomcat 5.5 that was standard. So whenever a newer war appeard it was auotodeployed if wanted. – Nils Sep 03 '16 at 13:58
  • Elastic File System could play a part in this. – Tim Sep 04 '16 at 01:27

2 Answers2

0

Same issue, after spending about more than two days, I realized that there's plugin named Publish artifacts over SSH for jenkins (if you have CI) and doing some configuration changes to allow NAT to talk to jenkins server, at least that did the trick for me.

I do have same setup, as servers in private subnet and NAT in public subnet, from where I ssh to deploy war's to the private machines.

Hope this helps!

p.shah
  • 1
0

Use CodeDeploy, you already have your war in a S3 bucket, so the only other thing you need is aws cli https://docs.aws.amazon.com/cli/latest/reference/deploy/create-deployment.html command.

For this to work, you need a CodeDeploy agent on the instance - you can follow this https://docs.aws.amazon.com/codedeploy/latest/userguide/codedeploy-agent-operations-install-linux.html

CodeDeploy agent needs one simple configuration file and at least one configuration script (cp .war file to Tomcat + reload the service).

CodeDeploy can update instances based on a tag or in an autoscaling group. I would also suggest to put your instances in an autoscaling group, but this can be done as a second step.

It is possible to remove the instance from ELB (standby) before update and put it back after the update, so you can have zero downtime. Together with healthchecks you can have a completely automated solution.

With CodeDeploy you can have similar features to Beanstalk but still keeping the flexibility of accessing the instance directly and using own templates. The instances also do not need to be accessible from internet at all.

Petr Chloupek
  • 254
  • 1
  • 6