0

Firstly, although knowing about AWS for a long time, this is the first time i've ever used it.

I gather that EBS volumes are for persistent data storage and instance store volumes are for one time use. I'm using the instances for data mining so they are launched on spot instances and use a lot of I/O while for filling requests for the master.

I cant use EBS because the I/O costs a fortune and i dont need the instance to survive, it gets a job from the master server, processes the job and returns it, no data needs to be saved on the instance. I feel that i need to use Instance Store but there doesn't seem to be a way to save an AMI and load it as an instance store. How am i meant to set this up ? I cant log into the instances every time to set them up, plus it takes 10 minutes to install required dependency. I assumed i'd be able setup the instance store instance, then save, and just load copys of that. Can anyone tell me where I'm going wrong or what i should do in this situation.

Thank you!

kjones1876
  • 213
  • 1
  • 2
  • 6

3 Answers3

0

I wasn't aware of EBS costing a fortune. While you can provision high IOPS you don't have to.

While you can save a snapshot of your system there is another method. You can use scripts to setup your EC2s automatically. You can supply a script via EC2 User-Data and have that install all your required dependancies. You can also take advantage of software like Puppet, Chef, SaltStack to automate things.

Drew Khoury
  • 4,569
  • 8
  • 26
  • 28
  • As for EBS i paid paying $0.50 in EBS fees for $0.05 worth of server fees in 1 hour. Thats 10x times the cost. Is there a way to make a snapshot of a running Instance Store volume ? I assumed thats how it would work but i couldn't find it. – kjones1876 Aug 03 '13 at 09:43
  • 50c p/h does sound high but I have no idea what services you paid for as I haven't had a look at the invoice. Something isn't right here. – Drew Khoury Aug 07 '13 at 04:07
  • http://aws.amazon.com/pricing/ebs/ Amazon EBS Standard volumes = $0.11 per GB-month of provisioned storage + $0.11 per 1 million I/O requests – Drew Khoury Aug 07 '13 at 04:08
0

We have a similar usage profile. This is what we do

  • Build an EBS-backed instance with just the OS/Application stuff.
    • Attach an Instance store to the new Instance and mount it where the ephemeral data is supposed to go.
  • Make an AMI out of it.
  • When spinning up a new instance, select the options to attach the ephemeral store to the correct spot in the file-system.

This allows us to have very small OS/app volumes (10GB) and very large working volumes for the ephemeral stuff we don't care about. The trick here is to be sure to make your instance-launch script maps the instance store to the right block device so the mount-scripts in the OS can handle it.

sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
0

I believe what you want to do is create your own AMI from an existing instance. Here's a brief overview of the steps:

  • I often use Ubuntu's Amazon EC2 AMI Locator to find the AMI I want. e.g. search for instance-store precise us-east to find the instance-store backed releases of 12.04 in us-east-1
  • Launch the instance, login as ubuntu and customize it as needed. Note: you must install java and the Amazon EC2 AMI Tools for the later steps.
  • Upload your Amazon EC2 private key and X.509 certificate to /tmp/cert on the instance. If you don't have them, you'll need to create and upload your signing cert
  • Use ec2-bundle-vol to create an image of the current system
  • Use ec2-upload-bundle to place the image on S3 for registration
  • Use ec2-register to turn the uploaded bundle into an AMI you can launch.

As an alternative, you can use a loopback device and ec2-bundle-image to install an OS from scratch.

dialt0ne
  • 3,027
  • 17
  • 27