11

Disclaimer: I am a front end dev in a server kind of world.

Hi guys. I have a server instance on EC2, the c5d.9xlarge, whose specs are:

  • System: Ubuntu 18.04
  • vCPU: 36 threads/cores
  • Memory: 72 GB
  • Storage: 900 GB NVMe SSD
  • Dedicated EBS Bandwidth: 7,000 Mbps
  • Network Performance: 10 Gbps

Scenario: I use this server to upload large videos (4K, 1+ hr) and process them using FFMPEG but compared to my previous UpCloud server with 12 cores and 48 GB of RAM, this EC2 server is taking 1.5 times longer to upload and process, which I think should not be the case.

Question: How do I max out the performance for what I'm paying for?

Martavis P.
  • 213
  • 1
  • 7
  • You should look at the cost of the [AWS Elastic Transcoder](https://aws.amazon.com/elastictranscoder/) to see if it works out better for you. – Tim Feb 05 '19 at 06:43
  • Thanks @Tim. I saw it a while ago but I actually have a couple more calls on this server so I figured I'd combine them. – Martavis P. Feb 05 '19 at 07:11

2 Answers2

19

Your c5d.9xlarge comes with 900GB instance storage (aka ephemeral storage) - are you using that for storing and processing the files? While your instance has some dedicated EBS bandwidth the on-instance SSD storage will still be much faster. I suggest you use that for all source and temporary files and only store the results to EBS.

There are some caveats with instance storage though:

  1. You have to format and mount it before it can be used. Refer to this answer for more details: Automatically mount SSD instance storage on AWS EC2 in Ubuntu 16.04

  2. The contents is wiped when you stop and re-start the instance. It survives reboot but not stop/start.

Update: By default the SSD isn't mounted - you'll have to follow the steps in the linked answer above to make use of it. In the standard config after boot you'll be using the EBS which is slower than SSD.

Then you will have to make sure that you're actually using it - set the upload, working and temporary directory to that SSD mount point.

Or even better - since you've got 72 GB of RAM - create a RAM disk and and use that for temporary files. That will be even faster than SSD (if the files fit).

First of all disregard the upload speed and optimise the processing - upload the file to EBS and time the processing, then upload to SSD and time it and then to RAM disk and time the processing. See how much they differ.

Upload speed will be affected by many aspects, including the distance and latency between you and the AWS region you're uploading to. Are you using an AWS region close to you?

Re CPU performance - you've got 36 CPUs available, however each single core may be slower than the cores in your previous 12-core machine. It depends on the CPU architecture and clock speed. However if you can parallelise the video processing into 36 threads you should be better off on this instance. If you rely on a single thread you may not get the performance you want.

Hope that helps :)

MLu
  • 23,798
  • 5
  • 54
  • 81
  • Yes, the hard drive is used for storing while the files are processing and the files are temp until the process finishes. Resulting files are sent to S3. I actually don't know if the SSD is mounted either. I'm starting to realize that AWS doesn't use the specs as a default setting. Does mounting an SSD make the upload faster? Follow up: any ideas how I can max out the allowed CPU spec as well? – Martavis P. Feb 05 '19 at 03:02
  • @MartavisP. added some updates to the answer. – MLu Feb 05 '19 at 03:33
  • Wow, thank you so much for the information! I will study and apply what you've written. – Martavis P. Feb 05 '19 at 04:06
  • 1
    Great answer. You need to benchmark to find the bottleneck, break it down to the upload time, the processing time, CPU usage, EBS / network usage, etc. It could be as simple as you have more latency to this server so the upload is slower, which you can work around by uploading data in parallel threads to EC2 / S3. Netflix has [this video](https://www.youtube.com/watch?v=89fYOo1V2pA) on youtube which might be interesting, but do your diagnosis steps before you bother with that kind of detail. – Tim Feb 05 '19 at 08:11
1

Is there any scope to look at alternatives? For the cost of a c5d.9xlarge, even if you're getting a substantial discount, in the dedicated server market you could have several equivalent or better machines

Using cloud for this kind of vertical-scaled problem is a recipe for overpaying and, as you've seen, poor performance

Sorry for the kind of non-answer, but I don't have enough rep to comment

Dark
  • 232
  • 1
  • 10
  • This was a discount for me, switching from UpCloud. So what do you recommend? – Martavis P. Feb 05 '19 at 18:45
  • 1
    It seems that the same compute in the cloud is always roughly 4x the price of a monthly rented server at some random web hoster. Then, there are enterprise discounts (~30%) and you can book for 3 years (2x cheaper but now much worse than that random web hoster). And that monthly rented server usually seems to be 1/12th the price of outright buying the hardware. That's what I see at least. So in the cloud you buy the hardware every 3 month but you don't own it. – usr Feb 05 '19 at 20:07
  • Yeah that's a good point, but you have to remember you're paying for the maintenance and convenience, like anything in life. I'd rather let AWS be my mechanic than spend days trying to figure out how to fix a server outage. It's good knowledge to have but ultimately my career is about the code so the cloud works for me. – Martavis P. Feb 06 '19 at 00:04