Running a Process That Needs 5GB RAM

I run a python process that needs roughly 5GB of RAM which takes about 5 minutes every day on my laptop. However, now, I would like to automate the process and run it on a server (to be more dependable). I looked around and found that there are VPSs that could do the job. Since I need the resources only for 5 minutes every day, not really certain whether I should subscribe to an entire VPS. Are there any other means to accomplishing the task? Like AWS Lambda? (when I checked, only about 1GB of RAM can be used with Lambda)

picmate 涅

Posted 2017-08-13T03:55:14.337

Reputation: 113

Lambda maxes out at 1.5GB of RAM. If you can reduce memory requirements it'd be perfect for this use case, and it would probably come within the free tier.

– Tim – 2017-08-13T22:27:41.820

Answers

A VPS is definitely overkill if your utilization rate is low. You might be able to swing a deal with a server operator over at LowEndTalk or use something like AWS EC2's spot instances or on-demand instances (depending on your exact requirements) or Microsoft Azure. You pay by the hour/minute for EC2 and Azure. I haven't spun up an EC2 instance because Microsoft provides my company with a bunch of free credits (obligatory plug for sponsor!) so I can't say how easy it is (it's straight forward to use, though -- I've used EC2 instances before). Having been accustomed to using VirtualBox's provisioning system, I found Azure's provisioning system to be a bit more complicated, but not overly difficult.

A standard Azure VM with 5GB of RAM would costs 28 cents/hour (billed by the minute). So that would set you back about $30/year for your 5 minutes of usage per day, plus a bit extra for the storage (I'm assuming you want persistent storage) and your initial setup. So maybe $40/year. This comes with a Windows license, should you want it.

I'm not familiar with all the gotchas for EC2, but on-demand pricing for 8 GB of RAM and two cores is 9 cents/hour if you don't need Windows (it's a few cents more per hour if you do). If you don't need much storage, I think this is much cheaper. And there's not that much bandwidth I expect you're going to consume in 5 minutes/day. So this probably works out to about $120/year if you need to run your job at 5 minutes per day (+ rounding up from Azure's per-hour billing) (or less than half of that using spot instance pricing, if you're flexible on timing).

lungj

Posted 2017-08-13T03:55:14.337

Reputation: 341

Thanks for the info there. Definitely useful. I am looking at EC2. My website is already on aws lightsail, so, EC2 should work well. – picmate 涅 – 2017-08-13T12:54:54.940

@picmate涅 -- Sorry, I was wrong -- EC2 bills by the hour. Azure is billed by the minute. – lungj – 2017-08-13T23:20:16.420

I think that means they will round up to the nearest hour when doing the billing – picmate 涅 – 2017-08-13T23:54:21.023

AWS EC2 is billed by the hour for the entire time the instance is running. That way, even when the instance is idling, I will get charged; even though I use it for 5 minutes a day. Therefore, I don't think the solutions you gave are quite right. – picmate 涅 – 2017-08-14T17:30:21.820

In Azure, what you do is you create a persistent disk to store the OS and then start/stop a virtual machine that uses the disk; you pay for the storage for the entire time (it's very cheap for a few gigabytes) and only for the on-time for the VM itself. You can also schedule starts and stops automatically https://blogs.msdn.microsoft.com/stevenedouard/2017/01/13/automate-azure-vm-startup-and-shutdown/ . I have a VM on Azure I use for backing up that I only start up to run a sync job and it's cheap to run.

– lungj – 2017-08-14T17:45:25.350

thanks. However, my use case is a little different. Throughout the day I download some content from the web (from a crone job). Each extract takes like 30 seconds. Then at the end of the day run the python process on the content that was downloaded. So, a little tricky to spin up a new instance every time. Also, there's some prep work to do before running anything (eg. loading required python modules etc.) – picmate 涅 – 2017-08-14T18:00:43.157

In that case, @Keltari's suggestion might be the one you want (if you have the space and inclination for maintaining a physical server and you can find one that fits into your budget). Or ask on LowEndTalk (above) to see if anyone can give you a good deal. People over there can and will create custom contracts to meet your needs. Your particular use case sounds like a good candidate if the 5GB of RAM is only needed on occasion. – lungj – 2017-08-14T18:09:04.273

Yeah. Btw, any idea about these people: https://www.ovh.com/us/vps/vps-ssd.xml. I don't mind investing $13/month for a VPS. Can use it for something else too.

– picmate 涅 – 2017-08-14T18:14:13.050

Why not just buy an old cheap PC? I just bought an old PC for $10 and shoved 10TB worth of storage in it and installed linux to be a file server. If all you are running is a python command, no need to break the bank.

Keltari

Posted 2017-08-13T03:55:14.337

Reputation: 57 019

A cheap old PC probably lacks the dependability OP mentions such as resistance to hardware failure, reliability of network/power, and theft. Also, to bring an "old PC" up with 5GB of RAM, provide power, and drops for it, I think per-minute billing is more cost effective. That's before factoring in the cost of replacing a failed component + time, especially if it's a disk failure. – lungj – 2017-08-13T05:29:14.083

@lungj cheap old pcs are no less reliable than any other pc. the things that are likely to fail are the same in all computers. In a case like this, i would use a $10 pc over any other solution. – Keltari – 2017-08-13T06:39:54.357

I dunno... we're comparing a cheap PC to a company that is guaranteeing uptime (three nines at the lowest tiers). Sure, a cheap old PC can last a long time (I recently took two machines offline that were running just fine and had been doing so 24/7 for just over four years; we those computers were manufactured in 2004 and we salvaged them), but some components simply age. Like fans, capacitors, and, of course, flash wear. Even excluding age, they are, statistically, a less reliable than the 2008-vintage $150 server I replaced them with that has 25 disks in a pool, redundant power, and ECC RAM. – lungj – 2017-08-13T06:53:12.973

Also, if you can direct me to a $10 PC shipped to Canada with 5GB (or more) of RAM, reliable or otherwise, that would be awesome. I would love to use it for virtualizing continuous integration testing environments. – lungj – 2017-08-13T07:00:06.427

@lungj goodwill, salvation army, or equivalent. youd be amazed what people give away. – Keltari – 2017-08-13T18:37:34.857