-2

I bough droplet on DigitalOcean 1GB and 1CPU. I'm running cronjob there. Currently I'm able to run 30 background tasks(processes) at once, once I go up server cannot handle that much.

Those background processes are sending requests to an endpoint, collect a data to array and then send it to another endpoint. Each task process 100 data.

Can you help me choose the right server. I'd like to run at least 100 tasks(processes) in the background at once. Should I buy more RAM or CPU?

Here is current usage, I don't see that server is overloaded with 40 tasks but it can't handle it: enter image description here

What am I doing wrong?

Misko Mali
  • 99
  • 2

1 Answers1

1

Neither one directly correlates to the amount of processes you can run at once, but I would go for more RAM.

Also, if you spread out the timing each process occurs, you can lower the peak load. So if you run each process once every 2 minutes, instead of all 100 constantly- you may not need an upgrade at all.

Exosylver
  • 26
  • 2
  • Great tip! What if I run each process once every 2 minutes, they will probably all run at the same time because it takes 15 minutes for each to finish. Isn't that the same? – Misko Mali Jun 08 '20 at 14:55
  • 1
    @MiskoMali I was under the assumption the script ran faster(a few minutes). But, what I said still applies. There are a few ways to space it out: 1) create 3 subgroups of 33 processes that all get run 15 minutes apart.(Only 33 processes will be run at one point, and every process will refresh every 45 minutes) 2) Start each process 30 seconds apart, one by one, so then by the time process #30 starts, #1 ended.(Only 30 processes on at once, refreshes every 50 minutes.)[Of course you can tweak the start time spacings to make it more condensed or more spread out.] – Exosylver Jun 08 '20 at 15:06
  • Awesome, that really helped me a lot. Thank you very much – Misko Mali Jun 08 '20 at 16:21