2

i have some R scripts, which only can run in sequential way, cannot be broken into chunks or any parallel library for R or any other language cannot be used.

Is there any way i can distribute the Sequential execution of code to multiple cores or may be multiple servers in network? to speed up execution ?

Farhan
  • 4,210
  • 9
  • 47
  • 76
  • You have said it cannot be done, but then you ask if it can be done? You have already answered your own question! – Michael Hampton Oct 28 '18 at 15:20
  • Which R implementation are you using, GNU, Microsoft, something else? – John Mahowald Oct 28 '18 at 15:51
  • @MichaelHampton: i did not said my Script cannot do it. i said it is limitation of script, is there any way that i can run this script without doing parellelization from inside R? – Farhan Oct 28 '18 at 17:21
  • If the script can only run sequentially and cannot be parallelized, then you already have your answer! – Michael Hampton Oct 28 '18 at 17:30
  • i have heard of some container, or some sort of technology, which paralelize execution of one script or program and distribute it to other cpus, cores or even network (cluster) – Farhan Oct 28 '18 at 18:04
  • Can the programs all be run at the same time, or do they have dependencies on the results of previous programs? – John Mahowald Oct 29 '18 at 13:22
  • Only workloads that can run in parallel can be parallelized! You have repeatedly said that your program cannot be parallelized. So I can't understand why you keep asking when you already know the answer! – Michael Hampton Oct 29 '18 at 14:33

1 Answers1

0

So let us assume you can run:

./my_script.R arg1

And you want to run the script on arg1..arg1000. Then you can use GNU Parallel:

parallel ./my_script.R {} ::: arg1 arg2 arg3 .. arg1000

This will start one my_script.R per CPU thread.

If you have multiple servers at your disposal that you can ssh to:

parallel -Sserver1 -Sserver2 ./my_script.R {} ::: arg1 arg2 arg3 .. arg1000

If this does not answer your question, please elaborate on your situation.

Ole Tange
  • 2,836
  • 5
  • 29
  • 45