2

I am new to the whole distributed computing / cloud thing. But I had an idea at work for our multimedia stuff like movie encoding / cpu intensive things tasks (which sometimes take a few hours).

Is there a 'free' (linux?) way to go about using a Windows machine, and offsetting those cpu cycles for that task to say 10 servers that are generally idle (cpu wise)?

I'm just curious if there is a way to do this or am I just grasping at straws here. My thought is that a 'cloud' setup would achieve this, however like I stated initially, I am a total newbie when it comes to it.

This is just an idea, looking for some thoughts? Anyone achieve this?

Jakub
  • 380
  • 1
  • 11

4 Answers4

3

Corporately we use grid computing for things similar to this. A great one to investigate is Condor.

Keith Pimmel
  • 46
  • 2
  • Also check out Sun/Oracle Grid Engine (similar product), available as open source too at http://gridengine.sunsource.net/ – Dominic Cleal May 04 '10 at 15:03
2

are those 'cpu intensive things'...

  • developed in-house? then you'll have to split and distribute the job yourself. there are some nice libraries, but everything is very low level.
  • prebuilt software? then talk to the developer/provider. either it's supported or it's not
  • same process applied to lots of individual items (for example, processing thousands of images)? then you might be able to set a simple system out of a DB table holding the subtask status + some batch scripts that: pick a task, mark as doing, do it, return the result, mark the task as done, repeat until all tasks are done.
Javier
  • 9,078
  • 2
  • 23
  • 24
1

I think that if you are talking about render farms, there are some examples of such projects.

sybreon
  • 7,357
  • 1
  • 19
  • 19
  • Not just render farms, but more like for large cpu intensive cycles, somehow 'funneling' that work through a server farm, but one that is not dedicated just for rendering. Not sure if I make sense... :P – Jakub Apr 30 '10 at 17:12
  • sure it makes sense. you're thinking in terms of mosix and beowulf clusters! but i think that application supported distribution is better than a generic solution. – sybreon May 01 '10 at 15:06
1

Ubuntu with Sun Grid Engine would work and is available in the repos. I think everyone's ideas are going to require you to wipe the host servers though, not sure if that's what you're looking for.

erimar77
  • 488
  • 2
  • 8