1

I have a grid cluster. (It's running SGE, but I don't think that's relevant.) All the machines are intended to be able to drop out and come back at any time without any significant issue.

However, my users need the ability to run cron jobs. Right now, they're just picking a server at random and plopping a cron job on it. I could assign a specific server, either in the grid or external, for them to run their jobs on, but that just seems wrong.

Does anyone have a solution where crontabs are stored centrally, but run (potentially) elsewhere? Or am I just overthinking this?

wfaulk
  • 6,828
  • 7
  • 45
  • 75

1 Answers1

1

Why don't you have the users set up their cron job on your submission server, and have it use qsub or qrsh to actually execute the task they need to run on the cluster? Isn't that the whole point of having a grid / cluster in the first place?

Kamil Kisiel
  • 11,946
  • 7
  • 46
  • 68
  • It's not that the cron jobs need to be clustered. More often than not, they're very simple jobs. It's just an issue of keeping track of where the cron jobs are located so that if I decide to retire or upgrade a particular machine, I don't have to worry about deleting someone's cron job. Like I said, though: I may be overthinking it. – wfaulk Feb 04 '10 at 07:31
  • That's exactly why you should put the crontabs on the submission host. You never have to worry about retiring the wrong node. Each night / iteration the cron job will just run on some host as decided by the scheduler. Doesn't matter if they're simple jobs, the scheduler already solves this problem. – Kamil Kisiel Feb 04 '10 at 17:19
  • Do you mean the qmaster? If so, I might as well just run the cron jobs directly on the qmaster, and that's back to simply assigning a specific server. To be clear, the jobs have no direct relationship to the cluster; it's just that every machine I have is part of the cluster. Or is there some cron feature built into SGE that I don't know about? – wfaulk Feb 04 '10 at 21:55
  • There's no cron feature specific to SGE. However, in your problem you stated: "Does anyone have a solution where crontabs are stored centrally, but run (potentially) elsewhere? Or am I just overthinking this?". In this solution the cron jobs would all be stored on the qmaster host, and run anywhere on the cluster (you don't even have to care where). You can remove any machine at any time without affecting the next run of the cron jobs. If you ever need to replace the qmaster, just move all the crontabs over to the new one. – Kamil Kisiel Feb 05 '10 at 00:35
  • Yeah, I guess by "centrally", I meant "in a cloud". My first thought was as an NIS map, or a shared config file. I *am* just overthinking it. – wfaulk Feb 05 '10 at 18:35