8

We have longish builds that typically we schedule our cron jobs around, but on occasion we have to re-run a build during a non-standard timeframe and can run into conflicts with cron jobs that are typically safe to run at those times.

We have multiple accounts that run both builds and cron jobs, so we can't suspend the crontab service for the entire machine and then restart it later.

I was wondering if anyone had a pattern or implementation. I imagine this working like

User creates a file: ~/block-crontab
user runs build The cron job looks for that file in the home dir of the user and if it's there is just skips all cron jobs. Otherwise it runs the jobs Then when build is done, user removes ~/block-crontab

Would that work? I'm guessing I'd need to modify the cron script somehow. I'm mostly wondering if there is a better/standard approach to this issue?

thanks.

Sean
  • 143
  • 5
  • What do you mean by `[the build] can run into conflicts with from jobs that are tipically safe to run at those times`? Are there non-build jobs that cannot run during the build? Are all the jobs mutually exclusive to each other? Or just regarding the build? – GnP Nov 27 '17 at 20:25
  • 1
    Have you looked intro `flock` or `run-one` (Debian/Ubuntu)? https://serverfault.com/questions/82857/prevent-duplicate-cron-jobs-running – Stefan Lasiewski Nov 27 '17 at 20:33
  • For example we run a big db update every AM. Then every hour through the pm we refresh the front page with news or random items. If run during the db update, the front page might have some missing elements. – Sean Nov 28 '17 at 16:06
  • Have not looked at flock or run-one. Thanks. – Sean Nov 28 '17 at 16:06

2 Answers2

10

Rather than messing with crond, I strongly suggest to implement some (even simple) form of locking inside your build scripts. For example, touch and check for a file in /var/run/: if your script find something, than another process is building the project. You obviously need to remove the lockfile when finished.

As @GnP noted in the comments, you can also use the flock utility for semi-automatically manage your lockfiles.

If you don't/can't rely on any locking mechanism, simple issue a service crond stop to shutdown the crond system.

shodanshok
  • 44,038
  • 6
  • 98
  • 162
  • 2
    The `flock` command would be a nice addition to this answer. It handles the lock file and all the little details there are to it. – GnP Nov 27 '17 at 17:45
  • @GnP Excellent suggestion! I've updated my answer accordingly – shodanshok Nov 27 '17 at 20:14
  • 1
    Just be aware that with `flock` the file descriptor is inherited with child processes unless you make some move to close it. That can sometimes cause unexpected behaviours especially if people start 'background jobs' that get executed in cron. – Matthew Ife Nov 28 '17 at 21:06
1

I tend to just wrap all long running commands in a screen and get cron to start the screen only if there is not one running already.

So the following line in crontab

*/2 * * * *  /bin/bash /path/to/LongRunningScript.bash

... turns into something like this:

*/2 * * * *  /usr/bin/screen -S MyUniqueName -Q select . || /usr/bin/screen -dmS MyUniqueName /bin/bash /path/to/LongRunningScript.bash

I like this because it also gives you the chance to attach to a running script and check its output / status.

In your scenario, you could get cron to check for another screen before executing a build, for example

0 3 * * *  /usr/bin/screen -S ManualBuild -Q select . || /usr/bin/screen -dmS AutomatedBuild /bin/bash /path/to/BuildScripts.bash
10 3 * * *  /usr/bin/screen -S ManualBuild -Q select . || /usr/bin/screen -dmS OtherAutomatedBuild /bin/bash /path/to/OtherBuildScripts.bash

When you run a manual build, just hop into a screen first before you run the script (please comment if you need hints how to connect / disconnect from screen. It is a useful utility - give it a shot if you have not started using it yet)

Type screen -S ManualBuild, hit [enter] and run whatever commands you want to run.

Note: If you use the example as supplied you might confuse cron if you have more than 1 screen session with a name of "ManualBuild" running.

trs
  • 111
  • 3
  • Sorry, just saw your "across multiple users" note -- this won't work across users without modification. Need to check if screen somehow supports connecting to other user's sessions. – trs Dec 04 '17 at 03:03
  • that's a nice idea. I'd have to break the devs the habit of leaving their term windows open for months at a time. :) – Sean Dec 04 '17 at 17:56
  • If they leave their session open you can attach to an attached screen with `screen -x ScreenName` and depending on your distro (suid settings for `screen`) you can make a screen session shareable to other users. Cleanest way would be to run these build commands under a specific user name I guess, same user that owns the cron job. – trs Dec 05 '17 at 00:25