2

If we update our systems with SaltStack this takes about 40 seconds.

During these 40 seconds the state of the system is not consistent.

If there are cron jobs which start during this period, it is likely that they produce strange errors.

Of course we could change our setup completely and use containers. This would make atomar updates possible. But this would need a lot of work which can't be handled today.

Is there a way to disable cron jobs on the minion if salt gets executed?

guettli
  • 3,113
  • 14
  • 59
  • 110

2 Answers2

1

To completely disable cron jobs, you can simply turn off the cron daemon (it's usually crond in CentOS and cron in Ubuntu server).
You can stop and start it directly in the salt run.

If you need to disable cron run just for some specific cron jobs, you can just create a script (let's call it run-crons) and use it to disable just specific cron runs. For example you can create a cron entry similar to:

* * * * * root run-crons && echo "this is my actual cron job"

with the run-crons script similar to

#!/bin/bash
set -e

[ -f /etc/disabled/crons ] && exit 10

exit 0

so if the /etc/disabled/crons exists, those specific cron lines preceded with run-crons won't run. You can create/remove the disable file as well during the salt run

To stop cron daemon when salt run, just include a state with something like:

stop_cron:
  service.dead:
    - name: cron
    - order: 1

start_cron:
  service.running:
    - name: cron
    - order: last

You may need to adjust the name of the cron daemon depending on your linux distribution

For the run-crons way, include a .sls with something like:

disable_cron:
  file.managed:
    - name: /etc/disabled/cron
    - replace: false
    - order: 1

enable_cron:
  file.absent:
    - name: /etc/disabled/cron
    - order: last
ProT-0-TypE
  • 491
  • 4
  • 8
0

There are actually two ways you can do this.

Modify the script method

Locate and modify the script that runs the SaltStack update. It could be something like bootstrap-salt.sh, but it could be anything. Try looking for it in your process list while the update is running (ps -ef |grep -i salt). it should be something with .sh at the end. Once you have located the script, you can insert at the top of the script a line that disables cron. This is the safest method.

This can be either of the following: 

/etc/init.d/crond stop

or 

crontab -l -u root > /root/cron.bak  
crontab -r -u root

Note: The later only works on the user you list and is dependent how you are managing cron. For example, this method will not disable any jobs listed in /etc/cron.daily/ cron.hourly/ cron.monthly/ or cron.weekly/. If you decide to use this method, it would be best to actually identify the process you do not want running at the same time as your SaltStack update and disable the cron which runs that process specifically.

Then at the end of the script one of the following: 

/etc/init.d/crond start

or 

crontab -u root /root/cron.bak

Run a daemon method

Write a script that runs as a daemon to check for a running process, but this is less reliable and would have a chance to miss during the time gap while the script loops. The only way to mitigate it would be to just loop it without a sleep, but again, that's not really advised. I would use this method only as a last resort if you are unable to locate or modify the script that launches the update for SaltStack.

The daemon script would look something like this: 

#!/bin/bash
while true
do
   # this might have to be tweaked based on the process you are looking for
   SaltUpdateRunning=$(ps -ef |grep -v grep |grep -i salt |grep -i update | wc -l)
   if [ $SaltUpdateRunning -gt 0 ]
   do
      /etc/init.d/crond stop
      sleep 3600 # 1 hour just to be safe - you can modify to 
                 # match the runtime of SaltStack update
      /etc/init.d/crond start
   done
   sleep 1 # this is for safety but you can remove and it will loop 
           # much faster with less chance to miss the running process
done

You should also look into SaltStack's ability to manage cron. I am not sure if there is a way for it to automatically disable cron or certain processes while the update is running, but it wouldn't hurt to dig a little deeper into it's abilities.

Bill
  • 126
  • 12