How can you make a salt execution module return a nonzero exit code on failure

Question

I'm trying to use Jenkins to run a Salt execution module command; if any minion fails to execute the command, I want the Jenkins job to fail. Jenkins just follows the general shell scripting practice of failing on a nonzero exit code, so to make it work Salt should too.

And that is where I'm stuck, running something like this works as expected:

root@salt-master:~# salt --batch-size 1 --failhard -G 'ec2_roles:stage' cmd.run 'exit 0'

Executing run on ['stage-12']

jid:
    20170209212325270060
retcode:
    0
stage-12:

Executing run on ['stage-13']

jid:
    20170209212325423735
retcode:
    0
stage-13:

Executing run on ['stage-197']

jid:
    20170209212325590982
retcode:
    0
stage-197:
root@salt-master:~# echo $?
0
root@salt-master:~# salt --batch-size 1 --failhard -G 'ec2_roles:stage' cmd.run 'exit 1'

Executing run on ['stage-12']

{'stage-12': {'jid': '20170209212334018054', 'retcode': 1, 'ret': ''}}
ERROR: Minions returned with non-zero exit code.
root@salt-master:~# echo $?
1

But when I try to run an execution module like the following test:

# mymodule.py
from salt.exceptions import CommandExecutionError

def testfailure():
    raise CommandExecutionError('fail!')

I get the following result:

root@salt-master:~# salt --batch-size 1 --failhard -G 'ec2_roles:stage' mymodule.testfailure

Executing run on ['stage-12']

jid:
    20170210023059009796
stage-12:
    ERROR: fail!

Executing run on ['stage-13']

jid:
    20170210023059179183
stage-13:
    ERROR: fail!

Executing run on ['stage-197']

jid:
    20170210023059426845
stage-197:
    ERROR: fail!
root@salt-master:~# echo $?
0

score 2 · Answer 1 · answered Feb 10 '17 at 12:42

I'm not sure how you handle errors in your module but anyway, I would like to shed some light on it.

There is available a dunder dictionary __context__. When you run an execution module the __context__ dictionary persists across all module executions until the modules are refreshed. State modules behaves similarly. The dictionary can ave a key 'retcode' which seems to refer to the return code salt minion/client should return and which you are missing.

I can see it used in some execution modules. One example from nspawn module:

def _make_container_root(name):
    '''
    Make the container root directory
    '''
    path = _root(name)
    if os.path.exists(path):
        __context__['retcode'] = salt.defaults.exitcodes.SALT_BUILD_FAIL
        raise CommandExecutionError(
            'Container {0} already exists'.format(name)
        )
    else:
        try:
            os.makedirs(path)
            return path
        except OSError as exc:
            raise CommandExecutionError(
                'Unable to make container root directory {0}: {1}'
                .format(name, exc)

Now, the bad things. I tested it on old SaltStack 2015.8.12 and it somehow works but without using exception:

def testfailure():
  __context__['retcode'] = 1

Executing the module returns error code higher than 0:

 salt my_minion mymodule.testfailure; echo $?
my_minion:
    None
ERROR: Minions returned with non-zero exit code
11

When you raise an exception it stops working and it always returns 0.

# mymodule.py
from salt.exceptions import CommandExecutionError

def testfailure():
  __context__['retcode'] = 1

  raise CommandExecutionError('fail')

Executing the module returns error code equal to 0 although it shouldn't:

salt my_minion mymodule.testfailure; echo $?
my_minion:
    ERROR: fail!
0

I also tested it on the latest available release 2016.11.3 and the behavior is the same. IMO, this is a bug. I reported it here.

Thanks, it looks like the main issue tracking this in GitHub is: https://github.com/saltstack/salt/issues/18510 -- I guess I can stop raising and use the `__context__` trick. — c4urself, Feb 10 '17 at 19:00

score 1 · Answer 2 · edited May 23 '17 at 12:41

AFAIK exit codes is a general issue with Salt. There's set of tickets in their Github bug tracker regarding this issue. The best way to find whether salt states were applied successfully or not I've seen is the one used by salt-kitchen. In a nutshell there's just a simple wrapper around salt command that greps output for the specific messages. The grep command is following:

grep -e Result.*False -e Data.failed.to.compile -e No.matching.sls.found.for

In your case you can also add match on string ERROR:. You also probably need to invert grep's exit code as it is 0 when the match is found. You could do this with a simple trick explained in this question. So in the end your salt command may look like:

salt <your options go here> | tee grep -q -e Result.*False -e Data.failed.to.compile -e No.matching.sls.found.for -e ERROR: ; test $? -eq 1

This will show the full output of salt, suppress grep output and return the inverted return code of grep meaning 1 if any of the error messages are found and 0 if not.

score 0 · Answer 3 · answered Jul 25 '17 at 21:02

0

For anyone still trying to solve this, you can do the following:

salt * state.highstate --retcode-passthrough

or

salt-call * state.highstate --retcode-passthrough

answered Jul 25 '17 at 21:02

kbuilds

151
1
6

How can you make a salt execution module return a nonzero exit code on failure

3 Answers3