rsync command works in terminal, fails in rundeck

Question

I'm trying to use rundeck to rsync files in a docker container volume to an offsite server. I can impersonate the rundeck user and run the rsync command in a terminal window just fine.

When rundeck executes the same command I will get an Unexpected Remote Arg error or an Unexpected Local Arg error depending on where I place the --rsync-path option in the command. If I leave out the --rsync-path I get a permission denied error because the files are in a docker volume and require root access.

Note: Commands and output have been redacted but are otherwise accurate

rsync command in terminal on rundeck server as user rundeck:

rsync -avP rundeck@remotehost:/var/lib/docker/volumes/dockervolumename/_data /srv/destination/backup_datetime --rsync-path="sudo rsync"

command output:

receiving incremental file list
created directory /srv/destination/backup_datetime
_data/
_data/storage/
_data/storage/upload/
_data/storage/uploads/
_data/storage/uploads/admin/
_data/storage/uploads/group/

sent 48 bytes  received 203 bytes  100.40 bytes/sec
total size is 0  speedup is 0.00

rundeck debug output of same command (Local Command):

13:56:50    [workflow] Begin step: 1,NodeDispatch
13:56:50    1: Workflow step executing: StepExecutionItem{type='NodeDispatch', keepgoingOnSuccess=false, hasFailureHandler=false}
13:56:50    preparing for sequential execution on 1 nodes
13:56:50    Executing command on node: localhost, NodeEntryImpl{tags=[], attributes={nodename=localhost, hostname=***, osVersion=8, osFamily=linux, osArch=x64, description=Offsite backup server @ ***, osName=CentOS}, project='null'}
13:56:50    [workflow] beginExecuteNodeStep(localhost): NodeDispatch: StepExecutionItem{type='NodeDispatch', keepgoingOnSuccess=false, hasFailureHandler=false}
13:56:50    LocalExecNodeStepPlugin, running command (6): 'rsync''-avP''rundeck@remotehost:/var/lib/docker/volumes/dockervolumename/_data''/srv/destination/backup_datetime''--rsync-path="sudo''rsync"'
13:56:50    Unexpected local arg: /srv/destination/backup_datetime
13:56:50    If arg is a remote file/dir, prefix it with a colon (:).
13:56:50    rsync error: syntax or usage error (code 1) at main.c(1368) [Receiver=3.1.3]
13:56:50    Failed: NonZeroResultCode: Result code was 1
13:56:50    [workflow] finishExecuteNodeStep(localhost): NodeDispatch: NonZeroResultCode: Result code was 1
13:56:50    1: Workflow step finished, result: Dispatch failed on 1 nodes: [localhost: NonZeroResultCode: Result code was 1 + {dataContext=MultiDataContextImpl(map={}, base=null)} ]
13:56:50    [workflow] Finish step: 1,NodeDispatch

I've verified that rundeck can interact with the destination server via ssh keys and run commands there just fine.

Is there some kind of interaction between rsync and rundeck that I'm missing?

I'm new to rundeck and rsync. From what I can tell the errors I'm getting usually occur because of a syntax error in the command. The original command runs fine in terminal so that leads me to believe that rundeck is somehow garbling it.

Source server is a VM running CentOS 7 in an OpenStack environment.

Destination server is a VM running CentOS 8 on an external network.

Answer 1

This may not be a perfect solution to this issue, but it appears to work while leaving SELinux in enforcing mode.

semanage permissive -a rsync_t

Essentially let rsync run under permissive settings while leaving the rest of SELinux policies in place.

Perhaps someone smarter can suggest a better answer.

Answer 2

It would be better to create a custom SELinux module for rsync.

User Edward said it beautifully over on the Unix and Linux channel in This post. See below for instructions on how to create your SELinux module.

This is done by letting the rsync daemon do it's work in permissive mode, capturing AVC denials as it goes along, and then converting the AVC denials to a policy, like so:

# put SELinux in permissive mode
setenforce 0

# --- do your rsync stuff ---

# get related AVC denials
# I'm using 'recent' here, depending on the rsync run time please adjust > accordingly
ausearch -m avc -ts recent --subject rsync_t

# go through the output. If you're satisfied, create the module
ausearch -m avc -ts recent --subject rsync_t | audit2allow -m roaima-rsync- custom-1 > roaima-rsync-custom-1.te
checkmodule -M -m -o roaima-rsync-custom-1.mod roaima-rsync-custom-1.te
semodule_package -o roaima-rsync-custom-1.pp -m roaima-rsync-custom-1.mod

# load the policy module
semodule -i roaima-rsync-custom-1.pp

# disable permissive mode
setenforce 1

# --- do your rsync stuff again --

Catching AVC denials that aren't audited ("dontaudit")

If for some reason, the "edge case" files still aren't accessible and the ausearch command doesn't yield results, you may be hitting a "dontaudit" rule.

To rebuild the SELinux policy ignoring all "dontaudit"-rules, run semodule -DB. The -D option disables "dontaudit" rules; the -B option rebuilds the policy.

Then try if you can trigger audit log events. If so, capture them like I showed above, create the SELinux module, and then re-enable "dontaudit" rules by running: semodule -B.

For a full list of "dontaudit" rules, run the sesearch --dontaudit command. Narrow down searches using the -s domain option and the grep command. For example: sesearch --dontaudit -s rsync_t.

I recommend heading over to his post and giving him credit for this.