ansible.posix.synchronize fails with unknown rsync option

Question

I am running an ansible playbook with the following task:

- name: Yada Yada Yada
  ansible.posix.synchronize:
    src: ../foo/bar/
    dest: /home/tada/tokio

When i run this using AWX i get the following error (see the original the json format at the bottom of this question):

rsync: on remote machine: --sparse-block=1024: unknown option
rsync error: syntax or usage error (code 1) at main.c(1749) [server=3.2.3]
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]

Information about versions:

• AWX: quay.io/ansible/awx:21.3
• remote target host: Ubuntu 22.04
• rsynch on remote host: version 3.2.3 protocol version 31 (i did also try with Ubuntu 20.04, that was running 3.1.3)
• custom executor (build with ansible-builder)
• rsync in executor: 3.1.3 protocol version 31

When running from my local machine it works, when running from AWX it fails.

The full error as reported by AWX

{
  "rc": 12,
  "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --rsh='/usr/bin/ssh -S none -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' --out-format='<<CHANGED>>%i %n%L' /runner/project/foo/bar/ root@xx.xx.xx.xx:/home/tada/tokio",
  "msg": "Warning: Permanently added 'xx.xx.xx.xx' (ECDSA) to the list of known hosts.\r\nrsync: on remote machine: --sparse-block=1024: unknown option\nrsync error: syntax or usage error (code 1) at main.c(1749) [server=3.2.3]\nrsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]\n",
  "invocation": {
    "module_args": {
      "src": "/runner/project/foo/bar/",
      "dest": "root@xx.xx.xx.xx:/home/tada/tokio",
      "_local_rsync_path": "rsync",
      "_local_rsync_password": null,
      "private_key": null,
      "rsync_path": null,
      "delete": false,
      "_substitute_controller": false,
      "archive": true,
      "checksum": false,
      "compress": true,
      "existing_only": false,
      "dirs": false,
      "copy_links": false,
      "set_remote_user": true,
      "rsync_timeout": 0,
      "rsync_opts": [],
      "ssh_connection_multiplexing": false,
      "partial": false,
      "verify_host": false,
      "delay_updates": true,
      "mode": "push",
      "dest_port": null,
      "recursive": null,
      "links": null,
      "perms": null,
      "times": null,
      "owner": null,
      "group": null,
      "ssh_args": null,
      "link_dest": null
    }
  },
  "_ansible_no_log": false,
  "changed": false
}