How to fetch multiple files from remote machine to local with Ansible

Question

I would like to copy files from remote directory to local directory with Ansible but fetch module allows me to copy only one file. I have many servers from which I need files (same directory each server) and I don't now how to do this with Ansible.

Any ideas?

Duncan Lock · Answer 1 · 2015-11-16T20:49:59.897

41

You should use the synchronise module to do this. This uses the awesome power of rsync. It will copy file & directory structures of any depth, is bulletproof and highly efficient - only copying the actual bytes that have changed:

- name: Fetch stuff from the remote and save to local
  synchronize:  src={{ item }} dest=/tmp/ mode=pull
  with_items:
    - "folder/one"
    - "folder/two"

The key is the mode parameter:

Specify the direction of the synchronization. In push mode the localhost or delegate is the source; In pull mode the remote host in context is the source.

edited Nov 16 '15 at 20:49

answered Aug 28 '15 at 00:42

Duncan Lock

1,762
1
16
18

2

I have found the `synchronise` module to be much more reliable and scalable than the other methods ansible has to copy files. – chicks Sep 30 '15 at 22:27
5

This is definitely a better way to do it than the accepted answer. – childofsoong Feb 02 '16 at 00:40
this module constantly crashes with permission denied (since this is rsync) – TheDESTROS Feb 11 '20 at 08:17
only bad/non bulletproof thing about this is it requires/assumes rsync is installed, when I tried it I got error rsync command not found, so I'll use the other method since I want this to work in internet disconnected environment where I don't control the hosts. – neokyle Dec 20 '20 at 19:14

Kęstutis · Accepted Answer · 2019-04-15T14:49:57.307

29

You will probably need to register remote content and than loop over it, something like this should work:

- shell: (cd /remote; find . -maxdepth 1 -type f) | cut -d'/' -f2
  register: files_to_copy

- fetch: src=/remote/{{ item }} dest=/local/
  with_items: "{{ files_to_copy.stdout_lines }}"

where /remote should be changed with directory path on your remote server and /local/ with directory on your master

edited Apr 15 '19 at 14:49

answered May 11 '15 at 10:13

Kęstutis

505
1
6
16

1

By the way, this goes only one level deep (leaves out subdirectories) and ignores directories in general so if that's not something you desire, just change the shell command acordingly. – Kęstutis May 11 '15 at 10:16
What will happen when I run on a bunch of servers? will each register it's own findings? and fetch the right ones? – Amir Mehler Apr 25 '16 at 13:10
any clue how to do this with win_find? I can't figure out how to extra path from the file list it returns – Peter Kahn Apr 21 '17 at 01:01
Is it also possible to save the fetched file in local in a different name? – vijay v Apr 23 '20 at 14:38

score 5 · Answer 3 · answered Dec 21 '16 at 17:47

i dont have enough rep to comment otherwise i would add it.

I used what Kęstutis posted. i had to make a slight modification

- shell: (cd /remote; find . -maxdepth 1 -type f) | cut -d'/' -f2
  register: files_to_copy

- fetch: src=/remote/{{ item }} dest=/local/
  with_items: "{{ files_to_copy.stdout_lines }}"

The with_items was the area i had to change. it was not able to locate the files otherwise.

Marketta · Answer 4 · 2017-02-21T13:44:21.457

3

Fixing the example above

- hosts: srv-test
  tasks:
    - find: paths="/var/tmp/collect" recurse=no patterns="*.tar"
      register: files_to_copy
    - fetch: src={{ item.path }} dest=/tmp
      with_items: "{{ files_to_copy.files }}"

edited Feb 21 '17 at 13:44

answered Feb 21 '17 at 12:00

Marketta

31
2

Z.Liu · Answer 5 · 2020-06-19T00:54:15.293

well, if you are using latest ansible version, like 2.9.9, I think we need quotes to the item

- name: use find to get the files list which you want to copy/fetch
  find: 
    paths: /etc/
    patterns: ".*passwd$"
    use_regex: True   
  register: file_2_fetch

- name: use fetch to get the files
  fetch:
    src: "{{ item.path }}"
    dest: /tmp/
    flat: yes
  with_items: "{{ file_2_fetch.files }}"

score 1 · Answer 6 · answered Jul 21 '20 at 08:40

i have prepared a playbook which will help to fetch the nmon performance report from server i have used fetch module with reference of shell module output and able to download the nmon data perfectly

---
#NMON PERFORMANCE REPORT FETCH
  - hosts: all
    gather_facts: yes
    become: yes
    vars_prompt:
      - name: "date_input"
        prompt: "Enter date to fetch nmon report (YYMMDD) = "
        private: no

    tasks:
      - pause:
          prompt: "\n\n Enter date to fetch nmon report?\n\n--------------------------------------\n\n1. Fetch nmon:\n2. Exist from Playbook:\n\nPlease select Action: \n--------------------------------------\n"
        register: menu

      - set_fact:
          option: "{{ menu.user_input }}"

      
      - name: Find-out nmon report for given date  {{ inventory_hostname }}
        shell: |
          ls -l /var/nmon/data/* | grep -i {{ date_input }} |awk '{print $NF}'
        register: nmon_files_to_copy
        when: option == "1"

      - name: 'Fetch the NMON Report {{ date_input }} output from server {{ inventory_hostname }}'
        fetch:
          src: "{{ item }}"
          dest: /tmp/
          flat: yes
        with_items: "{{ nmon_files_to_copy.stdout_lines }}"
        when: option == "1"

score 0 · Answer 7 · edited Dec 17 '21 at 08:36

0

Try this -

- shell: (cd /remote; find . -maxdepth 1 -type f) | cut -d'/' -f2
  register: files_to_copy

- fetch: src=/remote/{{ item }} dest=/local/ flat=yes
  with_items: "{{ files_to_copy.stdout_lines }}"

edited Dec 17 '21 at 08:36

Gerald Schneider

19,757
8
52
79

answered Dec 16 '21 at 05:36

Sagar bobade

1

1

Could you add an explanation why this is better than the answers already given? – Gerald Schneider Dec 17 '21 at 08:37
I observed that when i use [this](https://serverfault.com/a/1087454/946032) code "without" **flat=yes** then targeted files got copied from whole path. So i suggested to add **flat=yes** – Sagar bobade Dec 17 '21 at 10:14
@Sagarbobade Welcome to Server Fault! Please [edit your answer](https://serverfault.com/posts/1087454/edit) to include the additional information. – Paul Dec 17 '21 at 13:40

score 0 · Answer 8 · answered Jan 13 '22 at 07:36

you can compress your folder and fetch that. {{inventory_hostname}} good file name for 'tgz' file in your ansible server

- name: Compress folder
  community.general.archive:
    path: "/folder/dir"
    dest: "/dst/{{inventory_hostname}}.tgz"
- name: Fetch the file from server
  fetch: 
    src: "/dst/{{inventory_hostname}}.tgz"
    dest: ./backup/ 
    flat: yes

score 0 · Answer 9 · edited May 26 '16 at 02:15

0

- hosts: srv-test
  tasks:
    - find: paths="/var/tmp/collect" recurse=no patterns="*.tar"
      register: file_to_copy
    - fetch: src={{ item }} dest=/tmp
      with_items: files_to_copy.stdout_lines

edited May 26 '16 at 02:15

womble

95,029
29
173
228

answered May 26 '16 at 01:39

Diego Roberto Dos Santos

117
3

This doesn't work at all. Has this been tried? – Basil A Jul 08 '19 at 17:16

score 0 · Answer 10 · answered Jun 22 '19 at 07:17

I use this: 1. Pull directories from remote host to specific hosts

- name: Gather hosts stats from other hosts
  shell: " scp -r {{results_root_dir_src}} root@{{groups['profiling_server'][0]}}:{{results_root_dir_dest}}/abc/"
  when: "'profiling_server' not in group_names"
#It will not run on the node where the directories need to be copied.

Pull directories from node to localhost

- name: Gather from host to local
  delegate_to: 127.0.0.1
  run_once: true
  become: false
  shell: "scp -r root@{{groups['profiling_server'][0]}}:{{results_root_dir}} ./results_local_location "

inventory

[nodes]
server1
server2
server3
[profiling_server]
server1

How to fetch multiple files from remote machine to local with Ansible

10 Answers10