Appending to lists or adding keys to dictionaries in Ansible

Question

(Related to Callbacks or hooks, and reusable series of tasks, in Ansible roles):

Is there any better way to append to a list or add a key to a dictionary in Ansible than (ab)using a jina2 template expression?

I know you can do something like:

- name: this is a hack
  shell: echo "{% originalvar.append('x') %}New value of originalvar is {{originalvar}}"

but is there really no sort of meta task or helper to do this?

It feels fragile, seems to be undocumented, and relies on lots of assumptions about how variables work in Ansible.

My use case is multiple roles (database server extensions) that each need to supply some configuration to a base role (the database server). It's not as simple as appending a line to the db server config file; each change applies to the same line, e.g. the extensions bdr and pg_stat_statements must both appear on a target line:

shared_preload_libaries = 'bdr, pg_stat_statements'

Is the Ansible way to do this to just process the config file multiple times (once per extension) with a regexp that extracts the current value, parses it, and then rewrites it? If so, how do you make that idempotent across multiple runs?

What if the config is harder than this to parse and it's not as simple as appending another comma-separated value? Think XML config files.

Answer 1

Since Ansible v2.x you can do these:

# use case I: appending to LIST variable:
- name: my appender
  set_fact:
    my_list_var: '{{my_list_myvar + new_items_list}}'

# use case II: appending to LIST variable one by one:
- name: my appender
  set_fact:
    my_list_var: '{{my_list_var + [item]}}'
  with_items: '{{my_new_items|list}}'

# use case III: appending more keys DICT variable in a "batch":
- name: my appender
  set_fact:
    my_dict_var: '{{my_dict_var|combine(my_new_keys_in_a_dict)}}'

# use case IV: appending keys DICT variable one by one from tuples
- name: setup list of tuples (for 2.4.x and up
  set_fact:
    lot: >
      [('key1', 'value1',), ('key2', 'value2',), ..., ('keyN', 'valueN',)],
- name: my appender
  set_fact:
    my_dict_var: '{{my_dict_var|combine({item[0]: item[1]})}}'
  with_items: '{{lot}}'

# use case V: appending keys DICT variable one by one from list of dicts (thanks to @ssc)
- name: add new key / value pairs to dict
  set_fact:
    my_dict_var: "{{ my_dict_var | combine({item.key: item.value}) }}"
  with_items:
  - { key: 'key01', value: 'value 01' }
  - { key: 'key02', value: 'value 03' }
  - { key: 'key03', value: 'value 04' }

all the above is documented in: https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#combining-hashes-dictionaries

Answer 2

You can merge two lists in a variable with +. Say you have a group_vars file with this content:

---
# group_vars/all
pgsql_extensions:
  - ext1
  - ext2
  - ext3

And it's used in a template pgsql.conf.j2 like:

# {{ ansible_managed }}
pgsql_extensions={% for item in pgsql_extensions %}{{ item }}, {% endfor %}

You can then append extensions to the testing database servers like this:

---
# group_vars/testing_db
append_exts:
  - ext4
  - ext5
pgsql_extensions: "{{ pgsql_extensions + append_exts }}"

When the role is run in any of the testing servers, the aditional extensions will be added.

I'm not sure this works for dictionaries as well, and also be careful with spaces and leaving a dangling comma at the end of the line.

Answer 3

you need to split the loop into 2

--- 
- hosts: localhost
  tasks: 
    - include_vars: stacks
    - set_facts: roles={{stacks.Roles | split(' ')}}
    - include: addhost.yml
      with_items: "{{roles}}"

and addhost.yml

- set_facts: groupname={{item}}
- set_facts: ips={{stacks[item]|split(' ')}}
- local_action: add_host hostname={{item}} groupname={{groupname}}
  with_items: {{ips}}

Answer 4

Almost all answers here require changes in tasks, but I needed to dynamically merge dictionaries in vars definition, not during run.

E.g. I want to define some shared vars in all group_vars and then I want to extend them in some other group or host_vars. Very useful when working for roles.

If you try to use the combine or union filters overwriting the original variable in var files, you will end in infinite loop during templating, so I created this workaround (it's not solution).

You can define multiple variables based on some name pattern and then automatically load them in role.

group_vars/all.yml

dictionary_of_bla:
  - name: blabla
    value1 : blabla
    value2 : blabla

group_vars/group1.yml

dictionary_of_bla_group1:
  - name: blabla2
    value1 : blabla2
    value2 : blabla2

role code snippet

tasks:
  - name: Run for all dictionary_of_bla.* variations
    include_tasks: do_some_stuff.yml
    with_items: "{{ lookup('varnames','dictionary_of_bla.*').split(',') }}"
    loop_control:
      loop_var: _dictionary_of_bla

do_some_stuff.yml

- name: do magic
  magic:
    trick_name: item.name
    trick_value1: item.value1
    trick_value2: item.value2
  with_items: "{{ vars[_dictionary_of_bla] }}"

It's just a snippet, but you should get the idea how it works. note: lookup('varnames','') is available since ansible 2.8

I guess it would be also possible to merge all variables dictionary_of_bla.* into one dictionary during runtime using the same lookup.

The advantage of this approach is that you don't need to set exact lists of variable names, but only the pattern and user can set it dynamically.

Answer 5

Not sure when they added this, but at least for dictionaries/hashes (NOT lists/arrays), you can set the variable hash_behaviour, like so: hash_behaviour = merge in your ansible.cfg.

Took me quite a few hours to accidentally stumble upon this setting :S

Answer 6

Ansible is an automation system, and, concerning configuration file management, it's not very different from apt. The reason more and more software offers the feature to read configuration snippets from a conf.d directory is to enable such automation systems to have different packages/roles add configuration to the software. I believe that it is not the philosophy of Ansible to do what you have in mind, but instead to use the conf.d trick. If the software being configured does not offer this functionality, you may be in trouble.

Since you mention XML configuration files, I take the opportunity to do some whining. There's a reason for the Unix tradition of using plain text configuration files. Binary configuration files do not lend themselves well to system automation, so any kind of binary format will give you trouble and will likely require you to create a program to handle the configuration. (If anyone thinks XML is a plain text format, they should go have their brains examined.)

Now, on your specific PostgreSQL problem. PostgreSQL does support the conf.d trick. First, I would check whether shared_preload_libraries can be specified multiple times. I did not find any hint in the documentation that it can, but I would still try it. If it cannot be specified multiple times, I would explain my problem to the PostgreSQL guys in case they have ideas; this is a PostgreSQL issue and not an Ansible issue. If there is no solution and I really couldn't merge the different roles into one, I'd implement a system to compile the configuration on the managed host. In this case, I'd probably create a script /usr/local/sbin/update_postgresql_config which would compile /etc/postgresql/postgresql.conf.jinja into /etc/postgresql/9.x/main/postgresql.conf. The script would read the shared preload libraries from /etc/postgresql/shared_preload_libraries.txt, one library per line, and provide them to jinja.

It is not uncommon for automation systems to do this. An example is the Debian exim4 package.