Checking missing nodes in microk8s cluster with ansible

Question

I'm trying to automate servers configuration to deploy a microk8s cluster with Ansible

My main playbook :

  - name: Getting connected nodes
    ansible.builtin.shell: "microk8s.kubectl get nodes -o wide | awk '{ print $6, $2 }'"
    register: connected_nodes
    when: inventory_hostname == 'master_init'
    tags:
      microk8s

  - include_tasks: ./resources/microk8s/join2.yaml
    with_items: "{{ groups['workers'] }}"
    tags:
      - microk8s

The join2.yaml sub-playbook :

  - name: Check if node is connected
    #shell: "echo {{ hostvars['master_init']['connected_nodes']['stdout_lines'] }} | grep {{ item}} | grep {{ Ready }}"
    shell: "echo {{ connected_nodes }} | grep {{ item}} | grep {{ Ready }}"
    register: node_is_connected
    ignore_errors: yes
    failed_when: node_is_connected.rc == 2

  - name: Generating the join command including token for each missing node
    ansible.builtin.command: microk8s.add-node --format=short
    register: microk8s_join_cmd
    delegate_to: 'master_init'
    when: 
    - hostvars['master_init']['node_is_connected']['rc'] == 1
    tags:
      microk8s

  - name: Adding the worker node
    ansible.builtin.command: "{{ microk8s_join_cmd['stdout_lines'][0] }}"
    when: 
    - hostvars['master_init']['node_is_connected']['rc'] == 1
    tags:
      microk8s

The logic behing is to get the connected nodes on the master_init master node via the "Getting connected nodes" task, deduce the missing nodes via "Check if node is connected", generate a joining token on the master node ("Generating the join command including token for each missing node") and make the worker node join the cluster ("Adding the worker node").

I'm struggling with the condition for the last two tasks :

$ ansible-playbook playbook.yaml -t microk8s
PLAY [cluster] *******************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************
ok: [192.168.1.14]
ok: [master_init]
ok: [192.168.1.12]
ok: [192.168.1.13]

TASK [Getting connected nodes] ***************************************************************************************************************************************************
skipping: [192.168.1.12]
skipping: [192.168.1.13]
skipping: [192.168.1.14]
changed: [master_init]

TASK [include_tasks] *************************************************************************************************************************************************************
included: /Users/ben/ops/hydra/resources/microk8s/join2.yaml for master_init, 192.168.1.12, 192.168.1.13, 192.168.1.14 => (item=192.168.1.12)
included: /Users/ben/ops/hydra/resources/microk8s/join2.yaml for master_init, 192.168.1.12, 192.168.1.13, 192.168.1.14 => (item=192.168.1.13)
included: /Users/ben/ops/hydra/resources/microk8s/join2.yaml for master_init, 192.168.1.12, 192.168.1.13, 192.168.1.14 => (item=192.168.1.14)

TASK [Generating the join command including token for each missing node] *********************************************************************************************************
fatal: [master_init]: FAILED! => {"msg": "The conditional check 'hostvars['master_init']['node_is_connected']['rc'] == 1' failed. The error was: error while evaluating conditional (hostvars['master_init']['node_is_connected']['rc'] == 1): 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'node_is_connected'\n\nThe error appears to be in '/Users/ben/ops/hydra/resources/microk8s/join2.yaml': line 15, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n  - name: Generating the join command including token for each missing node\n    ^ here\n"}
fatal: [192.168.1.12 -> master_init]: FAILED! => {"msg": "The conditional check 'hostvars['master_init']['node_is_connected']['rc'] == 1' failed. The error was: error while evaluating conditional (hostvars['master_init']['node_is_connected']['rc'] == 1): 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'node_is_connected'\n\nThe error appears to be in '/Users/ben/ops/hydra/resources/microk8s/join2.yaml': line 15, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n  - name: Generating the join command including token for each missing node\n    ^ here\n"}
fatal: [192.168.1.13 -> master_init]: FAILED! => {"msg": "The conditional check 'hostvars['master_init']['node_is_connected']['rc'] == 1' failed. The error was: error while evaluating conditional (hostvars['master_init']['node_is_connected']['rc'] == 1): 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'node_is_connected'\n\nThe error appears to be in '/Users/ben/ops/hydra/resources/microk8s/join2.yaml': line 15, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n  - name: Generating the join command including token for each missing node\n    ^ here\n"}
fatal: [192.168.1.14 -> master_init]: FAILED! => {"msg": "The conditional check 'hostvars['master_init']['node_is_connected']['rc'] == 1' failed. The error was: error while evaluating conditional (hostvars['master_init']['node_is_connected']['rc'] == 1): 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'node_is_connected'\n\nThe error appears to be in '/Users/ben/ops/hydra/resources/microk8s/join2.yaml': line 15, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n  - name: Generating the join command including token for each missing node\n    ^ here\n"}

PLAY RECAP ***********************************************************************************************************************************************************************
192.168.1.12               : ok=5    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
192.168.1.13               : ok=5    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
192.168.1.14               : ok=5    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
master_init                : ok=6    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

I know that grep return a code error 1 when the output is empty, that's why I'm using the ignore_errors option, and relying my condition for the last two tasks on the Response Code == 1, but I guess I'm not using the facts in the right way.

Can you help please ?

The code is really a mess but here are some arbitrary notes that might make it work. The `grep {{ Ready }}` would expect an ansible var named "Ready" which seems wrong. The commented out command in "Check if node is connected" task seems more appropriate, since the `connected_nodes` var exists only in `master_init`. Use `stdout` instead of `stdout_lines` for the grep. For the last 2 tasks try to use `node_is_connected.rc` for the `when` clause (this var is node specific) — ttsakpc, Aug 17 '22 at 14:20
Thank you @ttsakpc , I'll give this a try ASAP. Could you develop more on the "mess" aspect please ? Is it regarding the structure, or the rookie mistake such as the `grep {{ Ready }}` statemetn ? — Ben, Aug 19 '22 at 07:53
Sorry, it sounded harsh :( I mostly mentioned it because all the tasks are shell/cmd which is bad practice for Ansible in general and modules should be used (there are modules for kubernetes, have a look at them). Also it seems to me that `cmd` module could be used in place of `shell` which is [safer](https://www.ansiblepilot.com/articles/ansible-modules-command-vs-shell/) — ttsakpc, Aug 19 '22 at 08:17
You can have a look at [this](https://www.d-nix.nl/2020/07/target-kubernetes-nodes-using-ansible/) that uses a [lookup](https://galaxy.ansible.com/kubernetes/core) to find the nodes connected to K8s cluster and then run the join command only on the nodes of your inventory group that do not exist in the nodes found by the lookup `when: inventory_hostname not in (nodes | json_query('[*].status.addresses[0].address'))` — ttsakpc, Aug 19 '22 at 08:59

Checking missing nodes in microk8s cluster with ansible

0 Answers0