Nomad nodes remove each other services from Consul

Question

I got two nodes Nomad cluster along with a Consul instance so that the jobs can register services to connect to.

However, the services keep getting synced and deregistered. Here is what I have from the Consul logs:

2021-01-26T14:49:59.174Z [INFO]  agent: Synced check: check=_nomad-check-dc23801467b8a65a4fd82311c2606724a180065c
2021-01-26T14:50:00.072Z [INFO]  agent: Synced check: check=_nomad-check-1783c554d9ee0a25d52532f4178c392e931e4bb1
2021-01-26T14:50:04.511Z [INFO]  agent: Synced service: service=_nomad-task-e8d2b77b-3bf5-96c1-8323-63b6151e2cf3-lb0-lb0-admin-admin
2021-01-26T14:50:09.962Z [INFO]  agent: Deregistered service: service=_nomad-task-e8d2b77b-3bf5-96c1-8323-63b6151e2cf3-lb0-lb0-admin-admin
2021-01-26T14:50:34.554Z [INFO]  agent: Synced service: service=_nomad-task-e8d2b77b-3bf5-96c1-8323-63b6151e2cf3-lb0-lb0-admin-admin
2021-01-26T14:50:39.984Z [INFO]  agent: Deregistered service: service=_nomad-task-e8d2b77b-3bf5-96c1-8323-63b6151e2cf3-lb0-lb0-admin-admin
2021-01-26T14:51:04.589Z [INFO]  agent: Synced service: service=_nomad-task-e8d2b77b-3bf5-96c1-8323-63b6151e2cf3-lb0-lb0-admin-admin
2021-01-26T14:51:10.009Z [INFO]  agent: Deregistered service: service=_nomad-task-e8d2b77b-3bf5-96c1-8323-63b6151e2cf3-lb0-lb0-admin-admin

Both nodes are started with the same configuration. However when I look at the logs at TRACE level, I have the following:

NodeA:

2021-01-28T15:58:55.519+0100 [DEBUG] consul.sync: sync complete: registered_services=3 deregistered_services=1 registered_checks=0 deregistered_checks=0

NodeB:

2021-01-28T15:58:59.037+0100 [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=3 registered_checks=0 deregistered_checks=0

Indeed, NodeA has got 3 jobs running while NodeB got 1. It seems both nodes is reverting the changes made by the other one.

Name               Address       Port  Status  Leader  Protocol  Build  Datacenter  Region
NodeA              198.51.100.1  4648  alive   false   2         1.0.2  us1         us
NodeB              198.51.100.2  4648  alive   true    2         1.0.2  us1         us

Did I miss something in my configuration? How to prevent this?

score 0 · Accepted Answer · answered Jan 29 '21 at 13:09

This behavior is actually displayed in the documentation. I just overlooked it:

An important requirement is that each Nomad agent talks to a unique Consul agent. Nomad agents should be configured to talk to Consul agents and not Consul servers. If you are observing flapping services, you may have multiple Nomad agents talking to the same Consul agent. As such avoid configuring Nomad to talk to Consul via DNS such as consul.service.consul

Nomad nodes remove each other services from Consul

1 Answers1