ETCD Cluster configuration for Kubernetes: Which one should be considered?

Question

I would like to know how to deploy a ETCD cluster for Kubernetes. It seems like there are two different documentation and I don't know which one must be considered or the impact of each of them.

From the Kubernetes documentation for a multi-cluster etcd, it suggested to start it like this

etcd --listen-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379 --advertise-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379

Here the --listen-client-urls-- has the list of all the ETCD endpoints and same thing for --advertise-client-urls and from the Kubernetes documentation that command is ran only once.

From the ETCD documentation that command must be run in each of the node

$ etcd --name infra0 --initial-advertise-peer-urls https://10.0.1.10:2380 \
  --listen-peer-urls https://10.0.1.10:2380 \
  --listen-client-urls https://10.0.1.10:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://10.0.1.10:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster infra0=https://10.0.1.10:2380,infra1=https://10.0.1.11:2380,infra2=https://10.0.1.12:2380 \
  --initial-cluster-state new \
  --client-cert-auth --trusted-ca-file=/path/to/ca-client.crt \
  --cert-file=/path/to/infra0-client.crt --key-file=/path/to/infra0-client.key \
  --peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt \
  --peer-cert-file=/path/to/infra0-peer.crt --peer-key-file=/path/to/infra0-peer.key

and we can notice that the --listen-client-urls-- contain only the IP address of the current node and the other parameters are not present in the Kubernetes documentation.

Why are they so different? Can you help me to understand? Which one is the good one? When does each of them must be used?

Answer 1

In short words I suggest following etcd documentation in terms of setting up etcd cluster.

---

In more words let's first look at what these flags mean.

--listen-client-urls - this is a member's flag (relevant on node's level):

List of URLs to listen on for client traffic. This flag tells the etcd to accept incoming requests from the clients on the specified scheme://IP:port combinations. Scheme can be either http or https. If 0.0.0.0 is specified as the IP, etcd listens to the given port on all interfaces. If an IP address is given as well as a port, etcd will listen on the given port and interface. Multiple URLs may be used to specify a number of addresses and ports to listen on. The etcd will respond to requests from any of the listed addresses and ports.

etcd - listen clients url

--advertise-client-url - this is a cluster scoped flag (it says for itself):

List of this member’s client URLs to advertise to the rest of the cluster. These URLs can contain domain names.

etcd - advertise-client-url

Also please find a short clarification in Q&A - difference between flags

---

As for kubernetes documentation you shared, this doesn't look logical because etcd command should run on each of the nodes, however in this example it's not clear how other nodes will get this information.

Also another option is for simplicity and quick start they offered such config and etcd should ignore IPs which are not used (for instance other nodes' IPs on local port to listen to client's traffic). And it shouldn't be bad if all etcd nodes can advertise IPs of all nodes.

However etcd docs state clearly to start etcd with only local address on each node - I assume it's the correct way.

I suggest referring to setting up the cluster. This docs cover how to set up HA cluster using kubeadm (probably not the most convenient way, however I was able to set it up and get work). As you can see in this example only etcd node's IP is presented in config which is aligned with etcd docs.

Useful links:

• etcd - configuration flags
• etcd - fault tolerance