I have learned a few things and would like to share them with you.
etcd is pronounced like "@ cee dee".
I have decided now to not use apt (etcd 3.2 on Ubuntu 18.04) for installation but to download the latest version (3.3.18) with wget. The first hit on Google worked.
Installation
cd /opt/
sudo wget https://github.com/etcd-io/etcd/releases/download/v3.3.18/etcd-v3.3.18-linux-amd64.tar.gz
sudo tar xvf etcd-v3.3.18-linux-amd64.tar.gz
cd etcd-v3.3.18-linux-amd64/
sudo mv etcd etcdctl /usr/local/bin/
sudo mkdir -p /var/lib/etcd/
sudo mkdir /etc/etcd/
sudo groupadd --system etcd
sudo useradd -s /sbin/nologin --system -g etcd etcd
sudo chown -R etcd:etcd /var/lib/etcd/
Reset
Remove all data in the member folders:
sudo rm /etc/etcd/*.etcd/member/ /opt/etcd-v*-linux-amd64/default.etcd/member/ /var/lib/etcd/member/ -fr
Or change argument --initial-cluster-token XYZ
(every five members the same) and start etcd
with parameter --force-new-cluster
.
When you restart your existing cluster or add more members: change from --initial-cluster-state new
to --initial-cluster-state existing
.
Drop all data (keys with values) from etcd cluster:
sudo ETCDCTL_API=3 etcdctl del "" --prefix
Configuration
sudo -u etcd etcd \
--name aaa \
--data-dir /var/lib/etcd/ \
--listen-peer-urls http://localhost:2380,http://localhost:7001,http://192.168.4.101:2380,http://192.168.4.101:7001 \
--listen-client-urls http://localhost:2379,http://localhost:4001,http://192.168.4.101:2379,http://192.168.4.101:4001 \
--initial-advertise-peer-urls http://192.168.4.101:2380 \
--initial-cluster aaa=http://192.168.4.101:2380,bbb=http://192.168.4.102:2380,ccc=http://192.168.4.103:2380,eee=http://192.168.4.105:2380,ddd=http://192.168.4.104:2380 \
--initial-cluster-state new \
--initial-cluster-token 2020-02-07T14:53 \
--advertise-client-urls http://192.168.4.101:2379
sudo -u etcd etcd \
--name bbb \
--data-dir /var/lib/etcd/ \
--listen-peer-urls http://localhost:2380,http://localhost:7001,http://192.168.4.102:2380,http://192.168.4.102:7001 \
--listen-client-urls http://localhost:2379,http://localhost:4001,http://192.168.4.102:2379,http://192.168.4.102:4001 \
--initial-advertise-peer-urls http://192.168.4.102:2380 \
--initial-cluster aaa=http://192.168.4.101:2380,bbb=http://192.168.4.102:2380,ccc=http://192.168.4.103:2380,eee=http://192.168.4.105:2380,ddd=http://192.168.4.104:2380 \
--initial-cluster-state new \
--initial-cluster-token 2020-02-07T14:53 \
--advertise-client-urls http://192.168.4.102:2379
sudo -u etcd etcd \
--name ccc \
--data-dir /var/lib/etcd/ \
--listen-peer-urls http://localhost:2380,http://localhost:7001,http://192.168.4.103:2380,http://192.168.4.103:7001 \
--listen-client-urls http://localhost:2379,http://localhost:4001,http://192.168.4.103:2379,http://192.168.4.103:4001 \
--initial-advertise-peer-urls http://192.168.4.103:2380 \
--initial-cluster aaa=http://192.168.4.101:2380,bbb=http://192.168.4.102:2380,ccc=http://192.168.4.103:2380,eee=http://192.168.4.105:2380,ddd=http://192.168.4.104:2380 \
--initial-cluster-state new \
--initial-cluster-token 2020-02-07T14:53 \
--advertise-client-urls http://192.168.4.103:2379
sudo -u etcd etcd \
--name ddd \
--data-dir /var/lib/etcd/ \
--listen-peer-urls http://localhost:2380,http://localhost:7001,http://192.168.4.104:2380,http://192.168.4.104:7001 \
--listen-client-urls http://localhost:2379,http://localhost:4001,http://192.168.4.104:2379,http://192.168.4.104:4001 \
--initial-advertise-peer-urls http://192.168.4.104:2380 \
--initial-cluster aaa=http://192.168.4.101:2380,bbb=http://192.168.4.102:2380,ccc=http://192.168.4.103:2380,eee=http://192.168.4.105:2380,ddd=http://192.168.4.104:2380 \
--initial-cluster-state new \
--initial-cluster-token 2020-02-07T14:53 \
--advertise-client-urls http://192.168.4.104:2379
sudo -u etcd etcd \
--name eee \
--data-dir /var/lib/etcd/ \
--listen-peer-urls http://localhost:2380,http://localhost:7001,http://192.168.4.105:2380,http://192.168.4.105:7001 \
--listen-client-urls http://localhost:2379,http://localhost:4001,http://192.168.4.105:2379,http://192.168.4.105:4001 \
--initial-advertise-peer-urls http://192.168.4.105:2380 \
--initial-cluster aaa=http://192.168.4.101:2380,bbb=http://192.168.4.102:2380,ccc=http://192.168.4.103:2380,eee=http://192.168.4.105:2380,ddd=http://192.168.4.104:2380 \
--initial-cluster-state new \
--initial-cluster-token 2020-02-07T14:53 \
--advertise-client-urls http://192.168.4.105:2379
You can put this into service:
/etc/systemd/system/etcd.service
(create if not exist) (example for first member aaa)
[Unit]
Description=etcd key-value store
Documentation=https://github.com/etcd-io/etcd
After=network.target
[Service]
User=etcd
Type=notify
Environment=ETCD_DATA_DIR=/var/lib/etcd
Environment=ETCD_NAME=%m
ExecStart=/usr/local/bin/etcd \
--name aaa \
--data-dir /var/lib/etcd/ \
--listen-peer-urls http://localhost:2380,http://localhost:7001,http://192.168.4.101:2380,http://192.168.4.101:7001 \
--listen-client-urls http://localhost:2379,http://localhost:4001,http://192.168.4.101:2379,http://192.168.4.101:4001 \
--initial-advertise-peer-urls http://192.168.4.101:2380 \
--initial-cluster aaa=http://192.168.4.101:2380,bbb=http://192.168.4.102:2380,ccc=http://192.168.4.103:2380,eee=http://192.168.4.105:2380,ddd=http://192.168.4.104:2380 \
--initial-cluster-state new \
--initial-cluster-token 2020-02-07T14:53 \
--advertise-client-urls http://192.168.4.101:2379
Restart=always
RestartSec=10s
LimitNOFILE=40000
sudo systemctl daemon-reload
# sudo systemctl enable etcd # for auto start after reboot
sudo systemctl restart etcd
If someone can make an example with encryption, i.e. with client certificates, I would be grateful.
Chrony
It is also important that all five machines have the same time. Otherwise you will see a lot of errors in your logs. For this I used chrony.
sudo timedatectl set-timezone Europe/Berlin
sudo timedatectl set-local-rtc 1 --adjust-system-clock
sudo timedatectl set-local-rtc 0
sudo systemctl stop systemd-timesyncd.service && sudo systemctl disable systemd-timesyncd.service
sudo apt update && sudo apt --yes install chrony
/etc/chrony/chrony.conf
# I use HTTP connect proxy and can't connect to external ntp servers:
local
bindcmdaddress 0.0.0.0
allow 192.168.0.0/16
cmdallow 192.168.0.0/16
# server 192.168.4.101 prefer iburst # himself
server 192.168.4.102 prefer iburst
server 192.168.4.103 prefer iburst
server 192.168.4.104 prefer iburst
server 192.168.4.105 prefer iburst
# ...
makestep 1 -1
# Show time etc.:
sudo timedatectl
# Show ntp network members:
sudo chronyc sources
Please remember to supply the worker nodes with the same time as well.
Checks
sudo etcdctl cluster-health
member eee6e5e8935fd1c9 is healthy: got healthy result from http://192.168.4.105:2379
member bbb7b0aca4c13cdc is healthy: got healthy result from http://192.168.4.102:2379
member aaac5ad73f7d224f is healthy: got healthy result from http://192.168.4.101:2379
member ccc20379b7c3a64e is healthy: got healthy result from http://192.168.4.103:2379
member ddd76f34bf32390e is healthy: got healthy result from http://192.168.4.104:2379
cluster is healthy
sudo etcdctl member list
eee6e5e8935fd1c9: name=eee peerURLs=http://192.168.4.105:2380 clientURLs=http://192.168.4.105:2379 isLeader=false
bbb7b0aca4c13cdc: name=bbb peerURLs=http://192.168.4.102:2380 clientURLs=http://192.168.4.102:2379 isLeader=false
aaac5ad73f7d224f: name=aaa peerURLs=http://192.168.4.101:2380 clientURLs=http://192.168.4.101:2379 isLeader=true
ccc20379b7c3a64e: name=ccc peerURLs=http://192.168.4.103:2380 clientURLs=http://192.168.4.103:2379 isLeader=false
ddd76f34bf32390e: name=ddd peerURLs=http://192.168.4.104:2380 clientURLs=http://192.168.4.104:2379 isLeader=false