How to switch from embedded etcd to external etcd cluster

Question

Is that possible to switch existing cluster with "embedded" etcd to external etcd?

Thanks

Answer 1

The devil's in the details, but for the most part, yes:

1. join your new external etcd members to the internal etcd cluster
2. update the kubeadm-config ConfigMap to indicate to future control plane members where etcd lives
3. patch the existing control plane yaml
4. remove the stacked etcd members
5. pray

etcd

Be Sure you have an understanding of this document, and have practiced it on a sample cluster, because if things go bad, unsticking an angry etcd cluster is painful. Make etcd snapshots early and often

kubeadm-config

kubectl -n kube-system edit configmap kubeadm-config

and replace the ClusterConfiguration etcd: key with something akin to

    etcd:
      external:
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/etcd/apiserver-etcd-client.crt
        endpoints:
        - https://your-new-etcd-url:2379
        keyFile: /etc/kubernetes/pki/etcd/apiserver-etcd-client.key

existing control plane pods

This is just the materialization of the yaml described above, but after provisioning, control plane Nodes don't watch that kubeadm-config for changes. You may actually be happier to just rotate all the control plane nodes if you have an autoscaling system in place, but if you have "pet" control plane nodes then:

  containers:
  - command:
    - kube-apiserver
    # ...
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/etcd/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/etcd/apiserver-etcd-client.key
    - --etcd-servers=https://your-new-etcd-url:2379

and ensure the new apiserver pod comes up a-ok

etcd member teardown

This step depends a great deal on how your current stacked members are running, whether through systemd, static pods, an operator, ... whatever, but you'll for sure need to remove their membership if the existing process doesn't do that as part of stopping them

export ETCDCTL_API=3
etcdctl member list
# find the memberid of the one to remove
bye_bye_member_id=cafebabedeadbeef
etcdctl member remove $bye_bye_member_id

and repeat that for every embedded etcd member as you shut them down