I have at home two CoreOS stable v1122.2.0 machines. both of them are registered at discovery.etcd.io properly. (i see both of them registered at https://ip:2380 when i browse to the discovery url).
the first server is 10.79.218.2, and the 2nd is 10.79.218.3.
I'm trying to configure etcd for tls. so I created the required certificates and I configured etcd2 for tls. all the configuration (of the first server with local ip of 10.79.218.2) is in the cloud-config.yml file:
#cloud-config
write_files:
- path: "/etc/ssl/etcd/ca.pem"
permissions: "0600"
owner: "etcd:etcd"
content: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
- path: "/etc/ssl/etcd/server1.pem"
permissions: "0600"
owner: "etcd:etcd"
content: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
- path: "/etc/ssl/etcd/server1-key.pem"
permissions: "0600"
owner: "etcd:etcd"
content: |
-----BEGIN EC PRIVATE KEY-----
...
-----END EC PRIVATE KEY-----
- path: "/etc/kubernetes/cni/net.d/10-calico.conf"
content: |
{
"name": "calico",
"type": "flannel",
"delegate": {
"type": "calico",
"etcd_endpoints": "https://10.79.218.2:2379,https://10.79.218.3:2379",
"log_level": "none",
"log_level_stderr": "info",
"hostname": "10.79.218.2",
"policy": {
"type": "k8s",
"k8s_api_root": "http://127.0.0.1:8080/api/v1/"
}
}
}
- path: "/etc/kubernetes/manifests/policy-controller.yaml"
content: |
apiVersion: v1
kind: Pod
metadata:
name: calico-policy-controller
namespace: calico-system
spec:
hostNetwork: true
containers:
# The Calico policy controller.
- name: k8s-policy-controller
image: calico/kube-policy-controller:v0.2.0
env:
- name: ETCD_ENDPOINTS
value: "https://10.79.218.2:2379,http://10.79.218.3:2379"
- name: K8S_API
value: "http://127.0.0.1:8080"
- name: LEADER_ELECTION
value: "true"
# Leader election container used by the policy controller.
- name: leader-elector
image: quay.io/calico/leader-elector:v0.1.0
imagePullPolicy: IfNotPresent
args:
- "--election=calico-policy-election"
- "--election-namespace=calico-system"
- "--http=127.0.0.1:4040"
- path: "/etc/kubernetes/manifests/kube-scheduler.yaml"
content: |
apiVersion: v1
kind: Pod
metadata:
name: kube-scheduler
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-scheduler
image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
command:
- /hyperkube
- scheduler
- --master=http://127.0.0.1:8080
- --leader-elect=true
livenessProbe:
httpGet:
host: 127.0.0.1
path: /healthz
port: 10251
initialDelaySeconds: 15
timeoutSeconds: 1
- path: "/etc/kubernetes/manifests/kube-controller-manager.yaml"
content: |
apiVersion: v1
kind: Pod
metadata:
name: kube-controller-manager
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-controller-manager
image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
command:
- /hyperkube
- controller-manager
- --master=http://127.0.0.1:8080
- --leader-elect=true
- --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --root-ca-file=/etc/kubernetes/ssl/ca.pem
livenessProbe:
httpGet:
host: 127.0.0.1
path: /healthz
port: 10252
initialDelaySeconds: 15
timeoutSeconds: 1
volumeMounts:
- mountPath: /etc/kubernetes/ssl
name: ssl-certs-kubernetes
readOnly: true
- mountPath: /etc/ssl/certs
name: ssl-certs-host
readOnly: true
volumes:
- hostPath:
path: /etc/kubernetes/ssl
name: ssl-certs-kubernetes
- hostPath:
path: /usr/share/ca-certificates
name: ssl-certs-host
- path: "/etc/kubernetes/manifests/kube-proxy.yaml"
content: |
apiVersion: v1
kind: Pod
metadata:
name: kube-proxy
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-proxy
image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
command:
- /hyperkube
- proxy
- --master=http://127.0.0.1:8080
- --proxy-mode=iptables
securityContext:
privileged: true
volumeMounts:
- mountPath: /etc/ssl/certs
name: ssl-certs-host
readOnly: true
volumes:
- hostPath:
path: /usr/share/ca-certificates
name: ssl-certs-host
- path: "/etc/kubernetes/manifests/kube-apiserver.yaml"
content: |
apiVersion: v1
kind: Pod
metadata:
name: kube-apiserver
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-apiserver
image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
command:
- /hyperkube
- apiserver
- --bind-address=0.0.0.0
- --etcd-servers=https://10.79.218.2:2379,https://10.79.218.3:2379
- --allow-privileged=true
- --service-cluster-ip-range=10.0.0.0/24
- --secure-port=443
- --advertise-address=10.79.218.2
- --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota
- --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
- --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --client-ca-file=/etc/kubernetes/ssl/ca.pem
- --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --runtime-config=extensions/v1beta1=true,extensions/v1beta1/networkpolicies=true
ports:
- containerPort: 443
hostPort: 443
name: https
- containerPort: 8080
hostPort: 8080
name: local
volumeMounts:
- mountPath: /etc/kubernetes/ssl
name: ssl-certs-kubernetes
readOnly: true
- mountPath: /etc/ssl/certs
name: ssl-certs-host
readOnly: true
volumes:
- hostPath:
path: /etc/kubernetes/ssl
name: ssl-certs-kubernetes
- hostPath:
path: /usr/share/ca-certificates
name: ssl-certs-host
- path: "/etc/flannel/options.env"
content: |
FLANNELD_IFACE=10.79.218.2
FLANNELD_ETCD_ENDPOINTS=https://10.79.218.2:2379,https://10.79.218.3:2379
- path: "/etc/kubernetes/ssl/ca.pem"
permissions: "0600"
owner: "root:root"
content: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
- path: "/etc/kubernetes/ssl/apiserver.pem"
permissions: "0600"
owner: "root:root"
content: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
- path: "/etc/kubernetes/ssl/apiserver-key.pem"
permissions: "0600"
owner: "root:root"
content: |
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
ssh_authorized_keys:
- "ssh-rsa ... ufk@ufk-osx-music"
coreos:
update:
reboot-strategy: etcd-lock
etcd2:
discovery: https://discovery.etcd.io/...
# multi-region and multi-cloud deployments need to use $public_ipv4
advertise-client-urls: https://10.79.218.2:2379
initial-advertise-peer-urls: https://10.79.218.2:2380
# listen on both the official ports and the legacy ports
# legacy ports can be omitted if your application doesn't depend on them
listen-client-urls: https://0.0.0.0:2379
listen-peer-urls: https://10.79.218.2:2380
units:
- name: 00-enp4s0.network
runtime: true
content: |
[Match]
Name=enp4s0
[Network]
Address=10.79.218.2/24
Gateway=10.79.218.232
DNS=8.8.8.8
- name: var-lib-rkt.mount
enable: true
command: start
content: |
[Mount]
What=/dev/disk/by-uuid/23cc3284-e121-4448-b755-b74b5a022251
Where=/var/lib/rkt
Type=btrfs
Options=loop,discard
- name: etcd2.service
command: start
drop-ins:
- name: 30-certs.conf
content: |
[Service]
Environment="ETCD_CERT_FILE=/etc/ssl/etcd/server1.pem"
Environment="ETCD_KEY_FILE=/etc/ssl/etcd/server1-key.pem"
Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
Environment="ETCD_CLIENT_CERT_AUTH=true"
Environment="ETCD_PEER_CERT_FILE=/etc/ssl/etcd/server1.pem"
Environment="ETCD_PEER_KEY_FILE=/etc/ssl/etcd/server1-key.pem"
Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
- name: flanneld.service
command: start
drop-ins:
- name: 50-network-config.conf
content: |
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}'
- name: 40-ExecStartPre-symlink.conf
content: |
[Service]
ExecStartPre=/usr/bin/ln -sf /etc/flannel/options.env /run/flannel/options.env
- name: kubelet.service
enable: false
content: |
[Service]
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
Environment=KUBELET_VERSION=v1.3.6_coreos.0
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--api-servers=http://127.0.0.1:8080 \
--network-plugin-dir=/etc/kubernetes/cni/net.d \
--network-plugin=cni \
--register-schedulable=false \
--allow-privileged=true \
--config=/etc/kubernetes/manifests \
--hostname-override=10.79.218.2 \
--cluster-dns=8.8.8.8 \
--cluster-domain=cluster.local
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
- name: calico-node.service
enable: false
content: |
[Unit]
Description=Calico per-host agent
Requires=network-online.target
After=network-online.target
[Service]
Slice=machine.slice
Environment=CALICO_DISABLE_FILE_LOGGING=true
Environment=HOSTNAME=10.79.218.2
Environment=IP=${ADVERTISE_IP}
Environment=FELIX_FELIXHOSTNAME=10.79.218.2
Environment=CALICO_NETWORKING=false
Environment=NO_DEFAULT_POOLS=true
Environment=ETCD_ENDPOINTS=https://10.79.218.2:2379,https://10.79.218.3:2379
ExecStart=/usr/bin/rkt run --inherit-env --stage1-from-dir=stage1-fly.aci \
--volume=modules,kind=host,source=/lib/modules,readOnly=false \
--mount=volume=modules,target=/lib/modules \
--trust-keys-from-https quay.io/calico/node:v0.19.0
KillMode=mixed
Restart=always
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
- i removed important stuff (like keys) and replaced them with
... - i have kubelet and calico configured but disabled for now till i'll get etcd2 up and running with tls.
so I'm trying to run etcdctl from my mac pro laptop to list the etcd2 members (while being connected to the same local network) using the command
./etcdctl --endpoints="https://10.79.218.2:2379,https://10.79.218.3:2379" --key-file="./cfssl/client-key.pem" --cert-file="./cfssl/client.pem" --ca-file="./cfssl/ca.pem" member list
and I get in response
Try '--no-sync' if you want to access non-published client endpoints(https://10.79.218.2:2379,https://10.79.218.3:2379).
Error: client: no endpoints available
running the same command with --no-sync results the following:
Failed to get leader: client: etcd cluster is unavailable or misconfigured
I created the certificates as described in https://github.com/coreos/docs/blob/master/os/generate-self-signed-certificates.md
what am I missing ?
thanks
update
so i found out about https://github.com/coreos/etcd/tree/master/hack/tls-setup. and I re-created the certificates using this script, and I configured my coreos etcd2 servers with these scripts.
it created the following certificates: ca-key.pem ca.csr ca.pem etcd1-key.pem etcd1.csr etcd1.pem etcd2-key.pem etcd2.csr etcd2.pem etcd3-key.pem etcd3.csr etcd3.pem proxy1-key.pem proxy1.csr proxy1.pem
the problem is that I don't see any client certificate.. so I don't quite understand what certificates to pass as parameters for etcdctl to work from my desktop. any ideas ?
update 2
so i found how to use etcdctl with tls.. i think. but still I get errors. I executed the following command:
/etcdctl --endpoint=https://10.79.218.2:2379 --ca-file=./certs/ca.pem --cert-file=./certs/etcd1.pem --key-file=./certs/etcd1-key.pem --no-sync ls
but I get the error
Error: client: etcd cluster is unavailable or misconfigured
error #0: x509: cannot validate certificate for 10.79.218.2 because it doesn't contain any IP SANs
