Really hoping somebody can help me out here. I had a CoreOS etcd2 member's disk fill up. After a reboot etcd2 was in a bad way. Ultimately I removed the /var/lib/etcd2/member data directory and went through the instructions to remove and re-add the machine to my cluster as detailed here: https://coreos.com/etcd/docs/latest/runtime-configuration.html#remove-a-member
However, while I can run etcd2 manually, attempting to start it with systemctl does not work. This is the etcd2.service:
[Unit]
Description=etcd2
Conflicts=etcd.service
[Service]
User=etcd
Type=notify
Environment=ETCD_DATA_DIR=/var/lib/etcd2
Environment=ETCD_NAME=%m
ExecStart=/usr/bin/etcd2
Restart=always
RestartSec=10s
LimitNOFILE=40000
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
# /run/systemd/system/etcd2.service.d/10-oem.conf
[Service]
Environment=ETCD_ELECTION_TIMEOUT=1200
# /run/systemd/system/etcd2.service.d/20-cloudinit.conf
[Service]
Environment="ETCD_ADVERTISE_CLIENT_URLS=http://172.31.9.22:2379,http://172.31.9.22:4001"
Environment="ETCD_DISCOVERY=https://discovery.etcd.io/567d080563e28e62cf886e48425f632b"
Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.31.9.22:2380"
Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379"
Environment="ETCD_LISTEN_PEER_URLS=http://172.31.9.22:2380"
Environment="ETCD_DEBUG=true"
I added the ETCD_DEBUG=true to try and improve the logging output. Speaking of which:
Feb 26 07:23:36 geo-coreos-database-02 systemd[1]: Starting etcd2...
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://172.31.9.22:2379,http://172.31.9.22:4001
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_DEBUG=true
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/567d080563e28e62cf886e48425f632b
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=1200
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.31.9.22:2380
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://172.31.9.22:2380
Feb 26 07:23:37 geo-coreos-database-02 systemd[1]: etcd2.service: Main process exited, code=exited, status=1/FAILURE
Feb 26 07:23:37 geo-coreos-database-02 systemd[1]: Failed to start etcd2.
Feb 26 07:23:37 geo-coreos-database-02 systemd[1]: etcd2.service: Unit entered failed state.
Feb 26 07:23:37 geo-coreos-database-02 systemd[1]: etcd2.service: Failed with result 'exit-code'.
Not very helpful. However when I run it manually, basing the config on the etcd2.service, the server starts and runs in the foreground without issue:
export ETCD_NAME="e0a8edc41f634fcf9451b5c68e3442bd"
export ETCD_DATA_DIR=/var/lib/etcd2
export ETCD_ADVERTISE_CLIENT_URLS=http://172.31.9.22:2379,http://172.31.9.22:4001
export ETCD_DISCOVERY=https://discovery.etcd.io/567d080563e28e62cf886e48425f632b
export ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.31.9.22:2380
export ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
export ETCD_LISTEN_PEER_URLS=http://172.31.9.22:2380
export ETCD_DEBUG=true
etcd2
This starts the server, with debugging as I would expect and I can even run etcdctl commands on the server. I am at an absolute loss as to how to debug this further. I absolutely don't want to create a new cluster for something so trivial, but this always seems the way with etcd, it's the only thing that every causes problems for us and they are often like this, obscure and difficult to fix.
