k8sp/etcd

Introduction to etcd

etcd's Issues

All etcd server exits with static mode.

Environment

bjlg-49p45-k8s-01 static-etcd2 # cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=1010.6.0
VERSION_ID=1010.6.0
BUILD_ID=2016-06-28-0910
PRETTY_NAME="CoreOS 1010.6.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
bjlg-49p45-k8s-01 static-etcd2 # /usr/bin/etcd2 -version
etcd Version: 2.3.1
Git SHA: 2b67f52
Go Version: go1.5.3
Go OS/Arch: linux/amd64

Deploy bash script, 3 etcd memory with named infra0, infra1 and infra2

export public_ipv4_infra0=192.168.49.45
export public_ipv4_infra1=192.168.49.46
export public_ipv4_infra2=192.168.49.47
export name=infra0
export NITIAL_CLUSTER="infra0=http://$public_ipv4_infra0:2380,infra1=http://$public_ipv4_infra1:2380,infra2=http://$public_ipv4_infra2:2380"
export ETCD_INITIAL_CLUSTER_STATE=new
/usr/bin/etcd2 --name $name --initial-advertise-peer-urls http://$public_ipv4_infra0:2380 \
  --listen-peer-urls http://$public_ipv4_infra0:2380 \
  --listen-client-urls http://$public_ipv4_infra0:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://$public_ipv4_infra0:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster infra0=http://$public_ipv4_infra0:2380,infra1=http://$public_ipv4_infra1:2380,infra2=http://$public_ipv4_infra2:2380 \
  --initial-cluster-state new \
  --data-dir /var/lib/etcd2/$name \

All process will be existed after a period of time. Here is the logs

infra0 log

2016-07-19 14:29:24.289951 E | etcdhttp: got unexpected response error (etcdserver: request timed out)
2016-07-19 14:29:24.859851 I | raft: f3aefe338854c13a is starting a new election at term 3419
2016-07-19 14:29:24.859887 I | raft: f3aefe338854c13a became candidate at term 3420
2016-07-19 14:29:24.859910 I | raft: f3aefe338854c13a received vote from f3aefe338854c13a at term 3420
2016-07-19 14:29:24.859923 I | raft: f3aefe338854c13a [logterm: 3243, index: 262126] sent vote request to e344a1396881242e at term 3420
2016-07-19 14:29:24.859935 I | raft: f3aefe338854c13a [logterm: 3243, index: 262126] sent vote request to eff15145dca03741 at term 3420
2016-07-19 14:29:26.000041 E | etcdhttp: got unexpected response error (etcdserver: request timed out) [merged 3 repeated lines in 1.71s]
2016-07-19 14:29:26.159758 I | raft: f3aefe338854c13a is starting a new election at term 3420
2016-07-19 14:29:26.159803 I | raft: f3aefe338854c13a became candidate at term 3421
2016-07-19 14:29:26.159814 I | raft: f3aefe338854c13a received vote from f3aefe338854c13a at term 3421
2016-07-19 14:29:26.159828 I | raft: f3aefe338854c13a [logterm: 3243, index: 262126] sent vote request to e344a1396881242e at term 3421
2016-07-19 14:29:26.159841 I | raft: f3aefe338854c13a [logterm: 3243, index: 262126] sent vote request to eff15145dca03741 at term 3421
2016-07-19 14:29:27.559756 I | raft: f3aefe338854c13a is starting a new election at term 3421
2016-07-19 14:29:27.559806 I | raft: f3aefe338854c13a became candidate at term 3422
2016-07-19 14:29:27.559818 I | raft: f3aefe338854c13a received vote from f3aefe338854c13a at term 3422
2016-07-19 14:29:27.559831 I | raft: f3aefe338854c13a [logterm: 3243, index: 262126] sent vote request to e344a1396881242e at term 3422
2016-07-19 14:29:27.559844 I | raft: f3aefe338854c13a [logterm: 3243, index: 262126] sent vote request to eff15145dca03741 at term 3422
2016-07-19 14:29:29.059778 I | raft: f3aefe338854c13a is starting a new election at term 3422
2016-07-19 14:29:29.059813 I | raft: f3aefe338854c13a became candidate at term 3423
2016-07-19 14:29:29.059824 I | raft: f3aefe338854c13a received vote from f3aefe338854c13a at term 3423
2016-07-19 14:29:29.059838 I | raft: f3aefe338854c13a [logterm: 3243, index: 262126] sent vote request to e344a1396881242e at term 3423
2016-07-19 14:29:29.059850 I | raft: f3aefe338854c13a [logterm: 3243, index: 262126] sent vote request to eff15145dca03741 at term 3423

infra1 log

2016-07-19 10:41:53.820550 I | fileutil: purged file /var/lib/etcd2/infra1/member/snap/0000000000000caa-0000000000029f36.snap successfully
2016-07-19 11:30:29.568414 I | raft: e344a1396881242e [term: 3242] received a MsgVote message with higher term from eff15145dca03741 [term: 3243]
2016-07-19 11:30:29.568457 I | raft: e344a1396881242e became follower at term 3243
2016-07-19 11:30:29.568474 I | raft: e344a1396881242e [logterm: 3242, index: 230649, vote: 0] voted for eff15145dca03741 [logterm: 3242, index: 230649] at term 3243
2016-07-19 11:30:29.568485 I | raft: raft.node: e344a1396881242e lost leader f3aefe338854c13a at term 3243
2016-07-19 11:30:29.578578 I | raft: raft.node: e344a1396881242e elected leader eff15145dca03741 at term 3243
2016-07-19 11:30:30.119368 I | raft: e344a1396881242e [term: 3243] ignored a MsgApp message with lower term from f3aefe338854c13a [term: 3242]
2016-07-19 11:30:30.119402 I | raft: e344a1396881242e [term: 3243] ignored a MsgApp message with lower term from f3aefe338854c13a [term: 3242]
2016-07-19 11:37:03.513093 I | etcdserver: start to snapshot (applied: 231836, lastsnap: 221835)
2016-07-19 11:37:03.545078 I | etcdserver: saved snapshot at index 231836
2016-07-19 11:37:03.545449 I | etcdserver: compacted raft log at 226836
2016-07-19 11:37:23.839884 I | fileutil: purged file /var/lib/etcd2/infra1/member/snap/0000000000000caa-000000000002c647.snap successfully
2016-07-19 12:32:38.217257 I | etcdserver: start to snapshot (applied: 241837, lastsnap: 231836)
2016-07-19 12:32:38.249246 I | etcdserver: saved snapshot at index 241837
2016-07-19 12:32:38.249525 I | etcdserver: compacted raft log at 236837
2016-07-19 12:32:53.859343 I | fileutil: purged file /var/lib/etcd2/infra1/member/snap/0000000000000caa-000000000002ed58.snap successfully
2016-07-19 13:28:11.867890 I | etcdserver: start to snapshot (applied: 251838, lastsnap: 241837)
2016-07-19 13:28:11.899176 I | etcdserver: saved snapshot at index 251838
2016-07-19 13:28:11.899461 I | etcdserver: compacted raft log at 246838
2016-07-19 13:28:23.876017 I | fileutil: purged file /var/lib/etcd2/infra1/member/snap/0000000000000caa-0000000000031469.snap successfully
2016-07-19 14:23:45.867896 I | etcdserver: start to snapshot (applied: 261839, lastsnap: 251838)
2016-07-19 14:23:45.900204 I | etcdserver: saved snapshot at index 261839
2016-07-19 14:23:45.900482 I | etcdserver: compacted raft log at 256839
2016-07-19 14:23:53.894054 I | fileutil: purged file /var/lib/etcd2/infra1/member/snap/0000000000000caa-0000000000033b7a.snap successfully

infra2 log

2016-07-19 14:25:00.608974 W | rafthttp: the connection to peer e344a1396881242e is unhealthy
2016-07-19 14:25:00.842435 W | etcdserver: failed to reach the peerURL(http://192.168.49.46:2380) of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:00.842488 W | etcdserver: cannot get the version of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:04.843764 W | etcdserver: failed to reach the peerURL(http://192.168.49.46:2380) of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:04.843825 W | etcdserver: cannot get the version of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:08.845047 W | etcdserver: failed to reach the peerURL(http://192.168.49.46:2380) of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:08.845106 W | etcdserver: cannot get the version of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:12.846185 W | etcdserver: failed to reach the peerURL(http://192.168.49.46:2380) of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:12.846248 W | etcdserver: cannot get the version of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:16.847425 W | etcdserver: failed to reach the peerURL(http://192.168.49.46:2380) of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:16.847487 W | etcdserver: cannot get the version of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:20.848772 W | etcdserver: failed to reach the peerURL(http://192.168.49.46:2380) of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:20.848834 W | etcdserver: cannot get the version of member e344a1396881242e (Get http://192.168.49.46:2380/version: dial tcp 192.168.49.46:2380: getsockopt: connection refused)
2016-07-19 14:25:23.777587 N | osutil: received terminated signal, shutting down...

"failed to send out heartbeat on time" when using static mode

logs:

Jun 12 03:04:01 hlg-wuyi-coreos-01 etcd2[4187]: server is likely overloaded
Jun 12 03:04:01 hlg-wuyi-coreos-01 etcd2[4187]: failed to send out heartbeat on time (deadline exceeded for 86.557859ms)
Jun 12 03:04:01 hlg-wuyi-coreos-01 etcd2[4187]: server is likely overloaded
Jun 12 03:04:32 hlg-wuyi-coreos-01 etcd2[4187]: failed to send out heartbeat on time (deadline exceeded for 10.632199ms)
Jun 12 03:04:32 hlg-wuyi-coreos-01 etcd2[4187]: server is likely overloaded
Jun 12 03:04:32 hlg-wuyi-coreos-01 etcd2[4187]: failed to send out heartbeat on time (deadline exceeded for 10.801399ms)
Jun 12 03:04:32 hlg-wuyi-coreos-01 etcd2[4187]: server is likely overloaded

etcdctl works fine:

# etcdctl member list
251125c4d7f71035: name=hlg-wuyi-coreos-03 peerURLs=http://172.24.3.221:2380 clientURLs=http://172.24.3.221:2379 isLeader=false
8daf20902656d961: name=hlg-wuyi-coreos-02 peerURLs=http://172.24.3.220:2380 clientURLs=http://172.24.3.220:2379 isLeader=false
e06797cebe158b25: name=hlg-wuyi-coreos-01 peerURLs=http://172.24.3.150:2380 clientURLs=http://172.24.3.150:2379 isLeader=true
# etcdctl ls
/coreos.com