ETCD真的折腾了很久，因为只想教一个人。
正文
大多数人都知道的两种部署方式：
一、静态部署（比较常用而且稳定，不过需要写的参数有点多）
二、动态发现（这个相对参数少一些，看着挺高大上，用过之后就知道了都一样。其中一种适用于Kubernetes集群部署，就是DNS那种）
动态发现其实细分能分为三种实际就是两种，因为动态发现分为自定义发现部署、公共发现部署以及DNS发现部署。
前两种目前自定义发现没有详细测试，公共发现很简单。博客内有相关文章简介（部分主要部署摘要），这三种实际第一种依赖于已有集群才能使用。不过集群起来之后还是要修改参数，很烦。
第二种因为是基于公共发现所以可以直接以新集群部署，不过这里开始发现的那个网址需要你能访问公网（https://discovery.etcd.io/new?size=3）而且这个链接一次性产品不能用二次重启报废，所以起来后也需要修改集群参数。
第三种可以理解为类似于静态不过实际不是，它与静态最大的差别就是。少了initial-cluster参数因为这个是静态使用，静态服务启动会根据这里的参数寻找其他集群节点。在不使用initial-cluster参数后，它其实增加了一个参数DNS专用的Discovery-srv（DNS SRV是DNS记录中一种，用来指定服务地址。与常见的A记录、cname不同的是，SRV中除了记录服务器的地址，还记录了服务的端口，并且可以设置每个服务地址的优先级和权重。访问服务的时候，本地的DNS resolver从DNS服务器查询到一个地址列表，根据优先级和权重，从中选取一个地址作为本次请求的目标地址。）
在使用DNS部署前一定要了解一个知识，这个很重要。
DNS SRV注意事项（具体写在了另一篇博客，不然太乱了）
上面都了解后可以开始部署了。
既然是集群发现那就先开始部署DNS服务吧，用dnsmasq可以很快速的部署起来。而且很方便
安装dnsmasq与测试域名解析的工具
yum install -y dnsmasq bind-utils
编写配置文件
[root@master etcd]# cat /etc/dnsmasq.conf
#额外加载配置文件的目录
conf-dir=/etc/dnsmasq.d,.rpmnew,.rpmsave,.rpmorig
#额外的hosts文件目录
addn-hosts=/etc/dns.hosts
#向所有上游服务器发送查询
all-servers
#定义dnsmasq监听的地址，默认是监控本机的所有网卡上。局域网内主机若要使用dnsmasq服务时，指定本机的IP地址。
listen-address=172.21.130.169,127.0.0.1
#指定额外的解析文件
resolv-file=/etc/resolv.dnsmasq.conf
#严格按照解析文件中的配置顺序进行查找
strict-order
配置SRV记录文件（分为两部分一部分是服务端或者直白的说就是member之间交互用的，另一部分就是接受客户端请求的）
[root@master etcd]# cat /etc/dnsmasq.d/dns_host.conf
srv-host=_etcd-server-ssl._tcp.linux.com,etcd1.linux.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.linux.com,etcd2.linux.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.linux.com,etcd3.linux.com,2380,0,100
srv-host=_etcd-client-ssl._tcp.linux.com,etcd1.linux.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.linux.com,etcd2.linux.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.linux.com,etcd3.linux.com,2379,0,100
配置指定额外的DNS解析文件
[root@master etcd]# cat /etc/resolv.dnsmasq.conf
nameserver 172.21.130.169
配置额外的hosts文件
[root@master etcd]# cat /etc/dns.hosts
172.21.130.169 etcd1.linux.com
172.21.130.168 etcd2.linux.com
172.28.17.85 etcd3.linux.com
启动服务&&查看状态
[root@master etcd]# systemctl restart dnsmasq.service && systemctl status dnsmasq.service
● dnsmasq.service - DNS caching server.
Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; disabled; vendor preset: disabled)
Active: active (running) since 二 2021-05-25 00:04:10 CST; 10ms ago
Main PID: 4874 (dnsmasq)
CGroup: /system.slice/dnsmasq.service
└─4874 /usr/sbin/dnsmasq -k -r /etc/resolv.dnsmasq.conf
5月 25 00:04:10 master systemd[1]: Stopped DNS caching server..
5月 25 00:04:10 master systemd[1]: Started DNS caching server..
5月 25 00:04:10 master dnsmasq[4874]: started, version 2.76 cachesize 150
5月 25 00:04:10 master dnsmasq[4874]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth net...t inotify
5月 25 00:04:10 master dnsmasq[4874]: reading /etc/resolv.dnsmasq.conf
5月 25 00:04:10 master dnsmasq[4874]: ignoring nameserver 172.21.130.169 - local interface #（忽略吧没事能用）因为把解析无限转给本地
5月 25 00:04:10 master dnsmasq[4874]: read /etc/hosts - 6 addresses
5月 25 00:04:10 master dnsmasq[4874]: read /etc/dns.hosts - 3 addresses
Hint: Some lines were ellipsized, use -l to show in full.
查看DNS SRV相关记录
[root@master etcd]# dig @172.21.130.169 +noall +answer SRV _etcd-server-ssl._tcp.linux.com
_etcd-server-ssl._tcp.linux.com. 0 IN SRV 0 100 2380 etcd2.linux.com.
_etcd-server-ssl._tcp.linux.com. 0 IN SRV 0 100 2380 etcd1.linux.com.
_etcd-server-ssl._tcp.linux.com. 0 IN SRV 0 100 2380 etcd3.linux.com.
[root@master etcd]# dig @172.21.130.169 +noall +answer SRV _etcd-client-ssl._tcp.linux.com
_etcd-client-ssl._tcp.linux.com. 0 IN SRV 0 100 2379 etcd2.linux.com.
_etcd-client-ssl._tcp.linux.com. 0 IN SRV 0 100 2379 etcd1.linux.com.
_etcd-client-ssl._tcp.linux.com. 0 IN SRV 0 100 2379 etcd3.linux.com.
[root@master etcd]# dig @172.21.130.169 +noall +answer etcd1.linux.com etcd2.linux.com etcd3.linux.com
etcd1.linux.com. 0 IN A 172.21.130.169
etcd2.linux.com. 0 IN A 172.21.130.168
etcd3.linux.com. 0 IN A 172.28.17.85
DNS SRV的记录格式：
_Service._Proto.Name TTL Class SRV Priority Weight Port Target
Service: 服务名称，前缀“_”是为防止与DNS Label（普通域名）冲突。
Proto: 服务使用的通信协议，_TCP、_UDP、其它标准协议或者自定义的协议。
Name: 提供服务的域名。
TTL: 缓存有效时间。
CLASS: 类别
Priority: 该记录的优先级，数值越小表示优先级越高，范围0-65535。
Weight: 该记录的权重，数值越高权重越高，范围0-65535。
Port: 服务端口号，0-65535。
Target: host 地址。
到这DNS SRV部分结束
考虑安全使用TLS部署
部署工具
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
利用CFSSL生成需要用到的证书
创建CA签名配置文件
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
#给服务端用的
"etcdserver": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
},
#给etcdctl查询信息用的
"etcdclient": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
#给member用的
"etcdpeer": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
创建签名请求文件
cat > ca-csr.json <<EOF
{
"CN": "ETCD CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
生成根证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
创建服务端的签名请求配置文件（提供给客户端进行校验）
[root@master etcd_ssl]# cat server.json
{
"CN": "ETCD CA",
"hosts": [
"*.linux.com",
"etcd1.linux.com",
"etcd2.linux.com",
"etcd3.linux.com"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
生成server证书与私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcdserver server.json | cfssljson -bare server
可以在生成一个member-csr.json给etcd成员之间使用，client就给etcdctl使用也可以（server-csr.json copy member-csr.json修改一下名字即可）
生成etcdctl客户端使用的证书
配置签名请求文件
[root@master etcd_ssl]# cat client.json
{
"CN": "ETCD CA",
"hosts": [
"*.linux.com",
"etcd1.linux.com",
"etcd2.linux.com",
"etcd3.linux.com"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcdclient client.json | cfssljson -bare client
查看证书内容
[root@master etcd_ssl]# openssl x509 -noout -text -in server.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
7d:5b:7f:2b:e0:13:90:f4:35:91:d2:5c:37:76:5d:a7:54:9e:bb:dc
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=CN, ST=Beijing, L=Beijing, CN=ETCD CA
Validity
Not Before: May 24 14:23:00 2021 GMT
Not After : May 22 14:23:00 2031 GMT
Subject: C=CN, ST=BeiJing, L=BeiJing, CN=ETCD CA
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:cf:b5:bf:ea:ad:02:26:62:09:9d:14:4a:70:c0:
65:55:70:9c:6c:1b:7e:7e:10:1b:31:85:a2:33:a9:
ed:5a:52:5a:e2:f5:bf:7f:15:c6:c2:25:45:11:12:
16:81:9b:a3:1c:3b:c9:94:d8:54:a4:4d:d6:74:0e:
3b:db:24:1d:63:c2:83:24:15:96:87:79:90:01:4f:
64:08:93:26:aa:34:74:32:57:bd:66:d1:83:88:f7:
cd:f7:37:86:c8:42:45:cb:c1:6e:25:0c:51:ae:a4:
08:d7:1c:81:70:d2:73:56:cb:f8:fa:c1:f5:73:79:
9c:8f:00:c7:39:e6:0f:f6:ba:07:52:73:e1:39:2f:
8e:70:1a:01:bd:f5:46:e6:6e:55:10:a2:bd:29:3d:
da:01:0c:f3:80:4b:8a:aa:4e:c5:75:6a:87:2f:50:
e7:5b:f7:61:17:28:14:f6:fb:f9:e5:97:07:bc:02:
1d:4f:6f:13:a3:71:2a:a9:61:e9:76:ff:05:f9:8d:
9c:f1:0f:12:f1:99:b2:0a:7b:fe:b8:0b:a5:59:29:
b8:83:09:71:fd:0e:26:67:7e:8a:e1:e4:76:2a:a2:
d0:d8:89:31:cc:b3:af:10:f2:92:04:4c:12:4e:d6:
98:67:2d:24:17:1d:c0:15:ac:19:7e:de:32:2b:33:
27:b1
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
16:5B:18:00:97:7E:1E:6D:84:9B:0E:1B:14:78:A7:C3:7B:C7:9C:D2
X509v3 Authority Key Identifier:
keyid:E4:E4:E6:CE:B2:61:8F:C9:54:F9:ED:29:A0:3B:12:E7:0D:C8:9D:EB
X509v3 Subject Alternative Name:
DNS:*.linux.com, DNS:etcd1.linux.com, DNS:etcd2.linux.com, DNS:etcd3.linux.com
Signature Algorithm: sha256WithRSAEncryption
ad:69:93:ad:67:0c:ce:71:57:58:8f:1d:d0:02:fa:6f:ac:f5:
d4:19:b2:b6:ae:5b:54:a5:47:dd:57:97:0d:06:7a:4c:eb:b8:
d2:63:56:21:50:b0:2f:a9:25:ee:ba:3f:95:c8:43:6a:85:ab:
46:f6:01:b6:4c:e6:ce:73:15:ca:e6:27:9a:fe:2f:d0:74:48:
40:97:ff:17:56:b2:ff:12:dd:02:17:4c:e5:0c:80:8f:31:ab:
da:6e:c6:00:80:3c:fc:66:89:33:7e:01:12:5c:0c:88:65:c3:
28:52:60:14:f8:a3:08:00:cc:ee:01:d0:7b:5b:6d:19:b2:55:
f0:a3:43:73:a4:fb:1f:43:92:ad:b6:f6:bc:3c:1c:4b:3e:d9:
9d:41:b3:2e:4f:27:9c:7c:ef:12:2c:57:0b:e9:c7:bf:10:70:
a0:9c:68:f3:b3:d0:4c:72:21:39:4b:a6:1e:53:5c:85:dd:df:
e6:bc:07:65:61:49:af:67:63:5b:e8:31:c4:97:dc:be:5f:d0:
c7:f1:f8:6b:d4:a1:a6:4a:2c:ef:52:1a:f5:49:56:af:9e:60:
c8:93:8f:92:21:fb:66:c9:d0:ea:51:db:6f:e6:d2:12:1e:6c:
8e:a1:14:c9:38:c7:4f:1a:1a:70:68:d9:8b:8a:1e:5a:f2:d9:
67:1e:4d:d0
[root@master etcd_ssl]#
查看所有证书文件
[root@master etcd_ssl]# pwd
/root/etcd_ssl
[root@master etcd_ssl]# ls
ca-config.json ca-csr.json ca.pem client.json client.pem member.json member.pem server.json server.pem
ca.csr ca-key.pem client.csr client-key.pem member.csr member-key.pem server.csr server-key.pem
证书到这，也就结束了
开始部署ETCD集群
配置文件结构
[root@master etcd_ssl]# tree /var/local/etcd
/var/local/etcd
├── bin
├── cfg
│ └── etcd.conf
├── data
└── ssl
├── ca-key.pem
├── ca.pem
├── client-key.pem
├── client.pem
├── member-key.pem
├── member.pem
├── server-key.pem
└── server.pem
8 directories, 12 files
bin就忽略吧，我为了不写环境变量给执行文件扔/usr/local/bin下了
编写配置文件
[root@master etcd_ssl]# cat /var/local/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/local/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"
#[Clustering]
ETCD_DISCOVERY_SRV="linux.com"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd1.linux.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd1.linux.com:2379"
ETCD_INITIAL_CLUSTER_TOKEN="my-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[security]
ETCD_CERT_FILE="/var/local/etcd/ssl/server.pem"
ETCD_KEY_FILE="/var/local/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/var/local/etcd/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/var/local/etcd/ssl/member.pem"
ETCD_PEER_KEY_FILE="/var/local/etcd/ssl/member-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/var/local/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CLIENT_CERT_AUTH="true"
[root@master2 ~]# cat /var/local/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/local/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"
#[Clustering]
ETCD_DISCOVERY_SRV="linux.com"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd2.linux.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd2.linux.com:2379"
ETCD_INITIAL_CLUSTER_TOKEN="my-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[security]
ETCD_CERT_FILE="/var/local/etcd/ssl/server.pem"
ETCD_KEY_FILE="/var/local/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/var/local/etcd/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/var/local/etcd/ssl/member.pem"
ETCD_PEER_KEY_FILE="/var/local/etcd/ssl/member-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/var/local/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CLIENT_CERT_AUTH="true"
[root@master1 ~]# cat /var/local/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/local/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"
#[Clustering]
ETCD_DISCOVERY_SRV="linux.com"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd3.linux.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd3.linux.com:2379"
ETCD_INITIAL_CLUSTER_TOKEN="my-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[security]
ETCD_CERT_FILE="/var/local/etcd/ssl/server.pem"
ETCD_KEY_FILE="/var/local/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/var/local/etcd/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/var/local/etcd/ssl/member.pem"
ETCD_PEER_KEY_FILE="/var/local/etcd/ssl/member-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/var/local/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CLIENT_CERT_AUTH="true"
编写启动文件
[root@master ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/var/local/etcd/cfg/etcd.conf
ExecStart=/usr/local/bin/etcd --auto-compaction-retention=1 \
--max-request-bytes=31457280 \
--quota-backend-bytes=1073741824 \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
到这ETCD配置也结束了，重启服务即可。
[root@master ~]# systemctl restart etcd && systemctl status etcd
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
Active: active (running) since 二 2021-05-25 00:45:43 CST; 5ms ago
Main PID: 5140 (etcd)
CGroup: /system.slice/etcd.service
└─5140 /usr/local/bin/etcd --auto-compaction-retention=1 --max-request-bytes=31457280 --quota-backend-bytes=1073741824 --logger=zap
5月 25 00:45:43 master1 etcd[5140]: {"level":"info","ts":"2021-05-25T00:45:43.267+0800","caller":"rafthttp/stream.go:425","msg":"established TC...111b52c"}
5月 25 00:45:43 master1 etcd[5140]: {"level":"info","ts":"2021-05-25T00:45:43.268+0800","caller":"raft/node.go:325","msg":"raft.node: 68359a3dd...term 85"}
5月 25 00:45:43 master1 etcd[5140]: {"level":"info","ts":"2021-05-25T00:45:43.269+0800","caller":"rafthttp/peer_status.go:51","msg":"peer becam...78e51f9"}
5月 25 00:45:43 master1 etcd[5140]: {"level":"info","ts":"2021-05-25T00:45:43.269+0800","caller":"rafthttp/stream.go:425","msg":"established TC...78e51f9"}
5月 25 00:45:43 master1 etcd[5140]: {"level":"info","ts":"2021-05-25T00:45:43.270+0800","caller":"rafthttp/stream.go:425","msg":"established TC...78e51f9"}
5月 25 00:45:43 master1 etcd[5140]: {"level":"info","ts":"2021-05-25T00:45:43.272+0800","caller":"rafthttp/stream.go:425","msg":"established TC...111b52c"}
5月 25 00:45:43 master1 etcd[5140]: {"level":"info","ts":"2021-05-25T00:45:43.288+0800","caller":"etcdserver/server.go:2037","msg":"published local memb...
5月 25 00:45:43 master1 etcd[5140]: {"level":"info","ts":"2021-05-25T00:45:43.289+0800","caller":"etcdserver/server.go:716","msg":"initialized peer conn...
5月 25 00:45:43 master1 etcd[5140]: {"level":"info","ts":"2021-05-25T00:45:43.290+0800","caller":"embed/serve.go:191","msg":"serving client tra...:]:2379"}
5月 25 00:45:43 master1 systemd[1]: Started Etcd Server.
Hint: Some lines were ellipsized, use -l to show in full.
查看状态
etcdctl环境变量
export ETCDCTL_ENDPOINTS=https://172.16.98.175:2379,https://172.16.98.176:2379,https://172.16.98.177:2379
export ETCDCTL_CACERT=/opt/etcd/ssl/ca.pem
export ETCDCTL_CERT=/opt/etcd/ssl/client.pem
export ETCDCTL_KEY=/opt/etcd/ssl/client-key.pem
或者可以这种
export ep=--endpoints=https://172.16.98.175:2379,https://172.16.98.176:2379,https://172.16.98.177:2379
[root@master etcd_ssl]# etcdctl ${epdns} endpoint status -w table
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://etcd1.linux.com:2379 | 414eb10a3111b52c | 3.4.16 | 20 kB | true | false | 88 | 19 | 19 | |
| https://etcd2.linux.com:2379 | b2f54def978e51f9 | 3.4.16 | 20 kB | false | false | 88 | 19 | 19 | |
| https://etcd3.linux.com:2379 | 68359a3ddf12fed8 | 3.4.16 | 20 kB | false | false | 88 | 19 | 19 | |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@master etcd_ssl]# cd
[root@master ~]# etcdctl ${epdns} endpoint status -w fields
"ClusterID" : 14982562468718563561
"MemberID" : 4705893317979780396
"Revision" : 1
"RaftTerm" : 88
"Version" : "3.4.16"
"DBSize" : 20480
"Leader" : 4705893317979780396
"IsLearner" : false
"RaftIndex" : 19
"RaftTerm" : 88
"RaftAppliedIndex" : 19
"Errors" : []
"Endpoint" : "https://etcd1.linux.com:2379"
"ClusterID" : 14982562468718563561
"MemberID" : 12895298799480492537
"Revision" : 1
"RaftTerm" : 88
"Version" : "3.4.16"
"DBSize" : 20480
"Leader" : 4705893317979780396
"IsLearner" : false
"RaftIndex" : 19
"RaftTerm" : 88
"RaftAppliedIndex" : 19
"Errors" : []
"Endpoint" : "https://etcd2.linux.com:2379"
"ClusterID" : 14982562468718563561
"MemberID" : 7509077544236416728
"Revision" : 1
"RaftTerm" : 88
"Version" : "3.4.16"
"DBSize" : 20480
"Leader" : 4705893317979780396
"IsLearner" : false
"RaftIndex" : 19
"RaftTerm" : 88
"RaftAppliedIndex" : 19
"Errors" : []
"Endpoint" : "https://etcd3.linux.com:2379"
[root@master ~]# etcdctl ${epdns} put ok 666
OK
[root@master ~]#
到此本文结束。
有一点小细节，其实DNS发现部署ETCD集群还有一个小特点，它其实定义了一个-discovery-srv-name（只针对member有效，可以用来区分同一个域下的多个集群使用，类似于ETCD_INITIAL_CLUSTER_TOKEN的作用）的参数，在启动服务的时候会联合你定义的-discovery-srv参数一起查询
实际就是在查询SRV的时候，前缀找的是两者集合，不然就会报错。
下面是一小段日志（可以看到有用的验证理论信息）
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://etcd2.linux.com:2379
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_CERT_FILE=/var/local/etcd/ssl/server.pem
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
5月 27 10:14:06 master2 etcd[11297]: {"level":"info","ts":"2021-05-27T10:14:06.846+0800","caller":"embed/etcd.go:117","msg":"configuring peer listeners","listen-peer-urls":["https://0.0.0.0:2380"]}
5月 27 10:14:06 master2 etcd[11297]: {"level":"info","ts":"2021-05-27T10:14:06.846+0800","caller":"embed/etcd.go:469","msg":"starting with peer TLS","tls-info":"cert = /var/local/etcd/ssl/member.pem, key = /var/local/etcd/ssl/member-key.pem, trusted-ca = /var/local/etcd/ssl/ca.pem, client-cert-auth = true, crl-file = ","cipher-suites":[]}
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_DATA_DIR=/var/local/etcd/data/default.etcd
5月 27 10:14:06 master2 etcd[11297]: {"level":"info","ts":"2021-05-27T10:14:06.846+0800","caller":"embed/etcd.go:127","msg":"configuring client listeners","listen-client-urls":["https://0.0.0.0:2379"]}
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_DISCOVERY_SRV=linux.com
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_DISCOVERY_SRV_NAME=my-etcd-cluster
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd2.linux.com:2380
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=my-etcd-cluster
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_KEY_FILE=/var/local/etcd/ssl/server-key.pem
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_NAME=etcd2
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_PEER_CERT_FILE=/var/local/etcd/ssl/member.pem
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_PEER_KEY_FILE=/var/local/etcd/ssl/member-key.pem
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/var/local/etcd/ssl/ca.pem
5月 27 10:14:06 master2 etcd[11297]: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/var/local/etcd/ssl/ca.pem
5月 27 10:14:06 master2 etcd[11297]: {"level":"info","ts":"2021-05-27T10:14:06.851+0800","caller":"embed/config.go:705","msg":"get cluster for etcd-server-ssl SRV","service-scheme":"https","service-name":"etcd-server-ssl-my-etcd-cluster","server-name":"etcd2","discovery-srv":"linux.com","advertise-peer-urls":["https://etcd2.linux.com:2380"],"found-cluster":[],"error":"error querying DNS SRV records for _etcd-server-ssl-my-etcd-cluster lookup _etcd-server-ssl-my-etcd-cluster._tcp.linux.com on 172.21.130.169:53: server misbehaving"}
5月 27 10:14:06 master2 etcd[11297]: {"level":"info","ts":"2021-05-27T10:14:06.853+0800","caller":"embed/config.go:722","msg":"get cluster for etcd-server SRV","service-scheme":"http","service-name":"etcd-server-my-etcd-cluster","server-name":"etcd2","discovery-srv":"linux.com","advertise-peer-urls":["https://etcd2.linux.com:2380"],"found-cluster":[],"error":"error querying DNS SRV records for _etcd-server-my-etcd-cluster lookup _etcd-server-my-etcd-cluster._tcp.linux.com on 172.21.130.169:53: server misbehaving"}
5月 27 10:14:06 master2 etcd[11297]: {"level":"warn","ts":"2021-05-27T10:14:06.853+0800","caller":"embed/config.go:652","msg":"failed to resolve during SRV discovery","error":"error querying DNS SRV records for _etcd-server-ssl-my-etcd-cluster lookup _etcd-server-ssl-my-etcd-cluster._tcp.linux.com on 172.21.130.169:53: server misbehaving"}
5月 27 10:14:06 master2 etcd[11297]: {"level":"info","ts":"2021-05-27T10:14:06.853+0800","caller":"embed/etcd.go:364","msg":"closing etcd server","name":"etcd2","data-dir":"/var/local/etcd/data/default.etcd","advertise-peer-urls":["https://etcd2.linux.com:2380"],"advertise-client-urls":["https://etcd2.linux.com:2379"]}
5月 27 10:14:06 master2 etcd[11297]: {"level":"info","ts":"2021-05-27T10:14:06.853+0800","caller":"embed/etcd.go:368","msg":"closed etcd server","name":"etcd2","data-dir":"/var/local/etcd/data/default.etcd","advertise-peer-urls":["https://etcd2.linux.com:2380"],"advertise-client-urls":["https://etcd2.linux.com:2379"]}
5月 27 10:14:06 master2 etcd[11297]: {"level":"warn","ts":"2021-05-27T10:14:06.853+0800","caller":"etcdmain/etcd.go:176","msg":"failed to start etcd","error":"error setting up initial cluster: error querying DNS SRV records for _etcd-server-ssl-my-etcd-cluster lookup _etcd-server-ssl-my-etcd-cluster._tcp.linux.com on 172.21.130.169:53: server misbehaving"}
5月 27 10:14:06 master2 etcd[11297]: {"level":"fatal","ts":"2021-05-27T10:14:06.853+0800","caller":"etcdmain/etcd.go:271","msg":"discovery failed","error":"error setting up initial cluster: error querying DNS SRV records for _etcd-server-ssl-my-etcd-cluster lookup _etcd-server-ssl-my-etcd-cluster._tcp.linux.com on 172.21.130.169:53: server misbehaving","stacktrace":"go.etcd.io/etcd/etcdmain.startEtcdOrProxyV2\n\t/tmp/etcd-release-3.4.16/etcd/release/etcd/etcdmain/etcd.go:271\ngo.etcd.io/etcd/etcdmain.Main\n\t/tmp/etcd-release-3.4.16/etcd/release/etcd/etcdmain/main.go:46\nmain.main\n\t/tmp/etcd-release-3.4.16/etcd/release/etcd/main.go:28\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
5月 27 10:14:06 master2 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
5月 27 10:14:06 master2 systemd[1]: Failed to start Etcd Server.
5月 27 10:14:06 master2 systemd[1]: Unit etcd.service entered failed state.
5月 27 10:14:06 master2 systemd[1]: etcd.service failed.
5月 27 10:14:07 master2 systemd[1]: etcd.service holdoff time over, scheduling restart.
5月 27 10:14:07 master2 systemd[1]: Stopped Etcd Server.
5月 27 10:14:07 master2 systemd[1]: start request repeated too quickly for etcd.service
5月 27 10:14:07 master2 systemd[1]: Failed to start Etcd Server.
5月 27 10:14:07 master2 systemd[1]: Unit etcd.service entered failed state.
5月 27 10:14:07 master2 systemd[1]: etcd.service failed.
下面是DNS SRV信息
[root@master ~]# dig @172.21.130.169 +noall +answer SRV _etcd-server-ssl-my-etcd-cluster._tcp.linux.com
_etcd-server-ssl-my-etcd-cluster._tcp.linux.com. 0 IN SRV 0 100 2380 etcd3.linux.com.
_etcd-server-ssl-my-etcd-cluster._tcp.linux.com. 0 IN SRV 0 100 2380 etcd2.linux.com.
_etcd-server-ssl-my-etcd-cluster._tcp.linux.com. 0 IN SRV 0 100 2380 etcd1.linux.com.
感谢观看
作者 K&。本博客都是作者手动编写，部分摘自网络。极大部分原创。
