kubernetes使用外部etcd kubernetes etcd集群

转载

mob64ca1401464d 2023-11-29 12:16:53

文章标签 kubernetes使用外部etcd kubernetes json Time linux 文章分类 kubernetes 云计算

一、集群规划

k8s有两种集群架构模式：单Master集群和多Master集群；实际生产环境中需部署多Master集群，以确保高可用；目前由于资源限制和仅作为学习测试用，下面就搭建单Master集群架构。

本人对单Master集群模式的机器规划：一台Master节点，三台Node工作节点，etcd数据库分别安装在Master、Node1和Node2节点上。

各角色的分配以及需要安装的组件如下图：

kubernetes使用外部etcd kubernetes etcd集群_kubernetes使用外部etcd

二、初始化操作

1.首先要关闭swap分区，这是从性能方面考虑的，因为当内存不足时，系统会自动调用swap分区，将部分内存数据存放到磁盘中，性能会下降。

kubernetes使用外部etcd kubernetes etcd集群_kubernetes_02

2.关闭防火墙：

[root@k8s-master-1 ~]# systemctl stop firewalld
[root@k8s-master-1 ~]# systemctl disable firewalld

3.关闭selinux：

[root@k8s-master-1 ~]# sed -i 's/enforcing/disabled/' /etc/selinux/config

4.集群内所有机器都要进行时间同步：

[root@k8s-master-1 ~]# yum install -y ntpdate
[root@k8s-master-1 ~]# ntpdate time.windows.com
14 Jan 10:09:17 ntpdate[2597]: adjust time server 20.189.79.72 offset -0.008674 sec
[root@k8s-master-1 ~]# date
Fri Jan 14 10:09:30 EST 2022

三、Etcd集群搭建

在Master、Node1和Node2上安装Etcd数据库。

1.下载cfssl工具来生成证书：

[root@k8s-master-1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
100   654  100   654    0     0    157      0  0:00:04  0:00:04 --:--:--   157
100  9.8M  100  9.8M    0     0   133k      0  0:01:15  0:01:15 --:--:-- 89828

[root@k8s-master-1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   658  100   658    0     0    452      0  0:00:01  0:00:01 --:--:--   452
100 2224k  100 2224k    0     0   162k      0  0:00:13  0:00:13 --:--:--  154k

[root@k8s-master-1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   663  100   663    0     0    566      0  0:00:01  0:00:01 --:--:--   566
100 6440k  100 6440k    0     0   313k      0  0:00:20  0:00:20 --:--:--  195k

给文件执行权限：

[root@k8s-master-1 ~]# chmod +x /usr/local/bin/cfssl*

2.自签Etcd SSL证书：

（1）需要创建以下目录：

/k8s/etcd/ssl（存放etcd的自签证书）
/k8s/etcd/cfg（存放etcd的配置文件）
/k8s/etcd/bin（存放etcd的执行程序）

[root@k8s-master-1 ~]# mkdir -p /k8s/etcd/{ssl,cfg,bin}

（2）创建CA配置文件：ca-config.json

[root@k8s-master-1 ~]# cd /k8s/etcd/ssl
[root@k8s-master-1 ssl]# vim ca-config.json
[root@k8s-master-1 ssl]# cat ca-config.json 
{
  "signing": {    #表示该证书可用于签名其它证书，生成的ca.pem证书中的CA=TRUE
    "default": {    #可以定义多个profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个profile
      "expiry": "87600h"    #证书过期时间
    },
    "profiles": {
      "etcd": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",    #表示client可以用该 CA 对server提供的证书进行验证
            "client auth"    #表示server可以用该CA对client提供的证书进行验证；
        ],
        "expiry": "87600h"
      }
    }
  }
}

（3）创建CA证书签名请求文件：ca-scr.json

[root@k8s-master-1 ssl]# vim ca-csr.json 
[root@k8s-master-1 ssl]# cat ca-csr.json 
{
  "CN": "etcd",    #Common Name，kube-apiserver从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法
  "key": {    #加密算法
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",    #国家
      "ST": "Shanghai",    #地区
      "L": "Shanghai",    #城市
      "O": "etcd",    #组织，kube-apiserver从证书中提取该字段作为请求用户所属的组(Group)
      "OU": "System"    #组织单位
    }
  ],
    "ca": {
       "expiry": "87600h"
    }
}

（4）生成CA证书和私钥

[root@k8s-master-1 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2022/01/15 04:05:40 [INFO] generating a new CA key and certificate from CSR
2022/01/15 04:05:40 [INFO] generate received request
2022/01/15 04:05:40 [INFO] received CSR
2022/01/15 04:05:40 [INFO] generating key: rsa-2048
2022/01/15 04:05:40 [INFO] encoded CSR
2022/01/15 04:05:40 [INFO] signed certificate with serial number 429705018602797242754573367192655674978633104892

[root@k8s-master-1 ssl]# ll
total 20
-rw-r--r-- 1 root root  286 Jan 15 04:00 ca-config.json
-rw-r--r-- 1 root root  997 Jan 15 04:05 ca.csr
-rw-r--r-- 1 root root  250 Jan 15 04:01 ca-csr.json
-rw------- 1 root root 1679 Jan 15 04:05 ca-key.pem（CA私钥）
-rw-r--r-- 1 root root 1350 Jan 15 04:05 ca.pem（CA数字证书）

（5）创建证书签名请求文件：etcd-csr.json

root@k8s-master-1 ssl]# cat > etcd-csr.json <<EOF
> {
>     "CN": "etcd",
>     "hosts": [    #需要指定授权使用该证书的主机IP或域名列表，这里配置所有Etcd的IP地址
>       "192.168.61.161",
>       "192.168.61.162",
>       "192.168.61.163"
>     ],
>     "key": {    #加密算法长度
>         "algo": "rsa",
>         "size": 2048
>     },
>     "names": [
>         {
>             "C": "CN",
>             "ST": "BeiJing",
>             "L": "BeiJing",
>             "O": "etcd",
>             "OU": "System"
>         }
>     ]
> }
> EOF

为Etcd生成证书和私钥：

[root@k8s-master-1 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd
2022/01/15 04:12:53 [INFO] generate received request
2022/01/15 04:12:53 [INFO] received CSR
2022/01/15 04:12:53 [INFO] generating key: rsa-2048
2022/01/15 04:12:53 [INFO] encoded CSR
2022/01/15 04:12:53 [INFO] signed certificate with serial number 169134289353272844556402247131396922483630866064
2022/01/15 04:12:53 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

etcd-key.pem：etcd的私钥
etcd.pem：etcd的数字证书

kubernetes使用外部etcd kubernetes etcd集群_kubernetes_03

在后面部署Etcd集群时会用到下面几个关键的证书文件：

[root@k8s-master-1 ssl]# ls *pem
ca-key.pem  ca.pem  etcd-key.pem  etcd.pem

3.Etcd数据库集群部署

Etcd数据库集群采用主从架构模式部署（一主多从），部署奇数台节点，集群会通过选举产生Leader。使用Raft一致性算法保证每个节点的一致性。

kubernetes使用外部etcd kubernetes etcd集群_Time_04

（1）下载etcd，从GitHub上下载合适版本的etcd，并且解压：

[root@k8s-master-1 ~]# cd /k8s/etcd/
[root@k8s-master-1 etcd]# wget https://github.com/etcd-io/etcd/releases/download/v3.2.28/etcd-v3.2.28-linux-amd64.tar.gz

[root@k8s-master-1 etcd]# tar zxf etcd-v3.2.28-linux-amd64.tar.gz

将etcd复制到/usr/local/bin目录下：

[root@k8s-master-1 etcd]# cp etcd-v3.2.28-linux-amd64/{etcd,etcdctl} /k8s/etcd/bin

（2）创建Etcd配置文件：etcd.conf

[root@k8s-master-1 ~]# cd /k8s/etcd/cfg/
[root@k8s-master-1 cfg]# vim etcd.conf 
[root@k8s-master-1 cfg]# cat etcd.conf 
# [member]
ETCD_NAME=etcd-1
ETCD_DATA_DIR=/k8s/data/default.etcd
ETCD_LISTEN_PEER_URLS=https://192.168.61.161:2380
ETCD_LISTEN_CLIENT_URLS=https://192.168.61.161:2379

# [cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.61.161:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.61.161:2379
ETCD_INITIAL_CLUSTER=etcd-1=https://192.168.61.161:2380,etcd-2=https://192.168.61.162:2380,etcd-3=https://192.168.61.163:2380
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_INITIAL_CLUSTER_STATE=new

# [security]
ETCD_CERT_FILE=/k8s/etcd/ssl/etcd.pem
ETCD_KEY_FILE=/k8s/etcd/ssl/etcd-key.pem
ETCD_TRUSTED_CA_FILE=/k8s/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/k8s/etcd/ssl/etcd.pem
ETCD_PEER_KEY_FILE=/k8s/etcd/ssl/etcd-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/k8s/etcd/ssl/ca.pem

（3）创建Etcd服务：etcd.service

[root@k8s-master-1 etcd]# vim etcd.service
[root@k8s-master-1 etcd]# cat etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/k8s/etcd/cfg/etcd.conf    #指定etcd.conf作为环境配置文件
WorkingDirectory=${ETCD_DATA_DIR}

ExecStart=/k8s/etcd/bin/etcd \
  --name=${ETCD_NAME} \
  --data-dir=${ETCD_DATA_DIR} \
  --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
  --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
  --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
  --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
  --initial-cluster=${ETCD_INITIAL_CLUSTER} \
  --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
  --initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
  --cert-file=${ETCD_CERT_FILE} \
  --key-file=${ETCD_KEY_FILE} \
  --trusted-ca-file=${ETCD_TRUSTED_CA_FILE} \
  --peer-cert-file=${ETCD_PEER_CERT_FILE} \
  --peer-key-file=${ETCD_PEER_KEY_FILE} \
  --peer-trusted-ca-file=${ETCD_PEER_TRUSTED_CA_FILE}

Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

（4）将Etcd目录拷贝到另外两个节点：

[root@k8s-master-1 etcd]# scp -r /k8s root@k8s-node-1:/k8s

[root@k8s-master-1 etcd]# scp -r /k8s root@k8s-node-2:/k8s

（5）修改两个节点的etcd.conf配置文件：

[root@k8s-node-1 cfg]# vim etcd.conf 
[root@k8s-node-1 cfg]# cat etcd.conf 
# [member]
ETCD_NAME=etcd-2
ETCD_DATA_DIR=/k8s/data/default.etcd
ETCD_LISTEN_PEER_URLS=https://192.168.61.162:2380
ETCD_LISTEN_CLIENT_URLS=https://192.168.61.162:2379

# [cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.61.162:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.61.162:2379
ETCD_INITIAL_CLUSTER=etcd-1=https://192.168.61.161:2380,etcd-2=https://192.168.61.162:2380,etcd-3=https://192.168.61.163:2380
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_INITIAL_CLUSTER_STATE=new

# [security]
ETCD_CERT_FILE=/k8s/etcd/ssl/etcd.pem
ETCD_KEY_FILE=/k8s/etcd/ssl/etcd-key.pem
ETCD_TRUSTED_CA_FILE=/k8s/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/k8s/etcd/ssl/etcd.pem
ETCD_PEER_KEY_FILE=/k8s/etcd/ssl/etcd-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/k8s/etcd/ssl/ca.pem



[root@k8s-node-2 cfg]# vim etcd.conf 
[root@k8s-node-2 cfg]# cat etcd.conf 
# [member]
ETCD_NAME=etcd-3
ETCD_DATA_DIR=/k8s/data/default.etcd
ETCD_LISTEN_PEER_URLS=https://192.168.61.163:2380
ETCD_LISTEN_CLIENT_URLS=https://192.168.61.163:2379

# [cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.61.163:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.61.163:2379
ETCD_INITIAL_CLUSTER=etcd-1=https://192.168.61.161:2380,etcd-2=https://192.168.61.162:2380,etcd-3=https://192.168.61.163:2380
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_INITIAL_CLUSTER_STATE=new

# [security]
ETCD_CERT_FILE=/k8s/etcd/ssl/etcd.pem
ETCD_KEY_FILE=/k8s/etcd/ssl/etcd-key.pem
ETCD_TRUSTED_CA_FILE=/k8s/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/k8s/etcd/ssl/etcd.pem
ETCD_PEER_KEY_FILE=/k8s/etcd/ssl/etcd-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/k8s/etcd/ssl/ca.pem

（6）启动Etcd服务：将etcd.service拷贝到/usr/lib/systemd/system/目录下，启动Etcd服务

[root@k8s-master-1 etcd]# cp /k8s/etcd/etcd.service /usr/lib/systemd/system/
[root@k8s-master-1 etcd]# systemctl daemon-reload
[root@k8s-master-1 etcd]# systemctl start etcd
[root@k8s-master-1 etcd]# systemctl enable etcd


[root@k8s-node-1 etcd]# cp /k8s/etcd/etcd.service /usr/lib/systemd/system/
[root@k8s-node-1 etcd]# systemctl daemon-reload
[root@k8s-node-1 etcd]# systemctl start etcd
[root@k8s-node-1 etcd]# systemctl enable etcd

[root@k8s-node-2 etcd]# cp /k8s/etcd/etcd.service /usr/lib/systemd/system/
[root@k8s-node-2 etcd]# systemctl daemon-reload
[root@k8s-node-2 etcd]# systemctl start etcd
[root@k8s-node-2 etcd]# systemctl enable etcd

查看Etcd的日志命令：tail -f /var/log/messages

查看Etcd集群状态命令：/k8s/etcd/bin/etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/etcd.pem --key-file=/k8s/etcd/ssl/etcd-key.pem --endpoints=https://192.168.61.161:2379,https://192.168.61.162:2379,https://192.168.61.163:2379 cluster-health

[root@k8s-master-1 etcd]# /k8s/etcd/bin/etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/etcd.pem --key-file=/k8s/etcd/ssl/etcd-key.pem --endpoints=https://192.168.61.161:2379,https://192.168.61.162:2379,https://192.168.61.163:2379 cluster-health
member 4166e968fa162f83 is healthy: got healthy result from https://192.168.61.161:2379
member cdade95107acd749 is healthy: got healthy result from https://192.168.61.163:2379
member cecb7c331cf85085 is healthy: got healthy result from https://192.168.61.162:2379
cluster is healthy

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。