基于containerd 部署 kubernetes 1.28集群

原创

KimiJW 2024-03-21 17:53:31 博主文章分类：Kubernetes ©著作权

文章标签 containerd runc kubernetes calico 文章分类 运维

©著作权归作者所有：来自51CTO博客作者KimiJW的原创作品，请联系作者获取转载授权，否则将追究法律责任

准备说明

8台Linux主机，安装Ubuntu 20.04系统，其中2台haproxy，3台master节点，3台work节点
每台主机不低于2GB 内存大小，CPU大于2核心
集群中的所有主机网络互通
节点中不能存在重复的主机名、mac地址或者product_uuid
交换分区配置。kubelet默认是在节点上检测到交换分区时，无法启动kubelet。kubelet从1.22版本开始支持交换分区。1.28版本，仅针对cgroup v2支持交换分区。如果kubelet没有配置好对swap分区的支持，则必须进行关闭交换分区swapoff -a，进行暂时关于交换分区。

确保节点的mac地址和product_uuid的唯一性

#每一个节点进行查看
root@K8SMS0001:~# ip link	#查看mac地址
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:b6:97:cd brd ff:ff:ff:ff:ff:ff

root@K8SMS0001:~# cat /sys/class/dmi/id/product_uuid	#查看uuid的唯一
42364795-73ca-edde-a606-9ab6b26b6516

主机系统参数配置

主机名配置说明

主机名	角色	ip地址
K8SHA0001	HA主节点	10.64.143.31
K8SHA0002	HA从节点	10.64.143.32
K8SMS0001	K8S-master01	10.64.143.33
K8SMS0002	K8S-master02	10.64.143.34
K8SMS0003	K8S-master03	10.64.143.35
K8SWK0001	K8S-worker01	10.64.143.36
K8SWK0002	K8S-worker02	10.64.143.37
K8SWK0003	K8S-worker03	10.64.143.38

hosts解析配置

# 每个节点进行配置
$ cat >> /etc/hosts << EOF
K8SHA0001 10.64.143.31
K8SHA0002 10.64.143.32
K8SMS0001 10.64.143.33
K8SMS0002 10.64.143.34
K8SMS0003 10.64.143.35
K8SWK0001 10.64.143.36
K8SWK0002 10.64.143.37
K8SWK0003 10.64.143.38

selinux、防火墙关闭

$ systemctl stop firewalld
$ firewall-cmd --state
$ setenforce 0
$ getenforce
$ sed -i 's/SELINUX=enforceing/SELINUX=disabled/g' /etc/selinux/config

时间同步

$ crontab -e
*/5 * * * * ntpdate time1.aliyun.com
$ ntpdate time1.aliyun.com

配置内核转发和网桥过滤

# 添加网桥过滤及内核转发配置文件
# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF

加载br_netfilter模块
# modprobe br_netfilter
# sysctl -p /etc/sysctl.d/k8s.conf

#查看是否加载	
# lsmod |grep br_netfilter	
br_netfilter           28672  0
bridge                176128  1 br_netfilter
# lsmod | grep overlay
overlay               118784  0

# sysctl -a |grep 'bridge-nf-call-ip\|ip_forward = 1'
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

安装ipset和ipvsadm

# yum install -y ipset ipvsadm

配置ipvsadm模块加载
# cat >> /etc/sysconfig/modules/ipvs.modules << EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

# 授权并加载模块
# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod |grep -e ip_vs -e nf_conntrack

关闭SWAP分区

# swappoff -a

HAproxy和keepalived安装配置

可以参考1.22.8的配置

# yum install haproxy keepalived -y

容器运行时安装

下载cri-containerd

github containerd地址 cri-containerd-1.7.14版本下载

# 下载cri-containerd 1.7.14版本
# wget https://github.com/containerd/containerd/releases/download/v1.7.14/cri-containerd-1.7.14-linux-amd64.tar.gz

二进制安装cri-containerd

# mkdir containerd && tar -zxf cri-containerd-1.7.14-linux-amd64.tar.gz -C containerd/ && cp ./usr/local/bin/* /usr/local/bin/ && cp ./etc/systemd/system/containerd.service /usr/lib/systemd/system/ && systemctl daemon-reload && containerd --version

生成并配置containerd默认配置文件

# 生成containerd配置文件，然后/etc/containerd/config.toml中修改如下配置项：
# mkdir /etc/containerd && containerd config default > /etc/containerd/config.toml
# vim /etc/containerd/config.toml
......          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
......
            SystemdCgroup = true	#修改SystemdCgroup
[plugins."io.containerd.grpc.v1.cri"]
......
    #修改pause镜像路径
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"	
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
......
    runtime_type = 'io.containerd.runc.v2'	#修改默认的runtime_type，否则后面crictl image存在报错信息

# 重启containerd服务
# systemctl restart containerd
# systemctl enable containerd

下载并安装runc

github runc下载地址 runc 1.1.12版本下载

# wget https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64
# mv runc.amd64 /usr/sbin/runc && chmod +x /usr/sbin/runc && runc -v

安装kubeadm、kubelet、kubectl

需要在每台机器上安装以下的软件包

kubeadm：用来初始化集群的指令。
kubelet：在集群中的每个节点上用来启动 Pod 和容器等。
kubectl：用来与集群通信的命令行工具。 kubeadm 无法进行安装或者管理 kubelet 或 kubectl，所以需要确保它们与通过 kubeadm 安装的控制平面的版本相匹配。如果不这样做，则存在发生版本偏差的风险，可能会导致一些预料之外的错误和问题。

增加阿里云的kubernetes源

# cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/rpm/repodata/repomd.xml.key
EOF

# yum clean all
# yum makecache

安装指定1.28.8版本kubeadm、kubectl、kubelet

# yum --showduplicate list kubeadm kubectl kublet

# yum install -y kubelet-1.28.8 kubeadm-1.28.8 kubectl-1.28.8

修改kubelet默认cgroup驱动

# vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"

# systemctl enable kubelet --now

初始化集群

查看初始化需要的镜像并下载

镜像所有的节点都先下好

# kubeadm config image list

# crictl pull registry.aliyuncs.com/google_containers/pause:3.9
# crictl pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.8
# crictl pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.8
# crictl pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.8
# crictl pull registry.aliyuncs.com/google_containers/kube-proxy:v1.28.8
# crictl pull registry.aliyuncs.com/google_containers/etcd:3.5.12-0

集群初始化

root@K8SMS0001:~# kubeadm init --control-plane-endpoint 10.64.143.30:6443 --kubernetes-version=v1.28.8 --pod-network-cidr 172.16.0.0/16 --service-cidr 10.96.0.0/16 --service-dns-domain cluster.local --image-repository registry.aliyuncs.com/google_containers --upload-certs
[init] Using Kubernetes version: v1.28.8
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local sphqjdk8sms0001] and IPs [10.96.0.1 10.64.143.33 10.64.143.30]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost sphqjdk8sms0001] and IPs [10.64.143.33 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost sphqjdk8sms0001] and IPs [10.64.143.33 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.037389 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node sphqjdk8sms0001 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node sphqjdk8sms0001 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: xy67nn.3yagma81us8dzatp
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 10.64.143.30:6443 --token xy67nn.3yagma81us8dzatp \
        --discovery-token-ca-cert-hash sha256:e4c4f8a7495c9f3a07e208200f355f1f6c97cfb7c5cd562f6758dbf172acef96 \
        --control-plane 

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.64.143.30:6443 --token xy67nn.3yagma81us8dzatp \
        --discovery-token-ca-cert-hash sha256:e4c4f8a7495c9f3a07e208200f355f1f6c97cfb7c5cd562f6758dbf172acef96

拷贝配置

# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config
# kubectl get node

master集群加入和work节点集群加入

master节点加入指令

  kubeadm join 10.64.143.30:6443 --token xy67nn.3yagma81us8dzatp \
        --discovery-token-ca-cert-hash sha256:e4c4f8a7495c9f3a07e208200f355f1f6c97cfb7c5cd562f6758dbf172acef96 \
        --control-plane

work节点加入指令

kubeadm join 10.64.143.30:6443 --token xy67nn.3yagma81us8dzatp \
        --discovery-token-ca-cert-hash sha256:e4c4f8a7495c9f3a07e208200f355f1f6c97cfb7c5cd562f6758dbf172acef96

calico网络插件部署

官方指导文档

安装calico自定义资源

# kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/tigera-operator.yaml

下载calico基础配置

# wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/custom-resources.yaml

# vim custom-resources.yaml
修改pod地址段，和初始化保持一致
cidr: 172.16.0.0/16

# watch kubectl get pods -n calico-system

calico相关pod都正常running后，集群就正常了。