k8s es加设置当前节点与其他节点交互的IP地址 k8s添加节点

一、Kubernetes集群部署方式

https://mp.weixin.qq.com/s/P90Z_rudmI548aOfD9aRtA  二进制版

https://github.com/easzlab/kubeasz   二进制部署工具

   ansible部署kubeadm版

二、Kubeadm部署k8s集群

https://kuboard.cn/install/install-kubernetes.html (kubeadm方式构建多master集群)

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/ (高可用)

三、环境准备(docker环境)

hostnamectl set-hostname k8s-master

cat /etc/hosts
192.168.40.137  k8s-master
192.168.40.138  k8s-node1
192.168.40.139  k8s-node2

systemctl disable firewalld
systemctl stop  firewalld

sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux

sed -i 's/.*swap.*/#&/' /etc/fstab
swapoff -a

update-alternatives --set iptables /usr/sbin/iptables-legacy

yum install -y yum-utils device-mapper-persistent-data lvm2 && yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && yum makecache && yum -y install docker-ce -y && systemctl enable docker.service && systemctl start docker# hostnamectl set-hostname k8s-master

所有机器安装kubeadm和kubelet

配置aliyun的yum源
# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装最新版kubeadm
yum makecache
yum install -y kubelet kubeadm kubectl ipvsadm

说明：如果想安装指定版本的kubeadmin
yum install kubelet-1.16.0-0.x86_64 kubeadm-1.16.0-0.x86_64 kubectl-1.16.0-0.x86_64

配置内核参数

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF

sysctl --system
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

加载ipvs相关内核模块
如果重新开机，需要重新加载(可以写在 /etc/rc.local 中开机自动加载)
vim 1.sh
#！/bin/bash
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4

查看是否加载成功
lsmod | grep ip_vs

四、获取镜像

特别说明：

• 三个节点都要下载
• 注意下载时把版本号修改到官方最新版，即使下载了最新版也可能版本不对应，需要按报错提示下载
• 每次部署都会有版本更新，具体版本要求，运行初始化过程失败会有版本提示
• kubeadm的版本和镜像的版本必须是对应的

用命令查看版本当前kubeadm对应的k8s镜像版本

# kubeadm config images list  
k8s.gcr.io/kube-apiserver:v1.17.2 
k8s.gcr.io/kube-controller-manager:v1.17.2 
k8s.gcr.io/kube-scheduler:v1.17.2 
k8s.gcr.io/kube-proxy:v1.17.2 
k8s.gcr.io/pause:3.1 
k8s.gcr.io/etcd:3.4.3-0 
k8s.gcr.io/coredns:1.6.5

使用下面的方法在aliyun拉取相应的镜像并重新打标

docker pull  registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.2

docker tag   registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.2  http://k8s.gcr.io/kube-apiserver:v1.17.2

docker  rmi 删除镜像

五、所有节点配置启动kubelet

配置kubelet使用国内pause镜像

获取docker的cgroups

DOCKER_CGROUPS=$( docker info | grep 'Cgroup' | awk -F':' '{print $2}')
echo $DOCKER_CGROUPS
cgroupfs

配置kubelet的cgroups

#cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS --pod-infra-container-image=k8s.gcr.io/pause:3.2"
EOF
注意这里的版本，和驱动！！！！！！

systemctl daemon-reload
systemctl enable kubelet && systemctl start kubelet

特别说明：在这里使用systemctl status kubelet，你会发现报错误信息
10月 11 00:26:43 node1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
10月 11 00:26:43 node1 systemd[1]: Unit kubelet.service entered failed state.
10月 11 00:26:43 node1 systemd[1]: kubelet.service failed.
运行journalctl -xefu kubelet 命令查看systemd日志才发现，真正的错误是：
 unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory
这个错误在运行kubeadm init 生成CA证书后会被自动解决，此处可先忽略。
简单地说就是在kubeadm init 之前kubelet会不断重启。

六、初始化集群

初始化完成必须要记录下初始化过程最后的命令

# kubeadm init --kubernetes-version=v1.17.4 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.40.137 --ignore-preflight-errors=Swap

--pod-network-cidr：如果创建集群时指定了该参数，那么 kube-proxy 的 cluster-cidr 就会被设置成该值(不指定该参数时，cluster-cidr 的默认值为空)。
cluster-cidr 主要用于帮助 kube-proxy 区分内外流量：当值为空时，kube-proxy 认为所有流量都是内部流量，不做 SNAT(MASQ)；当值非空时，来自 cluster-cidr 网络
(即 Pod 网络)的流量被当成内部流量，访问 Service 时不做 SNAT(MASQ)，来自其他网络的流量被当成外部流量，访问 Service 时需要做 SNAT(MASQ)。此外，kube-proxy 
还提供了一个独立参数 masquerade-all，如果设置了该参数，那所有访问 Service 的流量都会做 SNAT(MASQ)，不再区分内外流量。

后期修改--pod-network-cidr=10.244.0.0/16地址段，
kubectl -n kube-system edit cm kubeadm-config
vim /etc/kubernetes/manifests/kube-scheduler.yaml
通过 kubectl cluster-info dump | grep -m 1 cluster-cidr 命令可以检查配置是否生效

kubectl -n kube-system edit cm kubeadm-config
vim /etc/kubernetes/manifests/kube-scheduler.yaml
kubectl cluster-info dump | grep -m 1 cluster-cidr

失败执行：kubeadm reset
报错The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.

#vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false"

#systemctl daemon-reload
#systemctl restart kubelet

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
 https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:
 kubeadm join 192.168.1.200:6443 --token wip0ux.19q3dpudrnyc6q7i --discovery-token-ca-cert-hash sha256:e41c201f32d7aa6c57254cd78c13a5aa7242979f7152bf33ec25dde13c1dcc9a

报错The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.

#vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false"

#systemctl daemon-reload
#systemctl restart kubelet

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
 https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:
 kubeadm join 192.168.1.200:6443 --token wip0ux.19q3dpudrnyc6q7i --discovery-token-ca-cert-hash sha256:e41c201f32d7aa6c57254cd78c13a5aa7242979f7152bf33ec25dde13c1dcc9a

其中有以下关键内容：

[kubelet] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”
[certificates]生成相关的各种证书
[kubeconfig]生成相关的kubeconfig文件
[bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到

在master节点配置使用kubectl

rm -rf $HOME/.kube
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

 node节点使用kubectl工具

scp /etc/kubernetes/admin.conf  node1:/etc/kubernetes/

vim /etc/profile
export KUBECONFIG=/etc/kubernetes/admin.conf

source /etc/profile

kubectl get nodes
NAME   STATUS   ROLES   AGE   VERSION
master  NotReady  master  6m19s  v1.13.0

七、配置网络插件注意flannel镜像每个节点都要有)

master节点下载yaml配置文件

特别说明：版本会经常更新，如果配置成功，就手动去https://raw.githubusercontent.com/coreos/flannel/master/Documentation/ 下载最新版yaml文件

cd ~ && mkdir flannel && cd flannel
curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

修改配置文件kube-flannel.yml(此处要改几处镜像名)

说明：默认的镜像是http://quay.io/coreos/flannel:v0.10.0-amd64，如果你能pull下来就不用修改镜像地址，否则，修改yml中镜像地址为阿里镜像源,要修改所有的镜像版本，里面有好几条flannel镜像地址
image: http://registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64

指定启动网卡(配置文件部分截图)

containers:
  - name: kube-flannel
    image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64
    command:
    - /opt/bin/flanneld

    args:
    - --ip-masq
    - --kube-subnet-mgr
    - --iface=ens33
    - --iface=eth0
--iface=ens33 的值，是你当前的网卡,或者可以指定多网

启动
# kubectl apply -f ~/flannel/kube-flannel.yml

启动
# kubectl apply -f ~/flannel/kube-flannel.yml

启动
# kubectl apply -f ~/flannel/kube-flannel.yml

查看

# kubectl get pods -n kube-system -o wide  (集群搭建完毕才会全部是1)
NAME                             READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
coredns-66bff467f8-jzt7q         1/1     Running   3          3d23h   10.244.0.10      master   <none>           <none>
coredns-66bff467f8-rq87t         1/1     Running   3          3d23h   10.244.0.11      master   <none>           <none>
etcd-master                      1/1     Running   3          3d23h   192.168.40.137   master   <none>           <none>
kube-apiserver-master            1/1     Running   3          3d23h   192.168.40.137   master   <none>           <none>
kube-controller-manager-master   1/1     Running   3          3d23h   192.168.40.137   master   <none>           <none>
kube-flannel-ds-amd64-76tjs      1/1     Running   3          3d21h   192.168.40.139   node2    <none>           <none>
kube-flannel-ds-amd64-h84dp      1/1     Running   4          3d20h   192.168.40.138   node1    <none>           <none>
kube-flannel-ds-amd64-mnhbn      1/1     Running   3          3d21h   192.168.40.137   master   <none>           <none>
kube-proxy-5jjzr                 1/1     Running   2          3d21h   192.168.40.139   node2    <none>           <none>
kube-proxy-85hz4                 1/1     Running   3          3d20h   192.168.40.138   node1    <none>           <none>
kube-proxy-gbtgm                 1/1     Running   3          3d23h   192.168.40.137   master   <none>           <none>
kube-scheduler-master            1/1     Running   7          3d23h   192.168.40.137   master   <none>           <none>

#kubectl get pods -o wide 
NAME                                READY   STATUS             RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES
nginx-deployment-5bf87f5f59-js7nc   0/1     ImagePullBackOff   0          12m     10.244.5.12   node1   <none>           <none>
nginx-deployment-5bf87f5f59-nsb84   0/1     ImagePullBackOff   0          12m     10.244.5.13   node1   <none>           <none>

使用命令describ查看报错信息
# kubectl describ nginx-deployment-5bf87f5f59-js7nc 
.....省略大部分信息，关键如下
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  17m                default-scheduler  Successfully assigned default/nginx-deployment-5bf87f5f59-js7nc to node1
  Warning  Failed     12m                kubelet, node1     Failed to pull image "nginx:1.7.9": rpc error: code = Unknown desc = context canceled
  Warning  Failed     12m                kubelet, node1     Error: ErrImagePull
  Normal   BackOff    12m                kubelet, node1     Back-off pulling image "nginx:1.7.9"
  Warning  Failed     12m                kubelet, node1     Error: ImagePullBackOff
  Normal   Pulling    12m (x2 over 17m)  kubelet, node1     Pulling image "nginx:1.7.9"

# kubectl get service
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   14m

# kubectl get svc --namespace kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   15m

八、配置所有node节点加入集群

在所有node节点操作，此命令为初始化master成功后返回的结果

# kubeadm join 192.168.1.200:6443 --token ccxrk8.myui0xu4syp99gxu --discovery-token-ca-cert-hash sha256:e3c90ace969aa4d62143e7da6202f548662866dfe33c140095b020031bff2986

集群检测

# kubectl get pods -n kube-system
NAME                                    READY  STATUS       RESTARTS  AGE
coredns-6c66ffc55b-l76bq        1/1   Running       0      16m
coredns-6c66ffc55b-zlsvh        1/1   Running       0      16m
etcd-node1                          1/1   Running       0      16m
kube-apiserver-node1            1/1   Running       0      16m
kube-controller-manager-node1 1/1   Running         0      15m
kube-flannel-ds-sr6tq           0/1   CrashLoopBackOff  6      7m12s
kube-flannel-ds-ttzhv           1/1   Running       0      9m24s
kube-proxy-nfbg2                1/1   Running       0      7m12s
kube-proxy-r4g7b                1/1   Running       0      16m
kube-scheduler-node1            1/1   Running       0      16m

遇到异常状态0/1的pod长时间启动不了可删除它等待集群创建新的pod资源

# kubectl delete pod kube-flannel-ds-sr6tq -n kube-system
pod "kube-flannel-ds-sr6tq" deleted

删除后再次查看，发现状态为正常

再次查看节点状态

[root@master flannel]# kubectl get nodes -n kube-system
NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   19m     v1.17.2
node1    Ready    <none>   3m16s   v1.17.2
node2    Ready    <none>   103s    v1.17.2

到此集群配置完成

九、 测试kubernetes集群

$ kubectl create deployment nginx --image=nginx
$ kubectl expose deployment nginx --port=80 --type=NodePort
$ kubectl get pod,svc

十、kubelet(pod管家)

在kubernetes集群中，每个Node节点都会启动kubelet进程，用来处理Master节点下发到本节点的任务，管理Pod和其中的容器。kubelet会在API Server上注册节点信息，定期向Master汇报节点资源使用情况，并通过cAdvisor监控容器和节点资源。可以把kubelet理解成【Server-Agent】架构中的agent，是Node上的pod管家。更多kubelet配置参数信息可参考kubelet –help

当k8s节点出现 NotReady，可用如下命令查看报错信息

# journalctl   -f   -u   kubelet

-f:实时滚动更新  -u:指定某一个具体的服务

例如node节点的问题

# journalctl   -f   -u   kubelet
 Unable to update cni config: no networks found in /etc/cni/net.d

解决：
vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
添加
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/ --cni-bin-dir=/opt/cni/bin"

十一、小技巧

1、flannel  pod  强制删除不行
经排查，node2节点有问题(docker没有开机自启，重启docker)。节点正常后再强制删除

使用命令获取pod的名字
kubectl get podes -n NAMESPACE |grep Terminating  (NAMESPACE可替换为kube-system)
使用kubectl中的强制删除命令
kubectl delete pod podName -n NAMESPACE --force --grace-period=0

2、报错The connection to the server localhost:8080 was refused - did you specify the right host or port?

# export KUBECONFIG=/etc/kubernetes/admin.conf

3、集群初始化没有生成token或者忘记token

# kubeadm token list
会有一个token

获取ca证书sha256编码hash值
#openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

2cc3029123db737f234186636330e87b5510c173c669f513a9c0e0da395515b0

kubeadm token create --print-join-command --ttl=0 证书永不过期


node节点加入
#kubeadm join 10.167.11.153:6443 --token o4avtg.65ji6b778nyacw68 --discovery-token-ca-cert-hash sha256:2cc3029123db737f234186636330e87b5510c173c669f513a9c0e0da395515b0

 4、#  kubeadm config images list (列出k8s的镜像，然后下载阿里云的打成如下tag)

I0326 11:17:43.015885   47157 version.go:251] remote version is much newer: v1.18.0; falling back to: stable-1.17
W0326 11:17:46.061291   47157 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0326 11:17:46.061306   47157 validation.go:28] Cannot validate kubelet config - no validator is available
k8s.gcr.io/kube-apiserver:v1.17.4
k8s.gcr.io/kube-controller-manager:v1.17.4
k8s.gcr.io/kube-scheduler:v1.17.4
k8s.gcr.io/kube-proxy:v1.17.4
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.5

 5、在搭建好的k8s集群内创建的容器，只能在其所在的节点上curl可访问，但是在其他任何主机上无法访问容器占用的端口

解决方案1：你的系统可能没开路由

# vim /etc/sysctl.conf
找到这一行，放开注释
# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

重启主机(必须要重启才能生效)

解决方案2：

使用iptables打通网络

docker 从 1.13 版本开始，可能将 iptables FORWARD chain的默认策略设置为DROP，从而导致 ping 其它 Node 上的 Pod IP 失败，遇到这种情况时，需要手动设置策略为 ACCEPT：

# iptables -P FORWARD ACCEPT

并且把以下命令写入/etc/rc.local文件中，防止节点重启iptables FORWARD chain的默认策略又还原为DROP

# vim /etc/rc.local
sleep 60 && /sbin/iptables -P FORWARD ACCEPT
chmod +x /etc/rc.d/rc.local

6、kubectl命令报错

如果需要让普通用户可以运行 kubectl，请运行如下命令，其实这也是 kubeadm init 输出的一部分：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

或者，如果您是 root 用户，则可以运行：

export KUBECONFIG=/etc/kubernetes/admin.conf