k8s负载均衡失效的原因 k8s 负载均衡

文章目录

• 先具备单master节点部署环境
• 一、master2部署
• 二、k8s负载均衡部署

• 1、安装nginx服务
• 2、部署keepalived服务

多节点布属基于单节点布置结束，详细请看二进制部署单节点Kubernetes（K8S）

先具备单master节点部署环境

一、master2部署

优先关闭防火墙和selinux服务

1、在master01上操作

//复制kubernetes目录到master02

[root@localhost opt]#scp -r /opt/kubernetes/ root@192.168.182.44:/opt/

复制master中的三个组件启动脚本kube-apiserver.service kube-controller-manager.service kube-scheduler.service

scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.182.44:/usr/lib/systemd/system

master02上操作

修改配置文件kube-apiserver中的IP

[root@localhost ~]# cd /opt/kubernetes/cfg/
[root@localhost cfg]# vim kube-apiserver

特别注意：master02一定要有etcd证书
需要拷贝master01上已有的etcd证书给master02使用

master1操作

[root@localhost ~]#scp -r /opt/etcd/ root@192.168.182.44:/opt/

启动master02中的三个组件服务

[root@localhost cfg]# systemctl start kube-apiserver.service 
[root@localhost cfg]# systemctl start kube-controller-manager.service 
[root@localhost cfg]# systemctl start kube-scheduler.service

增加环境变量

[root@localhost cfg]# vim /etc/profile
#末尾添加
export PATH=$PATH:/opt/kubernetes/bin/
[root@localhost cfg]# source /etc/profile

[root@localhost ~]#kubectl get node
NAME             STATUS   ROLES    AGE   VERSION
192.168.182.22   Ready    <none>   11h   v1.12.3
192.168.182.33   Ready    <none>   10h   v1.12.3

此时做到这一步，master2只能查询到etcd，但是不能管理node1和node2

server指向的url是指向master1 的 master不能操作

为了解决这个问题，需要做一个负载均衡

二、k8s负载均衡部署

1、安装nginx服务

lb01和lb02操作

安装nginx服务，把nginx.sh和keepalived.conf脚本拷贝到家目录

[root@localhost ~]# systemctl stop firewalld.service 
[root@localhost ~]# setenforce 0
[root@localhost ~]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
[root@localhost ~]# yum install nginx -y

添加四层转发

[root@localhost ~]# vim /etc/nginx/nginx.conf 

events {
    worker_connections  1024;
}
stream {

   log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
        server 192.168.182.11:6443;
        server 192.168.182.44:6443;
    }
    server {
                listen 6443;
                proxy_pass k8s-apiserver;
    }
    }
http {

[root@localhost ~]# systemctl start nginx

2、部署keepalived服务

[root@localhost ~]# yum install keepalived -y
//修改配置文件
[root@localhost ~]# cp keepalived.conf /etc/keepalived/keepalived.conf 
cp：是否覆盖"/etc/keepalived/keepalived.conf"？ yes

注意：lb01是Mster配置如下：

需要修改配置文件的

成品如下

[root@localhost ~]# vim /etc/keepalived/keepalived.conf 

! Configuration File for keepalived

global_defs {
   # 接收邮件地址
   notification_email {
     [e-mail]
     [e-mail]
   } 
   notification_email_from [e-mail]
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30
   router_id NGINX_MASTER 
}  

vrrp_script check_nginx {
    script "/etc/nginx/check_nginx.sh"
}   

vrrp_instance VI_1 {
    state MASTER 
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
    priority 100    # 优先级，备服务器设置 90  
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication {  
        auth_type PASS
        auth_pass 1111
    }   
    virtual_ipaddress {
        192.168.182.100/24
    }   
    track_script {
        check_nginx
    }   
}

注意：lb02是Backup配置如下：

! Configuration File for keepalived 
 
global_defs { 
   # 接收邮件地址 
   notification_email { 
     [e-mail] 
     [e-mail] 
     [e-mail] 
   } 
   # 邮件发送地址 
   notification_email_from [e-mail]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER 
} 

vrrp_script check_nginx {
    script "/etc/nginx/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state BACKUP 
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 90    # 优先级，备服务器设置 90 
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress { 
        192.168.182.100/24 
    } 
    track_script {
        check_nginx
    } 
}

两个lb都创建脚本判断nginx是否运行

[root@localhost ~]# vim /etc/nginx/check_nginx.sh

count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi

赋权脚本文件启动keepalived服务

[root@localhost ~]# chmod +x /etc/nginx/check_nginx.sh
[root@localhost ~]# systemctl start keepalived

查看lb01地址信息

[root@localhost ~]# ip a

查看lb02地址信息

[root@localhost nginx]# ip a

//验证地址漂移（lb01中使用pkill nginx，再在lb02中使用ip a 查看）

//恢复操作（在lb01中先启动nginx服务，再启动keepalived服务）

//nginx站点/usr/share/nginx/html

开始修改node节点配置文件统一VIP（bootstrap.kubeconfig,kubelet.kubeconfig）

[root@localhost cfg]# vim /opt/kubernetes/cfg/bootstrap.kubeconfig
[root@localhost cfg]# vim /opt/kubernetes/cfg/kubelet.kubeconfig
[root@localhost cfg]# vim /opt/kubernetes/cfg/kube-proxy.kubeconfig

统统修改为VIP

server: https://192.168.182.100:6443
[root@localhost cfg]# systemctl restart kubelet.service 
[root@localhost cfg]# systemctl restart kube-proxy.service

替换完成直接自检

[root@localhost cfg]# grep 100 *
bootstrap.kubeconfig:    server: https://192.168.182.100:6443
kubelet.kubeconfig:    server: https://192.168.182.100:6443
kube-proxy.kubeconfig:    server: https://192.168.182.100:6443

在lb01上查看nginx的k8s日志

[root@localhost ~]# tail /var/log/nginx/k8s-access.log
192.168.182.33 k8s-apiserver - [15/Apr/2021:11:49:53 +0800] 502 0
192.168.182.33 k8s-apiserver - [15/Apr/2021:11:49:53 +0800] 502 0
192.168.182.33 k8s-apiserver - [15/Apr/2021:11:49:53 +0800] 502 0
192.168.182.33 k8s-apiserver - [15/Apr/2021:11:49:54 +0800] 502 0
192.168.182.22 k8s-apiserver - [15/Apr/2021:11:49:54 +0800] 502 0
192.168.182.22 k8s-apiserver - [15/Apr/2021:11:49:54 +0800] 502 0
192.168.182.22 k8s-apiserver - [15/Apr/2021:11:49:54 +0800] 502 0
192.168.182.33 k8s-apiserver - [15/Apr/2021:11:49:54 +0800] 502 0
192.168.182.22 k8s-apiserver - [15/Apr/2021:11:49:54 +0800] 502 0
192.168.182.22 k8s-apiserver - [15/Apr/2021:11:49:54 +0800] 502 0

在master01上操作

测试创建pod

[root@localhost ~]#kubectl run nginx --image=nginx
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx created

查看状态

[root@localhost ~]# kubectl get pods
NAME                    READY   STATUS              RESTARTS   AGE
nginx-dbddb74b8-nf9sk   0/1     ContainerCreating   0          33s   //正在创建中
[root@localhost ~]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-nf9sk   1/1     Running   0          80s  //创建完成，运行中

//注意日志问题
[root@localhost ~]# kubectl logs nginx-dbddb74b8-nf9sk
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-nf9sk)
这里想要直接查看日志是查看不了的，必须要先创建一个匿名用户，输入以下命令
[root@localhost ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created


//查看pod网络
[root@localhost ~]# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE              NOMINATED NODE
nginx-dbddb74b8-nf9sk   1/1     Running   0          11m   172.17.31.3   192.168.195.150   <none>

//在对应网段的node节点上操作可以直接访问
[root@localhost cfg]# curl 172.17.31.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
//访问就会产生日志
//回到master01操作
[root@localhost ~]# kubectl logs nginx-dbddb74b8-nf9sk
172.17.31.1 - - [05/Feb/2020:05:08:36 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"