kuberenetes 的多节点集群与高可用配置

标签(空格分隔): kubernetes系列


  • 一: kubernetes master 节点的 高可用
  • 二: 配置nginx 服务器
  • 三: 配置nginx 的LB 的 keepalived 高可用

一: kubernetes master 节点的 高可用

kubernetes 的多master集群架构

kubernetes 高可用主要在于apiserver


image_1dkhotd53bpl9cl1s82qt0g8e9.png-519.8kB

部署master01  IP 地址: 192.168.100.12 

在 192.168.100.12 部署与 master01 一样的 服务

scp -r /opt/kubernetes/ root@192.168.100.12:/opt/

scp -r /opt/etcd/ root@192.168.100.12:/opt/

scp /usr/bin/kubectl root@192.168.20.12:/usr/bin/

scp /usr/lib/systemd/system/kube-* root@192.168.100.12:/usr/lib/systemd/system/

image_1dkhpddb51u5g3s17ra1ktmbjs16.png-711kB

image_1dkhvjrhh2t1ilu56n1knh1ldm3n.png-271.7kB

image_1dki04i9s1inp11uo19l7qp0f074g.png-111.3kB

image_1dkhpdthu1da6tcarpb19mf1avk1j.png-214.9kB


修改 配置文件 
cd /opt/kubernetes/cfg 

vim kube-apiserver 

---
--bind-address=192.168.100.12

--advertise-address=192.168.100.12

image_1dkhpivse19oi49k1iie135c1tvt20.png-428.3kB

启动服务:

service kube-apiserver start

service kube-controller-manager start

service kube-scheduler start


image_1dkhpojvi150r10eltli5l19i42d.png-332.5kB

image_1dkhq05d01qfcldj1a5e9qt2u3a.png-558.4kB

1.png-490.1kB


login :
    192.168.20.12

kubectl get cs

image_1dki06f5p16mq1aq11j7l116gs1h4t.png-180.2kB

配置负载均衡nginx

nginx 服务器 地址: 192.168.100.15
nginx 的yum 包地址

http://nginx.org/en/linux_packages.html

cd /etc/yum.repos.d/

vim  nginx.repo
---
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key

---
yum install nignx 

image_1dki10vsl1g5f1tdj2q1i513tv5a.png-399.3kB


image_1dki139jv1vsg18d314m719vp1cec5n.png-404.7kB

cd /etc/nginx/
vim nginx.conf 

增加:
---
stream {
    log_format main "$remote_addr $upstream_addr $time_local $status";
    access_log /var/log/nginx/k8s-access.log main;
    upstream k8s-apiserver {

            server 192.168.100.11:6443;
            server 192.168.100.12:6443;

      }
    server {

           listen 192.168.100.15:6443;
           proxy_pass k8s-apiserver;

     }

}

2.png-180.6kB

image_1dki1l9jeuaotgklfprf81gan71.png-335.4kB

···

修改node 节点的 master 服务器指向
login : 192.168.20.13 
cd /opt/kubernetes/cfg/

vim bootstrap.kubeconfig

---
server: https://192.168.100.11:6443  改成:

server: https://192.168.100.15:6443
---

vim kubelet.kubeconfig

----
server: https://192.168.100.11:6443  改成:

server: https://192.168.100.15:6443
----

vim kube-proxy.kubeconfig
----
server: https://192.168.100.11:6443  改成:

server: https://192.168.100.15:6443
----

scp bootstrap.kubeconfig kubelet.kubeconfig kube-proxy.kubeconfig root@192.168.100.14:/opt/kubernetes/cfg/

image_1dki27qdl7341bb71d0gd8211357u.png-196.4kB

重新启动 node 节点的 kubelet 和 kube-proxy 

service kubelet restart 
service kube-proxy restart 

image_1dki2cv411gei28h1g501jdb17768b.png-130.9kB

image_1dki2g6sk72b1o0u1joplmk100h95.png-365.1kB image_1dki2erf51sl4t6885h14he1cpf8o.png-400.4kB

使用 主节点 配置 测试

kubectl get nodes 

image_1dki36rt7vef1tvjpbk150f1l2q12.png-246.5kB

image_1dki35tv1kp93cu9651bj415s29.png-345.8kB

三: 配置nginx 的LB 的 keepalived 高可用

login 192.168.100.16 

安装nginx 服务器 同上 192.168.100.15 一样

修改 :

vim /etc/nginx/nginx.conf

----
stream {
    log_format main "$remote_addr $upstream_addr $time_local $status";
    access_log /var/log/nginx/k8s-access.log main;

    upstream k8s-apiserver {

            server 192.168.100.11:6443;
            server 192.168.100.12:6443;

      }
    server {

           listen 192.168.100.16:6443;
           proxy_pass k8s-apiserver;

     }

}
---

service nginx start 

image_1dki55f8bh2dtjn10vclvv1bet2v.png-367.7kB

配置keepalive 高可用 

yum install keepalived 

keepalived 的配置文件

cd /etc/keepalived/

vim keepalived.conf 

---
! Configuration File for keepalived 
 
global_defs { 
   # 接收邮件地址 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   # 邮件发送地址 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER 
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的 
    priority 100    # 优先级,备192.168.100.16服务器设置 90   
    advert_int 1    # 指定VRRP 心跳包通告间隔时间,默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress { 
        192.168.100.70/24 
    } 
    track_script {
        check_nginx
    } 
}

----

配置 检查 nginx 进程 检查
cd /etc/keepalived/

vim check_nginx.sh

---

#!/bin/bash

count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    /etc/init.d/keepalived stop
fi

---
service keepalived start 

ps -ef |grep keepalived 

image_1dki6q3gm1qameqk1v5n1tr1nd23c.png-664.4kB

cd /etc/keepalived

scp check_nginx.sh keepalived.conf root@192.168.100.16:/etc/keepalived/



login: 192.168.100.16 

cd /etc/keepalived/

vim keepalived.conf
---
改 priority 100  为 priority 90

----


serivice keepalived start 

ps -ef |grep keepalived 

image_1dki71lik1prc1vhg1nlni7t3ja3s.png-311.9kB


改 node 节点的 IP 地址

修改node 节点的 master 服务器指向
login : 192.168.20.13 
cd /opt/kubernetes/cfg/

vim bootstrap.kubeconfig

---
server: https://192.168.100.15:6443  改成:

server: https://192.168.100.70:6443
---

vim kubelet.kubeconfig

----
server: https://192.168.100.15:6443  改成:

server: https://192.168.100.70:6443
----

vim kube-proxy.kubeconfig
----
server: https://192.168.100.15:6443  改成:

server: https://192.168.100.70:6443
----

scp bootstrap.kubeconfig kubelet.kubeconfig kube-proxy.kubeconfig root@192.168.100.14:/opt/kubernetes/cfg/


重新启动 node 节点的 kubelet 和 kube-proxy 

service kubelet restart 
service kube-proxy restart 


image_1dki82iapuip56n1o0aj3k5f949.png-169.5kB

image_1dki838n61jej1a13k0oa4is644m.png-173.4kB

login: 192.168.100.15
修改 nginx 的 配置文件

cd /etc/nginx/

vim nginx.conf 
----
将linsten 192.168.100.15:6443  改为: 0.0.0.0:6443

----

service nginx restart 

image_1dki8amqm8uo1dg24961t36cnp53.png-123.6kB image_1dki8b5tl141gglj1gkkr2v1g2p5g.png-140kB

login: 192.168.100.16
修改 nginx 的 配置文件

cd /etc/nginx/

vim nginx.conf 
----
将linsten 192.168.100.16:6443  改为: 0.0.0.0:6443

----
service nginx restart 

image_1dki8k1tj1q6b1oc817n8f1a10e46a.png-237.2kB

image_1dki8j2hc1c44mb17ht2pr1vdb5t.png-103.4kB

去master01 节点上面查看

kubectl get svc 

kubectl get svc,pods -n kube-system --all-namespaces 

image_1dki8nr8rol3n41pnu58510jr77.png-187.9kB

image_1dki8pe9b1ev0i601thf1v52ou37k.png-298kB

去master02 节点上面查看

kubectl get svc 

kubectl get svc,pods -n kube-system --all-namespaces 

image_1dki8rlou15q011e916vp11hq1cuk81.png-443kB