生产环境k8s集群方案主要由以下四部分组成


  • 1、F5 ,rancher ui及业务服务访问入口,所有域名访问解析到此F5
  • 2、nginx,k8s集群业务服务访问入口、HA负载
  • 3、rancher ui,k8s可视化管理平台
  • 4、k8s集群

整体架构图:

基于rancher的k8s集群搭建_rancher

准备:


  • 1、所有主机节点开通外网访问权限
  • 2、申请https://k8s-pro.jieyue.com rancher ui 域名解析
  • 3、申请xxx.k8spro.jieyue.com 业务服务 域名解析
  • 4、k8s相关命令文件地址:http://172.16.101.32/stgconfig/soft

一、安装环境

1、硬件环境

节点

IP

OS

角色

配置

nginx-115

172.19.101.115

centos7.6

nginx

2C,4G,100G

nginx-116

172.19.101.116

centos7.6

nginx

2C,4G,100G

rancher-117

172.19.101.117

centos7.6

rancher server

4C,8G,100G

rancher-134

172.19.101.134

centos7.6

rancher server

4C,8G,100G

k8s-135

172.19.101.135

centos7.6

controlplane,etcd,worker

4C,16G,300G

k8s-136

172.19.101.136

centos7.6

controlplane,etcd,worker

4C,16G,300G

k8s-137

172.19.101.137

centos7.6

controlplane,etcd,worker

4C,16G,300G

f5-138

172.19.101.138

f5

f5-140

172.19.101.140

f5

2、软件环境


  • 1、os:centos7.6
  • 2、rancher:2.3.2
  • 3、docker:17.03.2-ce , overlay2存储,extfs 文件系统

二、安装前的准备

1、主机环境配置

1.1、 关闭防火墙
sudo systemctl stop firewalld.service && sudo systemctl disable firewalld.service
1.2、关闭setlinx
sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
1.3、关闭swap

注释或删除swap交换分区:​​vi /etc/fstab​

#/dev/mapper/centos-swap swap swap defaults 0 0
1.4、配置主机时间、时区、系统语言
sudo  ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
sudo echo 'LANG="en_US.UTF-8"' >> /etc/profile;source /etc/profile
1.5、修改最大进程数 、 用户进程打开文件数
echo '1000000' > /proc/sys/kernel/pid_max

vim /etc/sysctl.conf

kernel.pid_max=1000000

# 保存生效
sysctl -p

vi /etc/security/limits.conf
*          soft    nproc     102400                                                                                                                                        
* hard nofile 65535
jyapp hard nofile 102400
jyapp soft nofile 102400
jyapp soft nproc 102400
root soft nproc unlimited
root soft nofile 102400
root hard nofile 102400
1.6、Kernel性能调优

​vi /etc/sysctl.conf​​ 并追加一下内容:

net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.neigh.default.gc_thresh1=4096
net.ipv4.neigh.default.gc_thresh2=6144
net.ipv4.neigh.default.gc_thresh3=8192

2、安装docker-ce并配置

2.1、添加用户(可选)
sudo adduser jyapp
2.2、为新用户设置密码
sudo passwd jyapp
2.3、为新用户添加sudo权限
sudo echo 'jyapp ALL=(ALL) ALL' >> /etc/sudoers
2.4、jyapp加用docker用户组
sudo usermod -a -G docker jyapp
2.5、编辑 daemon.json 文件
vi /etc/docker/daemon.json
{
"bip":"172.25.100.1/24"
,"insecure-registries":["172.16.101.43:5000","172.16.101.49","10.50.182.85","docker.jieyue.com"]
,"registry-mirrors": ["http://172.16.101.49","http://10.50.182.85","http://docker.jieyue.com","https://088aunhx.mirror.aliyuncs.com"]

,"log-driver": "json-file"
,"log-opts": {
"max-size": "100m",
"max-file": "3"
}
,"max-concurrent-downloads": 10
,"max-concurrent-uploads": 10
,"storage-driver": "overlay2"
,"storage-opts": ["overlay2.override_kernel_check=true"]
}
2.6、刷新配置重启 docker 服务
systemctl daemon-reload
systemctl restart docker

3、下载命令行工具

下载 ​​kubectl​​,​​rke​​,​​helm​​ 到 ​​172.19.101.135 /usr/local/bin​​ 下。

命令工具下载地址:http://172.16.101.32/stgconfig/soft/raw/master/k8s-kub.tar.gz

cd /usr/local/bin

#赋可执行权限
sudo chmod +x *

#修改profile
sudo vi /etc/profile

#追加
export PATH="/usr/local/bin:$PATH"

#保存退出,使之生效
sudo source /etc/profile

4、免密登录

在172.19.101.135用 jyapp 用户执行:

ssh-keygen -t rsa
ssh-copy-id -p 10022 -i ~/.ssh/id_rsa.pub jyapp@172.19.101.135
ssh-copy-id -p 10022 -i ~/.ssh/id_rsa.pub jyapp@172.19.101.136
ssh-copy-id -p 10022 -i ~/.ssh/id_rsa.pub jyapp@172.19.101.137

三、rke安装k8s

1、配置rke 创建rancher-cluster.yml文件
nodes:

- address: 172.19.101.135
user: jyapp
role: [controlplane,worker,etcd]
port: 10022
- address: 172.19.101.136
user: jyapp
role: [controlplane,worker,etcd]
port: 10022
- address: 172.19.101.137
user: jyapp
role: [controlplane,worker,etcd]
port: 10022
services:
etcd:
backup_config:
enabled: true
interval_hours: 6
retention: 12
extra_args:
quota-backend-bytes: '4294967296'
kubelet:
cluster_domain: cluster.local
cluster_dns_server: 10.43.0.10
fail_swap_on: true
extra_args:
max-pods: 50
kube-api-burst: '100'
kube-api-qps: '100'
#enforce-node-allocatable: 'pods'
#system-reserved: 'cpu=1,memory=4000Mi'
#kube-reserved: 'cpu=1,memory=4000Mi'
kube-controller:
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
extra_args:
node-cidr-mask-size: '24'
kube-api:
service_cluster_ip_range: 10.43.0.0/16
service_node_port_range: 30000-32767
pod_security_policy: false
always_pull_images: false
2、运行RKE命令
rke up --config rancher-cluster.yml

完成后,它应显示:Finished building Kubernetes cluster successfully。

在本目录会生成config文件:​​kube_config_rancher-cluster.yml​

如若操作失败,重新安装需要清理数据:

su - root
rm -rf /var/lib/rancher/etcd/*
rm -rf /etc/kubernetes/*
su - jyapp
rke remove --config ./rancher-cluster.yml
3、测试集群
mkdir ~/.kube
cp kube_config_rancher-cluster.yml ~/.kube/config
kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
kubectl get node
NAME STATUS ROLES AGE VERSION
172.19.101.135 Ready controlplane,etcd,worker 20d v1.13.5
172.19.101.136 Ready controlplane,etcd,worker 20d v1.13.5
172.19.101.137 Ready controlplane,etcd,worker 20d v1.13.5

四、安装 rancher server

1、分别在172.19.101.117、172.19.101.134上执行
docker run -d --restart=unless-stopped \
-p 80:80 -p 443:443 \
-v
/var/lib/rancher/:/var/lib/rancher/ \
-v /root/var/log/auditlog:/var/log/auditlog \
-e CATTLE_SYSTEM_CATALOG=bundled \
-e AUDIT_LEVEL=3 \
rancher/rancher:v2.3.2

访问:https://172.19.101.117


注:ui初始登录时需指访问地址为 k8s-pro.jieyue.com


2、通过rancher ui导入 rke k8s集群

基于rancher的k8s集群搭建_docker_02

五、 配置nginx

1、sudo vi /etc/nginx/nginx.conf

user nginx;
worker_processes 4;

error_log /var/log/nginx/error.log warn;

worker_rlimit_nofile 40000;

events {
worker_connections 8192;
}

http {

upstream rancher_web{
server 172.19.101.135:80;
server 172.19.101.136:80;
server 172.19.101.137:80;
}
server {

listen 80 default_server;
server_name _;
access_log logs/busiweb.access.log main;

client_max_body_size 300m;
client_body_buffer_size 128k;
proxy_connect_timeout 600;
proxy_read_timeout 600;
proxy_send_timeout 600;
proxy_buffer_size 64k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_pass http://rancher_web;
}

}

}

2、 或可以容器化nginx代替以上安装

下载附件nginx.tar.gz 并解压

略作修改:按上面nginx.conf 修改

执行 sh startNginx-80.sh 即可


其他

需开通网络访问权限

日志收集 elk

监控 promethus

配置和连接镜像仓库 harbor

根据需要可以构建本地 (dnsmasq | coredns)


(END)