搭建 Docker+Kubernetes 环境，然后部署 Spring 微服务，最后开放对外访问。
搭建 Docker+Kubernetes 环境，然后部署微服务，最后开放对外访问。
由于网络问题，可能下载安装过程中会失败，耐心多安装几次就 OK 了。
安装一定要有耐心！！！安装一定要有耐心！！！安装一定要有耐心！！！
环境
Centos：7.5
Docker：19.03
Kubernetes：1.1.18
准备两台 CentOS 机器，这里为了测试，只准备两台，1 台做 Master，1 台做 Worker。
安装 Docker
使用脚本安装 Docker
使用 root 登录 Centos
确保 yum 包更新到最新
yum update执行 Docker 安装脚本
curl -fsSL https://get.docker.com -o get-docker.shsh get-docker.sh执行这个脚本会添加 docker.repo 源并安装 Docker。
启动 Docker 并且设置为开机启动
systemctl start dockersystemctl enable docker至此 Docker 在 Centos 系统的安装已完成。
配置镜像加速器
鉴于国内网络问题，拉取 Docker 镜像十分缓慢，我们需要配置加速器来提速。
增加配置：
vim /etc/docker/daemon.json{ "registry-mirrors": ["https://847pb1vj.mirror.aliyuncs.com"], "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2"}生效配置、重启 Docker：
systemctl daemon-reloadsystemctl restart dockersystemctl enable docker关闭防火墙
systemctl stop firewalld # 关闭服务systemctl disable firewalld # 禁用服务禁用 SELinux
修改/etc/selinux/config，设置SELINUX=disabled，重启机器。
禁用交换分区
swapoff -avim /etc/sysconfig/kubelet修改为：KUBELETEXTRAARGS="--fail-swap-on=false"
安装 Kubernetes
安装 kubelet、kubeadm、kubectl、dashboard、添加 woker。
添加源
由于国内网络原因，官方文档中的地址不可用，本文替换为阿里云镜像地址：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=1repo_gpgcheck=1gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpgexclude=kube*EOF安装
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetessystemctl enable kubelet && systemctl start kubelet修改网络配置
cat <<EOF > /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1EOFsysctl --system注意： 至此，以上的全部操作， 在 Worker 机器上也需要执行，包括安装 Docker。
初始化 Master
生成初始化文件
kubeadm config print init-defaults > kubeadm-init.yaml该文件有 3 处需要修改：
- 将
advertiseAddress: 1.2.3.4修改为本机地址（注意是内网地址） - 将
imageRepository: k8s.gcr.io修改为imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers - nodeRegistration:name，改为 k8s-master（woker 节点的话，则修改为别的名称）
修改完毕之后：
apiVersion: kubeadm.k8s.io/v1beta2bootstrapTokens:- groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authenticationkind: InitConfigurationlocalAPIEndpoint: advertiseAddress: 192.168.0.237 bindPort: 6443nodeRegistration: criSocket: /var/run/dockershim.sock name: k8s-master taints: - effect: NoSchedule key: node-role.kubernetes.io/master---apiServer: timeoutForControlPlane: 4m0sapiVersion: kubeadm.k8s.io/v1beta2certificatesDir: /etc/kubernetes/pkiclusterName: kubernetescontrollerManager: {}dns: type: CoreDNSetcd: local: dataDir: /var/lib/etcdimageRepository: registry.cn-hangzhou.aliyuncs.com/google_containerskind: ClusterConfigurationkubernetesVersion: v1.15.0networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12scheduler: {}下载镜像
kubeadm config images pull --config kubeadm-init.yaml输入如下信息：
执行初始化
kubeadm init --config kubeadm-init.yaml等待执行完毕后, 会输出如下内容：
W0328 21:31:14.954124 17338 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io][init] Using Kubernetes version: v1.18.0[preflight] Running pre-flight checks [WARNING Hostname]: hostname "k8s-master" could not be reached [WARNING Hostname]: hostname "k8s-master": lookup k8s-master on 100.125.1.250:53: no such host[preflight] Pulling images required for setting up a Kubernetes cluster[preflight] This might take a minute or two, depending on the speed of your internet connection[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"[kubelet-start] Starting the kubelet[certs] Using certificateDir folder "/etc/kubernetes/pki"[certs] Generating "ca" certificate and key[certs] Generating "apiserver" certificate and key[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.237][certs] Generating "apiserver-kubelet-client" certificate and key[certs] Generating "front-proxy-ca" certificate and key[certs] Generating "front-proxy-client" certificate and key[certs] Generating "etcd/ca" certificate and key[certs] Generating "etcd/server" certificate and key[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.237 127.0.0.1 ::1][certs] Generating "etcd/peer" certificate and key[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.237 127.0.0.1 ::1][certs] Generating "etcd/healthcheck-client" certificate and key[certs] Generating "apiserver-etcd-client" certificate and key[certs] Generating "sa" key and public key[kubeconfig] Using kubeconfig folder "/etc/kubernetes"[kubeconfig] Writing "admin.conf" kubeconfig file[kubeconfig] Writing "kubelet.conf" kubeconfig file[kubeconfig] Writing "controller-manager.conf" kubeconfig file[kubeconfig] Writing "scheduler.conf" kubeconfig file[control-plane] Using manifest folder "/etc/kubernetes/manifests"[control-plane] Creating static Pod manifest for "kube-apiserver"[control-plane] Creating static Pod manifest for "kube-controller-manager"W0328 21:31:18.364428 17338 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"[control-plane] Creating static Pod manifest for "kube-scheduler"W0328 21:31:18.365138 17338 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s[apiclient] All control plane components are healthy after 21.501912 seconds[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster[upload-certs] Skipping phase. Please see --upload-certs[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule][bootstrap-token] Using token: abcdef.0123456789abcdef[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key[addons] Applied essential addon: CoreDNS[addons] Applied essential addon: kube-proxyYour Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/configYou should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 192.168.0.237:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:964c7b0c007ce17c979e631da17ad047dfa3bad76e407b6ee76d729ecf3cd9c7最后两行需要保存下来， kubeadm join ...是 Worker 节点加入所需要执行的命令。
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s[kubelet-check] Initial timeout of 40s passed.注意此处：可能安装时一直等待，最后超时，错误如下。
可能是因为 kubeadm-init.yaml 中 advertiseAddress 没配置成内网，导致网络问题。
接下来配置环境，让当前用户可以执行 kubectl 命令:
mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config测试一下：此处的NotReady是因为网络还没配置。
[root@ecs-babc-0004 kubernetes]$ kubectl get nodeNAME STATUS ROLES AGE VERSIONk8s-master NotReady master 2m25s v1.17.3配置网络
安装 calico 网络插件：
[root@ecs-babc-0004 ~]$ wget https://docs.projectcalico.org/v3.9/manifests/calico.yaml[root@ecs-babc-0004 ~]$ cat kubeadm-init.yaml | grep serviceSubnet:serviceSubnet: 10.96.0.0/12vim calico.yaml，将192.168.0.0/16修改为10.96.0.0/12
需要注意的是, calico.yaml 中的 IP 和 kubeadm-init.yaml 需要保持一致, 要么初始化前修改 kubeadm-init.yaml, 要么初始化后修改 calico.yaml.
执行kubectl apply -f calico.yaml初始化网络。
[root@ecs-babc-0004 kubernetes]# kubectl apply -f calico.yamlconfigmap/calico-config createdcustomresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org createdcustomresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org createdclusterrole.rbac.authorization.k8s.io/calico-kube-controllers createdclusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers createdclusterrole.rbac.authorization.k8s.io/calico-node createdclusterrolebinding.rbac.authorization.k8s.io/calico-node createddaemonset.apps/calico-node createdserviceaccount/calico-node createddeployment.apps/calico-kube-controllers createdserviceaccount/calico-kube-controllers created过几分钟再查看 node 信息，master 的状态已经是Ready了。
[root@ecs-babc-0004 kubernetes]# kubectl get nodeNAME STATUS ROLES AGE VERSIONk8s-master Ready master 11m v1.18.0此处有坑，半支烟被坑了好久，后来才明白，是因为拉取不到 calico/cni:v3.8.8 镜像，或者拉取很慢，找到坑之后，果断换版本
calico/cni:v3.9.5，分分钟就 OK 了。以下是排查过程：
首先执行kubectl get pods -n kube-system发现 pods 一直 pending，发现镜像一直在 pulling，有时 1 个小时都拉取不下来。
异常情况：
而正常情况应该是：
[root@ecs-babc-0004 kubernetes]# kubectl get pods -n kube-systemNAME READY STATUS RESTARTS AGEcalico-kube-controllers-5fc5dbfc47-4gvcb 0/1 Pending 0 43scalico-node-gvnnc 0/1 Init:2/3 0 43scoredns-546565776c-sx5rj 0/1 Pending 0 6m52scoredns-546565776c-xtb92 0/1 Pending 0 6m52setcd-k8s-master 1/1 Running 0 6m51skube-apiserver-k8s-master 1/1 Running 0 6m51skube-controller-manager-k8s-master 1/1 Running 0 6m51skube-proxy-7jk4h 1/1 Running 0 6m52skube-scheduler-k8s-master 1/1 Running 0 6m50s再执行 kubectl describe po calico-node-gvnnc -n kube-system 查看：
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m14s default-scheduler Successfully assigned kube-system/calico-node-gvnnc to k8s-master Normal Pulling 2m14s kubelet, k8s-master Pulling image "calico/cni:v3.9.5" Normal Pulled 99s kubelet, k8s-master Successfully pulled image "calico/cni:v3.9.5" Normal Created 98s kubelet, k8s-master Created container upgrade-ipam Normal Started 98s kubelet, k8s-master Started container upgrade-ipam Normal Pulled 97s kubelet, k8s-master Container image "calico/cni:v3.9.5" already present on machine Normal Created 97s kubelet, k8s-master Created container install-cni Normal Started 97s kubelet, k8s-master Started container install-cni Normal Pulling 96s kubelet, k8s-master Pulling image "calico/pod2daemon-flexvol:v3.9.5" Normal Pulled 74s kubelet, k8s-master Successfully pulled image "calico/pod2daemon-flexvol:v3.9.5" Normal Created 74s kubelet, k8s-master Created container flexvol-driver Normal Started 74s kubelet, k8s-master Started container flexvol-driver Normal Pulling 74s kubelet, k8s-master Pulling image "calico/node:v3.9.5" Normal Pulled 47s kubelet, k8s-master Successfully pulled image "calico/node:v3.9.5" Normal Created 47s kubelet, k8s-master Created container calico-node Normal Started 47s kubelet, k8s-master Started container calico-node安装 Dashboard
部署 Dashboard
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yamlkubectl apply -f recommended.yaml部署完毕后, 执行kubectl get pods --all-namespaces查看 pods 状态，过几分钟全部变成 Running
[root@ecs-babc-0004 kubernetes]# kubectl get pods --all-namespacesNAMESPACE NAME READY STATUS RESTARTS AGEkube-system calico-kube-controllers-5fc5dbfc47-4gvcb 1/1 Running 0 5m15skube-system calico-node-gvnnc 1/1 Running 0 5m15skube-system coredns-546565776c-sx5rj 1/1 Running 0 11mkube-system coredns-546565776c-xtb92 1/1 Running 0 11mkube-system etcd-k8s-master 1/1 Running 0 11mkube-system kube-apiserver-k8s-master 1/1 Running 0 11mkube-system kube-controller-manager-k8s-master 1/1 Running 0 11mkube-system kube-proxy-7jk4h 1/1 Running 0 11mkube-system kube-scheduler-k8s-master 1/1 Running 0 11mkubernetes-dashboard dashboard-metrics-scraper-66b49655d4-bb6px 0/1 ContainerCreating 0 8skubernetes-dashboard kubernetes-dashboard-74b4487bfc-5dw84 0/1 ContainerCreating 0 8s创建用户
创建一个用于登录 Dashboard 的用户， 创建文件dashboard-adminuser.yaml内容如下：
apiVersion: v1kind: ServiceAccountmetadata: name: admin-user namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: admin-userroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: admin-user namespace: kube-system执行命令kubectl apply -f dashboard-adminuser.yaml
生成证书
官方文档中提供了登录 1.7.X 以上版本的登录方式，但并不清晰， 半支烟没有完全按照该文档的方式进行操作。进入到.kube目录中执行 3 个命令：
[root@ecs-babc-0004 .kube]# grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt[root@ecs-babc-0004 .kube]# grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key[root@ecs-babc-0004 .kube]# openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"第三条命令生成证书时会提示输入密码，可以直接两次回车跳过，然后再查看文件列表，看到多了 3 个证书相关文件：
kubecfg.p12即需要导入客户端机器的证书，将证书拷贝到客户端机器上，导入即可，半支烟的电脑是 MAC，直接在~目录下执行以下命令即可，（xxx.xxx.xxx.xxx，是服务器的外网 ip）
scp root@xxx.xxx.xxx.xxx:/root/.kube/kubecfg.p12 ./MAC 电脑，下载完 p12 文件，需要双击安装证书。安装证书时输入证书密码即可。
此时我们可以登录面板了， 访问地址： https://{k8s-master-ip}:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login，登录时会提示选择证书，确认后会提示输入当前用户名密码(注意是电脑的用户名密码)。此处的{k8s-master-ip}是指服务器的外网 ip。
然后会弹出 dashboard 的登录界面：
登录 Dashboard
在服务器上的.kube目录下执行kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')，获取 Token。
[root@ecs-babc-0004 .kube]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')Name: admin-user-token-9gjgzNamespace: kube-systemLabels: <none>Annotations: kubernetes.io/service-account.name: admin-user kubernetes.io/service-account.uid: a70bac13-dc07-49d2-9f4d-4296654ad66fType: kubernetes.io/service-account-tokenData====ca.crt: 1025 bytesnamespace: 11 bytestoken: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWRoaGtiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiMjBkMTE0My1jZTk0LTQzNzktOWUxNC04ZjgwZjA2ZDg0NzkiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.f6IbPGwIdFZWStzBj8_vmF01oWW5ccaCpPuVQNLSK1pgEqn0kNVK_x0RYSuKEnujObzpQQdFiRYcI6ITHja2PIVc5Nv83VCn5IaLvZdYuGZWUYRw0efJUBMA4J4N8-pRkiw6fYAuWLeGYghLNXL_nDdC_JkG75ASqrr3U1MVaikOcfrEPaI-T_AJ3TMYhI8aFoKiERpumu5W1K6Jl80Am9pWDX0Ywis5SSUP1VYfu-coI48EXSptcaxEyv58PrHUd6t_oMVV9rpqSxrNtMZvMeXqe8Hnl21vR7ls5yTZegYtHXSc3PKvCaIalKhYXAuhogNcIXHaMzvLSbf-DSQkVw复制该 Token 到登录页，点击登录即可， 主界面如下：
添加 Worker
重复执行 安装 Docker ~ 安装 Kubernetes-修改网络配置 的全部操作，初始化一个 Worker 机器。
执行如下命令将 Worker 加入集群：
kubeadm join 192.168.0.237:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:964c7b0c007ce17c979e631da17ad047dfa3bad76e407b6ee76d729ecf3cd9c7- 注意: 此处的秘钥是初始化 Master 后生成的, 参考前文.
添加完毕后， 在 Master 上查看 Worker 节点状态：
[root@ecs-babc-0004 ~]# kubectl get nodesNAME STATUS ROLES AGE VERSIONecs-babc-0006 Ready <none> 93s v1.18.0k8s-master Ready master 174m v1.18.0当然在 Dashboard 也可查看:
总结
至此，在 Centos 上搭建 Docker 和 Kubernetes 已全部完成，步骤较多，只要有耐心，肯定能安装完毕。要知道半支烟在安装的时候，前后经历了十几次的重装系统，几乎崩溃，哈哈。
参考文献
感谢以下文献作者：
Linux 下安装 Docker
Kubernetes(一) 跟着官方文档从零搭建 K8S
错误链接：
k8s 报错解决思路
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
