手工离线部署k8s(v1.9)

 

1. 环境准备(采用一个master节点+两个node节点)
master 192.168.2.40
node-1 192.168.2.41
node-2 192.168.2.42

绑定hosts

2.master和node-1、node-2绑定hosts

#vi /etc/hosts

192.168.2.40   master
192.168.2.41   node-1
192.168.2.42   node-2

3. master节点与node节点ssh密码登录

[root@master ~]# ssh-keygen
[root@master ~]# ssh-copy-id node-1
[root@master ~]# ssh-copy-id node-2



4.关闭所有服务器防火墙和selinux

#systemctl stop firewalld.service
#systemctl disable firewalld.service
#sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
#grep SELINUX=disabled /etc/selinux/config
#setenforce 0

5.所有服务器关闭swap

#
swapoff -a && sed -i '/swap/d' /etc/fstab

6.所有服务器配置系统路由参数,防止kubeadm报路由警告

#echo -e "net.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nvm.swappiness = 0" >> /etc/sysctl.conf
#sysctl -p

注:

[root@master soft]# sysctl -p
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: 没有那个文件或目录
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: 没有那个文件或目录
[root@master soft]# modprobe bridge
[root@master soft]# lsmod|grep bridge
bridge                119562  0
stp                    12976  1 bridge
llc                    14552  2 stp,bridge
[root@master soft]# sysctl -p
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0


7.操作系统版本:centos7.2 

8.软件版本

kubernetes v1.9
docker:17.03
kubeadm:v1.9.0
kube-apiserver:v1.9.0
kube-controller-manager:v1.9.0
kube-scheduler:v1.9.0
k8s-dns-sidecar:1.14.7
k8s-dns-kube-dns:1.14.7
k8s-dns-dnsmasq-nanny:1.14.7
kube-proxy:v1.9.0
etcd:3.1.10
pause :3.0
flannel:v0.9.1
kubernetes-dashboard:v1.8.1

注意:采用kubeadm安装kubeadm为kubernetes官方推荐的自动化部署工具,他将kubernetes的组件以pod的形式部署在master和node节点上,并自动完成证书认证等操作。

因为kubeadm默认要从google的镜像仓库下载镜像,但目前国内无法访问google镜像仓库,所以提将镜像下好了,只需要将离线包的镜像导入到节点中就可以了。

4.安装步骤

1)所有服务器,下载相关包至/home/soft

链接:https://pan.baidu.com/s/1eUixGvo 密码:65yo

2)所有服务器,解压下载下来的离线包

#yum install -y bzip2

#tar -xjvf k8s_images.tar.bz2

3)所有服务器,安装docker-ce17.03(kubeadmv1.9最大支持docker-ce17.03)

安装依赖包

#yum install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm
#yum install -y ftp://ftp.icm.edu.pl/vol/rzm6/linux-slc/centos/7.1.1503/cr/x86_64/Packages/libseccomp-2.2.1-1.el7.x86_64.rpm
#yum install -y  http://rpmfind.net/linux/centos/7.4.1708/os/x86_64/Packages/libtool-ltdl-2.4.2-22.el7_3.x86_64.rpm
#cd k8s_images
#rpm -ihv docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm
#rpm -ivh docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm

注意:修改docker的镜像源为国内的daocloud的。

#
curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s 
http://3272dd08.m.daocloud.io

4)所有服务器,启动docker-ce

#systemctl start  docker.service &&systemctl enable  docker.service

5)所有服务器,导入镜像

docker load </home/soft/k8s_images/docker_images/k8s-dns-dnsmasq-nanny-amd64_v1.14.7.tar
docker load </home/soft/k8s_images/docker_images/k8s-dns-kube-dns-amd64_1.14.7.tar
docker load </home/soft/k8s_images/docker_images/k8s-dns-sidecar-amd64_1.14.7.tar
docker load </home/soft/k8s_images/docker_images/kube-apiserver-amd64_v1.9.0.tar
docker load </home/soft/k8s_images/docker_images/kube-controller-manager-amd64_v1.9.0.tar
docker load </home/soft/k8s_images/docker_images/kube-scheduler-amd64_v1.9.0.tar
docker load </home/soft/k8s_images/docker_images/flannel:v0.9.1-amd64.tar
docker load </home/soft/k8s_images/docker_images/pause-amd64_3.0.tar
docker load </home/soft/k8s_images/docker_images/kube-proxy-amd64_v1.9.0.tar
docker load </home/soft/k8s_images/kubernetes-dashboard_v1.8.1.tar
docker load </home/soft/k8s_images/docker_images/etcd-amd64_v3.1.10.tar

6)安装kubelet kubeadm kubectl包

rpm -ivh kubernetes-cni-0.6.0-0.x86_64.rpm --nodeps --force
yum localinstall -y socat-1.7.3.2-2.el7.x86_64.rpm
yum localinstall -y kubelet-1.9.9-9.x86_64.rpm
yum localinstall -y kubectl-1.9.0-0.x86_64.rpm
yum localinstall -y kubeadm-1.9.0-0.x86_64.rpm

5.master节点操作

1)启动kubelete

#systemctl start kubelet && systemctl enable kubelet

2)开始初始化master

#kubeadm init --kubernetes-version=v1.9.0 --pod-network-cidr=10.224.0.0/16 --token-ttl=0

注:kubernetes默认支持多重网络插件如flannel、weave、calico,这里使用flanne,就必须要设置

--pod-network-cidr参数,10.244.0.0/16是kube-flannel.yml里面配置的默认网段,如果需要修改的话,需要把kubeadm init的--pod-network-cidr参数和后面的kube-flannel.yml里面修改成一样的网段就可以了。

--kubernetes-version 最好指定版本,否则会请求 https://storage.googleapis.com/kubernetes-release/release/stable-1.9.txt ,如果没"翻""墙",就超时报错

--token-ttl 默认的token有效期24小时, 设置为0表示永不过期

 

3)发现kubelet启动不了,报错了,查看日志/var/log/messages如下:

kubelet: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

解决方法:发现原来是kubelet默认的cgroup的driver和docker的不一样,docker默认的cgroupfs,kubelet默认为systemd,可以用docker info | grep cgroup查看当前docker驱动方式
编辑 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd" 改为Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

重启reload

systemctl daemon-reload && systemctl restart kubelet
查看状态  #systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since 三 2018-04-11 15:11:22 CST; 22s ago
     Docs: http://kubernetes.io/docs/
 Main PID: 15942 (kubelet)
   Memory: 40.3M
   CGroup: /system.slice/kubelet.service
           └─15942 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kub...
 
4月 11 15:11:32 master kubelet[15942]: E0411 15:11:32.415152   15942 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubel...refused
4月 11 15:11:32 master kubelet[15942]: E0411 15:11:32.416006   15942 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubel...refused
4月 11 15:11:32 master kubelet[15942]: E0411 15:11:32.426454   15942 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/confi...refused
4月 11 15:11:34 master kubelet[15942]: E0411 15:11:34.653755   15942 eviction_manager.go:238] eviction manager: unexpected...t found
4月 11 15:11:34 master kubelet[15942]: W0411 15:11:34.657127   15942 cni.go:171] Unable to update cni config: No networks ...i/net.d
4月 11 15:11:34 master kubelet[15942]: E0411 15:11:34.657315   15942 kubelet.go:2105] Container runtime network not ready:...ialized
4月 11 15:11:35 master kubelet[15942]: I0411 15:11:35.238311   15942 kubelet_node_status.go:273] Setting node annotation t.../detach
4月 11 15:11:35 master kubelet[15942]: I0411 15:11:35.240636   15942 kubelet_node_status.go:82] Attempting to register node master
4月 11 15:11:39 master kubelet[15942]: W0411 15:11:39.658588   15942 cni.go:171] Unable to update cni config: No networks ...i/net.d
4月 11 15:11:39 master kubelet[15942]: E0411 15:11:39.658802   15942 kubelet.go:2105] Container runtime network not ready:...ialized
Hint: Some lines were ellipsized, use -l to show in full.

此时需要将环境reset一下,执行
#kubeadm reset
在重新执行
#kubeadm init --kubernetes-version=v1.9.0 --pod-network-cidr=10.224.0.0/16 --token-ttl=0

4)成功初始化如下:

[root@master k8s_images]# kubeadm init --kubernetes-version=v1.9.0 --pod-network-cidr=10.224.0.0/16 --token-ttl=0
[init] Using Kubernetes version: v1.9.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.40]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 29.003450 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node master as master by adding a label and a taint
[markmaster] Master master tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: d0c1ec.7d7a61a4e9ba83f8
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
 
Your Kubernetes master has initialized successfully!
 
To start using your cluster, you need to run the following as a regular user:
 
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
 
You can now join any number of machines by running the following on each node
as root:
 
  kubeadm join --token d0c1ec.7d7a61a4e9ba83f8 192.168.2.40:6443 --discovery-token-ca-cert-hash sha256:7b38dad17cd1378446121952632d78d041dfcddc27b4663d011113a3b6326a65


kubeadm join xxx保存下来,等下node节点需要使用,如果忘记了,可以在master上通过kubeadmin token list得到,也可以从新生成一个

当前生成token如下:

kubeadm join --token d0c1ec.7d7a61a4e9ba83f8 192.168.2.40:6443 --discovery-token-ca-cert-hash sha256:7b38dad17cd1378446121952632d78d041dfcddc27b4663d011113a3b6326a65

### 注意:kubeadm init 输出的 join 指令中 token 只有 24h 的有效期,如果过期后,需要重新生成,具体请参考:

# kubeadm token create --print-join-command

5)按照上面提示,此时root用户还不能使用kubelet控制集群需要,配置下环境变量
对于非root用户

#mkdir -p $HOME/.kube
#cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#chown $(id -u):$(id -g) $HOME/.kube/config


对于root用户

#export KUBECONFIG=/etc/kubernetes/admin.conf

也可以直接放到~/.bash_profile

#echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

source一下环境变量

source ~/.bash_profile

6)kubectl version测试

[root@master k8s_images]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}


6.安装网络,可以使用flannel、calico、weave、macvlan这里我们用flannel。

1)下载此文件

#wget https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

或直接使用离线包里面的
2)若要修改网段,修改配置文件kube-flannel.yml,需要kubeadm --pod-network-cidr=和这里同步,修改network项

net-conf.json: |

      {

        "Network": "10.244.0.0/16",

        "Backend": {

          "Type": "vxlan"

        }

      }

 

3)执行加载网络

#kubectl create  -f /home/soft/k8s_images/kube-flannel.yml


7.部署kubernetes-dashboard,kubernetes-dashboard是可选组件,因为,实在不好用,功能太弱了。 建议在部署master时一起把kubernetes-dashboard一起部署了,不然在node节点加入集群后,kubernetes-dashboard会被kube-scheduler调度node节点上,这样根kube-apiserver通信需要额外配置。

下载kubernetes-dashboard的配置文件或直接使用离线包里面的kubernetes-dashboard.yaml

1)创建kubernetes-dashboard

#kubectl create  -f /home/soft/k8s_images/kubernetes-dashboard.yaml


2) 如果想修改端口,或外部可访问

# ------------------- Dashboard Service ------------------- #
 
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 32666
  selector:
k8s-app: kubernetes-dashboard


 

注意:32666是映射端口,docker run -d xxx:xxx差不多映射出去即可。访问https://master_ip:32666

如果出现pod失败需要删除可使用以下命令,删除pod

kuberctl delete po -n kube-system <pod-name>

查看pod创建失败原因

# kubectl describe pod <pod-name> --namespace=kube-system

3)默认验证方式有kubeconfig和token,这里我们使用basicauth的方式进行apiserver的验证

创建/etc/kubernetes/manifests/pki/basic_auth_file 用于存放用户名和密码。basic_auth_file文件格式为user,password,userid

[root@master pki]# echo 'admin,admin,2' > /etc/kubernetes/pki/basic_auth_file


4)给kube-apiserver添加basic_auth验证

[root@master pki]# grep 'auth' /etc/kubernetes/manifests/kube-apiserver.yaml
- --enable-bootstrap-token-auth=true
    - --authorization-mode=Node,RBAC


添加

    - --basic_auth_file=/etc/kubernetes/pki/basic_auth_file

注意:!!!!如果这时直接kubectl apply -f xxxxxxxxx 执行更新kube-apiserver.yaml文件,会出现如下报错:

The connection to the server 192.168.2.40:6443 was refused - did you specify the right host or port?

解决方法:

kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml之前,先执行systemctl daemon-reload再执行systemctl restart kubelet,确认是否重启是否成功

# kubectl get node

# kubectl get pod --all-namespaces

 

5)更新应用/etc/kubernetes/manifests/kube-apiserver.yaml

[root@master manifests]# kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
pod "kube-apiserver" created


 

6)k8s1.6后版本都采用RBAC授权模型。默认情况下cluster-admin是拥有全部权限的,将admin和cluster-admin角色进行clusterrolebinding绑定,这样admin就有cluster-admin的权限。

[root@master ~]# kubectl create clusterrolebinding  login-on-dashboard-with-cluster-admin  --clusterrole=cluster-admin --user=admin
clusterrolebinding "login-on-dashboard-with-cluster-admin" created


检查是否正常获取到集群信息

# kubectl get clusterrolebinding/login-on-dashboard-with-cluster-admin -o yaml


 

7)查看所pod状态,已经都running

[root@master k8s_images]# kubectl get pod --all-namespaces
NAMESPACE     NAME                                   READY     STATUS    RESTARTS   AGE
kube-system   etcd-master                            1/1       Running   0          9m
kube-system   kube-apiserver-master                  1/1       Running   0          9m
kube-system   kube-controller-manager-master         1/1       Running   0          9m
kube-system   kube-dns-6f4fd4bdf-qj7s5               3/3       Running   0          37m
kube-system   kube-flannel-ds-4mvmz                  1/1       Running   0          9m
kube-system   kube-proxy-67jq2                       1/1       Running   0          37m
kube-system   kube-scheduler-master                  1/1       Running   0          9m
kube-system   kubernetes-dashboard-58f5cb49c-xsqf5   1/1       Running   0          32s


 

8)测试连接

[root@master ~]# curl --insecure https://master:6443 -basic -u admin:admin  
{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/admissionregistration.k8s.io",
    "/apis/admissionregistration.k8s.io/v1beta1",
    "/apis/apiextensions.k8s.io",
    "/apis/apiextensions.k8s.io/v1beta1",
    "/apis/apiregistration.k8s.io",
    "/apis/apiregistration.k8s.io/v1beta1",
    "/apis/apps",
    "/apis/apps/v1",
    "/apis/apps/v1beta1",
    "/apis/apps/v1beta2",
    "/apis/authentication.k8s.io",
    "/apis/authentication.k8s.io/v1",
    "/apis/authentication.k8s.io/v1beta1",
    "/apis/authorization.k8s.io",
    "/apis/authorization.k8s.io/v1",
    "/apis/authorization.k8s.io/v1beta1",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/autoscaling/v2beta1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/batch/v1beta1",
    "/apis/certificates.k8s.io",
    "/apis/certificates.k8s.io/v1beta1",
    "/apis/events.k8s.io",
    "/apis/events.k8s.io/v1beta1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/apis/networking.k8s.io",
    "/apis/networking.k8s.io/v1",
    "/apis/policy",
    "/apis/policy/v1beta1",
    "/apis/rbac.authorization.k8s.io",
    "/apis/rbac.authorization.k8s.io/v1",
    "/apis/rbac.authorization.k8s.io/v1beta1",
    "/apis/storage.k8s.io",
    "/apis/storage.k8s.io/v1",
    "/apis/storage.k8s.io/v1beta1",
    "/healthz",
    "/healthz/autoregister-completion",
    "/healthz/etcd",
    "/healthz/ping",
    "/healthz/poststarthook/apiservice-openapi-controller",
    "/healthz/poststarthook/apiservice-registration-controller",
    "/healthz/poststarthook/apiservice-status-available-controller",
    "/healthz/poststarthook/bootstrap-controller",
    "/healthz/poststarthook/ca-registration",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/healthz/poststarthook/kube-apiserver-autoregistration",
    "/healthz/poststarthook/rbac/bootstrap-roles",
    "/healthz/poststarthook/start-apiextensions-controllers",
    "/healthz/poststarthook/start-apiextensions-informers",
    "/healthz/poststarthook/start-kube-aggregator-informers",
    "/healthz/poststarthook/start-kube-apiserver-informers",
    "/logs",
    "/metrics",
    "/swagger-2.0.0.json",
    "/swagger-2.0.0.pb-v1",
    "/swagger-2.0.0.pb-v1.gz",
    "/swagger.json",
    "/swaggerapi",
    "/ui",
    "/ui/",
    "/version"
  ]


 

9) Firefox访问测试(不建议用谷歌),因为是自签的证书,所以浏览器会报证书未受信任问题。

:1.8版本的dashboard集成了运行命令(相当于执行了 kubectl exec -it etcd-vm1 -n kube-system /bin/sh ),使用起来还是挺方便的。 

 手工离线部署k8s(v1.9)_Kubernetes

8.node节点操作(2个node节点服务器需操作)

1)node-1、node-2修改kubelet配置文件cgroup的driver由systemd改为cgroupfs

#vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment=”KUBELET_CGROUP_ARGS=–cgroup-driver=cgroupfs”
#systemctl daemon-reload
#systemctl enable kubelet&&systemctl restart kubelet


2) node-1、node-2节点加入集群,使用master上面的kubeadm后的kubeadm join --xxx 命令加入

#kubeadm join --token d0c1ec.7d7a61a4e9ba83f8 192.168.2.40:6443 --discovery-token-ca-cert-hash sha256:7b38dad17cd1378446121952632d78d041dfcddc27b4663d011113a3b6326a65


3)在master节点上检查一下

[root@master k8s_images]# kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
master    Ready     master    1h        v1.9.0
node-1    Ready     <none>    1m        v1.9.0
node-2    Ready     <none>    58s       v1.9.0


4) 测试集群

master节点上发起个创建应用请求,建个名为httpd-app的应用,镜像为httpd,有两个副本pod

[root@master k8s_images]# kubectl run httpd-app --image=httpd --replicas=2
deployment "httpd-app" created
[root@master k8s_images]# kubectl get deployment
NAME        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
httpd-app   2         2         2            0           1m
[root@master k8s_images]# kubectl get pods -o wide
NAME                         READY     STATUS    RESTARTS   AGE       IP           NODE
httpd-app-5fbccd7c6c-5j5zb   1/1       Running   0          3m        10.224.2.2   node-2
httpd-app-5fbccd7c6c-rnkcm   1/1       Running   0          3m        10.224.1.2   node-1


 

因为创建的资源不是service所以不会调用kube-proxy,直接访问测试

#curl http://10.224.2.2

#curl http://10.224.1.2

 

删除应用httpd-app

[root@master ~]# kubectl delete deployment httpd-app
[root@master ~]# kubectl get pods


 

至此kubernetes基本集群安装完成。

手工离线部署k8s(v1.9)_k8s_02 

##问题总结###:

1、如果集群中主master进行重新初始化,并且之前已经加入过node节点,这时如果在原node节点执行kubeadm join --token xxxx时,会提示以下报错:

[root@node-1 ~]# kubeadm join --token 6540e9.c83615e67d622766 192.168.2.40:6443 --discovery-token-ca-cert-hash sha256:34dd77dc3b800a93ffb5fc27b9d7d1e28118f7bb51b0b630afe1153ebcd4f4b8
[preflight] Running pre-flight checks.
[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Some fatal errors occurred:
[ERROR Port-10250]: Port 10250 is in use
[ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
[ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`


解决方法:当集群重新初始化时,原有节点同样也要执行重置命令后,方可重新将节点加入集群

[root@node-1 ~]# kubeadm reset
[preflight] Running pre-flight checks.
[reset] Stopping the kubelet service.
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers.
[reset] No etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]


重新加入成功

[root@node-1 ~]# kubeadm join --token 6540e9.c83615e67d622766 192.168.2.40:6443 --discovery-token-ca-cert-hash sha256:34dd77dc3b800a93ffb5fc27b9d7d1e28118f7bb51b0b630afe1153ebcd4f4b8
[preflight] Running pre-flight checks.
[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "192.168.2.40:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.2.40:6443"
[discovery] Requesting info from "https://192.168.2.40:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.2.40:6443"
[discovery] Successfully established connection with API Server "192.168.2.40:6443"
 
This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.
 
Run 'kubectl get nodes' on the master to see this node join the cluster.