# kubernetes 1.9.0 kubeadm方式安装 # 1、安装rpm包 ``` yum localinstall -y kubeadm-1.9.0-0.x86_64.rpm kubectl-1.9.0-0.x86_64.rpm kubelet-1.9.0-0.x86_64.rpm kubernetes-cni-0.6.0-0.x86_64.rpm ``` # 2、修改内核参数 修改 /etc/sysctl.conf,添加以下内容 ``` net.ipv4.ip_forward=1 net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 ``` 修改后,及时生效 ``` sysctl -p ``` # 3、修改kubelet配置文件 kubelet和docker 的cgroup driver 有2种方式:cgroupfs和systemd.注意保持 2个应用的driver保持一致。 ## 3.1 docker是cgroupfs的,修改kubelet vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ``` #修改systemd为cgroupfs Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs" #新加一行 Environment="KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=foxchan/google_containers/pause-amd64:3.0" ``` 修改完成后 ``` systemctl daemon-reload ``` ## 3.2 docker是systemd的,kubelet无需修改 docker 启动命令添加如下内容,可以修改cgroup driver为systemd ``` --exec-opt native.cgroupdriver=systemd ``` # 4、安装 kubernetes ``` kubeadm init --kubernetes-version=1.9.0 --token-ttl 0 ``` 参数说明 - --kubernetes-version 最好指定版本,否则会请求 https://storage.googleapis.com/kubernetes-release/release/stable-1.9.txt ,如果没"翻""墙",就超时报错 - --token-ttl 默认的token有效期24小时, 设置为0表示永不过期 ## 因为镜像要"翻""墙",可以预先下载镜像,然后更改tag 镜像列表 - gcr.io/google_containers/kube-apiserver-amd64:v1.9.0 - gcr.io/google_containers/kube-controller-manager-amd64:v1.9.0 - gcr.io/google_containers/kube-scheduler-amd64:v1.9.0 - gcr.io/google_containers/pause-amd64:3.0 - gcr.io/google_containers/etcd-amd64:3.1.10 - gcr.io/google_containers/kube-proxy-amd64:v1.9.0 - gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7 - gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7 - gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7 - quay.io/calico/node:v3.0.1 - quay.io/coreos/etcd:v3.2.4 - quay.io/calico/cni:v2.0.0 - quay.io/calico/kube-controllers:v1.0.1 如果下载自己的或者dockerhub的镜像。可以利用脚本,批量替换镜像imagename ``` docker images | sed 's/foxchan/gcr.io\/google_containers/'| awk '{print "docker tag "$3" "$1":"$2}' ``` 安装信息 ``` [root@kvm-gs242024 ~]# kubeadm init --kubernetes-version=1.9.0 --token-ttl 0 --ignore-preflight-errors=all [init] Using Kubernetes version: v1.9.0 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks. [WARNING FileExisting-crictl]: crictl not found in system path [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [kvm-gs024 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.24] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated sa key and public key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki" [kubeconfig] Wrote KubeConfig file to disk: "admin.conf" [kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf" [controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml" [controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml" [controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml" [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml" [init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests". [init] This might take a minute or longer if the control plane images have to be pulled. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused. [apiclient] All control plane components are healthy after 78.502690 seconds [uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [markmaster] Will mark node kvm-gs242024 as master by adding a label and a taint [markmaster] Master kvm-gs242024 tainted and labelled with key/value: node-role.kubernetes.io/master="" [bootstraptoken] Using token: 1ac970.704ce2d03cc45382 [bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: kube-dns [addons] Applied essential addon: kube-proxy Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join --token 1ac970.704ce2d03cc45382 192.168.0.24:6443 --discovery-token-ca-cert-hash sha256:f70f07be83a7b2af2c41752b00def4389e3019006b3be643fe1ccf1c53368043 ``` # 安装后提示如何管理集群和添加node token要记得保存,当前版本 token 无法通过命令找回,否则无法添加node # master节点验证 ``` [root@kvm-master ~]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health": "true"} ``` # node节点验证 ``` [root@kvm-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kvm-node1 Ready 1h v1.9.0 kvm-master NotReady master 18h v1.9.0 kvm-node2 Ready 6m v1.9.0 ``` # 命令行验证 ``` curl --cacert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key https://k8smaster:6443 ``` ## 管理集群使用kubectl的报错 Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes") https://github.com/kubernetes/kubernetes/issues/48378 解决方式: export KUBECONFIG=/etc/kubernetes/kubelet.conf 这是普通用户,没有权限,会有报错 Error from server (Forbidden): daemonsets.extensions is forbidden: User "system:node:kvm-master" cannot list daemonsets.extensions in the namespace "default" export KUBECONFIG=/etc/kubernetes/admin.conf 管理用户 ###由于安全原因,默认情况下pod不会被schedule到master节点上,可以通过下面命令解除这种限制: ``` kubectl taint nodes --all node-role.kubernetes.io/master- ``` # calico网络安装 先装rbac kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/rbac.yaml 下载calico.yaml https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/calico.yaml 如果使用calico自带etcd,注意保证calico etcd的稳定 如果使用自有etcd集群,需要修改yaml里的etcd_endpoints 默认安装时,如果节点是多网卡会报错,导致网络不成功,calico 总是随机绑定网卡,导致注册失败 calico 报错日志 Skipping datastore connection test IPv4 address 10.96.0.1 discovered on interface kube-ipvs0 No AS number configured on node resource, using global value 需要修改calico.yaml,注意顺序 - name: IP value: "autodetect" - name: IP_AUTODETECTION_METHOD value: "can-reach=192.168.1.1" IP_AUTODETECTION_METHOD 参数说明 使用通过ip访问的interface can-reach=192.168.1.1 使用通过域名访问的interface can-reach=www.baidu.com 使用指定的interface interface=ethx ## 安装dashboard 下载yaml ``` wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml ``` 下载完成后修改imagename ``` name: kubernetes-dashboard image: foxchan/google_containers/kubernetes-dashboard-amd64:v1.8.0 ``` ``` kubectl proxy --address=masterip --accept-hosts='^*$' ``` 访问 ``` http://masterip:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy ``` ## 安装heapster 下载yaml ``` wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml ``` 下载完成后,修改image为自己的 ``` #grafana.yaml - name: grafana image: gcr.io/google_containers/heapster-grafana-amd64:v4.4.3 #heapster.yaml - name: heapster image: gcr.io/google_containers/heapster-amd64:v1.4.2 #influxdb.yaml - name: influxdb image: gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3 ``` 下载镜像 ``` docker pull foxchan/heapster-grafana-amd64:v4.4.3 docker pull foxchan/heapster-amd64:v1.4.2 docker pull foxchan/heapster-influxdb-amd64:v1.3.3 ``` 如果成功,dashboard页面 会有图形 # 将skydns替换为coredns 下载脚本 ``` wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed ``` 利用脚本 修改coredns.yaml.sed配置文件 ``` deploy.sh clusterip|kubectl apply ``` 我的clusterip是10.96.0.0/12 ``` ./deploy.sh 10.96.0.0/12|kubectl apply -f ``` 请确保coredns正常运行,然后就可以删除skydns ``` kubectl delete --namespace=kube-system deployment kube-dns ``` ### 另外 kubeadm 也添加了替换dns功能,需要kubeadm 1.9版本 参数如下: ``` kubeadm init --feature-gates=CoreDNS=true ``` # 总结 总的来说1.9 和1.8 没什么大的变化,以下是我关注的 - Bootstrap Tokens 从alpha 变为beta。 - etcd 版本从3.0 升级到3.1,因为etcd 版本数据是不兼容的,如果要升级etcd,需要提前备份数据。 - kube-proxy 的ipvs 从alpha变为beta - coredns 比较有前景,可以持续关注