k8s配置 ceph rbd存储类型storageClass

原创

草根追逐 2020-03-05 11:45:08 博主文章分类：容器编排|虚拟化 ©著作权

©著作权归作者所有：来自51CTO博客作者草根追逐的原创作品，请联系作者获取转载授权，否则将追究法律责任

k8s配置 ceph rbd存储类型storageClass

k8s的storageclass,实现pvc的动态创建,绑定

首先安装ceph参考https://blog.csdn.net/qq_42006894/article/details/88424199

1、部署rbd-provisioner git clone https://github.com/kubernetes-incubator/external-storage.git cd external-storage/ceph/rbd/deploy NAMESPACE=ceph sed -r -i "s/namespace: [^ ]+/namespace: $NAMESPACE/g" ./rbac/clusterrolebinding.yaml ./rbac/rolebinding.yaml kubectl -n $NAMESPACE apply -f ./rbac

2、创建secret 和 pool

###创建admin secret ceph auth get client.admin 2>&1 |grep "key = " |awk '{print $3'} |xargs echo -n > /tmp/key kubectl create secret generic ceph-admin-secret --from-file=/tmp/key --namespace=ceph --type=kubernetes.io/rbd ###创建ceph osd pool ceph osd pool create kube 128 128 ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kube' ###创建用户secret ceph auth get-key client.kube > /tmp/key1 kubectl create secret generic ceph-secret --from-file=/tmp/key1 --namespace=ceph --type=kubernetes.io/rbd

3、创建storageclass

cat ceph-sc-ceph.yml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-rbd-ceph namespace: ceph annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: ceph.com/rbd parameters: monitors: 172.16.13.198:6789,172.16.13.199:6789,172.16.13.200:6789 adminId: admin adminSecretName: ceph-admin-secret adminSecretNamespace: ceph pool: kubernetes userId: kubernetes userSecretName: ceph-secret userSecretNamespace: ceph fsType: ext4 imageFormat: "2" imageFeatures: "layering"

4、配置rdb-provisioner的rbac权限控制 cd external-storage/ceph/rbd/deploy 修改里面的默认命名空间就即可

cat clusterrolebinding.yaml kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: rbd-provisioner1 subjects:

kind: ServiceAccount name: rbd-provisioner1

roleRef: kind: ClusterRole name: rbd-provisioner1 apiGroup: rbac.authorization.k8s.io

cat clusterrole.yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: rbd-provisioner1 namespace: ceph rules:

apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"]
apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"]
apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"]
apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"]
apiGroups: [""] resources: ["services"] resourceNames: ["kube-dns","coredns"] verbs: ["list", "get"]
apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"]

cat deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: rbd-provisioner1 namespace: ceph spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: rbd-provisioner1 spec: containers: - name: rbd-provisioner1 image: "quay.io/external_storage/rbd-provisioner:latest" env: - name: PROVISIONER_NAME value: ceph.com/rbd serviceAccount: rbd-provisioner1

cat rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: rbd-provisioner1 namespace: ceph roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: rbd-provisioner1 subjects:

kind: ServiceAccount name: rbd-provisioner1 namespace: ceph

cat role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: rbd-provisioner1 namespace: ceph rules:

apiGroups: [""] resources: ["secrets"] verbs: ["get"]
apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"]

cat serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata: name: rbd-provisioner1 namespace: ceph

kubectl apply -f .

5、创建pvc cat ceph-pv-ceph.yml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-claim-ceph namespace: ceph spec: accessModes: - ReadWriteOnce storageClassName: ceph-rbd-ceph resources: requests: storage: 10Gi

kubectl apply -f ceph-pv-ceph.yml 查看，如果是bound 状态说明成功 #kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE claim1 Bound pvc-ff3e450b-4629-11e9-9740-080027a073ff 1Gi RWO rbd 51m

如果出错，定位错误方法 kubectl describe pvc/claim1
kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mysql-wayne-8744474d7-kz29c 1/1 Running 0 5d19h 10.244.6.85 node2 <none> <none> rbd-provisioner-67b4857bcd-8vsb8 1/1 Running 0 63m 10.244.3.7 node04 <none> <none> #查看到 rbd-provisioner 在 node04 上，然后再node04上查看容器log docker logs -f 6ed00b76cb55 # 6ed00b76cb55 是容器的id （docker ps 查看）

6,遇到的坑 error retrieving resource lock default/ceph.com-rbd: endpoints “ceph.com-rbd” is forbidden: User “system:serviceaccount:default:rbd-provisioner” cannot get endpoints in the namespace “default” 在 rbd-provisioner.yaml 中 clusterrole最后添加

apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"]

其他参考https://blog.csdn.net/qq_34857250/article/details/82562514

7、参考：https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd

PS设置默认storageclass： kubectl patch storageclass ceph-rbd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

错误提示: MountVolume.WaitForAttach failed for volume "pvc-23555713-da9e-11e9-931b-000c29158017" : fail to check rbd image status with: (executable file not found in $PATH), rbd output: () 解决办法在K8s各个节点执行 yum install -y ceph-common