k8s中使用nfs创建storageClass
k8s环境配置
服务器 | 角色 | 版本 |
---|---|---|
11.0.1.10 | master | v1.24.0 |
11.0.1.11 | worker | v1.24.0 |
11.0.1.12 | worker | v1.24.0 |
11.0.1.13 | nfs |
安装nfs服务器
11.0.1.13
安装nfs服务
yum install -y nfs-utils
配置nfs /etc/exports
/nfs *(rw,sync)
设置目录权限777
chmod 777 /nfs
重启服务
systemctl restart nfs
在其他服务器上也要安装nfs-utils
在k8s中安装nfs组件
NFS subdir external provisioner 是一个存储资源自动调配器,它可用将现有的 NFS 服务器通过持久卷声明来支持 Kubernetes 持久卷的动态分配
组件地址:https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/tree/master/deploy
创建rabc.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
kubectl apply -f rabc.yaml
创建deployment-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
# 镜像地址国内访问不到,需要更换为其他地址
#image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
image: willdockerhub/nfs-subdir-external-provisioner:v4.0.2
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
# provisioner的名字,可以自定义,要和class.yaml中保持一致
- name: PROVISIONER_NAME
value: k8s-sigs.io/nfs-subdir-external-provisioner
# nfs服务器的地址
- name: NFS_SERVER
value: 11.0.1.13
# nfs共享的目录
- name: NFS_PATH
value: /nfs
volumes:
- name: nfs-client-root
nfs:
# nfs服务器的地址
server: 11.0.1.13
# nfs共享的目录
path: /nfs
kubectl apply -f deployment-provisioner.yaml
# kubectl get pod
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-5bd7d76d8f-nrcgj 1/1 Running 1 (4h48m ago) 20h
创建storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-client
# provisioner名称要和deployment里面定义的保持一致
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
archiveOnDelete: "false"
kubectl apply -f storageclass.yaml
# kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-client k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 2d2h
测试
创建pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim
spec:
storageClassName: nfs-client
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Mi
创建pod,挂载pvc
kind: Pod
apiVersion: v1
metadata:
name: test-pod
spec:
containers:
- name: test-pod
image: busybox:stable
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: nfs-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: nfs-pvc
persistentVolumeClaim:
claimName: test-claim