Kubernetes 基于ceph rbd生成pv

Kubernetes 基于ceph rbd生成pv

1、创建ceph-secret这个k8s secret对象，这个secret对象用于k8s volume插件访问ceph集群，获取client.admin的keyring值，并用base64编码，在master1-admin（ceph管理节点）操作

[root@master1-admin ~]# ceph auth get-key client.admin | base64QVFDOWF4eGhPM0UzTlJBQUJZZnVCMlZISVJGREFCZHN0UGhMc3c9PQ==

2.创建ceph的secret，在k8s的控制节点操作：

[root@master ceph]# cat ceph-secret.yaml apiVersion: v1kind: Secretmetadata: name: ceph-secretdata: key: QVFDOWF4eGhPM0UzTlJBQUJZZnVCMlZISVJGREFCZHN0UGhMc3c9PQ==[root@master ceph]# kubectl apply -f ceph-secret.yaml secret/ceph-secret created

3.回到ceph 管理节点创建pool池，然后再创建一个块设备

[root@master1-admin ~]# ceph osd pool create k8stest 56pool 'k8stest' createdYou have new mail in /var/spool/mail/root[root@master1-admin ~]# rbd create rbda -s 1024 -p k8stest[root@master1-admin ~]# rbd feature disable k8stest/rbda object-map fast-diff deep-flatten[root@master1-admin ~]# ceph osd pool lsrbdcephfs_datacephfs_metadatak8srbd1k8stest

4、创建pv

[root@master ceph]# cat pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: ceph-pv spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce rbd: monitors: - '192.168.0.5:6789' - '192.168.0.6:6789' - '192.168.0.7:6789' pool: k8stest image: rbda user: admin secretRef: name: ceph-secret fsType: xfs readOnly: false persistentVolumeReclaimPolicy: Recycle[root@master ceph]# vim pv.yaml[root@master ceph]# kubectl apply -f pv.yaml persistentvolume/ceph-pv created[root@master ceph]# kubectl get pvNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM ceph-pv 1Gi RWO Recycle Available

[root@master ceph]# cat pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: ceph-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi[root@master ceph]# kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEceph-pvc Bound ceph-pv 1Gi RWO 4s[root@master ceph]# kubectl get pvNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGEceph-pv 1Gi RWO Recycle Bound default/ceph-pvc 5h23m

[root@master ceph]# cat pod-2.yaml apiVersion: apps/v1kind: Deploymentmetadata: name: nginx-deploymentspec: selector: matchLabels: app: nginx replicas: 2 # tells deployment to run 2 pods matching the template template: # create pods using pod definition in this template metadata: labels: app: nginx spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent ports: - containerPort: 80 volumeMounts: - mountPath: "/ceph-data" name: ceph-data volumes: - name: ceph-data persistentVolumeClaim: claimName: ceph-pvc[root@master ceph]# kubectl apply -f pod-2.yaml deployment.apps/nginx-deployment created[root@master ceph]# kubectl get pod -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESnginx-deployment-d9f89fd7c-v8rxb 1/1 Running 0 12s 10.233.90.179 node1 nginx-deployment-d9f89fd7c-zfwzj 1/1 Running 0 12s 10.233.90.178 node1

通过上面可以发现pod是可以以ReadWriteOnce共享挂载相同的pvc的

注意：ceph rbd块存储的特点

ceph rbd块存储能在同一个node上跨pod以ReadWriteOnce共享挂载ceph rbd块存储能在同一个node上同一个pod多个容器中以ReadWriteOnce共享挂载ceph rbd块存储不能跨node以ReadWriteOnce共享挂载如果一个使用ceph rdb的pod所在的node挂掉，这个pod虽然会被调度到其它node，但是由于rbd不能跨node多次挂载和挂掉的pod不能自动解绑pv的问题，这个新pod不会正常运行

#将之前所在的节点关闭，可以看到不能使用pvc[root@master ceph]# kubectl get podNAME READY STATUS RESTARTS AGEnfs-client-provisioner-867b44bd69-nhh4m 1/1 Running 0 9m30snfs-client-provisioner-867b44bd69-xscs6 1/1 Terminating 2 18dnginx-deployment-d9f89fd7c-4rzjh 1/1 Terminating 0 15mnginx-deployment-d9f89fd7c-79ncq 1/1 Terminating 0 15mnginx-deployment-d9f89fd7c-p54p9 0/1 ContainerCreating 0 9m30snginx-deployment-d9f89fd7c-prgt5 0/1 ContainerCreating 0 9m30sEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 11m default-scheduler Successfully assigned default/nginx-deployment-d9f89fd7c-p54p9 to node2 Warning FailedAttachVolume 11m attachdetach-controller Multi-Attach error for volume "ceph-pv" Volume is already used by pod(s) nginx-deployment-d9f89fd7c-79ncq, nginx-deployment-d9f89fd7c-4rzjh Warning FailedMount 25s (x5 over 9m26s) kubelet, node2 Unable to attach or mount volumes: unmounted volumes=[ceph-data], unattached volumes=[ceph-data default-token-cwbdx]: timed out waiting for the condition

Deployment更新特性：

deployment触发更新的时候，它确保至少所需 Pods 75% 处于运行状态（最大不可用比例为 25%）。故像一个pod的情况，肯定是新创建一个新的pod，新pod运行正常之后，再关闭老的pod。

默认情况下，它可确保启动的 Pod 个数比期望个数最多多出 25%

问题：

结合ceph rbd共享挂载的特性和deployment更新的特性，我们发现原因如下：

由于deployment触发更新，为了保证服务的可用性，deployment要先创建一个pod并运行正常之后，再去删除老pod。而如果新创建的pod和老pod不在一个node，就会导致此故障。

解决办法：

1，使用能支持跨node和pod之间挂载的共享存储，例如cephfs，GlusterFS等

2，给node添加label，只允许deployment所管理的pod调度到一个固定的node上。（不建议，这个node挂掉的话，服务就故障了）

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。