【Ceph】基于VMware Workstation虚拟机Ceph集群安装配置笔记

1.写在前面

本文主要内容为搭建三节点Ceph集群的过程。

这两天捣鼓了一下Ceph，准备做个底层基于ceph的nas存储，首先网上看了一些教程，还有对照了一下书籍，依然在安装配置时ceph还是遇到很多问题，所以写此贴记录一下自己遇到的问题和安装过程，同时也希望能够帮助一些小伙伴搭建Ceph集群。

本文使用 ceph-deploy 官方工具部署。

2.安装环境

linux版本，精简版

[root@node1 ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)

Ceph版本 Giant 

[root@node1 ~]# ceph -v
ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e)

VMware workstation版本

网络（桥接物理主机）

node1 192.168.68.11

node2 192.168.68.12

node3 192.168.68.13

网关 192.168.68.1

3.安装步骤

3.1 VMware Workstation 安装虚机步骤

3.2 centos7安装过程

node2、node3也是如此，局部将1改为2或3即可（或者使用Vmware Workstation克隆功能，这里不再演示）

给三个节点虚机分别增加3块硬盘

3.3 安装Ceph前虚机配置（三个节点都需要）

配置hosts访问

vi /etc/hosts

配置node1免密访问node2、3

ssh-keygen

ssh-copy-id node2

ssh-copy-id node3

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

关闭selinux

setenforce 0
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config

安装并配置ntp时钟服务

yum install -y ntp ntpdate -y
ntpdate pool.net.org
systemctl restart ntpdate.service
systemctl restart ntp.service
systemctl enable ntp.service
systemctl enable ntpdate.service

添加Ceph Giant 版本包 更新yum

rpm -Uhv http://download.ceph.com/rpm-giant/el7/noarch/ceph-release-1-0.el7.noarch.rpm

修改ceph.repo配置文件（提示':%d'可快速清除此页内容）

vi /etc/yum.repos.d/ceph.repo

[Ceph]
name=Ceph packages for $basearch
baseurl=http://download.ceph.com/rpm-giant/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://download.ceph.com/rpm-giant/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

[ceph-source]
name=Ceph source packages
baseurl=http://download.ceph.com/rpm-giant/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

更新yum

yum -y update

3.4 安装和配置Ceph

强烈建议此步骤前将三台虚机快照备份

3.4.1 在node1上创建Ceph集群

安装ceph-deploy

yum install ceph-deploy -y

创建一个ceph集群

mkdir /etc/ceph
cd /etc/ceph
ceph-deploy new node1

在node1节点给个节点安装ceph

ceph-deploy install node1 node2 node3

查看安装ceph版本

ceph -v

在node1上创建第一个ceph monitor

ceph-deploy mon create-initial

检查集群状态

ceph -s

在node1上创建OSD(在ceph目录下执行)

列出可用磁盘

ceph-deploy disk list node1

删除选择磁盘的分区和内容

ceph-deploy disk zap node1:sdb node1:sdc node1:sdd

创建osd

ceph-deploy osd create node1:sdb node1:sdc node1:sdd

检查集群状态

ceph -s

此时单节点配置完毕

3.4.2 拓展Ceph集群

在node1上将共有网络地址添加到文件/etc/ceph/ceph.conf

public network = 192.168.68.0/24

分别在node2、3上创建monitor

ceph-deploy mon create node2
ceph-deploy mon create node3

查看集群状态

ceph -s

此时另外两个节点已成功加入

处理node2、3节点的磁盘并创建OSD

ceph-deploy disk zap node2:sdb node2:sdc node2:sdd
ceph-deploy disk zap node3:sdb node3:sdc node3:sdd

ceph-deploy osd create node2:sdb node2:sdc node2:sdd
ceph-deploy osd create node3:sdb node3:sdc node3:sdd

调整rbd存储池pg_num和pgp_num的值，使我们的集群达到HEALTH_OK状态

ceph osd pool set rbd pg_num 256
ceph osd pool set rbd pgp_num 256

检查集群状态，如果为HEALTH_OK，集群已经完整搭建并且正常了

4.遇到的一些错误

原因：修改了ceph用户里的ceph.conf文件内容，但是没有把这个文件里的最新消息发送给其他节点，所有要推送消息

ceph-deploy --overwrite-conf config push node2

或

ceph-deploy --overwrite-conf mon create node2

5.一些常用命令

检查Ceph安装状态

ceph -s

ceph status

[root@node1 ceph]# ceph -s
    cluster 5e563d2f-94e6-4d9b-8aaf-6b2c76f856e4
     health HEALTH_OK
     monmap e3: 3 mons at {node1=192.168.68.11:6789/0,node2=192.168.68.12:6789/0,node3=192.168.68.13:6789/0}, election epoch 4, quorum 0,1,2 node1,node2,node3
     osdmap e53: 9 osds: 9 up, 9 in
      pgmap v122: 256 pgs, 1 pools, 0 bytes data, 0 objects
            318 MB used, 134 GB / 134 GB avail
                 256 active+clean
                 
[root@node1 ceph]# ceph -s
    cluster 5e563d2f-94e6-4d9b-8aaf-6b2c76f856e4
     health HEALTH_OK
     monmap e3: 3 mons at {node1=192.168.68.11:6789/0,node2=192.168.68.12:6789/0,node3=192.168.68.13:6789/0}, election epoch 4, quorum 0,1,2 node1,node2,node3
     osdmap e53: 9 osds: 9 up, 9 in
      pgmap v122: 256 pgs, 1 pools, 0 bytes data, 0 objects
            318 MB used, 134 GB / 134 GB avail
                 256 active+clean

查看Ceph版本

ceph -v

[root@node1 ceph]# ceph -v
ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e)

观察集群健康状况

ceph -

[root@node1 ceph]# ceph -w
    cluster 5e563d2f-94e6-4d9b-8aaf-6b2c76f856e4
     health HEALTH_OK
     monmap e3: 3 mons at {node1=192.168.68.11:6789/0,node2=192.168.68.12:6789/0,node3=192.168.68.13:6789/0}, election epoch 4, quorum 0,1,2 node1,node2,node3
     osdmap e53: 9 osds: 9 up, 9 in
      pgmap v122: 256 pgs, 1 pools, 0 bytes data, 0 objects
            318 MB used, 134 GB / 134 GB avail
                 256 active+clean

2022-04-25 10:27:09.830678 mon.0 [INF] pgmap v122: 256 pgs: 256 active+clean; 0 bytes data, 318 MB used, 134 GB / 134 GB avail

检查Ceph monitor仲裁状态

ceph quorum_status --format json-pretty

[root@node1 ceph]# ceph quorum_status --format json-pretty

{ "election_epoch": 4,
  "quorum": [
        0,
        1,
        2],
  "quorum_names": [
        "node1",
        "node2",
        "node3"],
  "quorum_leader_name": "node1",
  "monmap": { "epoch": 3,
      "fsid": "5e563d2f-94e6-4d9b-8aaf-6b2c76f856e4",
      "modified": "2022-04-25 10:06:48.209985",
      "created": "0.000000",
      "mons": [
            { "rank": 0,
              "name": "node1",
              "addr": "192.168.68.11:6789\/0"},
            { "rank": 1,
              "name": "node2",
              "addr": "192.168.68.12:6789\/0"},
            { "rank": 2,
              "name": "node3",
              "addr": "192.168.68.13:6789\/0"}]}}

列表PG

ceph pg dump

列表Ceph存储池

ceph osd lspools

[root@node1 ceph]# ceph osd lspools
0 rbd,

列表集群的认证密钥

ceph auth list

[root@node1 ceph]# ceph auth list
installed auth entries:

osd.0
  key: AQA7GGViMOKvBhAApRSMC8DDLnlOQXmAD7UUDQ==
  caps: [mon] allow profile osd
  caps: [osd] allow *
osd.1
  key: AQBEGGVioIu6IxAAtFI6GkzHH86f5DbZcFLP+Q==
  caps: [mon] allow profile osd
  caps: [osd] allow *
osd.2
  key: AQBMGGViMHbdKxAAlahPljoMpYC5gRoJBPwmcg==
  caps: [mon] allow profile osd
  caps: [osd] allow *
osd.3
  key: AQB9BWZiwNzrOhAADsHBX/QZgBgZ/5SbJ9wFlg==
  caps: [mon] allow profile osd
  caps: [osd] allow *
osd.4
  key: AQCHBWZiECT2IBAApALFn7F7IDMW/ctkL8BAsA==
  caps: [mon] allow profile osd
  caps: [osd] allow *
osd.5
  key: AQCPBWZisNn3NxAAiDcZGUPWY+e3lflW+7c6AQ==
  caps: [mon] allow profile osd
  caps: [osd] allow *
osd.6
  key: AQCdBWZiQLATHxAAc7z2NE3FmFUx28dIXeHN2g==
  caps: [mon] allow profile osd
  caps: [osd] allow *
osd.7
  key: AQCnBWZiIPh+CRAA/hDBfG/iwChiNcVvB4lw2Q==
  caps: [mon] allow profile osd
  caps: [osd] allow *
osd.8
  key: AQCvBWZiQE9vARAAroqxul/dQRsDnN7Cz9pkAA==
  caps: [mon] allow profile osd
  caps: [osd] allow *
client.admin
  key: AQCRF2ViOAyLMBAAip1+6gqV2wJmxYUlrBzdFQ==
  caps: [mds] allow
  caps: [mon] allow *
  caps: [osd] allow *
client.bootstrap-mds
  key: AQCSF2ViiMcjChAAncKTNo7o5sGaKFvoJHEFmA==
  caps: [mon] allow profile bootstrap-mds
client.bootstrap-osd
  key: AQCSF2ViUB/CABAAPXRQtcShw39kI6xYr51Cdw==
  caps: [mon] allow profile bootstrap-osd

检查集群使用状态

ceph df

[root@node1 ceph]# ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED 
    134G      134G         318M          0.23 
POOLS:
    NAME     ID     USED     %USED     MAX AVAIL     OBJECTS 
    rbd      0         0         0        45938M           0

检查OSD的Crush map

ceph osd tree

[root@node1 ceph]# ceph osd tree
# id  weight  type name up/down reweight
-1  0.08995 root default
-2  0.02998   host node1
0 0.009995      osd.0 up  1 
1 0.009995      osd.1 up  1 
2 0.009995      osd.2 up  1 
-3  0.02998   host node2
3 0.009995      osd.3 up  1 
4 0.009995      osd.4 up  1 
5 0.009995      osd.5 up  1 
-4  0.02998   host node3
6 0.009995      osd.6 up  1 
7 0.009995      osd.7 up  1 
8 0.009995      osd.8 up  1

查看osd黑名单ceph osd blacklist ls

[root@node1 ceph]# ceph osd blacklist ls
listed 0 entries

（作者水平有限，文章如有错误，烦请指正。）