ceph集群搭建

Ceph集群搭建

​​https://blog.csdn.net/xujiamin0022016/article/details/100788957​​参考文档​

ceph基础概念和知识​

ceph架构图​

Rados核心组件 ​

存储数据​

集群监视​

分担mon的部分功能​

存储池​

存储池中pg​

Librados (rados的访问接口)​

radowgw http协议 ​

rdb librbd​

cephfs cephfs​

rados api​

Ceph集群搭建​

ceph集群规划​

Ceph-Mon：3个+
Ceph-Mgr：2个+
Ceph-Mds：2个+​

Ceph-rgw：3个+​

ip​	主机名​	角色​
192.168.1.26​	K8s-master​	ceph-deploy ceph-mon1 ceph-mgr1 osd1 rgw1 mds01​
10.206.0.10​	ceph-mon1​	ceph-mon2 ceph-mgr2 osd2 rgw2 mds02​
10.206.0.13​	ceph-mon1​	ceph-mon3 ceph-mgr3 osd3 rgw3​

机器节点初始化​

hostnamectl set-hostname ceph-mon1 ​

vi /etc/hosts​

配置yum源​

yum install -y yum-utils && sudo yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/ && sudo yum install --nogpgcheck -y epel-release && sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 && sudo rm /etc/yum.repos.d/dl.fedoraproject.org*​

yum -y install wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release lrzsz openssh-server​

yum install deltarpm –y​

#vi /etc/yum.repos.d/ceph.repo​

[Ceph]​

name=Ceph packages for $basearch​

baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/$basearch​

enabled=1​

gpgcheck=1​

type=rpm-md​

gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc​

[Ceph-noarch]​

name=Ceph noarch packages​

baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/noarch​

enabled=1​

gpgcheck=1​

type=rpm-md​

gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc​

[ceph-source]​

name=Ceph source packages​

baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/SRPMS​

enabled=1​

gpgcheck=1​

type=rpm-md​

gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc​

各节点cephadm用户添加sudo权限​

adduser cephadm && echo "cephadm"|passwd --stdin cephadm​

echo "cephadm ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadm​

chmod 0440 /etc/sudoers.d/cephadm​

在管理节点上以cephadm上ssh免密登录​

ssh-keygen -t rsa 部署节点执行​

su cephadm​

cd .ssh​

for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02;do ssh-copy-id -i ~/.ssh/id_rsa.pub $i;done​

管理节点安装ceph-deploy​

yum install ceph-deploy pthon-setuptools python-subprocess32​

管理节点ceph-deploy命令安装mon​

Mon节点初始化相关配置​

在管理节点以cephadm用户运行​

su - cephadm ​

mkdir ceph-cluster && cd ceph-cluster​

ceph-deploy new ceph-mon1--在管理节点以cephadm连接到mon​

安装ceph包​

启动mon命令​

生成ceph.conf配置文件​

[cephadm@ceph-mon1 ceph-cluster]$ ls -ltr​

ceph.mon.keyring​

ceph-deploy-ceph.log​

ceph.conf​

在管理节点以cephadm用户ceph-deploy安装ceph软件​

--no-adjust-repos不调整ceph配置安装集群​

节点提前安装​

yum install ceph radosgw -y​

ceph-deploy install --no-adjust-repos ceph-mon1 ceph-mon2 ceph-mon3 ceph-osd4​

管理节点ceph-deploy初始化mon节点​

启动和激活mon节点，本地根据ceph文件生成秘钥​

ceph-deploy mon create-initial-----在管理节点以cephadm用户运行创建秘钥​

ceph-deply管理节点以将配置和client.admin秘钥环推送到其他所有的集群节点​

ceph-deploy admin ceph-mon1 ceph-mon2 ceph-mon3 ceph-osd4​

在所有节点以root的身份运行​

setfacl -m u:cephadm:r /etc/ceph/ceph.client.admin.keyring​

管理节点ceph-deploy安装Mgr​

Ceph必须要有一个mgr节点，所以我们在管理节点执行：​

su - cephadm​

ceph-deploy mgr create ceph-mon1拷贝文件到/etc/ceph/​

扩容mgr ceph-deploy mgr create ceph-mon2 ceph-mon-----添加mgr(需要添加的节点机器名)​

集群ceph -s ​

ceph集群添加osd节点​

查看​

ceph-deploy disk list ceph-mon1 ceph-mon2 ceph-mon3​

管理节点​

清空osd节点上用来作为osd设备的磁盘​

#要以cephadm用户在~/ceph-cluster/目录下执行 我这边的用来做存储的挂载盘都是dev/sdb 可以通过lsblk查看​

清空osd节点上用来作为osd设备的磁盘​

ceph-deploy disk zap ceph-mon1 /dev/vdb ​

ceph-deploy disk zap ceph-mon2 /dev/vdb ​

ceph-deploy disk zap ceph-mon3 /dev/vdb ​

ceph-deploy disk zap ceph-osd4 /dev/vdb​

创建OSD​

ceph-deploy osd create ceph-mon1 --data /dev/vdb​

ceph-deploy osd create ceph-mon2 --data /dev/vdb​

ceph-deploy osd create ceph-mon3 --data /dev/vdb​

ceph-deploy osd create ceph-osd4 --data /dev/vdb​

扩展Ceph集群​

添加mon节点​

管理节点上修改config​

fsid = d6e4b992-91d4-466b-a076-e6680f34f6e4​

mon_initial_members = ceph-mon1​

mon_host = 192.168.10.152,192.168.10.153,192.168.10.154​

auth_cluster_required = cephx​

auth_service_required = cephx​

auth_client_required = cephx​

public_network = 192.168.10.0/24​

推送​

ceph-deploy --overwrite-conf config push ceph-mon1 ceph-mon2 ceph-mon3 ceph-osd4​

ceph-deploy mon add ceph-mon2​

ceph-deploy mon add ceph-mon3​

ceph quorum_status --format json-pretty​

添加mgr​

ceph-deploy mgr create ceph-mon2 ceph-mon​

添加OSD​

ceph-deploy osd create ceph-mon1 --data /dev/vdb​

ceph-deploy osd create ceph-mon2 --data /dev/vdb​

ceph-deploy osd create ceph-mon3 --data /dev/vdb​

ceph-deploy osd create ceph-osd4 --data /dev/vdb​

安装mds​

su - cephadm​

cd ~/ceph-cluster/​

ceph-deploy mds create ceph-mon2 ceph-mon3​

安装rgw​

在admin节点安装Ceph Object Gateway包到client-node节点​

ceph-deploy install --rgw ceph-mon1 ceph-mon2 ceph-mon3​

修改配置文件并创建Ceph Object Gateway实例​

修改ceph.conf文件​

通过修改ceph.conf配置文件修改默认端口，添加标题为 [client.rgw.<client-node>]的部分，​

用ceph client节点hostname（hostname -s）替换<client-node>.​

如果节点name是client-node，那么在[global]部分之后，添加如下：​

[client.rgw.client-node]​

rgw_frontends = "civetweb port=80"​

注意：​

确保在rgw_frontends键/值对中的 端口=端口号 之间不留空白，​

如果您打算使用端口80，请确保Apache服务器未运行，否则将与Civetweb发生冲突。在这种情况下，我们建议删除Apache。​

推送配置​

ceph-deploy --overwrite-conf config push ceph-mon1 ceph-mon2 ceph-mon3 ceph-osd4​

创建Ceph Object Gateway实例​

ceph-deploy rgw create ceph-mon1 ceph-mon2 ceph-mon3​

Ceph卸载​

清除ceph存储​

清除安装包​

[cephuser@ceph-admin ~]$ ceph-deploy purge ceph1 ceph2 ceph3​

清除配置信息​

[cephuser@ceph-admin ~]$ ceph-deploy purgedata ceph1 ceph2 ceph3​

[cephuser@ceph-admin ~]$ ceph-deploy forgetkeys​

每个节点删除残留的配置文件​

[cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/osd/*​

[cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/mon/*​

[cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/mds/*​

[cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/bootstrap-mds/*​

[cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/bootstrap-osd/*​

[cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/bootstrap-mon/*​

[cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/tmp/*​

[cephuser@ceph-admin ~]$ sudo rm -rf /etc/ceph/*​

[cephuser@ceph-admin ~]$ sudo rm -rf /var/run/ceph/*​

Ceph使用​

Ceph常用维护命令​

ceph --help ​

Rbd操作​

创建存储池​

ceph osd pool create testpool（pool名称） 128（pg数量）​

通常在创建pool之前，需要覆盖默认的pg_num，​

官方推荐：​

若少于5个OSD， 设置pg_num为128。​

5~10个OSD，设置pg_num为512。​

10~50个OSD，设置pg_num为4096。​

超过50个OSD，可以参考pgcalc计算。​

查看存储池​

ceph osd lspools ​

创建rbd​

rbd create testpool/myimage --image-feature layering --size 1024​

testpool（pool名称）/myimage（image名）​

查看rdb​

rdb list ​

rbd info testpool/myimage​

映射块设备到自己机器​

rbd map testpool/myimage​

客户端服务器安装cephfs​

先修改/etc/yum.repos.d/ceph.repo如下​

[Ceph]​

name=Ceph packages for $basearch​

baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/$basearch​

enabled=1​

gpgcheck=1​

type=rpm-md​

gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc​

[Ceph-noarch]​

name=Ceph noarch packages​

baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/noarch​

enabled=1​

gpgcheck=1​

type=rpm-md​

gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc​

[ceph-source]​

name=Ceph source packages​

baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/SRPMS​

enabled=1​

gpgcheck=1​

type=rpm-md​

gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc​

关闭防火墙和selinux​

systemctl stop firewalld.service​

systemctl disable firewalld.service​

setenforce 0​

sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config​

安装ceph-fuse​

yum -y install epel-release​

yum clean all​

yum repolist​

yum -y install ceph-fuse​

复制ceph.client.admin.keyring和ceph.conf 到本机的/etc/ceph下 这里由于96没有改host就用了ip​

修改/etc/systemd/system/ceph-fuse.service 如下​

[root@localhost install]# cat /etc/systemd/system/ceph-fuse.service ​

[Unit]​

Descriptinotallow=Ceph FUSE client​

After=network-online.target local-fs.target time-sync.target​

Wants=network-online.target local-fs.target time-sync.target​

Cnotallow=umount.target​

PartOf=ceph-fuse.target​

[Service]​

Envirnotallow=-/etc/sysconfig/ceph​

Envirnotallow=CLUSTER=ceph​

#ExecStart=/usr/bin/ceph-fuse -f --cluster ${CLUSTER} %I​

ExecStart=/usr/bin/ceph-fuse -f -o rw,noexec,nosuid,nodev /mnt​

TasksMax=infinity​

Restart=on-failure​

StartLimitInterval=30min​

StartLimitBurst=3​

[Install]​

WantedBy=ceph-fuse.target​

重启服务​

systemctl start ceph-fuse.service​

systemctl status ceph-fuse.service​

mkdir -p /mnt/mycephfs2​

ceph-fuse -m 192.168.10.152:6789 /mnt/mycephfs2​

在/mnt下创建一个test文件​

在/mnt下创建一个test文件​

echo "123" > /mnt/mycephfs2/123abc.txt​

测试原来上传下载功能不小心手残没打tmp后没加/​

[root@localhost mycephfs2]# rados put 123abc.txt /mnt/mycephfs2/123abc.txt --pool=cephfs_data​

[root@localhost mycephfs2]# rados get 123abc.txt /tmp123abc.txt -p cephfs_data​

[root@localhost mycephfs2]# ll​

总用量​

-rw-r--r--. 1 root root 4 9月 7 16:40 123abc.txt​

[root@localhost mycephfs2]# cat /tmp123abc.txt ​

123​

卸载挂载块​

[root@localhost mycephfs2]# fusermount -zu /mnt/​

mycephfs/ mycephfs2/ test ​

[root@localhost mycephfs2]# fusermount -zu /mnt/mycephfs2​

[root@localhost mycephfs2]# df -h​

文件系统 容量已用可用已用% 挂载点​

/dev/mapper/centos-root 50G 19G 32G 37% /​

devtmpfs 1.9G 0 1.9G 0% /dev​

tmpfs 1.9G 0 1.9G 0% /dev/shm​

tmpfs 1.9G 8.9M 1.9G 1% /run​

tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup​

/dev/sda1 1014M 145M 870M 15% /boot​

/dev/mapper/centos-home 46G 33M 46G 1% /home​

tmpfs 379M 0 379M 0% /run/user/0​

文件被存到了ceph中 卸载挂载块文件已经不存在​

[root@localhost mycephfs2]# cd /mnt/mycephfs2/​

[root@localhost mycephfs2]# ll​

总用量​

再次下载文件测试​

[root@localhost mycephfs2]# rados get 123abc.txt /tmp/123abc.txt -p cephfs_data​

[root@localhost mycephfs2]# cat /tmp/123abc.txt ​

123​

Ceph-cephfs​

Ceph文件系统至少需要两个RADOS池，一个用于数据，一个用于元数据​

创建cephfs​

查看mds服务是否启用​

ceph mds stat​

创建数据pool和Metadata池​

ceph osd pool create cephfs_data 128 ​

ceph osd pool create cephfs_metadata 128​

启用pool​

ceph fs new cephfs cephfs_metadata cephfs_data ​

查看cephfs​

ceph fs ls​

挂载cephfs​

kernel driver挂载​

注意：检查是否启用cephx认证方法，如果值为none为禁用，cephx为启用​

关闭认证情况下挂载​

mount -t ceph 192.168.0.1:6789:/ /mnt/wyl​

cat ceph.client.admin.keyring​

mount -t ceph xxx.xxx.xxx.xxx:6789:/ /wyl -o name=admin,secret=xxxx​

fuse挂载​

安装挂载工具​

yum -y install ceph-fuse ceph​

将存储节点的admin秘钥拷贝到挂载客户端​

ceph-authtool -p /etc/ceph/ceph.client.admin.keyring > admin.key​

chmod 600 admin.key​

挂载​

mount -t ceph node1:6789:/ /mnt -o name=admin,secretfile=admin.key​