MongoDB分片群集
什么是分片
高数据量和吞吐量的数据库应用会对单机的性能造成较大的压力,大的查询量会把单机CPU耗尽,大的数据量对单机的储存压力大,最终会消耗系统的内存而将压力转移到磁盘IO上
MongoDB分片的优势
分片为应对高吞吐量与大数据量提供了方法
- 使用分片减少了对需要处理的请求,因此,通过水平拓展,群集可以提高自己的储存量和吞吐量。
- 使用分片减少了每个分片存储的数据
分片服务器分为以下几个服务器组成
-
路由服务器 (router)
-
配置服务器 (configsvr)
-
分片服务器 (shardsvr)
我们通过路由服务器发送请求给配置服务器,配置服务器调用下面的分片服务器
结构图如下
MongoDB分片群集的部署
====(注:分片群集在3.2和3.2以上的版本不同3.2版本在做分片服务器的时候可以一个分片服务器对应一台数据库,但是在3.2以上的版本它要求每个分片服务器必须做群集)====
这边我们用的是3.2版本的源代码包
实验步骤:做配置服务器
---------------------安装MongoDB 3.2----------------------------
#安装开发包
yum install openssl-devel -y
#解压源代码包
tar zxvf mongodb-linux-x86_64-3.2.1.tgz -C /opt/
#重命名并移动到/usr/local文件里面
mv mongodb-linux-x86_64-3.2.1/ /usr/local/mongodb
#创建数据存放位置
mkdir -p /data/mongodb/mongodb{1,2,3,4}
#创建日志存放位置
mkdir /data/mongodb/logs
创建以log为结尾的日志存放
touch /data/mongodb/logs/mongodb{1,2,3,4}.log
#给日志文件最大全权限
chmod -R 777 /data/mongodb/logs/*.log
#修改最大并发连接
ulimit -n 25000
ulimit -u 25000
上面是开始安装MongoDB,下面开始配置,配置服务器的配置文件
cd /usr/local/mongodb/bin/
vim mongodb1.conf
#配置服务器端口号
port=37017
#数据文件储存位置
dbpath=/data/mongodb/mongodb1
#日志文件储存位置
logpath=/data/mongodb/logs/mongodb1.log
logappend=true
fork=true
#同时承受的并发连接数
maxConns=5000
storageEngine=mmapv1
#配置为配置服务器
configsvr=true
某节点内存不足时,从其他节点分配内存
sysctl -w vm.zone_reclaim_mode=0
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
#建立软连接便于管理
ln -s /usr/local/mongodb/bin/mongo /usr/bin/mongo
ln -s /usr/local/mongodb/bin/mongod /usr/bin/mongod
#开启配置服务器
mongod -f /usr/local/mongodb/bin/mongodb1.conf
分片服务器 1和分片服务器 2同样配置改端口号数据文件日志文件位置加入shardsvr=true删除原有的配置服务器configsvr=true
#端口号改为47017
port=37017
#数据文件储存位置
dbpath=/data/mongodb/mongodb2
#日志文件储存位置
logpath=/data/mongodb/logs/mongodb.log
logappend=true
fork=true
#同时承受的并发连接数
maxConns=5000
storageEngine=mmapv1
#配置为分片服务器
shardsvr=true
#分别开启分片服务器1和2
启动路由服务器
cd /usr/local/mongodb/bin/
#启动mongos 路由服务器的端口号为27017日志存放目录在/usr/local/mongodb/bin/下面的route.log里面
配置服务IP:端口号 --chunkSize 1 块大小为1
./mongos --port 27017 --fork --logpath=/usr/local/mongodb/bin/route.log --configdb 192.168.235.204:37017 --chunkSize 1
直接启动mongo路由服务器
[root@localhost bin]# mongo
MongoDB shell version: 3.2.1
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Server has startup warnings:
2018-07-18×××4:52:06.622+0800 I CONTROL [main] ** WARNING: You are running this process as the root user, which is not recommended.
2018-07-18×××4:52:06.622+0800 I CONTROL [main]
mongos>
#查看数据库
mongos> show dbs
config 0.031GB
mongos> sh.status() #shards下为空,没有分片服务器
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5b4ee3961c914459f603260f")
}
shards:
active mongoses:
"3.2.1" : 1
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
#添加分片服务器
mongos> sh.addShard("192.168.32.207:47018")
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> sh.addShard("192.168.32.207:47017")
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5b4ee3961c914459f603260f")
}
shards: #下面两个是分片服务器
{ "_id" : "shard0000", "host" : "192.168.32.207:47018" }
{ "_id" : "shard0001", "host" : "192.168.32.207:47017" }
active mongoses:
"3.2.1" : 1
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
实验分片功能
添加两个分片服务器后,数据库和集合还未启用分片使用mongoimport命令导入sales.txt数据到kgc数据库的sales表
mongos> db.createCollection('users')
{ "ok" : 1 }
mongos> show collections
system.indexes
users
mongos> exit
bye
[root@localhost bin]# mongo
MongoDB shell version: 3.2.1
connecting to: test
Server has startup warnings:
2018-07-18×××4:52:06.622+0800 I CONTROL [main] ** WARNING: You are running this process as the root user, which is not recommended.
2018-07-18×××4:52:06.622+0800 I CONTROL [main]
mongos> show dbs
config 0.031GB
kgc 0.078GB
mongos> for (var i=1;i<=10000;i++)db.users.insert({"id":1,"name":"jaap"+i}) #插入10000条信息
WriteResult({ "nInserted" : 1 })
mongos> show tables
system.indexes
users
mongos> db.users.find() #查看信息
{ "_id" : ObjectId("5b4ef398b83d741f7c50491a"), "id" : 1, "name" : "zhang" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50491b"), "id" : 1, "name" : "jaap1" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50491c"), "id" : 1, "name" : "jaap2" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50491d"), "id" : 1, "name" : "jaap3" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50491e"), "id" : 1, "name" : "jaap4" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50491f"), "id" : 1, "name" : "jaap5" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504920"), "id" : 1, "name" : "jaap6" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504921"), "id" : 1, "name" : "jaap7" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504922"), "id" : 1, "name" : "jaap8" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504923"), "id" : 1, "name" : "jaap9" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504924"), "id" : 1, "name" : "jaap10" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504925"), "id" : 1, "name" : "jaap11" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504926"), "id" : 1, "name" : "jaap12" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504927"), "id" : 1, "name" : "jaap13" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504928"), "id" : 1, "name" : "jaap14" }
{ "_id" : ObjectId("5b4ef407b83d741f7c504929"), "id" : 1, "name" : "jaap15" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50492a"), "id" : 1, "name" : "jaap16" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50492b"), "id" : 1, "name" : "jaap17" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50492c"), "id" : 1, "name" : "jaap18" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50492d"), "id" : 1, "name" : "jaap19" }
Type "it" for more
mongos> db.users.find().limit(5) #查看指定信息
{ "_id" : ObjectId("5b4ef398b83d741f7c50491a"), "id" : 1, "name" : "zhang" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50491b"), "id" : 1, "name" : "jaap1" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50491c"), "id" : 1, "name" : "jaap2" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50491d"), "id" : 1, "name" : "jaap3" }
{ "_id" : ObjectId("5b4ef407b83d741f7c50491e"), "id" : 1, "name" : "jaap4" }
mongos> sh.status() #查看数据库分片信息
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5b4ee3961c914459f603260f")
}
shards:
{ "_id" : "shard0000", "host" : "192.168.32.207:47018" }
{ "_id" : "shard0001", "host" : "192.168.32.207:47017" }
active mongoses:
"3.2.1" : 1
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "kgc", "primary" : "shard0000", "partitioned" : false }
启用分片
mongos> sh.enableSharding("kgc")
{ "ok" : 1 }
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5b4ee3961c914459f603260f")
}
shards:
{ "_id" : "shard0000", "host" : "192.168.32.207:47018" }
{ "_id" : "shard0001", "host" : "192.168.32.207:47017" }
active mongoses:
"3.2.1" : 1
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "kgc", "primary" : "shard0000", "partitioned" : true
#对users表中id创建主键索引
mongos> db.users.createIndex({"id":1})
#表分片
mongos> sh.shardCollection("kgc.users",{"id":1})
#查看分片情况
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5b4ee3961c914459f603260f")
}
shards:
{ "_id" : "shard0000", "host" : "192.168.32.207:47018" }
{ "_id" : "shard0001", "host" : "192.168.32.207:47017" }
active mongoses:
"3.2.1" : 1
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "kgc", "primary" : "shard0000", "partitioned" : true }
kgc.users
shard key: { "id" : 1 }
unique: false
balancing: true
chunks:
shard0000 1
{ "id" : { "$minKey" : 1 } } -->> { "id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)
添加标签
#添加分片服务器
mongod -f mongodb4.conf
mongo
mongos> sh.addShard("192.168.235.205:47019")
mongos> sh.status()
chunks:
shard0000 4
shard0001 4
shard0002 3
#删除分片节点
mongos> use admin
mongos> db.runCommand({"removeshard":"192.168.235.205:47019"})