***mongodb的分片配置
md01 10.0.0.11
md02 10.0.0.12
md03 10.0.0.14

2、启动三台机器的mongod实例

根据Replica Set、Sharding策略部署mongod。将两个sharding组部署到三台服务器上,每个sharding组有三个replica set成员。

#Server1:
  1. mkdir -p /elain/data/mongodb/shard11
  2. mkdir -p /elain/data/mongodb/shard21
  3. /elain/apps/mongodb/bin/mongod --shardsvr --replSet shard1 --port 27017 --dbpath /elain/data/mongodb/shard11 --oplogSize 100 --logpath /elain/data/mongodb/shard11.log --logappend --fork --rest
  4. /elain/apps/mongodb/bin/mongod --shardsvr --replSet shard2 --port 27018 --dbpath /elain/data/mongodb/shard21 --oplogSize 100 --logpath /elain/data/mongodb/shard21.log --logappend --fork --rest
#Server2:
  1. mkdir -p /elain/data/mongodb/shard12/
  2. mkdir -p /elain/data/mongodb/shard22/
  3. /elain/apps/mongodb/bin/mongod --shardsvr --replSet shard1 --port 27017 --dbpath /elain/data/mongodb/shard12 --oplogSize 100 --logpath /elain/data/mongodb/shard12.log --logappend --fork --rest
  4. /elain/apps/mongodb/bin/mongod --shardsvr --replSet shard2 --port 27018 --dbpath /elain/data/mongodb/shard22 --oplogSize 100 --logpath /elain/data/mongodb/shard22.log --logappend --fork --rest
#Server3:
  1. mkdir -p /elain/data/mongodb/shard13/
  2. mkdir -p /elain/data/mongodb/shard23/
  3. /elain/apps/mongodb/bin/mongod --shardsvr --replSet shard1 --port 27017 --dbpath /elain/data/mongodb/shard13 --oplogSize 100 --logpath /elain/data/mongodb/shard13.log --logappend --fork --rest
  4. /elain/apps/mongodb/bin/mongod --shardsvr --replSet shard2 --port 27018 --dbpath /elain/data/mongodb/shard23 --oplogSize 100 --logpath /elain/data/mongodb/shard23.log --logappend --fork --rest
3、初始化Replica Set 通过命令行初始化两组Replica Set,通过mongo连接到一个mongod
  1. /elain/apps/mongodb/bin/mongo 10.0.0.11:27017
  2. config = {_id: 'shard1', members: [
  3. {_id: 0, host: '10.0.0.11:27017'},
  4. {_id: 1, host: '10.0.0.12:27017'},
  5. {_id: 2, host: '10.0.0.14:27017'}]};
  1. rs.initiate(config);
  2. /elain/apps/mongodb/bin/mongo 10.0.0.11:27018
  3. config = {_id: 'shard2', members: [
  4. {_id: 0, host: '10.0.0.11:27018'},
  5. {_id: 1, host: '10.0.0.12:27018'},
  6. {_id: 2, host: '10.0.0.14:27018'}]};
  7. rs.initiate(config);
4、启动并配置三台Config Server

#Server1、2、3:
  1. mkdir -p /elain/data/mongodb/config/
  2. /elain/apps/mongodb/bin/mongod --configsvr --dbpath /elain/data/mongodb/config/ --port 20000 --logpath /elain/data/mongodb/config1.log --logappend --fork
5、部署并配置三台Routing Server

指定所有的config sever地址参数,chunkSize是分割数据时每块(Chunk)的单位大小

#Server1、2、3:
  1. /elain/apps/mongodb/bin/mongos --configdb 10.0.0.11:20000,10.0.0.12:20000,10.0.0.14:20000 --port 30000 --chunkSize 100 --logpath /elain/data/mongodb/mongos.log --logappend --fork
6、命令行添加分片

连接到mongs服务器,并切换到admin
  1. /elain/apps/mongodb/bin/mongo 10.0.0.11:30000/admin
  1. db.runCommand( {
  2. addshard : "shard1/10.0.0.11:27017,10.0.0.12:27017,10.0.0.14:27017",
  3. name:"shard1",
  4. maxsize:2048,
  5. allowLocal:true } );
  6. db.runCommand( {
  7. addshard : "shard2/10.0.0.11:27018,10.0.0.12:27018,10.0.0.14:27018",
  8. name:"shard2",
  9. maxsize:2048,
  10. allowLocal:true } );
  11. db.runCommand( { listshards : 1 } );
如果列出(sharding)了以上二个你加的shards,表示shards已经配置成功 #激活数据库分片
  1. db.runCommand( { enablesharding : "elain" } );
要使单个collection也分片存储,需要给collection指定一个分片key,通过以下命令操作:
  1. db.runCommand( { shardcollection : “<namespace>”,key : <shardkeypatternobject});
注:
a. 分片的collection系统会自动创建一个索引(也可用户提前创建好)
b. 分片的collection只能有一个在分片key上的唯一索引,其它唯一索引不被允许
One note: a sharded collection can have only one unique index, which must exist on the shard key. No other unique indexes can exist on the collection.
  1. mongoimport -d elain -c test --type csv --headerline --file test.csv --host 10.0.0.11:30000
分片collection例子
  1. db.runCommand( { shardcollection : "elain.test",key : {id: 1} } )
  2. mongos> db.chujq.t1.stats();
  3. {
  4. "sharded" : true,
  5. "flags" : 1,
  6. "ns" : "elain.test",
  7. "count" : 4058977,
  8. "numExtents" : 34,
  9. "size" : 675039816,
  10. "storageSize" : 818757632,
  11. "totalIndexSize" : 131854352,
  12. "indexSizes" : {
  13. "_id_" : 131854352
  14. },
  15. "avgObjSize" : 166.30786919955446,
  16. "nindexes" : 1,
  17. "nchunks" : 14,
  18. "shards" : {
  19. "shard1" : {
  20. "ns" : "elain.test",
  21. "count" : 1860365,
  22. "size" : 309376352,
  23. "avgObjSize" : 166.29873815084673,
  24. "storageSize" : 408920064,
  25. "numExtents" : 16,
  26. "nindexes" : 1,
  27. "lastExtentSize" : 77955072,
  28. "paddingFactor" : 1,
  29. "flags" : 1,
  30. "totalIndexSize" : 60371584,
  31. "indexSizes" : {
  32. "_id_" : 60371584
  33. },
  34. "ok" : 1
  35. },
  36. "shard2" : {
  37. "ns" : "elain.test",
  38. "count" : 2198612,
  39. "size" : 365663464,
  40. "avgObjSize" : 166.31559547569103,
  41. "storageSize" : 409837568,
  42. "numExtents" : 18,
  43. "nindexes" : 1,
  44. "lastExtentSize" : 74846208,
  45. "paddingFactor" : 1,
  46. "flags" : 1,
  47. "totalIndexSize" : 71482768,
  48. "indexSizes" : {
  49. "_id_" : 71482768
  50. },
  51. "ok" : 1
  52. }
  53. },
  54. "ok" : 1
  55. }
删除片操作
  1. mongos> db.runCommand({"removeshard" : "10.0.0.11:27018"});
  2. {
  3. "msg" : "draining started successfully",
  4. "state" : "started",
  5. "shard" : "shard2",
  6. "ok" : 1
  7. }
再执行,可看到removeshard的挪动进度
  1. mongos> db.runCommand({"removeshard" : "10.0.0.11:27018"});
  2. {
  3. "msg" : "draining ongoing",
  4. "state" : "ongoing",
  5. "remaining" : {
  6. "chunks" : NumberLong(3),
  7. "dbs" : NumberLong(0)
  8. },
  9. "ok" : 1
  10. }
例:
  1. /elain/apps/mongodb/bin/mongod --config /elain/data/mongodb/shard1/shard1.properties --rest
***MongoDB高可用集群设计实战部署(一)
环境:
CentOS 6.0 x64

md01: 10.0.0.11
md02: 10.0.0.12
md03: 10.0.0.14
md04: 10.0.0.15
md05: 10.0.0.16
md06: 10.0.0.17

设计思路:

md01、md02、md03 做一组复制集
md04、md05、md06 做一组复制集
再把这两组复制集用分片做成 shard1、shard2 用LVS 调用

下载安装mongodb
  1. cd /root/tools
  2. wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.0.tgz
  3. tar zxvf mongodb-linux-x86_64-2.0.0.tgz
  4. mv mongodb-linux-x86_64-2.0.0 /elain/apps/mongodb-linux-x86_64-2.0.0
  5. ln -s /elain/apps/mongodb-linux-x86_64-2.0.0 /elain/apps/mongodb
  6. ln -s /elain/apps/mongodb/bin/* /bin/
#添加用户组
  1. /usr/sbin/groupadd -g 690 mongodb
  2. /usr/sbin/useradd -g mongodb mongodb -u 690 -s /sbin/nologin
各节点hosts文件 添加:
  1. true > /etc/hosts
  2. echo -ne "
  3. 10.0.0.11 md01
  4. 10.0.0.12 md02
  5. 10.0.0.14 md03
  6. 10.0.0.15 md04
  7. 10.0.0.16 md05
  8. 10.0.0.17 md06
  9. " >>/etc/hosts
副本集配置: 启动各节点: md01
  1. mkdir -p /elain/logs/mongodb/
  2. mkdir -p /elain/data/mongodb/db
  3. /elain/apps/mongodb/bin/mongod --rest --replSet elain/md01:27017 --fork --port 27017 --dbpath /elain/data/mongodb/db/ --logpath /elain/logs/mongodb/mongodb.log
md02
  1. mkdir -p /elain/logs/mongodb/
  2. mkdir -p /elain/data/mongodb/db
  3. /elain/apps/mongodb/bin/mongod --rest --replSet elain/md02:27017 --fork --port 27017 --dbpath /elain/data/mongodb/db/ --logpath /elain/logs/mongodb/mongodb.log
md03
  1. mkdir -p /elain/logs/mongodb/
  2. mkdir -p /elain/data/mongodb/db
  3. /elain/apps/mongodb/bin/mongod --rest --replSet elain/md03:27017 --fork --port 27017 --dbpath /elain/data/mongodb/db/ --logpath /elain/logs/mongodb/mongodb.log
#启动仲裁节点
在md02上
  1. mkdir /elain/data/mongodb/arb
  2. /elain/apps/mongodb/bin/mongod --fork --rest --replSet elain --dbpath /elain/data/mongodb/arb --port 27015 --logpath /elain/logs/mongodb/mongodb.log
初始化节点:
md01:(登录其中任何一个节点操作皆可)
  1. > rs.initiate({
  2. _id : "elain",
  3. members : [
  4. {_id : 1, host : "md01:27017"},
  5. {_id : 2, host : "md02:27017"},
  6. {_id : 3, host : "md03:27017"},
  7. {_id : 4, host : "md02:27015", "arbiterOnly": true},
  8. ]
  9. });
#验证
  1. PRIMARY> rs.status()
也可浏览:http://10.0.0.11:28017/_replSet 查看状态

查看副本集状态
  1. >rs.status()
  2. >user local;
  3. >rs.isMaster()
  4. >db.system.replset.find()
查看当前主库:
  1. >db.$cmd.findOne({ismaster:1});
另一组副本集同理操作即可;这里为省篇幅就不再写出,但切记两复制集名称不可以重复
启动并配置三台Config Server

#md01,03,05上执行
  1. mkdir -p /elain/data/mongodb/config/
  2. /elain/apps/mongodb/bin/mongod --configsvr --dbpath /elain/data/mongodb/config/ --port 20000 --logpath /elain/logs/mongodb/config.log --logappend --fork
5、部署并配置三台Routing Server

指定所有的config sever地址参数,chunkSize是分割数据时每块(Chunk)的单位大小

#md02,md04,md06
  1. /elain/apps/mongodb/bin/mongos --configdb md01:20000,md03:20000,md05:20000 --port 30000 --chunkSize 100 --logpath /elain/logs/mongodb/mongos.log --logappend --fork
6、命令行添加分片

连接到mongs服务器,并切换到admin
  1. /elain/apps/mongodb/bin/mongo 10.0.0.11:30000/admin
  1. db.runCommand( {
  2. addshard : "elain/md01:27017",
  3. name:"shard1",
  4. maxsize:2048,
  5. allowLocal:true } );
  6. db.runCommand( {
  7. addshard : "chujq/md04:27017",
  8. name:"shard2",
  9. maxsize:2048,
  10. allowLocal:true } );
注:添加复制集elain,其中包含一个服务器md01:27017(还饿别的服务器,如md02、md03),如果md01挂了,mongos会知道它所连接的是一个复制集,并会使用新的主节点(md02或md03)
  1. db.runCommand( { listshards : 1 } );
如果列出(sharding)了以上二个你加的shards,表示shards已经配置成功

导入数据:
  1. mongoimport -d elain -c elain --type csv --headerline --file /root/bak/test.csv --host md02:30000
注:数据自备,我这里是从生产环境下MYSQL里导出的一些真实数据来做测试

#激活数据库分片
  1. db.runCommand( { enablesharding : "elain" } );
  2. mongos> db.runCommand( { enablesharding : "elain" } );
  3. { "ok" : 1 }
  4. mongos> show dbs
  5. config 0.1875GB
  6. elain 0.453125GB
  7. test (empty)
#进入数据库,建立索引,为分片做准备
  1. /elain/apps/mongodb/bin/mongo 10.0.0.12:30000/admin
  2. use elain;
  3. db.elain.ensureIndex({"client_userid":1});
查看索引:
  1. mongos> db.elain.find({"client_userid" : 151512}).explain();
  2. {
  3. "cursor" : "BtreeCursor client_userid_1",
  4. "nscanned" : 3,
  5. "nscannedObjects" : 3,
  6. "n" : 3,
  7. "millis" : 0,
  8. "nYields" : 0,
  9. "nChunkSkips" : 0,
  10. "isMultiKey" : false,
  11. "indexOnly" : false,
  12. "indexBounds" : {
  13. "client_userid" : [
  14. [
  15. 151512,
  16. 151512
  17. ]
  18. ]
  19. }
  20. }
#添加分片:
  1. use admin;
  2. db.runCommand( { shardcollection : "elain.elain",key : {client_userid: 1} } )
  3. mongos> db.runCommand( { shardcollection : "elain.elain",key : {client_userid: 1} } )
  4. { "collectionsharded" : "elain.elain", "ok" : 1 }
#查看分片状态
  1. use elain;
  2. db.elain.stats();
  3. mongos> db.elain.stats();
  4. {
  5. "sharded" : true,
  6. "flags" : 1,
  7. "ns" : "elain.elain",
  8. "count" : 507372,
  9. "numExtents" : 10,
  10. "size" : 84375328,
  11. "storageSize" : 97849344,
  12. "totalIndexSize" : 29253728,
  13. "indexSizes" : {
  14. "_id_" : 16474640,
  15. "client_userid_1" : 12779088
  16. },
  17. "avgObjSize" : 166.29874727024747,
  18. "nindexes" : 2,
  19. "nchunks" : 1,
  20. "shards" : {
  21. "shard2" : {
  22. "ns" : "elain.elain",
  23. "count" : 507372,
  24. "size" : 84375328,
  25. "avgObjSize" : 166.29874727024747,
  26. "storageSize" : 97849344,
  27. "numExtents" : 10,
  28. "nindexes" : 2,
  29. "lastExtentSize" : 26099712,
  30. "paddingFactor" : 1,
  31. "flags" : 1,
  32. "totalIndexSize" : 29253728,
  33. "indexSizes" : {
  34. "_id_" : 16474640,
  35. "client_userid_1" : 12779088
  36. },
  37. "ok" : 1
  38. }
  39. },
  40. "ok" : 1
  41. }
在分片后新写数据第一次:
  1. mongos> db.elain.stats();
  2. {
  3. "sharded" : true,
  4. "flags" : 1,
  5. "ns" : "elain.elain",
  6. "count" : 676496,
  7. "numExtents" : 12,
  8. "size" : 112500436,
  9. "storageSize" : 129179648,
  10. "totalIndexSize" : 47551616,
  11. "indexSizes" : {
  12. "_id_" : 21968912,
  13. "client_userid_1" : 25582704
  14. },
  15. "avgObjSize" : 166.29874529930703,
  16. "nindexes" : 2,
  17. "nchunks" : 8,
  18. "shards" : {
  19. "shard1" : {
  20. "ns" : "elain.elain",
  21. "count" : 0,
  22. "size" : 0,
  23. "storageSize" : 8192,
  24. "numExtents" : 1,
  25. "nindexes" : 2,
  26. "lastExtentSize" : 8192,
  27. "paddingFactor" : 1,
  28. "flags" : 1,
  29. "totalIndexSize" : 16352,
  30. "indexSizes" : {
  31. "_id_" : 8176,
  32. "client_userid_1" : 8176
  33. },
  34. "ok" : 1
  35. },
  36. "shard2" : {
  37. "ns" : "elain.elain",
  38. "count" : 676496,
  39. "size" : 112500436,
  40. "avgObjSize" : 166.29874529930703,
  41. "storageSize" : 129171456,
  42. "numExtents" : 11,
  43. "nindexes" : 2,
  44. "lastExtentSize" : 31322112,
  45. "paddingFactor" : 1,
  46. "flags" : 1,
  47. "totalIndexSize" : 47535264,
  48. "indexSizes" : {
  49. "_id_" : 21960736,
  50. "client_userid_1" : 25574528
  51. },
  52. "ok" : 1
  53. }
  54. },
  55. "ok" : 1
  56. }
分片后新写数据第二次:
  1. mongos> db.elain.stats();
  2. {
  3. "sharded" : true,
  4. "flags" : 1,
  5. "ns" : "elain.elain",
  6. "count" : 1189194,
  7. "numExtents" : 23,
  8. "size" : 194533928,
  9. "storageSize" : 252874752,
  10. "totalIndexSize" : 87262448,
  11. "indexSizes" : {
  12. "_id_" : 43692544,
  13. "client_userid_1" : 43569904
  14. },
  15. "avgObjSize" : 163.58468677103988,
  16. "nindexes" : 2,
  17. "nchunks" : 8,
  18. "shards" : {
  19. "shard1" : {
  20. "ns" : "elain.elain",
  21. "count" : 396370,
  22. "size" : 62195348,
  23. "avgObjSize" : 156.91234957236924,
  24. "storageSize" : 86114304,
  25. "numExtents" : 11,
  26. "nindexes" : 2,
  27. "lastExtentSize" : 20881408,
  28. "paddingFactor" : 1,
  29. "flags" : 1,
  30. "totalIndexSize" : 35949872,
  31. "indexSizes" : {
  32. "_id_" : 17954496,
  33. "client_userid_1" : 17995376
  34. },
  35. "ok" : 1
  36. },
  37. "shard2" : {
  38. "ns" : "elain.elain",
  39. "count" : 792824,
  40. "size" : 132338580,
  41. "avgObjSize" : 166.9205019020615,
  42. "storageSize" : 166760448,
  43. "numExtents" : 12,
  44. "nindexes" : 2,
  45. "lastExtentSize" : 37588992,
  46. "paddingFactor" : 1,
  47. "flags" : 1,
  48. "totalIndexSize" : 51312576,
  49. "indexSizes" : {
  50. "_id_" : 25738048,
  51. "client_userid_1" : 25574528
  52. },
  53. "ok" : 1
  54. }
  55. },
  56. "ok" : 1
  57. }
分片后新写数据第三次:
  1. mongos> db.elain.stats();
  2. {
  3. "sharded" : true,
  4. "flags" : 1,
  5. "ns" : "elain.elain",
  6. "count" : 1376876,
  7. "numExtents" : 23,
  8. "size" : 225576604,
  9. "storageSize" : 252874752,
  10. "totalIndexSize" : 100826432,
  11. "indexSizes" : {
  12. "_id_" : 50249696,
  13. "client_userid_1" : 50576736
  14. },
  15. "avgObjSize" : 163.83218532387812,
  16. "nindexes" : 2,
  17. "nchunks" : 10,
  18. "shards" : {
  19. "shard1" : {
  20. "ns" : "elain.elain",
  21. "count" : 494202,
  22. "size" : 77551984,
  23. "avgObjSize" : 156.92365469990003,
  24. "storageSize" : 86114304,
  25. "numExtents" : 11,
  26. "nindexes" : 2,
  27. "lastExtentSize" : 20881408,
  28. "paddingFactor" : 1,
  29. "flags" : 1,
  30. "totalIndexSize" : 42057344,
  31. "indexSizes" : {
  32. "_id_" : 21600992,
  33. "client_userid_1" : 20456352
  34. },
  35. "ok" : 1
  36. },
  37. "shard2" : {
  38. "ns" : "elain.elain",
  39. "count" : 882674,
  40. "size" : 148024620,
  41. "avgObjSize" : 167.70021548159343,
  42. "storageSize" : 166760448,
  43. "numExtents" : 12,
  44. "nindexes" : 2,
  45. "lastExtentSize" : 37588992,
  46. "paddingFactor" : 1,
  47. "flags" : 1,
  48. "totalIndexSize" : 58769088,
  49. "indexSizes" : {
  50. "_id_" : 28648704,
  51. "client_userid_1" : 30120384
  52. },
  53. "ok" : 1
  54. }
  55. },
  56. "ok" : 1
  57. }
分片后新写数据第四次(DOWN 掉md04的mongo服务)
  1. mongos> db.elain.stats();
  2. {
  3. "sharded" : true,
  4. "flags" : 1,
  5. "ns" : "elain.elain",
  6. "count" : 1686310,
  7. "numExtents" : 26,
  8. "size" : 275761376,
  9. "storageSize" : 353116160,
  10. "totalIndexSize" : 129033632,
  11. "indexSizes" : {
  12. "_id_" : 63265888,
  13. "client_userid_1" : 65767744
  14. },
  15. "avgObjSize" : 163.52946729842083,
  16. "nindexes" : 2,
  17. "nchunks" : 10,
  18. "shards" : {
  19. "shard1" : {
  20. "ns" : "elain.elain",
  21. "count" : 740264,
  22. "size" : 116213588,
  23. "avgObjSize" : 156.98938216636228,
  24. "storageSize" : 141246464,
  25. "numExtents" : 13,
  26. "nindexes" : 2,
  27. "lastExtentSize" : 30072832,
  28. "paddingFactor" : 1,
  29. "flags" : 1,
  30. "totalIndexSize" : 61810560,
  31. "indexSizes" : {
  32. "_id_" : 32556832,
  33. "client_userid_1" : 29253728
  34. },
  35. "ok" : 1
  36. },
  37. "shard2" : {
  38. "ns" : "elain.elain",
  39. "count" : 946046,
  40. "size" : 159547788,
  41. "avgObjSize" : 168.64696642657967,
  42. "storageSize" : 211869696,
  43. "numExtents" : 13,
  44. "nindexes" : 2,
  45. "lastExtentSize" : 45109248,
  46. "paddingFactor" : 1,
  47. "flags" : 1,
  48. "totalIndexSize" : 67223072,
  49. "indexSizes" : {
  50. "_id_" : 30709056,
  51. "client_userid_1" : 36514016
  52. },
  53. "ok" : 1
  54. }
  55. },
  56. "ok" : 1
  57. }
总结:通过以上四次的写数据测试,我们可以看到分片是成功的,每次写数据,shard1、shard2都有数据写入,且,在下面的复制集中DOWN 掉任意一台,不影响整个架构的正常服务。

删除片操作
  1. mongos> db.runCommand({"removeshard" : "10.0.0.11:27011"});
  2. {
  3. "msg" : "draining started successfully",
  4. "state" : "started",
  5. "shard" : "shard2",
  6. "ok" : 1
  7. }
再执行,可看到removeshard的挪动进度
  1. mongos> db.runCommand({"removeshard" : "10.0.0.11:27011"});
  2. {
  3. "msg" : "draining ongoing",
  4. "state" : "ongoing",
  5. "remaining" : {
  6. "chunks" : NumberLong(3),
  7. "dbs" : NumberLong(0)
  8. },
  9. "ok" : 1
  10. }