mongodb3.0 单机数据迁移分片复制集

原创

mb644789379f291 2023-04-26 01:11:15 博主文章分类：mongodb ©著作权

©著作权归作者所有：来自51CTO博客作者mb644789379f291的原创作品，请联系作者获取转载授权，否则将追究法律责任

1,启用分片

1,启用分片

mongos> use admin
switched to db admin
mongos> db.runCommand({enablesharding:"test"})
{ "ok" : 1 }
mongos> db.printShardingStatus()
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("555adbbecc86e0433d1c531c")
}
  shards:
	{  "_id" : "firstset",  "host" : "firstset/127.0.0.1:10001,127.0.0.1:10002,127.0.0.1:10003" }
	{  "_id" : "firstset1",  "host" : "firstset1/127.0.0.1:30001,127.0.0.1:30002,127.0.0.1:30003" }
  balancer:
	Currently enabled:  yes
	Currently running:  no
	Failed balancer rounds in last 5 attempts:  0
	Migration Results for the last 24 hours: 
		No recent migrations
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "fensme",  "partitioned" : true,  "primary" : "firstset" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "firstset" }

2，设置片键(根据file_id与chunck n )

mongos> db.runCommand( { shardCollection : "test.fs.chunks" , key : { files_id : 1 , n : 1 } } )
{ "collectionsharded" : "test.fs.chunks", "ok" : 1 }

3，导入数据

[root@localhost ~]# mongorestore -h '127.0.0.1:27038' -d 'test' --dir='/usr/mongodb/bak/mogo_data2/test' --noIndexRestore
2015-05-27T22:34:42.859-0400	building a list of collections to restore from /usr/mongodb/bak/mogo_data2/test dir
2015-05-27T22:34:42.871-0400	reading metadata file from /usr/mongodb/bak/mogo_data2/test/fs.chunks.metadata.json
2015-05-27T22:34:42.872-0400	reading metadata file from /usr/mongodb/bak/mogo_data2/test/fs.files.metadata.json
2015-05-27T22:34:42.872-0400	reading metadata file from /usr/mongodb/bak/mogo_data2/test/blog.metadata.json
2015-05-27T22:34:42.885-0400	restoring test.fs.chunks from file /usr/mongodb/bak/mogo_data2/test/fs.chunks.bson
2015-05-27T22:34:42.889-0400	restoring test.fs.files from file /usr/mongodb/bak/mogo_data2/test/fs.files.bson
2015-05-27T22:34:42.889-0400	restoring test.blog from file /usr/mongodb/bak/mogo_data2/test/blog.bson
2015-05-27T22:34:42.892-0400	error: E11000 duplicate key error index: test.blog.$_id_ dup key: { : ObjectId('55595022e26fd5794a58bdd3') }
2015-05-27T22:34:42.893-0400	no indexes to restore
2015-05-27T22:34:42.893-0400	finished restoring test.blog
2015-05-27T22:34:42.907-0400	error: E11000 duplicate key error index: test.fs.files.$_id_ dup key: { : ObjectId('5565395645ce131ca0a3a66b') }
2015-05-27T22:34:42.907-0400	no indexes to restore
2015-05-27T22:34:42.907-0400	finished restoring test.fs.files
2015-05-27T22:34:44.485-0400	error: E11000 duplicate key error index: test.fs.chunks.$_id_ dup key: { : ObjectId('5565395645ce131ca0a3a66c') }
2015-05-27T22:34:44.485-0400	no indexes to restore
2015-05-27T22:34:44.485-0400	finished restoring test.fs.chunks
2015-05-27T22:34:44.486-0400	done

4，查看导入后mogodb的状态

[root@localhost ~]# mongo   '127.0.0.1:27038'
MongoDB shell version: 3.0.3
connecting to: 127.0.0.1:27038/test
Server has startup warnings: 
2015-05-27T06:42:00.545-0400 I CONTROL  ** WARNING: You are running this process as the root user, which is not recommended.
2015-05-27T06:42:00.545-0400 I CONTROL  
mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("555adbbecc86e0433d1c531c")
}
  shards:
	{  "_id" : "firstset",  "host" : "firstset/127.0.0.1:10001,127.0.0.1:10002,127.0.0.1:10003" }
	{  "_id" : "firstset1",  "host" : "firstset1/127.0.0.1:30001,127.0.0.1:30002,127.0.0.1:30003" }
  balancer:
	Currently enabled:  yes
	Currently running:  no
	Failed balancer rounds in last 5 attempts:  0
	Migration Results for the last 24 hours: 
		51 : Success
		1 : Failed with error 'migration already in progress', from firstset to firstset1
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "fensme",  "partitioned" : true,  "primary" : "firstset" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "firstset" }
		test.fs.chunks
			shard key: { "files_id" : 1, "n" : 1 }
			chunks:
				firstset	17
				firstset1	17
			too many chunks to print, use verbose if you want to force print

5，查看导入成功后的数据

mongos> use test
switched to db test
mongos> db.fs.files.find()
{ "_id" : ObjectId("55501599b26cf1ed2482ae09"), "chunkSize" : NumberLong(261120), "length" : NumberLong(5277), "md5" : "4031e801545605ecea9d3b79e39bed4c", "filename" : "test_001.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-11T02:36:09.474Z"), "aliases" : null }
{ "_id" : ObjectId("555168feb26ca6bbc2e42ff5"), "chunkSize" : NumberLong(261120), "length" : NumberLong(19929), "md5" : "343905146f1b83c6fc74e5055be0e686", "filename" : "你好.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-12T02:44:14.213Z"), "aliases" : null }
{ "_id" : ObjectId("555169f3b26ca6bbc2e42ff7"), "chunkSize" : NumberLong(261120), "length" : NumberLong(19929), "md5" : "343905146f1b83c6fc74e5055be0e686", "filename" : "你好1.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-12T02:48:19.182Z"), "aliases" : null }
{ "_id" : ObjectId("555169f3b26ca6bbc2e42ff9"), "chunkSize" : NumberLong(261120), "length" : NumberLong(19929), "md5" : "343905146f1b83c6fc74e5055be0e686", "filename" : "你好2- 副本.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-12T02:48:19.226Z"), "aliases" : null }
{ "_id" : ObjectId("55516fccb26c7664a4f2cbf1"), "chunkSize" : NumberLong(261120), "length" : NumberLong(19929), "md5" : "343905146f1b83c6fc74e5055be0e686", "filename" : "你好1.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-12T03:13:16.816Z"), "aliases" : null }
{ "_id" : ObjectId("555171f5b26c7664a4f2cbf3"), "chunkSize" : NumberLong(261120), "length" : NumberLong(38226), "md5" : "e94d9fb6cea34a6cd7e9556700bbc26e", "filename" : "你好1.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-12T03:22:29.024Z"), "aliases" : null }
{ "_id" : ObjectId("5551c1feb26cbff27a40da0f"), "chunkSize" : NumberLong(261120), "length" : NumberLong(25205), "md5" : "b9d53340719d3f663e78d10fa2954158", "filename" : "dada.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-12T09:03:58.285Z"), "aliases" : null }
{ "_id" : ObjectId("5551d64bb26cae38b47ae4d9"), "chunkSize" : NumberLong(261120), "length" : NumberLong(12468), "md5" : "0fe1936bf88a5e58f1fe92b1badfd3db", "filename" : "mingxing0323.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-12T10:30:35.706Z"), "aliases" : null }
{ "_id" : ObjectId("5551db4bb26c36a5b5de55ac"), "chunkSize" : NumberLong(261120), "length" : NumberLong(87424), "md5" : "b5ad982d77b2addcfc4d3c76a6249f64", "filename" : "info.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-12T10:51:55.890Z"), "aliases" : null }
{ "_id" : ObjectId("5551dbbab26c36a5b5de55ae"), "chunkSize" : NumberLong(261120), "length" : NumberLong(12468), "md5" : "0fe1936bf88a5e58f1fe92b1badfd3db", "filename" : "mingxing0323.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-12T10:53:46.459Z"), "aliases" : null }
{ "_id" : ObjectId("5552d394b26cc26a231afd2a"), "chunkSize" : NumberLong(261120), "length" : NumberLong(5277), "md5" : "4031e801545605ecea9d3b79e39bed4c", "filename" : "test_001.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-13T04:31:16.176Z"), "aliases" : null }
{ "_id" : ObjectId("5552d3b4b26cc26a231afd2c"), "chunkSize" : NumberLong(261120), "length" : NumberLong(5277), "md5" : "4031e801545605ecea9d3b79e39bed4c", "filename" : "test_001.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-13T04:31:48.860Z"), "aliases" : null }
{ "_id" : ObjectId("5552d44fb26cc26a231afd2e"), "chunkSize" : NumberLong(261120), "length" : NumberLong(5277), "md5" : "4031e801545605ecea9d3b79e39bed4c", "filename" : "test_001.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-13T04:34:23.688Z"), "aliases" : null }
{ "_id" : ObjectId("555555b4fab7abcdc665a9a4"), "chunkSize" : NumberLong(261120), "length" : NumberLong(19929), "md5" : "343905146f1b83c6fc74e5055be0e686", "filename" : "你好2- 副本.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-15T02:11:00.457Z"), "aliases" : null }
{ "_id" : ObjectId("55559a69b26c02df63403ee8"), "chunkSize" : NumberLong(261120), "length" : NumberLong(12468), "md5" : "0fe1936bf88a5e58f1fe92b1badfd3db", "filename" : "mingxing0323.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-15T07:04:09.168Z"), "aliases" : null }
{ "_id" : ObjectId("5555a85045ce30364e4f23e8"), "chunkSize" : NumberLong(261120), "length" : NumberLong(40269), "md5" : "5682ba73dddf743c987070adab2af9a6", "filename" : "new_head.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-15T08:03:28.184Z"), "aliases" : null }
{ "_id" : ObjectId("5555ab9245ce30364e4f23ea"), "chunkSize" : NumberLong(261120), "length" : NumberLong(29140), "md5" : "9b9ba7c3f0fba6b29da831aed88b7764", "filename" : "new_head.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-15T08:17:22.838Z"), "aliases" : null }
{ "_id" : ObjectId("5555ad73174a642d26ac3cf4"), "chunkSize" : NumberLong(261120), "length" : NumberLong(40269), "md5" : "5682ba73dddf743c987070adab2af9a6", "filename" : "new_head.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-15T08:25:23.251Z"), "aliases" : null }
{ "_id" : ObjectId("5555adfc174a642d26ac3cf6"), "chunkSize" : NumberLong(261120), "length" : NumberLong(40269), "md5" : "5682ba73dddf743c987070adab2af9a6", "filename" : "new_head.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-15T08:27:40.384Z"), "aliases" : null }
{ "_id" : ObjectId("5555ae21174a642d26ac3cf8"), "chunkSize" : NumberLong(261120), "length" : NumberLong(40269), "md5" : "5682ba73dddf743c987070adab2af9a6", "filename" : "new_head.jpg", "contentType" : ".jpg", "uploadDate" : ISODate("2015-05-15T08:28:17.713Z"), "aliases" : null }

6，查看集群stats

mongos> db.stats()
{
	"raw" : {
		"firstset/127.0.0.1:10001,127.0.0.1:10002,127.0.0.1:10003" : {
			"db" : "test",
			"collections" : 5,
			"objects" : 419,
			"avgObjSize" : 26246.606205250595,
			"dataSize" : 10997328,
			"storageSize" : 22667264,
			"numExtents" : 12,
			"indexes" : 6,
			"indexSize" : 81760,
			"fileSize" : 67108864,
			"nsSizeMB" : 16,
			"extentFreeList" : {
				"num" : 0,
				"totalSize" : 0
			},
			"dataFileVersion" : {
				"major" : 4,
				"minor" : 22
			},
			"ok" : 1,
			"$gleStats" : {
				"lastOpTime" : Timestamp(1432779346, 1),
				"electionId" : ObjectId("55659f093b83cbe0a661535c")
			}
		},
		"firstset1/127.0.0.1:30001,127.0.0.1:30002,127.0.0.1:30003" : {
			"db" : "test",
			"collections" : 3,
			"objects" : 141,
			"avgObjSize" : 51637.90070921986,
			"dataSize" : 7280944,
			"storageSize" : 11161600,
			"numExtents" : 7,
			"indexes" : 2,
			"indexSize" : 16352,
			"fileSize" : 67108864,
			"nsSizeMB" : 16,
			"extentFreeList" : {
				"num" : 0,
				"totalSize" : 0
			},
			"dataFileVersion" : {
				"major" : 4,
				"minor" : 22
			},
			"ok" : 1,
			"$gleStats" : {
				"lastOpTime" : Timestamp(1432779346, 1),
				"electionId" : ObjectId("55659f3fc2161dd3e935a9c7")
			}
		}
	},
	"objects" : 560,
	"avgObjSize" : 32639.09107142857,
	"dataSize" : 18278272,
	"storageSize" : 33828864,
	"numExtents" : 19,
	"indexes" : 8,
	"indexSize" : 98112,
	"fileSize" : 134217728,
	"extentFreeList" : {
		"num" : 0,
		"totalSize" : 0
	},
	"ok" : 1
}

7，查看chunks分片信息

mongos> use test
switched to db test
mongos> show collections
blog
fs.chunks
fs.files
system.indexes
mongos> db.fs.files.stats()
{
	"sharded" : false,
	"primary" : "firstset",
	"ns" : "test.fs.files",
	"count" : 266,
	"size" : 63840,
	"avgObjSize" : 240,
	"numExtents" : 3,
	"storageSize" : 172032,
	"lastExtentSize" : 131072,
	"paddingFactor" : 1,
	"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
	"userFlags" : 1,
	"capped" : false,
	"nindexes" : 3,
	"totalIndexSize" : 57232,
	"indexSizes" : {
		"_id_" : 8176,
		"filename_1_uploadDate_1" : 24528,
		"_filename" : 24528
	},
	"ok" : 1,
	"$gleStats" : {
		"lastOpTime" : Timestamp(1432779653, 271),
		"electionId" : ObjectId("55659f093b83cbe0a661535c")
	}
}
mongos> db.fs.chunks.stats()
{
	"sharded" : true,
	"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
	"userFlags" : 1,
	"capped" : false,
	"ns" : "test.fs.chunks",
	"count" : 271,
	"numExtents" : 11,
	"size" : 18212624,
	"storageSize" : 33624064,
	"totalIndexSize" : 32704,
	"indexSizes" : {
		"_id_" : 16352,
		"files_id_1_n_1" : 16352
	},
	"avgObjSize" : 67205.25461254612,
	"nindexes" : 2,
	"nchunks" : 34,
	"shards" : {
		"firstset" : {
			"ns" : "test.fs.chunks",
			"count" : 136,
			"size" : 10932096,
			"avgObjSize" : 80383,
			"numExtents" : 6,
			"storageSize" : 22474752,
			"lastExtentSize" : 11325440,
			"paddingFactor" : 1,
			"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
			"userFlags" : 1,
			"capped" : false,
			"nindexes" : 2,
			"totalIndexSize" : 16352,
			"indexSizes" : {
				"_id_" : 8176,
				"files_id_1_n_1" : 8176
			},
			"ok" : 1,
			"$gleStats" : {
				"lastOpTime" : Timestamp(1432779346, 1),
				"electionId" : ObjectId("55659f093b83cbe0a661535c")
			}
		},
		"firstset1" : {
			"ns" : "test.fs.chunks",
			"count" : 135,
			"size" : 7280528,
			"avgObjSize" : 53929,
			"numExtents" : 5,
			"storageSize" : 11149312,
			"lastExtentSize" : 8388608,
			"paddingFactor" : 1,
			"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
			"userFlags" : 1,
			"capped" : false,
			"nindexes" : 2,
			"totalIndexSize" : 16352,
			"indexSizes" : {
				"_id_" : 8176,
				"files_id_1_n_1" : 8176
			},
			"ok" : 1,
			"$gleStats" : {
				"lastOpTime" : Timestamp(1432779346, 1),
				"electionId" : ObjectId("55659f3fc2161dd3e935a9c7")
			}
		}
	},
	"ok" : 1
}

8，分别查看两个分片的信息

connecting to: 127.0.0.1:10001/test
Server has startup warnings: 
2015-05-27T06:37:28.619-0400 I CONTROL  ** WARNING: --rest is specified without --httpinterface,
2015-05-27T06:37:28.619-0400 I CONTROL  **          enabling http interface
2015-05-27T06:37:28.708-0400 I STORAGE  [initandlisten] 
2015-05-27T06:37:28.708-0400 I STORAGE  [initandlisten] ** WARNING: Readahead for /usr/mongodb/slave1/firstset1 is set to 4096KB
2015-05-27T06:37:28.708-0400 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
2015-05-27T06:37:28.708-0400 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
2015-05-27T06:37:28.712-0400 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-05-27T06:37:28.712-0400 I CONTROL  [initandlisten] 
2015-05-27T06:37:28.712-0400 I CONTROL  [initandlisten] 
2015-05-27T06:37:28.712-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-05-27T06:37:28.712-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-27T06:37:28.712-0400 I CONTROL  [initandlisten] 
2015-05-27T06:37:28.712-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-05-27T06:37:28.712-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-27T06:37:28.712-0400 I CONTROL  [initandlisten] 
firstset:PRIMARY> show dbs
local  1.078GB
test   0.078GB
firstset:PRIMARY> use test
switched to db test
firstset:PRIMARY> show collections
blog
fs.chunks
fs.files
system.indexes
firstset:PRIMARY> fs.chunks.count()
2015-05-28T02:39:37.531-0400 E QUERY    ReferenceError: fs is not defined
    at (shell):1:1
firstset:PRIMARY> db.fs.chunks.count()
136
firstset:PRIMARY> 



[root@localhost ~]# mongo 127.0.0.1:30001
MongoDB shell version: 3.0.3
connecting to: 127.0.0.1:30001/test
Server has startup warnings: 
2015-05-27T06:40:48.686-0400 I CONTROL  ** WARNING: --rest is specified without --httpinterface,
2015-05-27T06:40:48.686-0400 I CONTROL  **          enabling http interface
2015-05-27T06:40:48.849-0400 I STORAGE  [initandlisten] 
2015-05-27T06:40:48.849-0400 I STORAGE  [initandlisten] ** WARNING: Readahead for /usr/mongodb/slave2/firstset1 is set to 4096KB
2015-05-27T06:40:48.849-0400 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
2015-05-27T06:40:48.849-0400 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
2015-05-27T06:40:48.850-0400 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-05-27T06:40:48.850-0400 I CONTROL  [initandlisten] 
2015-05-27T06:40:48.850-0400 I CONTROL  [initandlisten] 
2015-05-27T06:40:48.850-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-05-27T06:40:48.850-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-27T06:40:48.850-0400 I CONTROL  [initandlisten] 
2015-05-27T06:40:48.850-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-05-27T06:40:48.850-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-27T06:40:48.850-0400 I CONTROL  [initandlisten] 
firstset1:PRIMARY> use test
switched to db test
firstset1:PRIMARY> show collections
fs.chunks
system.indexes
firstset1:PRIMARY> db.fs.chunks.count()
135
firstset1:PRIMARY>

注意：1，连接mongos 必须用mongo 127.0.0.1:27038，否则进入mongos自身的mongodb。

2，导入数据到分片复制集，必须先创建库，然后启用分片，接着设置键值

3，导入数据的时候，必须--noIndexRestore否则会报错索引已存在

add by joeyon1985