前言：
今天遇到主从表不一致的情况，很奇怪为什么会出现不一致的情况，因为复制状态一直都是正常的。最后检查出现不一致的数据都是主键，原来是当时初始化数据的时候导致的。现在分析记录下这个问题，避免以后再遇到这个"坑"。
背景：
主从服务器，MIXED复制模式。
分析：
表：SPU
Table: SPU
Create Table: CREATE TABLE `SPU` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`trademark` varchar(255) NOT NULL COMMENT '品牌',
`item_code` varchar(255) NOT NULL COMMENT '货号',
`product_id` int(10) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `trademark` (`trademark`),
KEY `item_code` (`item_code`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='SPU'
当时的初始化操作的SQL：
INSERT INTO SPU(trademark, item_code)
SELECT * FROM
(
SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2
FROM
product_property a LEFT JOIN product_property b
ON
a.product_id = b.product_id
WHERE
a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> ''
AND
b.value IS NOT NULL AND b.value <> ''
) as aa
上面的SQL执行完之后，主从表的COUNT数量一样，但SPU表有个自增主键，就在这里出现主从插入SPU的顺序不一样，即主从SELECT出来的结果顺序不一样。由于上面的结果集很大，所以就取前10条记录看看：
主：
现在表中数据的顺序：
zjy@192.168.10.23 : tt 04:38:48>select id,trademark col1,item_code col2 from SPU limit 10;
+----+-----------+-------------+
| id | col1 | col2 |
+----+-----------+-------------+
| 1 | 鼓浪屿 | gd rg |
| 2 | 山松 | 10020122 |
| 3 | coulter | 无 |
| 4 | oricell | mubmd-01101 |
| 5 | oricell | huxma-01101 |
| 6 | oricell | huxmf-01001 |
| 7 | oricell | rawmx-01001 |
| 8 | oricell | rasmx-01001 |
| 9 | oricell | rafmx-01101 |
| 10 | oricell | rbxmx-01001 |
+----+-----------+-------------+
10 rows in set (0.01 sec)
当时初始化的sql读出数据的顺序：
zjy@192.168.10.23 : tt 04:40:25>SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2
-> FROM
-> product_property a LEFT JOIN product_property b
-> ON
-> a.product_id = b.product_id
-> WHERE
-> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> ''
-> AND
-> b.value IS NOT NULL AND b.value <> '' limit 10;
+-----------+-------------+
| col1 | col2 |
+-----------+-------------+
| 鼓浪屿 | gd rg |
| 山松 | 10020122 |
| coulter | 无 |
| oricell | mubmd-01101 |
| oricell | huxma-01101 |
| oricell | huxmf-01001 |
| oricell | rawmx-01001 |
| oricell | rasmx-01001 |
| oricell | rafmx-01101 |
| oricell | rbxmx-01001 |
+-----------+-------------+
10 rows in set (0.01 sec)
上面结果col1,col2的顺序一模一样。
从：
现在表中数据的顺序：
zjy@192.168.10.8 : tt 04:38:03>select id,trademark col1,item_code col2 from SPU limit 10;
+----+--------------------------------+-----------------+
| id | col1 | col2 |
+----+--------------------------------+-----------------+
| 1 | bio-rad | 125-0140 |
| 2 | oricell(tm) | huxma-90011 |
| 3 | oricell(tm) | huxmf-90011 |
| 4 | oricell(tm) | rawmx-90011 |
| 5 | oricell | rasmx-90011 |
| 6 | oricell | rafmx-90011 |
| 7 | oricell | rbxmx-90011 |
| 8 | 上海科技有限公司 | bsm03011 |
| 9 | oricell | caxmx-90011 |
| 10 | oricell(tm) | tedta-10001-100 |
+----+--------------------------------+-----------------+
10 rows in set (0.00 sec)
当时初始化的sql读出数据的顺序：
zjy@192.168.10.8 : tt 04:38:58>SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2
-> FROM
-> product_property a LEFT JOIN product_property b
-> ON
-> a.product_id = b.product_id
-> WHERE
-> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> ''
-> AND
-> b.value IS NOT NULL AND b.value <> '' limit 10;
+--------------------------------+-----------------+
| col1 | col2 |
+--------------------------------+-----------------+
| bio-rad | 125-0140 |
| oricell(tm) | huxma-90011 |
| oricell(tm) | huxmf-90011 |
| oricell(tm) | rawmx-90011 |
| oricell | rasmx-90011 |
| oricell | rafmx-90011 |
| oricell | rbxmx-90011 |
| 上海科技有限公司 | bsm03011 |
| oricell | caxmx-90011 |
| oricell(tm) | tedta-10001-100 |
+--------------------------------+-----------------+
10 rows in set (0.01 sec)
上面结果col1,col2的顺序一模一样。
到此为止，大家就清楚为什么SPU表的数据不一致了，准确来说是主键对应的数据不一致。要是自增主键只是提升INNODB的性能，没有业务上的意义，那么对于产品来说是没有影响的，可以忽略这个问题。否则，就需要好好的处理这个问题了。从另一个方面来说，也是因为复制的模式是STATEMENT引发这个问题的，因为同一个QUERY在2个地方执行出的结果不一样；要是ROW的复制模式，主会把所有字段的记录全部传送给从，就不会出现这个问题。
进一步分析： 为什么一样的SQL在主从上跑出来的数据顺序不一样呢？
通过EXPLAIN 看到主上的QUERY 先读 b表，再读a表；而从上的则是先扫描a表，再读b表。出现这样的情况，就是数据在块里面分布不一致，导致索引利用的方式也不一样，最终影响优化器的选择。因为INNODB是索引组织表的，一旦走的索引不一样，就会导致数据以不同的顺序被扫描出来。上面的这些结果刚好被验证。SQL执行计划如下：
主：先b再a
zjy@192.168.10.23 : tt 09:18:04>explain SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 FROM product_property a LEFT JOIN product_property b ON a.product_id = b.product_id WHERE a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' AND b.value IS NOT NULL AND b.value <> '';
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+
| 1 | SIMPLE | b | ref | property_id,product_id | property_id | 4 | const | 4145932 | Using where; Using temporary |
| 1 | SIMPLE | a | ref | property_id,product_id | product_id | 4 | tt.b.product_id | 1 | Using where |
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+
2 rows in set (0.01 sec)
从：先a再b
zjy@192.168.10.8 : tt 09:18:13>explain SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 FROM product_property a LEFT JOIN product_property b ON a.product_id = b.product_id WHERE a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' AND b.value IS NOT NULL AND b.value <> '';
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+
| 1 | SIMPLE | a | ref | property_id,product_id | property_id | 4 | const | 17079464 | Using where; Using temporary |
| 1 | SIMPLE | b | ref | property_id,product_id | product_id | 4 | tt.a.product_id | 5 | Using where |
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+
2 rows in set (0.13 sec)
主从对比发现，他们的执行计划和各表走的索引都不一样，导致最后出来的顺序也不一样的（结果集是一样的），这就验证了分析说的情况。那要是执行计划和索引一致呢？接下来继续验证下：
进一步验证：
因为INNODB是索引组织表的，索引就是数据，要是主从的执行计划一样，则他们的结果会是？
主的执行计划：
zjy@192.168.10.23 : tt 05:42:15>explain SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2
-> FROM
-> product_property a LEFT JOIN product_property b
-> ON
-> a.product_id = b.product_id
-> WHERE
-> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> ''
-> AND
-> b.value IS NOT NULL AND b.value <> '';
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+
| 1 | SIMPLE | b | ref | property_id,product_id | property_id | 4 | const | 3771874 | Using where; Using temporary |
| 1 | SIMPLE | a | ref | property_id,product_id | product_id | 4 | tt.b.product_id | 1 | Using where |
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+
2 rows in set (0.01 sec)
从的执行计划：
zjy@192.168.10.8 : tt 05:42:07>explain SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2
-> FROM
-> product_property a LEFT JOIN product_property b
-> ON
-> a.product_id = b.product_id
-> WHERE
-> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> ''
-> AND
-> b.value IS NOT NULL AND b.value <> '';
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+
| 1 | SIMPLE | b | ref | property_id,product_id | property_id | 4 | const | 17375012 | Using where; Using temporary |
| 1 | SIMPLE | a | ref | property_id,product_id | product_id | 4 | tt.b.product_id | 5 | Using where |
+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+
2 rows in set (0.00 sec)
执行计划一样，都是先b表再a表，再重新执行初始化的SQL：
主：
zjy@192.168.10.23 : tt 05:42:26>SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2
-> FROM
-> product_property a LEFT JOIN product_property b
-> ON
-> a.product_id = b.product_id
-> WHERE
-> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> ''
-> AND
-> b.value IS NOT NULL AND b.value <> '' limit 10
-> ;
+-----------+-------------+
| col1 | col2 |
+-----------+-------------+
| 鼓浪屿 | gd rg |
| 山松 | 10020122 |
| coulter | 无 |
| oricell | mubmd-01101 |
| oricell | huxma-01101 |
| oricell | huxmf-01001 |
| oricell | rawmx-01001 |
| oricell | rasmx-01001 |
| oricell | rafmx-01101 |
| oricell | rbxmx-01001 |
+-----------+-------------+
10 rows in set (0.00 sec)
从：
zjy@192.168.10.8 : tt 05:42:32>SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2
-> FROM
-> product_property a LEFT JOIN product_property b
-> ON
-> a.product_id = b.product_id
-> WHERE
-> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> ''
-> AND
-> b.value IS NOT NULL AND b.value <> '' limit 10
-> ;
+-----------+-------------+
| col1 | col2 |
+-----------+-------------+
| 鼓浪屿 | gd rg |
| 山松 | 10020122 |
| coulter | 无 |
| oricell | mubmd-01101 |
| oricell | huxma-01101 |
| oricell | huxmf-01001 |
| oricell | rawmx-01001 |
| oricell | rasmx-01001 |
| oricell | rafmx-01101 |
| oricell | rbxmx-01001 |
+-----------+-------------+
10 rows in set (0.01 sec)
好了，要是执行计划一样，结果是：SQL在主从上跑出来的结果一致了。
PS：另一个方法就是用ROW模式，有兴趣的可以测试下。
总结：
主从复制在STATEMENT下面确实被忽略了一些问题，可以用ROW模式代替，但也要知道ROW模式有哪些问题，可以参考MySQL Binlog 【ROW】和【STATEMENT】选择 。也要清楚数据在磁盘块里面分布不一致，影响优化器的选择而导致索引利用的方式也不一样，最终也影响到数据的顺序。
总之，在主从上执行一些比较大的数据量的操作（批量、初始化）的时候，尽可能的先去主从上查看他们的执行计划是否一样，走的索引是否一致，确保查询出来的结果一样。另：这篇文章：blog.xupeng.me/2013/10/11/mysql-replace-into-trap/(MySQL "replace into" 的坑) 也在一定程度上说明别用自增主键当成有意义的数据。
~~~~~~~~~~~~~~~
万物之中,希望至美
~~~~~~~~~~~~~~~
