canal.instance.filter.regex:
mysql 数据解析关注的表，Perl正则表达式.
多个正则之间以逗号(,)分隔，转义符需要双斜杠(\\)
常见例子：
1. 所有表：.* or .*\\..*
2. canal schema下所有表： canal\\..*
3. canal下的以canal打头的表：canal\\.canal.*
4. canal schema下的一张表：canal\\.test1
5. 多个规则组合使用：canal\\..*,mysql.test1,mysql.test2 (逗号分隔)
canal.instance.filter.black.regex : mysql 数据解析表的黑名单，表达式规则见白名单的规则
https://github.com/alibaba/canal/wiki/AdminGuide
几点说明：
1. mysql链接时的起始位置
canal.instance.master.journal.name + canal.instance.master.position : 精确指定一个binlog位点，进行启动
canal.instance.master.timestamp : 指定一个时间戳，canal会自动遍历mysql binlog，找到对应时间戳的binlog位点后，进行启动
不指定任何信息：默认从当前数据库的位点，进行启动。(show master status)
2. mysql解析关注表定义
标准的Perl正则，注意转义时需要双斜杠：\\
3. mysql链接的编码
目前canal版本仅支持一个数据库只有一种编码，如果一个库存在多个编码，需要通过filter.regex配置，将其拆分为多个canal instance，为每个instance指定不同的编码
在介绍instance配置之前，先了解一下canal如何维护一份增量订阅&消费的关系信息：
解析位点 (parse模块会记录，上一次解析binlog到了什么位置，对应组件为：CanalLogPositionManager)
消费位点 (canal server在接收了客户端的ack后，就会记录客户端提交的最后位点，对应的组件为：CanalMetaManager)
canal-admin模式，canal.serverMode必须配，不然会报NPE
2020-06-23 00:00:14.984 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] WARN c.a.o.c.p.inbo
und.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2020-06-23 00:00:15.024 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] WARN c.a.o.c.p.inbo
und.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just last position
{"identity":{"slaveId":-1,"sourceAddress":{"address":"mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local","port":3306}},"postion":{"gtid":"","included":false,"journa
lName":"mysql-uat-new-mysqlha-0-bin.001768","position":345523555,"serverId":100,"timestamp":1585193491000}}
2020-06-23 00:00:15.042 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] WARN c.a.o.c.p.inbo
und.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-uat-new-mysqlha-0-bin.001768,position=345523
555,serverId=100,gtid=,timestamp=1585193491000] cost : 58ms , the next step is binlog dump
2020-06-23 00:00:15.056 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] ERROR c.a.o.canal.pa
rse.inbound.mysql.dbsync.DirectLogFetcher - I/O error while reading from client socket
java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:102) ~[canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:235) [canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:257) [canal.parse-1.1.3.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2020-06-23 00:00:15.056 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] ERROR c.a.o.c.p.inbo
und.mysql.rds.RdsBinlogEventParserProxy - dump address mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 has an error, retrying. caused by
java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:102) ~[canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:235) ~[canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:257) ~[canal.parse-1.1.3.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2020-06-23 00:00:15.057 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] ERROR com.alibaba.ot
ter.canal.common.alarm.LogAlarmHandler - destination:omssaps[java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log fi
le name in binary log index file
at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:102)
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:235)
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:257)
at java.lang.Thread.run(Thread.java:748)
]
http://whitesock.iteye.com/blog/1329616
http://www.iteye.com/topic/1129002 七锋已经fix了canal的，要合到dbsync里面。
http://www.bitscn.com/pdb/mysql/201404/228411.html
查询资料发现mysql版本为5.6时，
这个错误一般出现在master5.6,slave在低版本的情况下。这是由于5.6使用了crc32做binlog的checksum；
当一个event被写入binary log(二进制日志)的时候，checksum也同时写入binary log，然后在event通过网络传输到从服务器(slave)之后，再在从服务器中对其进行验证并写入从服务器的relay log.
由于每一步都记录了event和checksum，所以我们可以很快地找出问题所在。
在master1中设置binlog_checksum =none；
mysql> show variables like "%sum%";
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| binlog_checksum | CRC32 |
| innodb_checksum_algorithm | innodb |
| innodb_checksums | ON |
| master_verify_checksum | OFF |
| slave_sql_verify_checksum | ON |
+---------------------------+--------+
5 rows in set (0.00 sec)
mysql> set global binlog_checksum='NONE'
Query OK, 0 rows affected (0.09 sec)
mysql> show variables like "%sum%";
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| binlog_checksum | NONE |
| innodb_checksum_algorithm | innodb |
| innodb_checksums | ON |
| master_verify_checksum | OFF |
| slave_sql_verify_checksum | ON |
+---------------------------+--------+
\
2020-06-23 20:36:04.987 [destination = omssaps , address = mysql.zkh360.com/120.27.222.64:13306 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandl
er - destination:omssaps[com.google.common.collect.ComputationException: com.alibaba.fastjson.JSONException: deserialize inet adress error
at com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:889)
at com.alibaba.otter.canal.meta.MemoryMetaManager.getCursor(MemoryMetaManager.java:91)
at com.alibaba.otter.canal.meta.PeriodMixedMetaManager.getCursor(PeriodMixedMetaManager.java:150)
at com.alibaba.otter.canal.parse.index.MetaLogPositionManager.getLatestIndexBy(MetaLogPositionManager.java:57)
at com.alibaba.otter.canal.parse.index.FailbackLogPositionManager.getLatestIndexBy(FailbackLogPositionManager.java:68)
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.findStartPositionInternal(MysqlEventParser.java:424)
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.findStartPosition(MysqlEventParser.java:366)
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:186)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.fastjson.JSONException: deserialize inet adress error
at com.alibaba.fastjson.serializer.MiscCodec.deserialze(MiscCodec.java:316)
at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:642)
at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:610)
at com.alibaba.fastjson.serializer.MiscCodec.deserialze(MiscCodec.java:189)
at com.alibaba.fastjson.parser.deserializer.FastjsonASMDeserializer_3_LogIdentity.deserialze(Unknown Source)
at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:184)
at com.alibaba.fastjson.parser.deserializer.FastjsonASMDeserializer_2_LogPosition.deserialze(Unknown Source)
at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:184)
at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:557)
at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:188)
at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:184)
at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:642)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:350)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:318)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:281)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:381)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:361)
at com.alibaba.otter.canal.common.utils.JsonUtils.unmarshalFromByte(JsonUtils.java:36)
at com.alibaba.otter.canal.meta.ZooKeeperMetaManager.getCursor(ZooKeeperMetaManager.java:153)
at com.alibaba.otter.canal.meta.PeriodMixedMetaManager$3.apply(PeriodMixedMetaManager.java:65)
at com.alibaba.otter.canal.meta.PeriodMixedMetaManager$3.apply(PeriodMixedMetaManager.java:62)
at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:356)
at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:182)
at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:151)
at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:67)
at com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:885)
... 8 more
Caused by: java.net.UnknownHostException: mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at java.net.InetAddress.getByName(InetAddress.java:1076)
at com.alibaba.fastjson.serializer.MiscCodec.deserialze(MiscCodec.java:314)
... 33 more
]
canal和otter共用一个zk，在/otter/canal/destinations/ 节点上会复用的
otter中关于canal在zk上的标记：
com.alibaba.otter.shared.arbitrate.impl.ArbitrateViewServiceImpl
/**
* 查询当前的仲裁器的一些运行状态视图
*
* @author jianghang 2011-9-27 下午05:27:38
* @version 4.0.0
*/
public class ArbitrateViewServiceImpl implements ArbitrateViewService {
private static final String CANAL_PATH = "/otter/canal/destinations/%s";
private static final String CANAL_DATA_PATH = CANAL_PATH + "/%s";
private static final String CANAL_CURSOR_PATH = CANAL_PATH + "/%s/cursor";
private ZkClientx zookeeper = ZooKeeperClient.getInstance();
public MainStemEventData mainstemData(Long channelId, Long pipelineId) {
String path = ManagePathUtils.getMainStem(channelId, pipelineId);
try {
byte[] bytes = zookeeper.readData(path);
return JsonUtils.unmarshalFromByte(bytes, MainStemEventData.class);
} catch (ZkException e) {
return null;
}
}
public Long getNextProcessId(Long channelId, Long pipelineId) {
String processRoot = ManagePathUtils.getProcessRoot(channelId, pipelineId);
IZkConnection connection = zookeeper.getConnection();
// zkclient会将获取stat信息和正常的操作分开，使用原生的zk进行优化
ZooKeeper orginZk = ((ZooKeeperx) connection).getZookeeper();
Stat processParentStat = new Stat();
// 获取所有的process列表
try {
orginZk.getChildren(processRoot, false, processParentStat);
return (Long) ((processParentStat.getCversion() + processParentStat.getNumChildren()) / 2L);
} catch (Exception e) {
return -1L;
}
}
public List<ProcessStat> listProcesses(Long channelId, Long pipelineId) {
List<ProcessStat> processStats = new ArrayList<ProcessStat>();
String processRoot = ManagePathUtils.getProcessRoot(channelId, pipelineId);
IZkConnection connection = zookeeper.getConnection();
// zkclient会将获取stat信息和正常的操作分开，使用原生的zk进行优化
ZooKeeper orginZk = ((ZooKeeperx) connection).getZookeeper();
// 获取所有的process列表
List<String> processNodes = zookeeper.getChildren(processRoot);
List<Long> processIds = new ArrayList<Long>();
for (String processNode : processNodes) {
processIds.add(ManagePathUtils.getProcessId(processNode));
}
Collections.sort(processIds);
for (int i = 0; i < processIds.size(); i++) {
Long processId = processIds.get(i);
// 当前的process可能会有变化
ProcessStat processStat = new ProcessStat();
processStat.setPipelineId(pipelineId);
processStat.setProcessId(processId);
List<StageStat> stageStats = new ArrayList<StageStat>();
processStat.setStageStats(stageStats);
try {
String processPath = ManagePathUtils.getProcess(channelId, pipelineId, processId);
Stat zkProcessStat = new Stat();
List<String> stages = orginZk.getChildren(processPath, false, zkProcessStat);
Collections.sort(stages, new StageComparator());
StageStat prev = null;
for (String stage : stages) {// 循环每个process下的stage
String stagePath = processPath + "/" + stage;
Stat zkStat = new Stat();
StageStat stageStat = new StageStat();
stageStat.setPipelineId(pipelineId);
stageStat.setProcessId(processId);
byte[] bytes = orginZk.getData(stagePath, false, zkStat);
if (bytes != null && bytes.length > 0) {
// 特殊处理zookeeper里的data信息，manager没有对应node中PipeKey的对象，所以导致反序列化会失败，需要特殊处理，删除'@'符号
String json = StringUtils.remove(new String(bytes, "UTF-8"), '@');
EtlEventData data = JsonUtils.unmarshalFromString(json, EtlEventData.class);
stageStat.setNumber(data.getNumber());
stageStat.setSize(data.getSize());
Map exts = new HashMap();
if (!CollectionUtils.isEmpty(data.getExts())) {
exts.putAll(data.getExts());
}
exts.put("currNid", data.getCurrNid());
exts.put("nextNid", data.getNextNid());
exts.put("desc", data.getDesc());
stageStat.setExts(exts);
}
if (prev != null) {// 对应的start时间为上一个节点的结束时间
stageStat.setStartTime(prev.getEndTime());
} else {
stageStat.setStartTime(zkProcessStat.getMtime()); // process的最后修改时间,select
// await成功后会设置USED标志位
}
stageStat.setEndTime(zkStat.getMtime());
if (ArbitrateConstants.NODE_SELECTED.equals(stage)) {
stageStat.setStage(StageType.SELECT);
} else if (ArbitrateConstants.NODE_EXTRACTED.equals(stage)) {
stageStat.setStage(StageType.EXTRACT);
} else if (ArbitrateConstants.NODE_TRANSFORMED.equals(stage)) {
stageStat.setStage(StageType.TRANSFORM);
// } else if
// (ArbitrateConstants.NODE_LOADED.equals(stage)) {
// stageStat.setStage(StageType.LOAD);
}
prev = stageStat;
stageStats.add(stageStat);
}
// 添加一个当前正在处理的
StageStat currentStageStat = new StageStat();
currentStageStat.setPipelineId(pipelineId);
currentStageStat.setProcessId(processId);
if (prev == null) {
byte[] bytes = orginZk.getData(processPath, false, zkProcessStat);
if (bytes == null || bytes.length == 0) {
continue; // 直接认为未使用，忽略之
}
ProcessNodeEventData nodeData = JsonUtils.unmarshalFromByte(bytes, ProcessNodeEventData.class);
if (nodeData.getStatus().isUnUsed()) {// process未使用,直接忽略
continue; // 跳过该process
} else {
currentStageStat.setStage(StageType.SELECT);// select操作
currentStageStat.setStartTime(zkProcessStat.getMtime());
}
} else {
// 判断上一个节点，确定当前的stage
StageType stage = prev.getStage();
if (stage.isSelect()) {
currentStageStat.setStage(StageType.EXTRACT);
} else if (stage.isExtract()) {
currentStageStat.setStage(StageType.TRANSFORM);
} else if (stage.isTransform()) {
currentStageStat.setStage(StageType.LOAD);
} else if (stage.isLoad()) {// 已经是最后一个节点了
continue;
}
currentStageStat.setStartTime(prev.getEndTime());// 开始时间为上一个节点的结束时间
}
if (currentStageStat.getStage().isLoad()) {// load必须为第一个process节点
if (i == 0) {
stageStats.add(currentStageStat);
}
} else {
stageStats.add(currentStageStat);// 其他情况都添加
}
} catch (NoNodeException e) {
// ignore
} catch (KeeperException e) {
throw new ArbitrateException(e);
} catch (InterruptedException e) {
// ignore
} catch (UnsupportedEncodingException e) {
// ignore
}
processStats.add(processStat);
}
return processStats;
}
public PositionEventData getCanalCursor(String destination, short clientId) {
String path = String.format(CANAL_CURSOR_PATH, destination, String.valueOf(clientId));
try {
IZkConnection connection = zookeeper.getConnection();
// zkclient会将获取stat信息和正常的操作分开，使用原生的zk进行优化
ZooKeeper orginZk = ((ZooKeeperx) connection).getZookeeper();
Stat stat = new Stat();
byte[] bytes = orginZk.getData(path, false, stat);
PositionEventData eventData = new PositionEventData();
eventData.setCreateTime(new Date(stat.getCtime()));
eventData.setModifiedTime(new Date(stat.getMtime()));
eventData.setPosition(new String(bytes, "UTF-8"));
return eventData;
} catch (Exception e) {
return null;
}
}
public void removeCanalCursor(String destination, short clientId) {
String path = String.format(CANAL_CURSOR_PATH, destination, String.valueOf(clientId));
zookeeper.delete(path);
}
@Override
public void removeCanal(String destination, short clientId) {
String path = String.format(CANAL_DATA_PATH, destination, String.valueOf(clientId));
zookeeper.deleteRecursive(path);
}
public void removeCanal(String destination) {
String path = String.format(CANAL_PATH, destination);
zookeeper.deleteRecursive(path);
}
}
cannal中在zk上的标识：
/**
* 存储结构：
*
* <pre>
* /otter
* canal
* cluster
* destinations
* dest1
* running (EPHEMERAL)
* cluster
* client1
* running (EPHEMERAL)
* cluster
* filter
* cursor
* mark
* 1
* 2
* 3
* </pre>
*
* @author zebin.xuzb @ 2012-6-21
* @version 1.0.0
*/
public class ZookeeperPathUtils {
public static final String ZOOKEEPER_SEPARATOR = "/";
public static final String OTTER_ROOT_NODE = ZOOKEEPER_SEPARATOR + "otter";
public static final String CANAL_ROOT_NODE = OTTER_ROOT_NODE + ZOOKEEPER_SEPARATOR
+ "canal";
public static final String DESTINATION_ROOT_NODE = CANAL_ROOT_NODE + ZOOKEEPER_SEPARATOR
+ "destinations";
public static final String FILTER_NODE = "filter";
public static final String BATCH_MARK_NODE = "mark";
public static final String PARSE_NODE = "parse";
public static final String CURSOR_NODE = "cursor";
public static final String RUNNING_NODE = "running";
public static final String CLUSTER_NODE = "cluster";
public static final String DESTINATION_NODE = DESTINATION_ROOT_NODE
+ ZOOKEEPER_SEPARATOR + "{0}";
public static final String DESTINATION_PARSE_NODE = DESTINATION_NODE + ZOOKEEPER_SEPARATOR
+ PARSE_NODE;
public static final String DESTINATION_CLIENTID_NODE = DESTINATION_NODE + ZOOKEEPER_SEPARATOR
+ "{1}";
public static final String DESTINATION_CURSOR_NODE = DESTINATION_CLIENTID_NODE
+ ZOOKEEPER_SEPARATOR + CURSOR_NODE;
public static final String DESTINATION_CLIENTID_FILTER_NODE = DESTINATION_CLIENTID_NODE
+ ZOOKEEPER_SEPARATOR + FILTER_NODE;
public static final String DESTINATION_CLIENTID_BATCH_MARK_NODE = DESTINATION_CLIENTID_NODE
+ ZOOKEEPER_SEPARATOR + BATCH_MARK_NODE;
public static final String DESTINATION_CLIENTID_BATCH_MARK_WITH_ID_PATH = DESTINATION_CLIENTID_BATCH_MARK_NODE
+ ZOOKEEPER_SEPARATOR + "{2}";
/**
* 服务端当前正在提供服务的running节点
*/
public static final String DESTINATION_RUNNING_NODE = DESTINATION_NODE + ZOOKEEPER_SEPARATOR
+ RUNNING_NODE;
/**
* 客户端当前正在工作的running节点
*/
public static final String DESTINATION_CLIENTID_RUNNING_NODE = DESTINATION_CLIENTID_NODE
+ ZOOKEEPER_SEPARATOR + RUNNING_NODE;
/**
* 整个canal server的集群列表
*/
public static final String CANAL_CLUSTER_ROOT_NODE = CANAL_ROOT_NODE + ZOOKEEPER_SEPARATOR
+ CLUSTER_NODE;
public static final String CANAL_CLUSTER_NODE = CANAL_CLUSTER_ROOT_NODE
+ ZOOKEEPER_SEPARATOR + "{0}";
/**
* 针对某个destination的工作的集群列表
*/
public static final String DESTINATION_CLUSTER_ROOT = DESTINATION_NODE + ZOOKEEPER_SEPARATOR
+ CLUSTER_NODE;
public static final String DESTINATION_CLUSTER_NODE = DESTINATION_CLUSTER_ROOT
+ ZOOKEEPER_SEPARATOR + "{1}";
public static String getDestinationPath(String destinationName) {
return MessageFormat.format(DESTINATION_NODE, destinationName);
}
public static String getClientIdNodePath(String destinationName, short clientId) {
return MessageFormat.format(DESTINATION_CLIENTID_NODE, destinationName, String.valueOf(clientId));
}
public static String getFilterPath(String destinationName, short clientId) {
return MessageFormat.format(DESTINATION_CLIENTID_FILTER_NODE, destinationName, String.valueOf(clientId));
}
public static String getBatchMarkPath(String destinationName, short clientId) {
return MessageFormat.format(DESTINATION_CLIENTID_BATCH_MARK_NODE, destinationName, String.valueOf(clientId));
}
public static String getBatchMarkWithIdPath(String destinationName, short clientId, Long batchId) {
return MessageFormat.format(DESTINATION_CLIENTID_BATCH_MARK_WITH_ID_PATH,
destinationName,
String.valueOf(clientId),
getBatchMarkNode(batchId));
}
public static String getCursorPath(String destination, short clientId) {
return MessageFormat.format(DESTINATION_CURSOR_NODE, destination, String.valueOf(clientId));
}
public static String getCanalClusterNode(String node) {
return MessageFormat.format(CANAL_CLUSTER_NODE, node);
}
/**
* 服务端当前正在提供服务的running节点
*/
public static String getDestinationServerRunning(String destination) {
return MessageFormat.format(DESTINATION_RUNNING_NODE, destination);
}
/**
* 客户端当前正在工作的running节点
*/
public static String getDestinationClientRunning(String destination, short clientId) {
return MessageFormat.format(DESTINATION_CLIENTID_RUNNING_NODE, destination, String.valueOf(clientId));
}
public static String getDestinationClusterNode(String destination, String node) {
return MessageFormat.format(DESTINATION_CLUSTER_NODE, destination, node);
}
public static String getDestinationClusterRoot(String destination) {
return MessageFormat.format(DESTINATION_CLUSTER_ROOT, destination);
}
public static String getParsePath(String destination) {
return MessageFormat.format(DESTINATION_PARSE_NODE, destination);
}
/**
* 将batchNode转换为Long
*/
public static short getClientId(String clientNode) {
return Short.valueOf(clientNode);
}
/**
* 将batchNode转换为Long
*/
public static long getBatchMarkId(String batchMarkNode) {
return Long.valueOf(batchMarkNode);
}
/**
* 将batchId转化为zookeeper中的node名称
*/
public static String getBatchMarkNode(Long batchId) {
return StringUtils.leftPad(String.valueOf(batchId.intValue()), 10, '0');
}
}
com.alibaba.otter.canal.common.zookeeper.ZookeeperPathUtils
node启动时连接manager的重试操作：
/**
* 通讯交互的client的默认实现实现
*
* @author jianghang
*/
public class DefaultCommunicationClientImpl implements CommunicationClient {
private static final Logger logger = LoggerFactory.getLogger(DefaultCommunicationClientImpl.class);
private CommunicationConnectionFactory factory = null;
private int poolSize = 10;
private ExecutorService executor = null;
private int retry = 3;
private int retryDelay = 1000;
private boolean discard = false;
public DefaultCommunicationClientImpl(){
}
public DefaultCommunicationClientImpl(CommunicationConnectionFactory factory){
this.factory = factory;
}
public void initial() {
RejectedExecutionHandler handler = null;
if (discard) {
handler = new ThreadPoolExecutor.DiscardPolicy();
} else {
handler = new ThreadPoolExecutor.AbortPolicy();
}
executor = new ThreadPoolExecutor(poolSize, poolSize, 60 * 1000L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(10 * 1000),
new NamedThreadFactory("communication-async"), handler);
}
public void destory() {
executor.shutdown();
}
public Object call(final String addr, final Event event) {
Assert.notNull(this.factory, "No factory specified");
CommunicationParam params = buildParams(addr);
CommunicationConnection connection = null;
int count = 0;
Throwable ex = null;
while (count++ < retry) {
try {
connection = factory.createConnection(params);
return connection.call(event);
} catch (Exception e) {
logger.error(String.format("call[%s] , retry[%s]", addr, count), e);
try {
Thread.sleep(count * retryDelay);
} catch (InterruptedException e1) {
// ignore
}
ex = e;
} finally {
if (connection != null) {
connection.close();
}
}
}
logger.error("call[{}] failed , event[{}]!", addr, event.toString());
throw new CommunicationException("call[" + addr + "] , Event[" + event.toString() + "]", ex);
}
public void call(final String addr, final Event event, final Callback callback) {
Assert.notNull(this.factory, "No factory specified");
submit(new Runnable() {
@Override
public void run() {
Object obj = call(addr, event);
callback.call(obj);
}
});
}
public Object call(final String[] addrs, final Event event) {
Assert.notNull(this.factory, "No factory specified");
if (addrs == null || addrs.length == 0) {
throw new IllegalArgumentException("addrs example: 127.0.0.1:1099");
}
ExecutorCompletionService completionService = new ExecutorCompletionService(executor);
List<Future<Object>> futures = new ArrayList<Future<Object>>(addrs.length);
List result = new ArrayList(10);
for (final String addr : addrs) {
futures.add(completionService.submit((new Callable<Object>() {
@Override
public Object call() throws Exception {
return DefaultCommunicationClientImpl.this.call(addr, event);
}
})));
}
Exception ex = null;
int errorIndex = 0;
while (errorIndex < futures.size()) {
try {
Future future = completionService.take();// 它也可能被打断
future.get();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
ex = e;
break;
} catch (ExecutionException e) {
ex = e;
break;
}
errorIndex++;
}
if (errorIndex < futures.size()) {
for (int index = 0; index < futures.size(); index++) {
Future<Object> future = futures.get(index);
if (future.isDone() == false) {
future.cancel(true);
}
}
} else {
for (int index = 0; index < futures.size(); index++) {
Future<Object> future = futures.get(index);
try {
result.add(future.get());
} catch (InterruptedException e) {
// ignore
Thread.currentThread().interrupt();
} catch (ExecutionException e) {
// ignore
}
}
}
if (ex != null) {
throw new CommunicationException(String.format("call addr[%s] error by %s", addrs[errorIndex],
ex.getMessage()), ex);
} else {
return result;
}
}
public void call(final String[] addrs, final Event event, final Callback callback) {
Assert.notNull(this.factory, "No factory specified");
if (addrs == null || addrs.length == 0) {
throw new IllegalArgumentException("addrs example: 127.0.0.1:1099");
}
submit(new Runnable() {
@Override
public void run() {
Object obj = call(addrs, event);
callback.call(obj);
}
});
}
/**
* 直接提交一个异步任务
*/
public Future submit(Runnable call) {
Assert.notNull(this.factory, "No factory specified");
return executor.submit(call);
}
/**
* 直接提交一个异步任务
*/
public Future submit(Callable call) {
Assert.notNull(this.factory, "No factory specified");
return executor.submit(call);
}
// ===================== helper method ==================
private CommunicationParam buildParams(String addr) {
CommunicationParam params = new CommunicationParam();
String[] strs = StringUtils.split(addr, ":");
if (strs == null || strs.length != 2) {
throw new IllegalArgumentException("addr example: 127.0.0.1:1099");
}
InetAddress address = null;
try {
address = InetAddress.getByName(strs[0]);
} catch (UnknownHostException e) {
throw new CommunicationException("addr_error", "addr[" + addr + "] is unknow!");
}
params.setIp(address.getHostAddress());
params.setPort(Integer.valueOf(strs[1]));
return params;
}
}
com.alibaba.otter.shared.communication.core.impl.DefaultCommunicationClientImpl
2020-08-04 17:48:35.353 [main] ERROR c.a.o.s.c.core.impl.DefaultCommunicationClientImpl - call[otter-manager.pro.svc.cluster.local:1099] , retry[2]
com.alibaba.dubbo.rpc.RpcException: Failed to invoke remote method: acceptEvent, provider: dubbo://172.25.3.187:1099/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&iothreads=4&lazy=true&pa
yload=8388608&serialization=java&threads=50, cause: client(url: dubbo://172.25.3.187:1099/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&heartbeat=60000&iothreads=4&lazy=true&payload=83886
08&send.reconnect=true&serialization=java&threads=50) failed to connect to server /172.25.3.187:1099, error message is:Connection refused
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:101) ~[dubbo-2.5.3.jar:2.5.3]
at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144) ~[dubbo-2.5.3.jar:2.5.3]
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52) ~[dubbo-2.5.3.jar:2.5.3]
at com.alibaba.dubbo.common.bytecode.proxy0.acceptEvent(proxy0.java) ~[na:2.5.3]
last position【运行时位点存在哪？】
①destinations，节点读取目录，在conf目录下创建一个目录，这边创建的节点名字为vads0
②默认配置是file-instance.xml 这个就是各种信息会使用文件的形式记录，我选择使用这边写的default-instance.xml ，因为我不想去看文件。default有一行配置将游标记录在ZK服务上面。
区别就是cursor文件有没有在ZK上面记录。
21. 链接方式（参考：http://www.importnew.com/25189.html）
1. HA配置架构图
2. 单连
3. 两个client+两个instance+1个mysql
当mysql变动时，两个client都能获取到变动
4. 一个server+两个instance+两个mysql+两个client
5. instance****的standby配置
Standby：备库
22. 总结
这里总结了一下Canal的一些点，仅供参考：
- 原理：模拟mysql slave的交互协议，伪装自己为mysql slave，向mysql master发送dump协议；mysql master收到dump请求，开始推送binary log给slave(也就是canal)；解析binary log对象(原始为byte流)
- 重复消费问题：在消费端解决。
- 采用开源的open-replicator来解析binlog
- canal需要维护EventStore，可以存取在Memory, File, zk
- canal需要维护客户端的状态，同一时刻一个instance只能有一个消费端消费
- 数据传输格式：protobuff
- 支持binlog format 类型:statement, row, mixed. 多次附加功能只能在row下使用，比如otter
- binlog position可以支持保存在内存，文件，zk中
- instance启动方式：rpc/http; 内嵌
- 有ACK机制
- 无告警，无监控，这两个功能都需要对接外部系统
- 方便快速部署。
23. 我调试成功的代码地址
https://gitee.com/zhiqishao/canal-client
