缓存和数据库一致性问题中写道:无论是更新缓存还是删除缓存,只要第二步发生失败,那么就会导致数据库和缓存不一致。
1.重试
2.异步重试
3.订阅binlog日志
具体来讲就是,我们的业务应用在修改数据时,「只需」修改数据库,无需操作缓存。
那什么时候操作缓存呢?这就和数据库的「变更日志」有关了。
拿 MySQL 举例,当一条数据发生修改时,MySQL 就会产生一条变更日志(Binlog),我们可以订阅这个日志,拿到具体操作的数据,然后再根据这条数据,去删除对应的缓存。
使用canal中间件,订阅数据库变更日志,将变更传递到下游消息队列.
环境准备:
mysql数据库: 1.开启binlog,row模式,server-id = 1不与canal的slaveId重复
2.账号授权
canal下载安装配置:
1.下载安装
2.配置 - rocketMq为例
conf/canal.properties
#################################################
######### common argument #############
#################################################
# tcp bind ip
canal.ip = 192.168.43.231
# register ip to zookeeper
#canal.register.ip =
canal.port = 11111
#canal.metrics.pull.port = 11112
# canal instance user/passwd
canal.user = canal
canal.passwd = E3619321C1A937C46A0D8BD1DAC39F93B27D4458
# canal admin config
#canal.admin.manager = 127.0.0.1:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
# admin auto register
#canal.admin.register.auto = true
#canal.admin.register.cluster =
#canal.admin.register.name =
#canal.zkServers =
# flush data to zk
#canal.zookeeper.flush.period = 1000
#canal.withoutNetty = false
# tcp, kafka, rocketMQ, rabbitMQ
#需要修改的地方
canal.serverMode = rocketMQ
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE
canal.instance.memory.rawEntry = true
## detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false
# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size = 1024
# mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60
# network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30
# binlog filter config
canal.instance.filter.druid.ddl = true
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false
canal.instance.filter.transaction.entry = false
canal.instance.filter.dml.insert = false
canal.instance.filter.dml.update = false
canal.instance.filter.dml.delete = false
# binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB
# binlog ddl isolation
canal.instance.get.ddl.isolation = false
# parallel parser config
canal.instance.parser.parallel = true
## concurrent thread number, default 60% available processors, suggest not to exceed Runtime.getRuntime().availableProcessors()
#canal.instance.parser.parallelThreadSize = 16
## disruptor ringbuffer size, must be power of 2
canal.instance.parser.parallelBufferSize = 256
# table meta tsdb info
canal.instance.tsdb.enable = true
canal.instance.tsdb.dir = ${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url = jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
canal.instance.tsdb.dbUsername = canal
canal.instance.tsdb.dbPassword = canal
# dump snapshot interval, default 24 hour
canal.instance.tsdb.snapshot.interval = 24
# purge snapshot expire , default 360 hour(15 days)
canal.instance.tsdb.snapshot.expire = 360
#################################################
######### destinations #############
#################################################
#canal.destinations = example
canal.destinations = example
# conf root dir
canal.conf.dir = ../conf
# auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5
# set this value to 'true' means that when binlog pos not found, skip to latest.
# WARN: pls keep 'false' in production env, or if you know what you want.
canal.auto.reset.latest.pos.mode = false
canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml
#canal.instance.tsdb.spring.xml = classpath:spring/tsdb/mysql-tsdb.xml
canal.instance.global.mode = spring
canal.instance.global.lazy = false
canal.instance.global.manager.address = ${canal.admin.manager}
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
canal.instance.global.spring.xml = classpath:spring/file-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml
##################################################
######### MQ Properties #############
##################################################
# aliyun ak/sk , support rds/mq 要修改的
canal.aliyun.accessKey = LTAIs4e56kBVE9
canal.aliyun.secretKey = pJPGAtvJKGWsS
canal.aliyun.uid=
canal.mq.servers = 192.168.56.1:9876
canal.mq.flatMessage = true
canal.mq.canalBatchSize = 50
canal.mq.canalGetTimeout = 100
# Set this value to "cloud", if you want open message trace feature in aliyun.
canal.mq.accessChannel = local
canal.mq.database.hash = true
canal.mq.send.thread.size = 30
canal.mq.build.thread.size = 8
##################################################
######### Kafka #############
##################################################
# kafka.bootstrap.servers = 127.0.0.1:9092
# kafka.acks = all
# kafka.compression.type = none
# kafka.batch.size = 16384
# kafka.linger.ms = 1
# kafka.max.request.size = 1048576
# kafka.buffer.memory = 33554432
# kafka.max.in.flight.requests.per.connection = 1
# kafka.retries = 0
# kafka.kerberos.enable = false
# kafka.kerberos.krb5.file = "../conf/kerberos/krb5.conf"
# kafka.kerberos.jaas.file = "../conf/kerberos/jaas.conf"
##################################################
######### RocketMQ #############
##################################################
rocketmq.producer.group = redis-test-producer
rocketmq.enable.message.trace = false
rocketmq.customized.trace.topic =
rocketmq.namespace =
#需要修改的地方,多个值用分号隔开
rocketmq.namesrv.addr = 192.168.56.1:9876
rocketmq.retry.times.when.send.failed = 0
rocketmq.vip.channel.enabled = false
rocketmq.tag = canal_tag
##################################################
######### RabbitMQ #############
##################################################
# rabbitmq.host =
# rabbitmq.virtual.host =
# rabbitmq.exchange =
# rabbitmq.username =
# rabbitmq.password =
# rabbitmq.deliveryMode =
instance.properties
## mysql serverId
canal.instance.mysql.slaveId = 5
#position info,需要改成自己的数据库信息
canal.instance.master.address = 127.0.0.1:3306
canal.instance.master.journal.name =
canal.instance.master.position =
canal.instance.master.timestamp =
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#username/password,需要改成自己的数据库信息
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8
#table regex
canal.instance.filter.regex =.*\\..*
canal.mq.topic = canal_topic
canal.mq.partition = 0
3.启动
rocket相关
测试:
<!--canal-->
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.client</artifactId>
<version>1.1.4</version>
</dependency>
package com.cloud.redis.consumer.binlog;
rocket监听
@Slf4j
@Component
@RocketMQMessageListener(topic = "canal_topic",
selectorExpression = "*",
consumerGroup = "canal_consumer",
messageModel = MessageModel.CLUSTERING, //广播...
consumeMode = ConsumeMode.ORDERLY) //同时,顺序
public class MyRocketMQListener implements RocketMQListener {
@Override
public void onMessage(Object message) {
System.out.println("MQListener1..."+message);
}
}
问题
1.mysql服务报错:MySQL启动失败提示:本地计算机上的Mysql服务启动后停止 - 查看data下.err日志解决2.连接问题:D:\software\canal\canal\logs\user_date - caching_sha2_password Auth failed -select host, user, authentication_string, plugin from userALTER USER 'canal'@'%' IDENTIFIED WITH mysql_native_password BY 'canal';
3.c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
4. The specified topic is blank - 设置canal.mq.topic
5.Connection refused - canal.serverMode = tcp
6.NoClassDefFoundError:org/apache/rocketmq/client/consumer/DefaultLitePullConsumer - 导包jar
7.Error[UPDATE command denied to user 'canal'@'localhost' for table 'canal_node_server'] - 1. GRANT SELECT, INSERT, UPDATE, DELETE, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; 2.flush privileges; 3.show grants for 'canal'@'%';