Flink 和Flink CDC

转载

deanyuancn 2024-09-23 20:50:04

文章标签 Flink 和Flink CDC mysql flink 数据 文章分类 架构后端开发

Flink CDC

1、CDC 简介

1.1 什么是CDC

CDC 是Change Data Capture（变更数据获取）的简称。核心思想是，监测并捕获数据库 的变动（包括数据或数据表的插入、更新以及删除等），将这些变更按发生的顺序完整记录下来，写入到消息中间件中以供其他服务进行订阅及消费。

1.2 CDC 的种类

CDC 主要分为基于查询和基于 Binlog 两种方式，我们主要了解一下这两种之间的区别：

	基于查询的 CDC	基于 Binlog 的CDC
开源产品	Sqoop、Kafka JDBC Source	Canal、Maxwell、Debezium（Flink里内置了）
执行模式	Batch	Streaming
是否可以捕获所有数据变化	否	是
延迟性	高延迟	低延迟
是否增加数据库压力	是	否

1.3 Flink-CDC

Flink 社区开发了 flink-cdc-connectors 组件，这是一个可以直接从 MySQL、PostgreSQL 等数据库直接读取全量数据和增量变更数据的 source 组件。

目前也已开源，开源地址：https://github.com/ververica/flink-cdc-connectors

2、Flink CDC案例实操

2.1 环境配置

# 1.进入MySql配置文件
vim /etc/my.cnf

# 2.添加如下信息,cdc_test为要开启binlog的数据库名
server_id=1 #mysql5.7版本开启binlog强制需要添加该参数
log_bin=mysql-bin #表示开启binlog并指定binglog文件名
binlog_format=row #默认
expire_logs_days=7 #binlog保留天数
binlog-do-db=cdc_test #设置对哪个数据库开启binlog

# 3.保存退出后，重启MySQL(下面命令为CentOS6.9)
service mysqld restart

# 4.进入到/var/lib/mysql 目录下，输入以下命令，查看是否有结果
ll | grep mysql-bin

# 5.出现结果表示已经开启了binlog

2.2 DataStream 方式的应用

2.2.1 导入依赖

<dependencies>
	<dependency>
 		<groupId>org.apache.flink</groupId>
 		<artifactId>flink-java</artifactId>
 		<version>1.12.0</version>
 	</dependency>
 	
    <dependency>
 		<groupId>org.apache.flink</groupId>
 		<artifactId>flink-streaming-java_2.12</artifactId>
 		<version>1.12.0</version>
 	</dependency>
 
    <dependency>
 		<groupId>org.apache.flink</groupId>
 		<artifactId>flink-clients_2.12</artifactId>
 		<version>1.12.0</version>
 	</dependency>
 
    <dependency>
 		<groupId>org.apache.hadoop</groupId>
 		<artifactId>hadoop-client</artifactId>
 		<version>3.1.3</version>
 	</dependency>
 
    <dependency>
 		<groupId>mysql</groupId>
 		<artifactId>mysql-connector-java</artifactId>
 		<version>5.1.49</version>
	</dependency>

    <dependency>
 		<groupId>org.apache.flink</groupId>
 		<artifactId>flink-table-planner-blink_2.12</artifactId>
 		<version>1.12.0</version>
	</dependency>

    <!-- mysql-cdc -->
    <dependency>
    	<groupId>com.ververica</groupId>
        <artifactId>flink-connector-mysql-cdc</artifactId>
 		<version>2.0.0</version>
	</dependency>
 
    <dependency>
        <groupId>com.alibaba</groupId>
     	<artifactId>fastjson</artifactId>
     	<version>1.2.75</version>
 	</dependency>
</dependencies>
<build>
 	<plugins>
 		<plugin>
 			<groupId>org.apache.maven.plugins</groupId>
 			<artifactId>maven-assembly-plugin</artifactId>
 			<version>3.0.0</version>
 			<configuration>
 				<descriptorRefs>
 					<descriptorRef>jar-with-dependencies</descriptorRef>
 				</descriptorRefs>
 			</configuration>
 			<executions>
 				<execution>
 					<id>make-assembly</id>
 					<phase>package</phase>
 					<goals>
						<goal>single</goal>
 					</goals>
 				</execution>
 			</executions>
 		</plugin>
 	</plugins>
</build>

2.2.2 编写代码

简单测试

package com.pzb;

import com.ververica.cdc.connectors.mysql.MySqlSource;
import com.ververica.cdc.connectors.mysql.table.StartupOptions;
import com.ververica.cdc.debezium.DebeziumSourceFunction;
import com.ververica.cdc.debezium.StringDebeziumDeserializationSchema;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

/**
 * @author 海绵先生
 * @Description TODO
 * @date 2023/1/12-20:03
 */
public class FlinkCDC {
    public static void main(String[] args) throws Exception {
        //1.获取Flink 执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);

        //2.通过FlinkCDC构建SourceFunction
        /*
        * 格式：MySqlSource.<String>builder().[中间各参数].build()
        * */
        DebeziumSourceFunction<String> sourceFunction = MySqlSource.<String>builder()//<String>为最终返回类型，官方提供的返回类型为String
                .hostname("hadoop111")
                .port(3306)
                .username("root")
                .password("xxxx")
                .databaseList("cdc_test")// 监控的数据库，若只选了数据库参数，则监控所有表
                .tableList("cdc_test.user_info")//监控那张表，格式：数据库名.表名。因为databaseList参数是可以监控多个数据库的
                .deserializer(new StringDebeziumDeserializationSchema())
                .startupOptions(StartupOptions.initial())//startupOptions参数有5种方式：initial、earliest、latest、specificOffset、timestamp
                .build();

        // 获取数据
        DataStreamSource<String> dataStreamSource = env.addSource(sourceFunction);

        //打印数据
        dataStreamSource.print();

        env.execute("FlinkCDC");
    }
}

initial：第一次启动时读取原表已有的历史数据, 操作类型为READ, 之后不断做检查点存储；第二次启动时一定要指明检查点文件的具体位置, 这样就可以断点续传; 即使Flink宕机了, 重启后是从上次offset开始读, 而不是latest检查点在打包部署后才有用, 因为那样才可以指明检查点的具体位置
earliest：从BinLog第一行数据开始读, 最好先给这个数据库加上BinLog后, 再去读取创建数据库
latest：读取最新变更数据, 从Flink程序启动后开始算
specificOffset：指明BinLog文件位置和从哪个offset开始读;这个一般来说不怎么用, 因为本地没存offset的信息, 很难知道offset读到哪了
可以从BinLog某一时刻的数据开始读

原文链接：

往监控的数据表里添加数据，查看结果：

<!-- 读取数据 -->
SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={ts_sec=1673530792, file=mysql-bin.000001, pos=444, snapshot=true}} ConnectRecord{topic='mysql_binlog_source.cdc_test.user_info', kafkaPartition=null, key=Struct{id=1001}, keySchema=Schema{mysql_binlog_source.cdc_test.user_info.Key:STRUCT}, value=Struct{after=Struct{id=1001,name=zhangsan,sex=male},source=Struct{version=1.5.2.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1673530792529,snapshot=true,db=cdc_test,table=user_info,server_id=0,file=mysql-bin.000001,pos=444,row=0},op=r,ts_ms=1673530792532}, valueSchema=Schema{mysql_binlog_source.cdc_test.user_info.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}

<!-- 此处省略一条数据 -->

<!-- 插入数据 -->
SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={transaction_id=null, ts_sec=1673531033, file=mysql-bin.000001, pos=509, row=1, server_id=1, event=2}} ConnectRecord{topic='mysql_binlog_source.cdc_test.user_info', kafkaPartition=null, key=Struct{id=1003}, keySchema=Schema{mysql_binlog_source.cdc_test.user_info.Key:STRUCT}, value=Struct{after=Struct{id=1003,name=wangwu,sex=famale},source=Struct{version=1.5.2.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1673531033000,db=cdc_test,table=user_info,server_id=1,file=mysql-bin.000001,pos=649,row=0},op=c,ts_ms=1673531029606}, valueSchema=Schema{mysql_binlog_source.cdc_test.user_info.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}

<!-- 修改数据 -->
SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={transaction_id=null, ts_sec=1673531342, file=mysql-bin.000001, pos=803, row=1, server_id=1, event=2}} ConnectRecord{topic='mysql_binlog_source.cdc_test.user_info', kafkaPartition=null, key=Struct{id=1003}, keySchema=Schema{mysql_binlog_source.cdc_test.user_info.Key:STRUCT}, value=Struct{before=Struct{id=1003,name=wangwu,sex=famale},after=Struct{id=1003,name=wangwu,sex=male},source=Struct{version=1.5.2.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1673531342000,db=cdc_test,table=user_info,server_id=1,file=mysql-bin.000001,pos=943,row=0},op=u,ts_ms=1673531338605}, valueSchema=Schema{mysql_binlog_source.cdc_test.user_info.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}

<!-- 删除数据 -->
SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={transaction_id=null, ts_sec=1673531548, file=mysql-bin.000001, pos=1119, row=1, server_id=1, event=2}} ConnectRecord{topic='mysql_binlog_source.cdc_test.user_info', kafkaPartition=null, key=Struct{id=1002}, keySchema=Schema{mysql_binlog_source.cdc_test.user_info.Key:STRUCT}, value=Struct{before=Struct{id=1002,name=lisi,sex=male},source=Struct{version=1.5.2.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1673531548000,db=cdc_test,table=user_info,server_id=1,file=mysql-bin.000001,pos=1259,row=0},op=d,ts_ms=1673531544601}, valueSchema=Schema{mysql_binlog_source.cdc_test.user_info.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}

Flink 和Flink CDC_mysql

修改则是u，删除为d

采用initial时，当程序重启时，历史数据还是会被消费，并且都是读取形式

2.3 Flink SQL 方式应用

//2.2通过FlinkSQL构建SourceFunction
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
        tableEnv.executeSql("CREATE TABLE user_info ( " +
                " id STRING PRIMARY KEY, " +
                " name STRING, " +
                " sex STRING " +
                " ) WITH ( " +
                " 'connector' = 'mysql-cdc', " +
                " 'scan.startup.mode' = 'initial', " +
                " 'hostname' = 'hadoop111', " +
                " 'port' = '3306', " +
                " 'username' = 'root', " +
                " 'password' = '1234', " +
                " 'database-name' = 'cdc_test', " +
                " 'table-name' = 'user_info' " +
                ")");
        //3.查询数据并转换为流输出
        Table table = tableEnv.sqlQuery("select * from user_info");
        DataStream<Tuple2<Boolean, Row>> retractStream = tableEnv.toRetractStream(table, Row.class);
        retractStream.print();

操作结果：

/**********读取************/
(true,+I[1001, zhangsan, male])
(true,+I[1003, wangwu, female])
/*********插入***********/
(true,+I[1002, lisi, male])
/**********更新**********/
(false,-U[1002, lisi, male])
(true,+U[1002, lisi, female])
/*********删除**********/
(false,-D[1002, lisi, female])

2.3 自定义反序列化器

package com.pzb;

import com.alibaba.fastjson.JSONObject;
import com.ververica.cdc.debezium.DebeziumDeserializationSchema;
import io.debezium.data.Envelope;
import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.util.Collector;
import org.apache.kafka.connect.data.Field;
import org.apache.kafka.connect.data.Schema;
import org.apache.kafka.connect.data.Struct;
import org.apache.kafka.connect.source.SourceRecord;

import java.util.List;


/**
 * @author 海绵先生
 * @Description TODO 关于DataStream方式的CDC，自定义反序列化
 * @date 2023/1/13-21:26
 */
/*官方默认String类型的数据样式
SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={transaction_id=null, ts_sec=1673531342, file=mysql-bin.000001, pos=803, row=1, server_id=1, event=2}} ConnectRecord{topic='mysql_binlog_source.cdc_test.user_info', kafkaPartition=null, key=Struct{id=1003}, keySchema=Schema{mysql_binlog_source.cdc_test.user_info.Key:STRUCT}, value=Struct{before=Struct{id=1003,name=wangwu,sex=famale},after=Struct{id=1003,name=wangwu,sex=male},source=Struct{version=1.5.2.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1673531342000,db=cdc_test,table=user_info,server_id=1,file=mysql-bin.000001,pos=943,row=0},op=u,ts_ms=1673531338605}, valueSchema=Schema{mysql_binlog_source.cdc_test.user_info.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}
* */
public class CustomerDeserializationSchema implements DebeziumDeserializationSchema<String> {

    /*TODO 明确自己想要的数据格式
    * (
    *   "db":"",
    *   "tableName":"",
    *   "before":{"id":"1001","name":""...},
    *   "after":{"id":"1001","name":""...},
    *   "op":""
    * )
    * */

    @Override
    public void deserialize(SourceRecord sourceRecord, Collector<String> collector) throws Exception {
        //创建JSON 对象用于封装结果数据
        JSONObject result = new JSONObject();

        //获取库名&表名
        String topic = sourceRecord.topic();//根据上面样式通过sourceRecord.键值，获取对应值
        //获取结果：topic='mysql_binlog_source.cdc_test.user_info'
        String[] fields = topic.split("\\.");//安装`.`进行分割（.需要转义）

        //添加对应的库名和表名键值对
        result.put("db",fields[1]);
        result.put("tableName",fields[2]);

        //获取before数据
        Struct value = (Struct) sourceRecord.value();//需要进行强转下,注意导的是：org.apache.kafka.connect.data.Struct 这个包
        Struct before = value.getStruct("before");//通过指定before，获取before字段的数据
        JSONObject beforeJson = new JSONObject();
        if (before != null){// before字段是有可能为空的(比如读取[op=r]、插入操作[op=c]...)，所以要进行判断
            Schema schema = before.schema();//获取before 的schema信息
            List<Field> fieldList = schema.fields();//获取before里的全部字段

            for (Field field : fieldList){
                // 通过field.name()获取对应的字段名,before.get(field)根据字段名,获取对应的值
                beforeJson.put(field.name(), before.get(field));
            }
        }
        result.put("before",beforeJson);//把before信息添加进去

        //同理获取after
        Struct after = value.getStruct("after");
        JSONObject afterJson = new JSONObject();
        if (after != null){
            Schema schema = after.schema();
            List<Field> fieldList = schema.fields();

            for (Field field : fieldList){
                afterJson.put(field.name(), after.get(field));
            }
        }
        result.put("after",afterJson);

        //获取操作类型(operation不能直接通过sourceRecord获取)
        Envelope.Operation operation = Envelope.operationFor(sourceRecord);//注意导包：io.debezium.data.Envelope
        //将operation信息添加进去
        result.put("op",operation);

        //输出数据
        collector.collect(result.toJSONString());
    }

    @Override
    public TypeInformation<String> getProducedType() {
        // 返回类型
        return BasicTypeInfo.STRING_TYPE_INFO;
    }
}

自定义后的结果：

{"op":"READ","before":{},"after":{"sex":"male","name":"zhangsan","id":"1001"},"db":"cdc_test","tableName":"user_info"}
{"op":"READ","before":{},"after":{"sex":"male","name":"wangwu","id":"1003"},"db":"cdc_test","tableName":"user_info"}
{"op":"UPDATE","before":{"sex":"male","name":"wangwu","id":"1003"},"after":{"sex":"female","name":"wangwu","id":"1003"},"db":"cdc_test","tableName":"user_info"}
{"op":"CREATE","before":{},"after":{"sex":"male","name":"lisi","id":"1002"},"db":"cdc_test","tableName":"user_info"}
{"op":"DELETE","before":{"sex":"male","name":"lisi","id":"1002"},"after":{},"db":"cdc_test","tableName":"user_info"}

在自定义前一定要明白自己想要的是什么数据

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。

上一篇：spring security入口

下一篇：koa2 typescript 使用ctx

提问和评论都可以，用心的回复会被更多人看到评论

发布评论

相关文章

官方博客	全部文章	热门标签	班级博客
了解我们	网站地图	意见反馈

鸿蒙开发者社区	51CTO学堂
51CTO	软考资讯

Flink 和Flink CDC

Flink 和Flink CDC

Flink CDC

1、CDC 简介

1.1 什么是CDC

1.2 CDC 的种类

1.3 Flink-CDC

2、Flink CDC案例实操

2.1 环境配置

2.2 DataStream 方式的应用

2.2.1 导入依赖

2.2.2 编写代码

2.3 Flink SQL 方式应用

2.3 自定义反序列化器

51CTO博客