flink消费kafka 消费线程 flink消费kafka并行度

转载

mob64ca1405d568 2024-02-21 14:12:26

文章标签 flink消费kafka 消费线程数据 kafka 并行度 文章分类 架构后端开发

Flink 版本： 1.15.0

问题

在社区看到以下问题：

请教个问题哈，sink 到 kafka，采用默认的分区器，是不是每个并行度都会与kafka的partition维护一个连接

比如 10 个并行度，3个 partition，那么维护的连接数总共为 10*3 个  ？  是的

还是一个taskManager建立一个生产者 一个生产者对应多个分区

一个taskManager里面多个slot共享一个生产者？ no

刚好想起，之前有个分析程序，用 FlinkKafkaProducer 写数据到 kafka，sink 只有一个并行度，sink 的 topic 有多个分区，数据永远只往分区 0 发送数据

测试程序

来一个简单的测试程序：

读取 kafka 数据
来个 map 算子处理一下，在数据上加入当前的 subtask 的 index，标明数据是在哪个并行度处理的
sink 数据到 kafka
写个简单的 java 消费者程序，讲数据的分区和数据内容输出

通过调整 flink 任务的并行度和 sink 的 topic 的分区数，测试以上问题

flink 程序

读取kafka，map算子给数据加上当前 subtask 数，标明数据是那个并行度的，最后 sink 到 kafka

object KafkaSinkTest {

  val LOG = LoggerFactory.getLogger("KafkaSinkTest")

  def main(args: Array[String]): Unit = {

    val topic = "user_log"
    val sinkTopic = "user_log_sink"

    // env
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    // global parllelism
    val parallelism = 1
    env.setParallelism(parallelism)

    // kafka source
    val kafkaSource = KafkaSource.builder[String]()
      .setBootstrapServers(Common.BROKER_LIST)
      .setTopics(topic)
      .setGroupId("KafkaSinkTest")
      .setStartingOffsets(OffsetsInitializer.latest())
      .setValueOnlyDeserializer(new SimpleStringSchema())
      .build();

    // kafka sink
    val kafkaSink = KafkaSink
      .builder[String]()
      .setBootstrapServers(bootstrapServer)
      .setKafkaProducerConfig(Common.getProp)
      .setRecordSerializer(KafkaRecordSerializationSchema.builder[String]()
        .setTopic(sinkTopic)
        // 不指定 key 的序列号器，key 会为 空
//        .setKeySerializationSchema(new SimpleStringSchema())
        .setValueSerializationSchema(new SimpleStringSchema())
        .build()
      )
      .build()

    // add source，读取数据
    val sourceStream = env.fromSource(kafkaSource, WatermarkStrategy.noWatermarks(), "kafkaSource")

    // map, add current subtask index
    val mapStream = sourceStream
     // rebalance data to all parallelisn
      .rebalance
      .flatMap(new RichFlatMapFunction[String, String] {
        override def flatMap(element: String, out: Collector[String]): Unit = {
          val parallelism = getRuntimeContext.getIndexOfThisSubtask
          out.collect(parallelism + "," + element)

        }
      })
      .name("flatMap")
      .uid("flatMap")

    // sink to kafka, new api
    mapStream.sinkTo(kafkaSink)

    // sink to kafka, old api
    //    val kafkaProducer = new FlinkKafkaProducer[String](bootstrapServer,sinkTopic, new SimpleStringSchema())
    //    mapStream.addSink(kafkaProducer)
    //      .setParallelism(parallelism)

    env.execute("KafkaSinkTest")
  }

任务流图如下：

flink消费kafka 消费线程 flink消费kafka并行度_kafka

消费者

打印从 kafka 消费出来的数据，并打印数据在 topic 的分区编号

private static final String topic = "user_log_sink";

    public static void main(String[] args) throws InterruptedException {

        Properties prop = KafkaUtils.getProp();

        KafkaConsumer kafkaConsumer = new KafkaConsumer(prop);
        kafkaConsumer.subscribe(Arrays.asList(topic));
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

        while (true) {

            ConsumerRecords<String, String> consumerRecords = kafkaConsumer.poll(100);
            String key;
            String time;
            String value = null;

            for (ConsumerRecord<String, String> record : consumerRecords) {
                value = record.value();
                // 读取数据所在的分区
                int partition = record.partition();

                System.out.println( "partition: " +partition+ ", value : " + value);
            }

            Thread.sleep(5 * 1000);
   }
}

场景测试

场景1： 1 个并行度对 3 个分区

kakfa 消费结果：

partition: 2, value : 0,{"category_id":10,"user_id":"349","item_id":"10507","behavior":"buy","ts":"2022-05-28 15:34:51.774"}
partition: 0, value : 0,{"category_id":10,"user_id":"351","item_id":"10941","behavior":"buy","ts":"2022-05-28 15:34:53.774"}
partition: 1, value : 0,{"category_id":10,"user_id":"348","item_id":"10354","behavior":"buy","ts":"2022-05-28 15:34:50.774"}

结论：数据一对多，一个并行度的数据往 3 个分区发

场景2： 3 个并行度对 1 个分区

kakfa 消费结果：

partition: 0, value : 1,{"category_id":10,"user_id":"72","item_id":"10433","behavior":"buy","ts":"2022-05-27 14:19:02.437"}
partition: 0, value : 2,{"category_id":10,"user_id":"73","item_id":"1093","behavior":"buy","ts":"2022-05-27 14:19:03.437"}
partition: 0, value : 0,{"category_id":10,"user_id":"74","item_id":"10217","behavior":"buy","ts":"2022-05-27 14:19:04.437"}

结论：数据多对一，多个并行度的数据往 1 个分区发

场景3： 3 个并行度对 3 个分区

kakfa 消费结果：

partition: 0, value : 1,{"category_id":10,"user_id":"567","item_id":"10900","behavior":"buy","ts":"2022-05-27 14:27:17.437"}
partition: 2, value : 1,{"category_id":10,"user_id":"573","item_id":"10241","behavior":"buy","ts":"2022-05-27 14:27:23.437"}
partition: 1, value : 1,{"category_id":10,"user_id":"570","item_id":"10943","behavior":"buy","ts":"2022-05-27 14:27:20.437"}

partition: 0, value : 2,{"category_id":10,"user_id":"568","item_id":"1050","behavior":"buy","ts":"2022-05-27 14:27:18.437"}
partition: 1, value : 2,{"category_id":10,"user_id":"574","item_id":"1021","behavior":"buy","ts":"2022-05-27 14:27:24.437"}

partition: 1, value : 0,{"category_id":10,"user_id":"566","item_id":"1028","behavior":"buy","ts":"2022-05-27 14:27:16.437"}
partition: 2, value : 0,{"category_id":10,"user_id":"569","item_id":"10801","behavior":"buy","ts":"2022-05-27 14:27:19.437"}
partition: 0, value : 0,{"category_id":10,"user_id":"578","item_id":"10759","behavior":"buy","ts":"2022-05-27 14:27:28.437"}

结论：数据多对多，多个并行度的数据往 3 个分区发

场景4： 2 个并行度对 4 个分区

kakfa 消费结果：

partition: 0, value : 0,{"category_id":10,"user_id":"397","item_id":"10669","behavior":"buy","ts":"2022-05-28 15:35:39.774"}
partition: 2, value : 0,{"category_id":10,"user_id":"399","item_id":"10940","behavior":"buy","ts":"2022-05-28 15:35:41.774"}
partition: 1, value : 0,{"category_id":10,"user_id":"427","item_id":"1032","behavior":"buy","ts":"2022-05-28 15:36:09.774"}

partition: 2, value : 1,{"category_id":10,"user_id":"401","item_id":"10798","behavior":"buy","ts":"2022-05-28 15:35:43.774"}
partition: 0, value : 1,{"category_id":10,"user_id":"400","item_id":"10654","behavior":"buy","ts":"2022-05-28 15:35:42.774"}
partition: 1, value : 1,{"category_id":10,"user_id":"398","item_id":"10627","behavior":"buy","ts":"2022-05-28 15:35:40.774"}

结论：数据多对多，多个并行度的数据往 3 个分区发

场景5： 2 个并行度对 3 个分区

kakfa 消费结果：

partition: 2, value : 1,{"category_id":10,"user_id":"708","item_id":"10929","behavior":"buy","ts":"2022-05-27 14:29:38.437"}
partition: 0, value : 1,{"category_id":10,"user_id":"706","item_id":"10161","behavior":"buy","ts":"2022-05-27 14:29:36.437"}

partition: 2, value : 0,{"category_id":10,"user_id":"709","item_id":"10438","behavior":"buy","ts":"2022-05-27 14:29:39.437"}
partition: 0, value : 0,{"category_id":10,"user_id":"707","item_id":"10870","behavior":"buy","ts":"2022-05-27 14:29:37.437"}
partition: 1, value : 0,{"category_id":10,"user_id":"711","item_id":"10789","behavior":"buy","ts":"2022-05-27 14:29:41.437"}

结论：数据一对多，一个并行度的数据往 3 个分区发

场景6： 4 个并行度对 2 个分区

kakfa 消费结果：

partition: 0, value : 0,{"category_id":10,"user_id":"264","item_id":"10261","behavior":"buy","ts":"2022-05-27 14:22:14.437"}
partition: 1, value : 0,{"category_id":10,"user_id":"271","item_id":"10784","behavior":"buy","ts":"2022-05-27 14:22:21.437"}

partition: 1, value : 1,{"category_id":10,"user_id":"261","item_id":"10206","behavior":"buy","ts":"2022-05-27 14:22:11.437"}
partition: 0, value : 1,{"category_id":10,"user_id":"268","item_id":"10781","behavior":"buy","ts":"2022-05-27 14:22:18.437"}

partition: 1, value : 2,{"category_id":10,"user_id":"262","item_id":"10429","behavior":"buy","ts":"2022-05-27 14:22:12.437"}
partition: 0, value : 2,{"category_id":10,"user_id":"269","item_id":"10893","behavior":"buy","ts":"2022-05-27 14:22:19.437"}

partition: 1, value : 3,{"category_id":10,"user_id":"263","item_id":"10578","behavior":"buy","ts":"2022-05-27 14:22:13.437"}
partition: 0, value : 3,{"category_id":10,"user_id":"267","item_id":"10790","behavior":"buy","ts":"2022-05-27 14:22:17.437"}

结论：数据多对多，多个并行度的数据往 2 个分区发

场景7： 3 个并行度对 2 个分区

kakfa 消费结果：

partition: 0, value : 2,{"category_id":10,"user_id":"475","item_id":"10947","behavior":"buy","ts":"2022-05-27 14:25:45.437"}
partition: 1, value : 2,{"category_id":10,"user_id":"478","item_id":"10605","behavior":"buy","ts":"2022-05-27 14:25:48.437"}

partition: 1, value : 0,{"category_id":10,"user_id":"479","item_id":"10749","behavior":"buy","ts":"2022-05-27 14:25:49.437"}
partition: 0, value : 0,{"category_id":10,"user_id":"476","item_id":"10856","behavior":"buy","ts":"2022-05-27 14:25:46.437"}

partition: 0, value : 1,{"category_id":10,"user_id":"477","item_id":"10418","behavior":"buy","ts":"2022-05-27 14:25:47.437"}
partition: 1, value : 1,{"category_id":10,"user_id":"474","item_id":"1078","behavior":"buy","ts":"2022-05-27 14:25:44.437"}

结论：数据多对多，多个并行度的数据往 2 个分区发

场景8：老的 kafka api FlinkKafkaProducer 1 个并行度对 3 个分区

kakfa 消费结果：

partition: 0, value : 0,{"category_id":10,"user_id":"520","item_id":"10207","behavior":"buy","ts":"2022-05-28 15:37:42.774"}

结论：数据一对多，一个并行度的数据往 1 个分区发

场景9：老的 kafka api FlinkKafkaProducer 2 个并行度对 4 个分区

kakfa 消费结果：

partition: 0, value : 0,{"category_id":10,"user_id":"580","item_id":"10577","behavior":"buy","ts":"2022-05-28 15:38:42.774"}
partition: 1, value : 1,{"category_id":10,"user_id":"577","item_id":"10481","behavior":"buy","ts":"2022-05-28 15:38:39.774"}

结论：数据一对多，一个并行度的数据只会往 1 个分区发，另外两个分区无数据

场景9：老的 kafka api FlinkKafkaProducer 4 个并行度对 2 个分区

kakfa 消费结果：

partition: 0, value : 2,{"category_id":10,"user_id":"785","item_id":"10883","behavior":"buy","ts":"2022-05-28 15:42:07.774"}
partition: 0, value : 0,{"category_id":10,"user_id":"787","item_id":"102","behavior":"buy","ts":"2022-05-28 15:42:09.774"}
partition: 1, value : 1,{"category_id":10,"user_id":"788","item_id":"10618","behavior":"buy","ts":"2022-05-28 15:42:10.774"}
partition: 1, value : 3,{"category_id":10,"user_id":"790","item_id":"10627","behavior":"buy","ts":"2022-05-28 15:42:12.774"}

结论：数据多对一，一个并行度的数据只会往 1 个分区发

结论

KafkaSink api

随机策略，数据随机发往所有分区

Flink 的 kafka connector 并没有设置分区策略，直接使用的 kafka 客户端

Flink 中调用 kafka 生产者发送数据：

flink消费kafka 消费线程 flink消费kafka并行度_并行度_02

Flink 1.15.0 使用的 kafka-client 版本是 2.8.1 的默认设置: 随机选择一个和上一次不一样的分区

分区源码： StickyPartitionCache.nextPartition

public int nextPartition(String topic, Cluster cluster, int prevPartition) {
        List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
        Integer oldPart = indexCache.get(topic);
        Integer newPart = oldPart;
        // Check that the current sticky partition for the topic is either not set or that the partition that 
        // triggered the new batch matches the sticky partition that needs to be changed.
        if (oldPart == null || oldPart == prevPartition) {
            List<PartitionInfo> availablePartitions = cluster.availablePartitionsForTopic(topic);
            if (availablePartitions.size() < 1) {
                Integer random = Utils.toPositive(ThreadLocalRandom.current().nextInt());
                newPart = random % partitions.size();
            } else if (availablePartitions.size() == 1) {
                newPart = availablePartitions.get(0).partition();
            } else {
                while (newPart == null || newPart.equals(oldPart)) {
                    int random = Utils.toPositive(ThreadLocalRandom.current().nextInt());
                    newPart = availablePartitions.get(random % availablePartitions.size()).partition();
                }
            }
            // Only change the sticky partition if it is null or prevPartition matches the current sticky partition.
            if (oldPart == null) {
                indexCache.putIfAbsent(topic, newPart);
            } else {
                indexCache.replace(topic, prevPartition, newPart);
            }
            return indexCache.get(topic);
        }
        return indexCache.get(topic);
    }

注：如果指定 key 的序列化器，会将数据用指定的序列化器序列化后生成 kafka 的 key

源码：

@Override
public ProducerRecord<byte[], byte[]> serialize(
        IN element, KafkaSinkContext context, Long timestamp) {
    final String targetTopic = topicSelector.apply(element);
    final byte[] value = valueSerializationSchema.serialize(element);
    byte[] key = null;
    // key 的 序列化器，讲数据序列化为 key，实际就是 key 和 value 一样
    if (keySerializationSchema != null) {
        key = keySerializationSchema.serialize(element);
    }
    final OptionalInt partition =
            partitioner != null
                    ? OptionalInt.of(
                            partitioner.partition(
                                    element,
                                    key,
                                    value,
                                    targetTopic,
                                    context.getPartitionsForTopic(targetTopic)))
                    : OptionalInt.empty();

    return new ProducerRecord<>(
            targetTopic,
            partition.isPresent() ? partition.getAsInt() : null,
            timestamp == null || timestamp < 0L ? null : timestamp,
            key,
            value);
}

注：这样有个小问题，

FlinkKafkaProducer api

默认采用 sink 算子并行度 % kafka 分区数，所以每个并行度的数据永远都只会往一个分区发

@Override
public int partition(T record, byte[] key, byte[] value, String targetTopic, int[] partitions) {
    Preconditions.checkArgument(
            partitions != null && partitions.length > 0,
            "Partitions of the target topic is empty.");

    return partitions[parallelInstanceId % partitions.length];
}

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。