kafka 顺序消费 多线程 kafka如何实现顺序消费

topic: "topic_query_p3r1" 分配了三个partition分区

实现顺序性原理：

设置相同的key会把消息投递到同一个分区的topic中，再由一个消费者来消费该分区topic。

投递顺序消息

   同一组行为设置相同的key，会把这组数据投递到同一分区topic中。

/**
     * 投递顺序性消息，根据用户id做取模推送到不同分区的topic中
     * 相同的key推送到同一分区中
     */
    @RequestMapping("/kafka2")
    public String testKafka2() {
        for (int userId = 0; userId < 300; userId++) {
            kafkaTemplate.send("topic_query_p3r1", userId + "", "insert" + userId);
            kafkaTemplate.send("topic_query_p3r1", userId + "", "update" + userId);
            kafkaTemplate.send("topic_query_p3r1", userId + "", "delete" + userId);
        }
        return null;
    }

消费顺序消息

方式1 - 直接进行消费

   因为投递的相同行为的消息是有序的，所以直接消费也不会有问题。

/**
     * 消费topic_query_p3r1主题，ConsumerGroupId1消费组
     */
    @KafkaListener(topics = "topic_query_p3r1", groupId = "ConsumerGroupId1")
    public void p3r2ConsumerGroupId0(ConsumerRecord<?, ?> consumer) throws InterruptedException {
        System.out.println("消费者A topic名称:" + consumer.topic() +
                ", key:" + consumer.key() +
                ", value:" + consumer.value() +
                ", 分区位置:" + consumer.partition() +
                ", 下标" + consumer.offset()+"    "+Thread.currentThread().getId());
        Thread.sleep(10);
    }

方式2.1 - 一个消费者来指定具体分区进行消费

   指定具体分区来进行消费。

/**
     * 消费者，解决消息顺序性
     * 注解参数：partitions=0表示：只消费该主题中0分区的数据。
     */
    @KafkaListener(topicPartitions = {@TopicPartition(topic = "topic_query_p3r1", partitions = {"0"})}, groupId = "ConsumerGroupId1")
    public void receive(ConsumerRecord<?, ?> consumer) {
        System.out.println("消费者C topic名称:" + consumer.topic() +
                ",key:" + consumer.key() + "," +
                ",value:" + consumer.value() + "," +
                "分区位置:" + consumer.partition() +
                ", 下标" + consumer.offset());
    }

方式2.2 - 多个消费者来指定不同分区进行消费。

   写多个消费者方法来分别指向不同分区，提高消费速度，但是此方法不灵活。

/**
     * 消费0分区的topic_query_p3r1主题消费者，ConsumerGroupId1消费组
     */
    @KafkaListener(topicPartitions = {@TopicPartition(topic = "topic_query_p3r1", partitions = {"0"})}, groupId = "ConsumerGroupId1")
    public void p3r2ConsumerGroupId0(ConsumerRecord<?, ?> consumer) throws InterruptedException {
        System.out.println("消费者A topic名称:" + consumer.topic() +
                ", key:" + consumer.key() +
                ", value:" + consumer.value() +
                ", 分区位置:" + consumer.partition() +
                ", 下标" + consumer.offset()+"    "+Thread.currentThread().getId());
        Thread.sleep(10);
    }
    /**
     * 消费1分区的topic_query_p3r1主题消费者，ConsumerGroupId1消费组
     */
    @KafkaListener(topicPartitions = {@TopicPartition(topic = "topic_query_p3r1", partitions = {"1"})}, groupId = "ConsumerGroupId1")
    public void p3r2ConsumerGroupId1(ConsumerRecord<?, ?> consumer) throws InterruptedException {
        System.out.println("消费者A topic名称:" + consumer.topic() +
                ", key:" + consumer.key() +
                ", value:" + consumer.value() +
                ", 分区位置:" + consumer.partition() +
                ", 下标" + consumer.offset()+"    "+Thread.currentThread().getId());
        Thread.sleep(10);
    }
    /**
     * 消费2分区的topic_query_p3r1主题消费者，ConsumerGroupId1消费组
     */
    @KafkaListener(topicPartitions = {@TopicPartition(topic = "topic_query_p3r1", partitions = {"2"})}, groupId = "ConsumerGroupId1")
    public void p3r2ConsumerGroupId2(ConsumerRecord<?, ?> consumer) throws InterruptedException {
        System.out.println("消费者A topic名称:" + consumer.topic() +
                ", key:" + consumer.key() +
                ", value:" + consumer.value() +
                ", 分区位置:" + consumer.partition() +
                ", 下标" + consumer.offset()+"    "+Thread.currentThread().getId());
        Thread.sleep(10);
    }

多线程顺序消费

// 使用两个内存队列
    final int queueLingth = 2;

    // 创建两个内存队列
    Queue<Map> queueA = new ConcurrentLinkedQueue<>();
    Queue<Map> queueB = new ConcurrentLinkedQueue<>();

    /**
     * 投递顺序性消息，根据用户id做取模推送到不同分区的topic中
     * 相同的key推送到相同的分区中
     */
    @RequestMapping("/kafka2")
    public String testKafka2() {
        for (int userId = 0; userId < 300; userId++) {
            kafkaTemplate.send("topic_query_p3r1", userId + "", "insert" + userId);
            kafkaTemplate.send("topic_query_p3r1", userId + "", "update" + userId);
            kafkaTemplate.send("topic_query_p3r1", userId + "", "delete" + userId);
        }
        return null;
    }

    /**
     * 主题消费者-把相同行为的数据放到同一内存队列中
     */
    @KafkaListener(topics = "topic_query_p3r1", groupId = "ConsumerGroupId1")
    public void p3r2ConsumerGroupId0(ConsumerRecord<?, ?> consumer){
        // 1.封装消息参数
        Map param = new HashMap();
        param.put("topic", consumer.topic());
        param.put("key", consumer.key());
        param.put("value", consumer.value());
        param.put("p", consumer.partition());

        // 2.把相同行为(key)数据添加到同一内存队列中
        int queueHash = consumer.key().hashCode() % queueLingth;
        if (queueHash == 0) {
            queueA.add(param);
        }
        if (queueHash == 1) {
            queueB.add(param);
        }
    }


    // 开启两个线程消费内存队列中的消息
    @Override
    public void run(ApplicationArguments args) throws Exception {
        new Thread() {
            @Override
            public void run() {
                while (true) {
                    if (queueA.size() > 0) {
                        Map poll = queueA.poll();
                        System.out.println("Thrend-Id: "+ Thread.currentThread().getId() +
                                "  topic：" + poll.get("topic") +
                                "  key:" + poll.get("key") +
                                "  value:" + poll.get("value") +
                                "  partition:" + poll.get("p"));

                        try {
                            Thread.sleep(10);
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                        }
                    }
                }
            }
        }.start();

        new Thread() {
            @Override
            public void run() {
                while (true) {
                    if (queueB.size() > 0) {
                        Map poll = queueB.poll();
                        System.out.println("Thrend-Id: "+ Thread.currentThread().getId() + "  topic：" + poll.get("topic") + "  key:" + poll.get("key") + "  value:" + poll.get("value") + "  partition:" + poll.get("p"));

                        try {
                            Thread.sleep(10);
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                        }
                    }
                }
            }
        }.start();

    }

打印：insert、update、delete都是有序的。相同行为都在同一线程下执行。