Kafka
kafka是一个高吞吐的分布式消息队列系统。特点是生产者消费者模式,先进先出(FIFO)保证顺序,自己不丢数据,默认每隔7天清理数据。消息列队常见场景:系统之间解耦合、峰值压力缓冲、异步通信。
- producer : 消息生产者
- consumer : 消息消费之
- broker : kafka集群的server,负责处理消息读、写请求,存储消息,在kafka cluster这一层这里,其实里面是有很多个broker
- topic : 消息队列/分类相当于队列,里面有生产者和消费者模型
- zookeeper : 元数据信息存在zookeeper中,包括:存储消费偏移量,topic话题信息,partition信息
- 1、一个topic分成多个partition
- 2、每个partition内部消息强有序, 其中的每个消息都有一个序号交offset
- 3、一个partition 只对应一个broker, 一个broker 可以管理多个partition
- 4、 消息直接写入文件,并不保存在内存中
- 5、按照时间策略, 默认一周删除, 而不是消息消费完就删除
- 6、producer自己决定网那个partition写消息,可以是轮询的负载均衡,或者是基于hash的partition策略
kafka 的消息消费模型
- consumer 自己维护消费到哪个offset
- 每个consumer都有对应的group
- group 内是queue消费模型
– 各个consumer消费不同的partition
– 一个消息在group内只消费一次 - 各个group各自独立消费,互不影响
kafka 特点
- 生存者消费模型:FIFO; partition内部是FIFO的, partition之间不是FIFO
- 高性能:单节点支持上千个客户端,百MB/s 吞吐
- 持久性:直接持久在普通的磁盘上,性能比较好; 直接append 方式追加到磁盘,数据不会丢
- 分布式:数据副本冗余,流量负载均衡、可扩展; 数据副本,也就是同一份数据可以到不同的broker上面去,也就是当一份数据, 磁盘坏掉,数据不亏丢失
- 很灵活: 消息长时间持久化+Cilent维护消费状态; 1、持久花时间长,可以是一周、一天,2、可以自定义消息偏移量
kafka 安装
- https://www.apache.org/dyn/closer.cgi?path=/kafka/2.0.1/kafka_2.11-2.0.1.tgz 下载
- 解压压缩包,修改config 文件夹下 server.properties
// 节点编号:(不同节点按0,1,2,3整数来配置)
broker.id = 0
// 数据存放目录
log.dirs = /log
// zookeeper 集群配置
zookeeper.connect=node1:2181,node2:2181,node3:2181
- 启动
bin/kafka-server-start.sh config/server.properties
可以单独配置一个启动文件
vim start-kafka.sh
nohup bin/kafka-server-start.sh config/server.properties > kafka.log 2>&1 &
授权 chmod 755 start-kafka.sh
kafka基础命令
创建topic./kafka-topics.sh --zookeeper node1:2181,node2:2181,node3:2181 --create --topic t0315 --partitions 3 --replication-factor 3
查看topic: ./kafka-topics.sh --zookeeper node1:2181,node2:2181,node3:2181 --list
生产者:./kafka-console-producer.sh --topic t0315 --broker-list node1:9092,node2:9092,node3:9092
消费者:./kafka-console-consumer.sh --bootstrap-server node1:9092,node2:9092,node3:9092 --topic t0315
获取描述: ./kafka-topics.sh --describe --zookeeper node1:2181,node2:2181,node3:2181 --topic t0315
kafka中有一个被称为优先副本(preferred replicas)的概念。如果一个分区有3个副本,且这3个副本的优先级别分别为0,1,2,根据优先副本的概念,0会作为leader 。当0节点的broker挂掉时,会启动1这个节点broker当做leader。当0节点的broker再次启动后,会自动恢复为此partition的leader。不会导致负载不均衡和资源浪费,这就是leader的均衡机制。
在配置文件conf/ server.properties中配置开启(默认就是开启):auto.leader.rebalance.enable true
Code 部分
sparkStreaming 的direact 方式
<properties>
<spark.version>2.2.0</spark.version>
</properties>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
<version>${spark.version}</version>
<!-- <exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>-->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
producer 部分:
import kafka.serializer.StringEncoder;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
/**
*@Author PL
*@Date 2018/12/27 10:59
*@Description TODO
**/
public class KafkaProducer {
public static void main(String[] args) throws InterruptedException {
Properties pro = new Properties();
pro.put("bootstrap.servers","node1:9092,node2:9092,node3:9092");
pro.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
pro.put("value.serializer","org.apache.kafka.common.serialization.StringSerializer");
//Producer<String,String> producer = new Producer<String, String>(new ProducerConfig(pro));
//org.apache.kafka.clients.producer.KafkaProducer producer1 = new Kafka
org.apache.kafka.clients.producer.KafkaProducer<String,String> producer = new org.apache.kafka.clients.producer.KafkaProducer<String, String>(pro);
System.out.println("11");
String topic = "t0315";
String msg = "hello word";
for (int i =0 ;i <100;i++) {
producer.send(new ProducerRecord<String, String>(topic, "hello", msg));
System.out.println(msg);
}
producer.close();
}
}
customer
import kafka.serializer.StringDecoder;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.streaming.Durations;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaPairInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.KafkaUtils;
import scala.Tuple2;
import java.util.*;
/**
*@Author PL
*@Date 2018/12/26 13:28
*@Description TODO
**/
public class SparkStreamingForkafka {
public static void main(String[] args) throws InterruptedException {
SparkConf sc = new SparkConf().setMaster("local[2]").setAppName("test");
JavaStreamingContext jsc = new JavaStreamingContext(sc, Durations.seconds(5));
Map<String,String> kafkaParam = new HashMap<>();
kafkaParam.put("metadata.broker.list","node1:9092,node2:9092,node3:9092");
//kafkaParam.put("t0315",1);
HashSet<String> topic = new HashSet<>();
topic.add("t0315");
//JavaPairInputDStream<String, String> line = KafkaUtils.createStream(jsc,"node1:9092,node2:9092,node3:9092","wordcountGrop",kafkaParam);
JavaPairInputDStream<String, String> line = KafkaUtils.createDirectStream(jsc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParam, topic);
JavaDStream<String> flatLine = line.flatMap(new FlatMapFunction<Tuple2<String, String>, String>() {
@Override
public Iterator<String> call(Tuple2<String, String> tuple2) throws Exception {
return Arrays.asList(tuple2._2.split(" ")).iterator();
}
});
JavaPairDStream<String, Integer> pair = flatLine.mapToPair(new PairFunction<String, String, Integer>() {
@Override
public Tuple2<String, Integer> call(String s) throws Exception {
return new Tuple2<String, Integer>(s, 1);
}
});
JavaPairDStream<String, Integer> count = pair.reduceByKey(new Function2<Integer, Integer, Integer>() {
@Override
public Integer call(Integer integer, Integer integer2) throws Exception {
return integer + integer2;
}
});
count.print();
jsc.start();
jsc.awaitTermination();
jsc.close();;
}
}
上述方式为一个SparkStreaming 的消费者, direct方式就是把kafka当成一个存储数据的库,spark 自己维护offset。假设,driver 端宕机了, 之后再重启,会从offset 那一部分开始取?
所以我们需要将kafka 的offset 保存在文件中, 宕机之后在启动时去恢复文件中的offset 读取数据。
import kafka.serializer.StringDecoder;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function0;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.streaming.Durations;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaPairInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.KafkaUtils;
import scala.Tuple2;
import java.util.*;
/**
*@Author PL
*@Date 2018/12/26 13:28
*@Description TODO
**/
public class KafkaCheckPoint {
public static void main(String[] args) throws InterruptedException {
final String checkPoint = "./checkPoint";
Function0<JavaStreamingContext> scFunction = new Function0<JavaStreamingContext>() {
@Override
public JavaStreamingContext call() throws Exception {
return createJavaStreamingContext();
}
};
// 如果存在checkport 就恢复数据,不存在就直接运行
JavaStreamingContext jsc = JavaStreamingContext.getOrCreate(checkPoint, scFunction);
jsc.start();
jsc.awaitTermination();
jsc.close();;
}
public static JavaStreamingContext createJavaStreamingContext(){
System.out.println("初始化"); // 第一次会执行,宕机之后重启执行数据恢复时不执行
final SparkConf sc = new SparkConf().setMaster("local").setAppName("test");
JavaStreamingContext jsc = new JavaStreamingContext(sc, Durations.seconds(5));
/**
* checkpoint 保存
* 1、 配置信息
* 2、Dstream 执行逻辑
* 3、Job 的执行进度
* 4、offset
*/
jsc.checkpoint("./checkPoint");
Map<String,String> kafkaParam = new HashMap<>();
kafkaParam.put("metadata.broker.list","node1:9092,node2:9092,node3:9092");
HashSet<String> topic = new HashSet<>();
topic.add("t0315");
JavaPairInputDStream<String, String> line = KafkaUtils.createDirectStream(jsc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParam, topic);
JavaDStream<String> flatLine = line.flatMap(new FlatMapFunction<Tuple2<String, String>, String>() {
@Override
public Iterator<String> call(Tuple2<String, String> tuple2) throws Exception {
return Arrays.asList(tuple2._2.split(" ")).iterator();
}
});
JavaPairDStream<String, Integer> pair = flatLine.mapToPair(new PairFunction<String, String, Integer>() {
@Override
public Tuple2<String, Integer> call(String s) throws Exception {
return new Tuple2<String, Integer>(s, 1);
}
});
JavaPairDStream<String, Integer> count = pair.reduceByKey(new Function2<Integer, Integer, Integer>() {
@Override
public Integer call(Integer integer, Integer integer2) throws Exception {
return integer + integer2;
}
});
count.print();
return jsc;
}
}
这次我们启动的时候会发现先从checkpoint中恢复数据, 从上次宕机的数据开始读取并执行。但是,当我们更改功能时,发现新修改的部分没有执行, 还是执行的上次保存的代码。。。。。。。
这时候可以把offset 保存至zookeeper中
主方法
import com.pl.data.offset.getoffset.GetTopicOffsetFromKafkaBroker;
import com.pl.data.offset.getoffset.GetTopicOffsetFromZookeeper;
import kafka.common.TopicAndPartition;
import org.apache.log4j.Logger;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import java.util.Map;
public class UseZookeeperManageOffset {
/**
* 使用log4j打印日志,“UseZookeeper.class” 设置日志的产生类
*/
static final Logger logger = Logger.getLogger(UseZookeeperManageOffset.class);
public static void main(String[] args) throws InterruptedException {
/**
* 从kafka集群中得到topic每个分区中生产消息的最大偏移量位置
*/
Map<TopicAndPartition, Long> topicOffsets = GetTopicOffsetFromKafkaBroker.getTopicOffsets("node1:9092,node2:9092,node3:9092", "t0315");
/**
* 从zookeeper中获取当前topic每个分区 consumer 消费的offset位置
*/
Map<TopicAndPartition, Long> consumerOffsets =
GetTopicOffsetFromZookeeper.getConsumerOffsets("node1:2181,node2:2181,node3:2181","pl","t0315");
/**
* 合并以上得到的两个offset ,
* 思路是:
* 如果zookeeper中读取到consumer的消费者偏移量,那么就zookeeper中当前的offset为准。
* 否则,如果在zookeeper中读取不到当前消费者组消费当前topic的offset,就是当前消费者组第一次消费当前的topic,
* offset设置为topic中消息的最大位置。
*/
if(null!=consumerOffsets && consumerOffsets.size()>0){
topicOffsets.putAll(consumerOffsets);
}
/**
* 如果将下面的代码解开,是将topicOffset 中当前topic对应的每个partition中消费的消息设置为0,就是从头开始。
*/
/*for(Map.Entry<TopicAndPartition, Long> item:topicOffsets.entrySet()){
item.setValue(0l);
}*/
/**
* 构建SparkStreaming程序,从当前的offset消费消息
*/
JavaStreamingContext jsc = SparkStreamingDirect.getStreamingContext(topicOffsets,"pl");
jsc.start();
jsc.awaitTermination();
jsc.close();
}
}
获取kafka中当前的offset 偏移量(kafka API)
import kafka.api.PartitionOffsetRequestInfo;
import kafka.cluster.Broker;
import kafka.common.TopicAndPartition;
import kafka.javaapi.OffsetRequest;
import kafka.javaapi.OffsetResponse;
import kafka.javaapi.PartitionMetadata;
import kafka.javaapi.TopicMetadata;
import kafka.javaapi.TopicMetadataRequest;
import kafka.javaapi.TopicMetadataResponse;
import kafka.javaapi.consumer.SimpleConsumer;
/**
* 测试之前需要启动kafka
* @author root
*
*/
public class GetTopicOffsetFromKafkaBroker {
public static void main(String[] args) {
Map<TopicAndPartition, Long> topicOffsets = getTopicOffsets("node1:9092,node2:9092,node3:9092", "t0315");
Set<Entry<TopicAndPartition, Long>> entrySet = topicOffsets.entrySet();
for(Entry<TopicAndPartition, Long> entry : entrySet) {
TopicAndPartition topicAndPartition = entry.getKey();
Long offset = entry.getValue();
String topic = topicAndPartition.topic();
int partition = topicAndPartition.partition();
System.out.println("topic = "+topic+",partition = "+partition+",offset = "+offset);
}
}
/**
* 从kafka集群中得到当前topic,生产者在每个分区中生产消息的偏移量位置
* @param KafkaBrokerServer
* @param topic
* @return
*/
public static Map<TopicAndPartition,Long> getTopicOffsets(String KafkaBrokerServer, String topic){
Map<TopicAndPartition,Long> retVals = new HashMap<TopicAndPartition,Long>();
// 遍历kafka集群,并拆分
for(String broker:KafkaBrokerServer.split(",")){
SimpleConsumer simpleConsumer = new SimpleConsumer(broker.split(":")[0],Integer.valueOf(broker.split(":")[1]), 64*10000,1024,"consumer");
TopicMetadataRequest topicMetadataRequest = new TopicMetadataRequest(Arrays.asList(topic));
TopicMetadataResponse topicMetadataResponse = simpleConsumer.send(topicMetadataRequest);
List<TopicMetadata> topicMetadataList = topicMetadataResponse.topicsMetadata();
// 遍历每个topic下的元数据
for (TopicMetadata metadata : topicMetadataList) {
// 遍历元数据下的分区
for (PartitionMetadata part : metadata.partitionsMetadata()) {
Broker leader = part.leader();
if (leader != null) {
TopicAndPartition topicAndPartition = new TopicAndPartition(topic, part.partitionId());
PartitionOffsetRequestInfo partitionOffsetRequestInfo = new PartitionOffsetRequestInfo(kafka.api.OffsetRequest.LatestTime(), 10000);
OffsetRequest offsetRequest = new OffsetRequest(ImmutableMap.of(topicAndPartition, partitionOffsetRequestInfo), kafka.api.OffsetRequest.CurrentVersion(), simpleConsumer.clientId());
OffsetResponse offsetResponse = simpleConsumer.getOffsetsBefore(offsetRequest);
if (!offsetResponse.hasError()) {
long[] offsets = offsetResponse.offsets(topic, part.partitionId());
retVals.put(topicAndPartition, offsets[0]);
}
}
}
}
simpleConsumer.close();
}
return retVals;
}
}
获取zookeeper中上次的消费的offset
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.RetryUntilElapsed;
import com.fasterxml.jackson.databind.ObjectMapper;
import kafka.common.TopicAndPartition;
public class GetTopicOffsetFromZookeeper {
public static Map<TopicAndPartition,Long> getConsumerOffsets(String zkServers,String groupID, String topic) {
Map<TopicAndPartition,Long> retVals = new HashMap<TopicAndPartition,Long>();
// 连接 zookeeper
ObjectMapper objectMapper = new ObjectMapper();
CuratorFramework curatorFramework = CuratorFrameworkFactory.builder()
.connectString(zkServers).connectionTimeoutMs(1000)
.sessionTimeoutMs(10000).retryPolicy(new RetryUntilElapsed(1000, 1000)).build();
curatorFramework.start();
try{
String nodePath = "/consumers/"+groupID+"/offsets/" + topic;
if(curatorFramework.checkExists().forPath(nodePath)!=null){
List<String> partitions=curatorFramework.getChildren().forPath(nodePath);
for(String partiton:partitions){
int partitionL=Integer.valueOf(partiton);
Long offset=objectMapper.readValue(curatorFramework.getData().forPath(nodePath+"/"+partiton),Long.class);
TopicAndPartition topicAndPartition=new TopicAndPartition(topic,partitionL);
retVals.put(topicAndPartition, offset);
}
}
}catch(Exception e){
e.printStackTrace();
}
curatorFramework.close();
return retVals;
}
public static void main(String[] args) {
Map<TopicAndPartition, Long> consumerOffsets = getConsumerOffsets("node1:2181,node2:2181,node3:2181","pl","t0315");
Set<Entry<TopicAndPartition, Long>> entrySet = consumerOffsets.entrySet();
for(Entry<TopicAndPartition, Long> entry : entrySet) {
TopicAndPartition topicAndPartition = entry.getKey();
String topic = topicAndPartition.topic();
int partition = topicAndPartition.partition();
Long offset = entry.getValue();
System.out.println("topic = "+topic+",partition = "+partition+",offset = "+offset);
}
}
}
读取kafka中指定offset开始的消息
import com.fasterxml.jackson.databind.ObjectMapper;
import kafka.common.TopicAndPartition;
import kafka.message.MessageAndMetadata;
import kafka.serializer.StringDecoder;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.RetryUntilElapsed;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.VoidFunction;
import org.apache.spark.streaming.Durations;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.HasOffsetRanges;
import org.apache.spark.streaming.kafka.KafkaUtils;
import org.apache.spark.streaming.kafka.OffsetRange;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
public class SparkStreamingDirect {
public static JavaStreamingContext getStreamingContext(Map<TopicAndPartition, Long> topicOffsets,final String groupID){
SparkConf conf = new SparkConf().setMaster("local[*]").setAppName("SparkStreamingOnKafkaDirect");
conf.set("spark.streaming.kafka.maxRatePerPartition", "10");
JavaStreamingContext jsc = new JavaStreamingContext(conf, Durations.seconds(5));
// jsc.checkpoint("/checkpoint");
Map<String, String> kafkaParams = new HashMap<String, String>();
kafkaParams.put("metadata.broker.list","node1:9092,node2:9092,node3:9092");
// kafkaParams.put("group.id","MyFirstConsumerGroup");
for(Map.Entry<TopicAndPartition,Long> entry:topicOffsets.entrySet()){
System.out.println(entry.getKey().topic()+"\t"+entry.getKey().partition()+"\t"+entry.getValue());
}
JavaInputDStream<String> message = KafkaUtils.createDirectStream(
jsc,
String.class,
String.class,
StringDecoder.class,
StringDecoder.class,
String.class,
kafkaParams,
topicOffsets,
new Function<MessageAndMetadata<String,String>,String>() {
private static final long serialVersionUID = 1L;
public String call(MessageAndMetadata<String, String> v1)throws Exception {
return v1.message();
}
}
);
final AtomicReference<OffsetRange[]> offsetRanges = new AtomicReference<>();
JavaDStream<String> lines = message.transform(new Function<JavaRDD<String>, JavaRDD<String>>() {
private static final long serialVersionUID = 1L;
@Override
public JavaRDD<String> call(JavaRDD<String> rdd) throws Exception {
OffsetRange[] offsets = ((HasOffsetRanges) rdd.rdd()).offsetRanges();
offsetRanges.set(offsets);
return rdd;
}
}
);
message.foreachRDD(new VoidFunction<JavaRDD<String>>(){
/**
*
*/
private static final long serialVersionUID = 1L;
@Override
public void call(JavaRDD<String> t) throws Exception {
ObjectMapper objectMapper = new ObjectMapper();
CuratorFramework curatorFramework = CuratorFrameworkFactory.builder()
.connectString("node1:2181,node2:2181,node3:2181").connectionTimeoutMs(1000)
.sessionTimeoutMs(10000).retryPolicy(new RetryUntilElapsed(1000, 1000)).build();
curatorFramework.start();
for (OffsetRange offsetRange : offsetRanges.get()) {
long fromOffset = offsetRange.fromOffset();
long untilOffset = offsetRange.untilOffset();
final byte[] offsetBytes = objectMapper.writeValueAsBytes(offsetRange.untilOffset());
String nodePath = "/consumers/"+groupID+"/offsets/" + offsetRange.topic()+ "/" + offsetRange.partition();
System.out.println("nodePath = "+nodePath);
System.out.println("fromOffset = "+fromOffset+",untilOffset="+untilOffset);
if(curatorFramework.checkExists().forPath(nodePath)!=null){
curatorFramework.setData().forPath(nodePath,offsetBytes);
}else{
curatorFramework.create().creatingParentsIfNeeded().forPath(nodePath, offsetBytes);
}
}
curatorFramework.close();
}
});
lines.print();
return jsc;
}
}