Knative接收Kafka消息
knative可以通过POST接收cloudevents格式的HTTP请求,同时knative提供kafkaSource,kafkaChannel,kafkaBinding的方式接收普通kafka消息并转换成cloudevents发到knative service。这里提供基于Java的实现以及kafkaSource的实现。
Java
项目搭建
首先下载cloudevents-java SDK ,然后利用maven将这4个包进行打包,然后手动引入项目
接下来pom这么写,因为向Knative Broker发送HTTP请求,所以这里使用OKHttp3
<dependency>
<groupId>io.cloudevents</groupId>
<artifactId>cloudevents-kafka</artifactId>
<version>2.0.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>io.cloudevents</groupId>
<artifactId>cloudevents-json-jackson</artifactId>
<version>2.0.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.11</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.25</version>
<scope>compile</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp -->
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>4.8.0</version>
</dependency>
在打包方式上按下面
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<!--这里要替换成jar包main方法所在类 -->
<mainClass>com.aaron.consumer.MultiTopicListener</mainClass>
</manifest>
<manifestEntries>
<Class-Path>.</Class-Path>
</manifestEntries>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id> <!-- this is used for inheritance merges -->
<phase>package</phase> <!-- 指定在打包节点执行jar包合并操作 -->
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
准备工作完成后,开始解决Java如何监听
Java监听多topic
首先得解决如何获取bootstrapServer
以及topic
,因为Java程序是会放到k8s中运行的,所以在yaml中进行参数编写。标签是env
具体deployment.yaml
如下
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-cloudevent-test-deploy
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: kafka-cloudevent-test
release: stabel
template:
metadata:
labels:
app: kafka-cloudevent-test
release: stabel
env: test
spec:
containers:
- name: kafka-cloudevent-test
image: docker.io/kafka-cloudevents-test:1.0
imagePullPolicy: Always
ports:
- name: http
containerPort: 80
env:
- name: broker
value: http://broker-ingress.knative-eventing.svc.cluster.local/default/kafka-broker
- name: kafka.configs
value: "[\"localhost:9092#raw\",\"localhost:9092#test\"]"
这里为了简便topic和bootstrapServer的编写工作,将其以bootstrapServer#topic
的形式组成一个字符串。上面env的作用就是为app设置环境参数,这样Java可以使用System.getenv()
获取到kafka集群参数。
这里为了优化代码结构,创建了KafkaConfig
和KnativeConfig
两个POJO
class KafkaConfig{
private String bootStrapServer;
private String topic;
}
class KnativeConfig{
public static final String BROKER = "broker";
private String broker;
}
这里broker是指在knative中broker通过ingress接收HTTP请求,因此这个broker地址就是ingress的url,可以通过kubectl get broker
查看
KafkaUtil
public class KafkaUtil {
private static Logger logger = LoggerFactory.getLogger(KafkaUtil.class);
private static final String BOOTSTRAP_SERVERS_CONFIG = "bootstrap.servers";
private static final String TOPIC = "topic";
private static final String KAFKA_CONFIGS = "kafka.configs";
private static Gson gson = new GsonBuilder().create();
public static KnativeConfig getKnativeConfig(){
Map<String,String> env = System.getenv();
KnativeConfig config = new KnativeConfig();
String broker = env.get(KnativeConfig.BROKER);
if (StringUtils.isNotBlank(broker)){
config.setBroker(broker);
return config;
}
throw new IllegalArgumentException("broker can not be null");
}
public static List<KafkaConfig> getMultiKafkaConfig(){
Map<String,String> env = System.getenv();
String configsJson = env.get(KAFKA_CONFIGS);
if (StringUtils.isNotBlank(configsJson)){
String[] kafkaConfigs = gson.fromJson(configsJson,String[].class);
return Arrays.stream(kafkaConfigs).map(KafkaUtil::split).collect(Collectors.toList());
}
throw new IllegalArgumentException("kafka.configs can not be null");
}
/**
* 将bootstrapServer#topic字符串分割
* @param strs 字符串
* @return 0为bootstrapServer 1为topic
*/
public static KafkaConfig split(String strs){
String[] args = strs.split("#");
return new KafkaConfig(args[0],args[1]);
}
public static KafkaConsumer<String,String> createConsumer(KafkaConfig config){
Properties properties = new Properties();
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,config.getBootStrapServer());
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "sample-cloudevents-consumer");
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
// 这里对Kafka消息解析
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
KafkaConsumer<String,String> consumer = new KafkaConsumer<>(properties);
consumer.subscribe(Collections.singletonList(config.getTopic()));
return consumer;
}
}
CloudEventUtil
public class CloudEventUtil {
private static Logger logger = LoggerFactory.getLogger(KafkaCloudEventsConverter.class);
private static final String TYPE = "dev.knative.kafka.event";
private static final String JSON = "application/json";
private static final String CE_JSON = "application/cloudevents+json; charset=UTF-8";
private static final String SUBJECT_PATTERN = "partition:%s#%s";
private static final String SOURCE = "%s#%s";
private static Gson gson = new GsonBuilder().create();
public static Request buildCloudEventRequest(ConsumerRecord<String,String> record, String url){
Request.Builder builder = new Request.Builder();
String subject = String.format(SUBJECT_PATTERN,record.partition(),record.offset());
String source = String.format(SOURCE,KafkaUtil.getKafkaConfig()[0],record.topic());
MediaType jsonType = MediaType.parse("application/json");
Headers.Builder headerBuilder = new Headers.Builder();
RequestBody requestBody = RequestBody.create(jsonType,gson.toJson(record.value()));
headerBuilder.add("Ce-Specversion","1.0");
headerBuilder.add("Ce-Type",TYPE);
headerBuilder.add("Ce-Source",source);
headerBuilder.add("Content-Type",JSON);
headerBuilder.add("Ce-Subject",subject);
return builder.headers(headerBuilder.build()).post(requestBody).url(url).build();
}
public static void sendMsgToBroker(ConsumerRecord<String,String> record, String broker){
OkHttpClient client = HttpUtil.getClient();
Request brokerRequest = CloudEventUtil.buildCloudEventRequest(record,broker);
logger.info("发送---");
try (Response response = client.newCall(brokerRequest).execute()){
} catch (IOException e) {
logger.error("发送失败:{}",e.getMessage());
}
}
}
public class HttpUtil {
private static OkHttpClient client;
/**
* 单线程操作不需要DCL
* @return OkHttpClient
*/
public static OkHttpClient getClient() {
if (client == null){
client = new OkHttpClient.Builder().build();
}
return client;
}
}
编写监听代码
public class MultiTopicListener implements Runnable{
Logger logger = LoggerFactory.getLogger(MultiTopicListener.class);
static Gson gson = new GsonBuilder().create();
@Override
public void run(){
String broker = KafkaUtil.getKnativeConfig().getBroker();
List<KafkaConfig> kafkaConfigs = KafkaUtil.getMultiKafkaConfig();
List<KafkaConsumer> consumers = kafkaConfigs.stream().map(KafkaUtil::createConsumer).collect(Collectors.toList());
while (true){
consumers.forEach(consumer -> {
logger.info("开始监听");
ConsumerRecords<String,String> record = consumer.poll(Duration.ofMillis(1000));
record.forEach(r -> {
CloudEventUtil.sendMsgToBroker(r,broker);
});
});
}
}
public static void main(String[] args) {
Thread thread = new Thread(new MultiTopicListener());
thread.start();
}
}
打包以POD方式运行在K8S上
这里使用Docker进行镜像制作,通过dockerhub管理镜像,没有的同学自己装下。Dockerfile
FROM openjdk:8-jdk-alpine
#容器中创建目录
RUN mkdir -p /home/user/kafka-cloudevent-test
#编译后的jar包copy到容器中创建到目录内
COPY kafka-cloudevent-test-1.0-SNAPSHOT-jar-with-dependencies.jar /home/user/kafka-cloudevent-test/kafka-cloudevent-test-1.0-SNAPSHOT-jar-with-dependencies.jar
#指定容器启动时要执行的命令
ENTRYPOINT ["java","-jar","/home/user/kafka-cloudevent-test/kafka-cloudevent-test-1.0-SNAPSHOT-jar-with-dependencies.jar"]
再把maven pkg
做好的jar放到服务器上。使用下面命令制作镜像并上传到dockerhubdocker build -f Dockerfile -t kafka-cloudevents-test:1.0 .
docker push kafka-cloudevents-test
编写deployment和service
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-cloudevent-test-deploy
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: kafka-cloudevent-test
release: stabel
template:
metadata:
labels:
app: kafka-cloudevent-test
release: stabel
env: test
spec:
containers:
- name: kafka-cloudevent-test
image: docker.io/kafka-cloudevents-test:1.0
imagePullPolicy: Always
ports:
- name: http
containerPort: 80
env:
- name: broker
value: http://broker-ingress.knative-eventing.svc.cluster.local/default/kafka-broker
- name: kafka.configs
value: "[\"localhost:9092#raw\",\"localhost:9092#test\"]"
apiVersion: v1
kind: Service
metadata:
name: kafka-cloudevent-test
namespace: default
spec:
type: NodePort
selector:
app: kafka-cloudevent-test
release: stabel
ports:
- name: http
port: 8080
targetPort: 8080
nodePort: 30080
ref:
apiVersion: eventing.knative.dev/v1beta1
kind: Broker
name: kafka-source-broker
使用kubectl apply -f deployment
和kubectl apply -f service.yaml
运行
到这里就结束了,可以测试一波。
KafkaSource
kafkaSource的deployment.yaml如下
apiVersion: sources.knative.dev/v1alpha1
kind: KafkaSource
metadata:
name: kafka-source
spec:
consumerGroup: knative-group
bootstrapServers: ["localhost:9092","localhost:9092"]
topics: ["raw","test"]
sink:
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: event-display
kubectl apply -f deployment
执行,这个sink
标签就是指EventSource向具体哪个应用发数据。这里采用了默认的event-display进行数据展示。
参考资料及命令
https://knative.dev/v0.15-docs/eventing/samples/kafka/source/
kubectl get po
获取podkubectl get deployment
获取deploymentkubectl describe pod xxxxx
获取xxxxx pod的状态kubectl delete deployment xxxxx
删除xxxxx的部署,这时对应的pod也会被删除kubectl delete -f deployment.yaml
删除该deployment对应的部署。