kafka 配置 kerberos认证 kafka权限认证acl

Kafka SASL/SCRAM+ACL实现动态创建用户及权限控制**

使用SASL / SCRAM进行身份验证

请先在不配置任何身份验证的情况下启动Kafka

1. 创建SCRAM Credentials

Kafka中的SCRAM实现使用Zookeeper作为凭证(credential)存储。 可以使用kafka-configs.sh在Zookeeper中创建凭据。 对于启用的每个SCRAM机制，必须通过添加具有机制名称的配置来创建凭证。 必须在启动Kafka broker之前创建代理间通信的凭据。 可以动态创建和更新客户端凭证，并使用更新的凭证来验证新连接。

1.创建broker建通信用户(或称超级用户)

bin/kafka-configs.sh --zookeeper 10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181 --alter --add-config 'SCRAM-SHA-256=[password=admin-sec],SCRAM-SHA-512=[password=admin-sec]' --entity-type users --entity-name admin

2.创建客户端用户producer

bin/kafka-configs.sh --zookeeper 10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181 --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=prod-sec],SCRAM-SHA-512=[password=prod-sec]' --entity-type users --entity-name producer

3.创建客户端用户consumer

bin/kafka-configs.sh --zookeeper 10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181 --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=prod-sec],SCRAM-SHA-512=[password=cons-sec]' --entity-type users --entity-name consumer

4.查看SCRAM证书

[root@node002229 kafka]# bin/kafka-configs.sh --zookeeper 10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181 --describe --entity-type users --entity-name producer

Configs for user-principal 'fanboshi' are SCRAM-SHA-512=salt=MWwwdWJqcjBncmUwdzY1Mzdoa2NwNXppd3A=,stored_key=mGCJy5k3LrE2gs6Dp4ALRhgy37l1WYPUIdoOncCF+B3Ti3wL2sQNmzg8oEz3tUs9DFsclFCygjbysb0S0BU9bA==,server_key=iTyX0U0Jt02dkddUm6QrVwNf3lJk72dBNs9EDHTqe8kLlNGIp9ypzRkcgkc+WVMd1bkAF3cg8vk9Q1LrJ/2i/A==,iterations=4096,SCRAM-SHA-256=salt=ZDg5MHVlYW40dW9jbXJ6MndvZDVlazd3ag==,stored_key=cgX1ldpXnDL1+TlLHJ3IHn7tAQS/7pQ7BVZUtECpQ3A=,server_key=i7Mcnb5sPUqfIFs6qKWWHZ2ortoKiRc7oabHOV5dawI=,iterations=8192

5.删除SCRAM证书

[root@node002229 kafka]# bin/kafka-configs.sh --zookeeper 10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181 --alter  --delete-config 'SCRAM-SHA-512' --entity-type users --entity-name producer

2. 配置Kafka Brokers

1.在每个Kafka broker的config目录中添加一个类似下面的JAAS文件，我们称之为kafka_server_jaas.conf：

[root@node002229 config]# cat kafka_server_jaas.conf
KafkaServer {
  org.apache.kafka.common.security.scram.ScramLoginModule required
  username="admin"
  password="admin-secret";
};

注：不要少写了分号

2.将JAAS配置文件位置作为JVM参数传递给每个Kafka broker：
修改 /usr/local/kafka/bin/kafka-server-start.sh
将exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@" 注释, 增加下面的内容

#exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS -Djava.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf kafka.Kafka "$@"

或者不修改kafka-server-start.sh脚本, 而是将下面的内容添加到~/.bashrc

export KAFKA_PLAIN_PARAMS="-Djava.security.auth.login.config=/usr/local/kafka/config/kafka_server_jaas.conf"
export KAFKA_OPTS="$KAFKA_PLAIN_PARAMS $KAFKA_OPTS"

注：未尝试过，有兴趣的老铁可以试试

3.如此处所述，在server.properties中配置SASL端口和SASL机制。 例如：

# 认证配置
hostname=ip地址
listeners=SASL_PLAINTEXT://ip地址:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
sasl.enabled.mechanisms=SCRAM-SHA-256

# ACL配置
allow.everyone.if.no.acl.found=true
super.users=User:admin
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

在官方文档中写的是

listeners=SASL_SSL://host.name:port
security.inter.broker.protocol=SASL_SSL

我们可以根据自己的需求选择SSL或PLAINTEXT, 我这里选择PLAINTEXT不加密明文传输, 省事, 性能也相对好一些。

3.重启ZK/Kafka

到这一步整个kafka集群已经启动了SASL/SCRAM身份认证及权限控制功能。

4.客户端配置

1.修改 config/consumer.properties文件，增加

[root@node002229 kafka]# vim config/consumer.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256

2.修改 config/producer.properties文件，增加

[root@node002229 kafka]# vim config/producer.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256

3.创建config/kafka_client_producer_jaas.conf文件用于设置生产者客户端的用户名及密码

[root@node002229 kafka]# vim config/kafka_client_producer_jaas.conf
KafkaClient {
  org.apache.kafka.common.security.scram.ScramLoginModule required
    username="producer"
    password="prod-sec"
};

4.修改kafka-console-producer.sh脚本

vim bin/kafka-console-producer.sh
#exec $(dirname $0)/kafka-run-class.sh kafka.tools.ConsoleProducer "$@"
exec $(dirname $0)/kafka-run-class.sh -Djava.security.auth.login.config=$(dirname $0)/../config/kafka_client_producer_jaas.conf kafka.tools.ConsoleProducer "$@"

注：文件路径也可使用绝对路径

5.创建config/kafka_client_consumer_jaas.conf文件用于设置消费者客户端的用户名及密码

[root@node002229 kafka]# vim config/kafka_client_consumer_jaas.conf
KafkaClient {
  org.apache.kafka.common.security.scram.ScramLoginModule required
    username="consumer"
    password="cons-sec"
};

6.修改kafka-console-consumer.sh脚本

vim bin/kafka-console-consumer.sh
#exec $(dirname $0)/kafka-run-class.sh kafka.tools.ConsoleProducer "$@"
exec $(dirname $0)/kafka-run-class.sh -Djava.security.auth.login.config=$(dirname $0)/../config/kafka_client_consumer_jaas.conf kafka.tools.ConsoleProducer "$@"

注：文件路径也可使用绝对路径

7.创建测试topic

bin/kafka-topics.sh -create --zookeeper 10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181 -replication-factor 3 --partitions 3 --topic test

8.测试生产消息

bin/kafka-console-producer.sh --bootstrap-server 10.2.36.42:9092,10.2.36.149:9092,10.2.36.168:9092 --topic test --producer.config config/producer.properties

注：需给用户授权才可进行发布消息

9.测试消费消息

bin/kafka-console-consumer.sh --bootstrap-server 10.2.36.42:9092,10.2.36.149:9092,10.2.36.168:9092 --topic test --from-beginning --consumer.config config/consumer.properties

注：需给用户授权才可消费消息

ACL配置
1.授予producer用户对test topic 写权限, 只允许 192.168.2.* 网段

bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181 --add --allow-principal User:producer --operation Write --topic test --allow-host 192.168.2.*

注：网段限制可以不加

授予consumer用户对test topic 读权限, 只允许 192.168.2.* 网段

bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181 --add --allow-principal User:consumer--operation Read --topic test --allow-host 192.168.2.*

授予consumer用户, consumer-group 消费者组 对test topic 读权限, 只允许 192.168.2.* 网段

bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181 --add --allow-principal User:consumer--operation Read --group consumer-group --allow-host 192.168.2.*

查看acl配置

[root@node002229 kafka]# bin/kafka-acls.sh --list --authorizer-properties zookeeper.connect=10.2.36.42:2181,10.2.36.149:2181,10.2.36.168:2181
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=test1, patternType=LITERAL)`: 
 	(principal=User:producer, host=*, operation=WRITE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=consumer-group, patternType=LITERAL)`: 
 	(principal=User:consumer, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:abcconsumer, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=10000082, patternType=LITERAL)`: 
 	(principal=User:producer, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:abcconsumer, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=test, patternType=LITERAL)`: 
 	(principal=User:producer, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:consumer, host=*, operation=READ, permissionType=ALLOW)

删除配置

bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.c