CDH spark集成kerberos

文章目录

• 相关版本信息：
• 一、安装kerberos服务

• 1、yum安装
• 2、配置kerberos

• 2.1 修改/etc/krb5.conf配置
• 2.2 修改/var/kerberos/krb5kdc/kadm5.acl
• 2.3 修改/var/kerberos/krb5kdc/kdc.conf配置

• 3、创建Kerberos数据库
• 4、创建Kerberos的管理账号
• 5、启动服务并自启
• 6、测试kerberos

• 二、为CDH集群启用kerberos

• 1、安装额外包
• 2、在KDC中给Cloudera Manager添加管理员账号
• 3、 进入Cloudera Manager的“管理”->“安全”界面，进行配置

• 3.1进入安全界面
• 3.2 点击启用kerberos
• 3.3 全部勾选, 点击“继续”
• 3.4 修改KDC配置, 点击“继续”
• 3.5 不建议让Cloudera Manager来管理krb5.conf, 点击“继续”
• 3.6 输入Cloudera Manager的Kerbers管理员账号，一定得和之前创建的账号一致，点击“继续”
• 集群kerberos开启完成

• 三、测试使用kerberos

• 1、创建测试账户，并登录
• 2、测试beeline登录
• 3、对hive进行插入数据

相关版本信息：

CDH版本：6.2.0(单节点)
Linux版本：CentOS7.6
操作用户：root

一、安装kerberos服务

1、yum安装

yum -y install krb5-server krb5-libs krb5-workstation

2、配置kerberos

2.1 修改/etc/krb5.conf配置

#Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
 default_realm = LIUCZ.COM //需要修改
#default_ccache_name = KEYRING:persistent:%{uid}

[realms] //需要修改
 LIUCZ.COM = {
  kdc = l01.liucz.com
  admin_server = l01.liucz.com
 }

[domain_realm] //需要修改
.l01.liucz.com = LIUCZ.COM
 l01.liucz.com = LIUCZ.COM

2.2 修改/var/kerberos/krb5kdc/kadm5.acl

*/[e-mail]       *

2.3 修改/var/kerberos/krb5kdc/kdc.conf配置

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 LIUCZ.COM = { //需要修改
  #master_key_type = aes256-cts
  max_renewable_life= 7d 0h 0m 0s //需要修改
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }

3、创建Kerberos数据库

kdb5_util create -r LIUCZ.COM -s

4、创建Kerberos的管理账号

[root@l01 ~]# kadmin.local 
Authenticating as principal root/[e-mail] with password.
kadmin.local:  addprinc admin/[e-mail]
WARNING: no policy specified for admin/[e-mail]; defaulting to no policy
Enter password for principal "admin/[e-mail]": 
Re-enter password for principal "admin/[e-mail]": 
Principal "admin/[e-mail]" created.

5、启动服务并自启

[root@l01 ~]# systemctl start krb5kdc.service kadmin.service
[root@l01 ~]# systemctl enable krb5kdc.service kadmin.service

6、测试kerberos

[root@l01 ~]# kinit admin/[e-mail]
Password for admin/[e-mail]: 
[root@l01 ~]# klist 
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin/[e-mail]

Valid starting       Expires              Service principal
02/26/2020 03:24:09  02/27/2020 03:24:09  krbtgt/[e-mail]
	renew until 03/04/2020 03:24:09

二、为CDH集群启用kerberos

1、安装额外包

[root@l01 ~]# yum install -y openldap-clients.x86_64

2、在KDC中给Cloudera Manager添加管理员账号

[root@l01 ~]# kadmin.local 
Authenticating as principal admin/[e-mail] with password.
kadmin.local:  addprinc scm/[e-mail]
WARNING: no policy specified for scm/[e-mail]; defaulting to no policy
Enter password for principal "scm/[e-mail]": 
Re-enter password for principal "scm/[e-mail]": 
Principal "scm/[e-mail]" created.
kadmin.local:  q

3、 进入Cloudera Manager的“管理”->“安全”界面，进行配置

3.1进入安全界面

3.2 点击启用kerberos

3.3 全部勾选, 点击“继续”

3.4 修改KDC配置, 点击“继续”

3.5 不建议让Cloudera Manager来管理krb5.conf, 点击“继续”

3.6 输入Cloudera Manager的Kerbers管理员账号，一定得和之前创建的账号一致，点击“继续”

集群kerberos开启完成

三、测试使用kerberos

1、创建测试账户，并登录

[root@l01 ~]# kadmin.local 
Authenticating as principal admin/[e-mail] with password.
kadmin.local:  addprinc [e-mail]
WARNING: no policy specified for [e-mail]; defaulting to no policy
Enter password for principal "[e-mail]": 
Re-enter password for principal "[e-mail]": 
Principal "[e-mail]" created.
kadmin.local:  q
[root@l01 ~]# kdestroy 
[root@l01 ~]# kinit [e-mail]
Password for [e-mail]: 
[root@l01 ~]# klist 
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: [e-mail]

Valid starting       Expires              Service principal
02/26/2020 04:01:26  02/27/2020 04:01:26  krbtgt/[e-mail]
	renew until 03/04/2020 04:01:26

2、测试beeline登录

[root@l01 ~]# beeline 
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 2.1.1-cdh6.2.0 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000/;principal=hive/[e-mail]
Connecting to jdbc:hive2://localhost:10000/;principal=hive/[e-mail]
Connected to: Apache Hive (version 2.1.1-cdh6.2.0)
Driver: Hive JDBC (version 2.1.1-cdh6.2.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:10000/>

3、对hive进行插入数据

测试中如果出现目录权限问题，可以通过hdfs相关命令对相应目录授权：

这个指令需要以hdfs身份执行

我们需要先用kadmin.loca创建一个hdfs账号，kinit该账号，再进行hdfs目录权限的修改

其中setfacl需要

hdfs dfs -setfacl -m user:hive:rwx /user

修改完成后kinit之前创建的账户[e-mail]

即可对hive表进行查询，插入操作；

因为电脑配置较低，采用hive的本地执行模式进行数据的插入；