cdh 制作镜像制作cdr镜像

转载

mob64ca140651e5 2024-07-20 08:35:14

文章标签 cdh 制作镜像 hadoop cloudera hive 大数据 文章分类 云原生云计算

前言

根据CDH安装流程，将分别安装cloudera-manager-server节点和 cloudera-manager-agent节点，两者安装步骤和安装包不一样，也是考虑到后续方便扩展节点，所以单独将这两个节点安装到不同的容器内

接下来先安装cloudera-manager-server节点的内容

1. 创建并运行基础容器

创建运行容器参数说明：

参数	说明
-d	后台运行容器
-name	容器名
-hostname	容器主机名
--privileged=true	使容器具有root权限
server/bashimage	镜像库名:标签(也可直接填IMAGE ID)，指定该容器使用的哪个镜像

启动基础容器，在此基础上安装master-server所需的软件

docker run -d \
--name server001 \
--hostname server001 \
--privileged=true \
server/bashimage \
/usr/sbin/init  \
&& docker ps

结果：
8ac7901ae94ccc85cbf9768cec10adc4f9a4e2e9034cd9cf60d3ef2367fe1a11
CONTAINER ID   IMAGE              COMMAND            CREATED                  STATUS                  PORTS     NAMES
8ac7901ae94c   server/bashimage   "/usr/sbin/init"   Less than a second ago   Up Less than a second   22/tcp    server001

2. 初始化容器环境

把容器当作一台全新的物理机器，还是需要重新配置基本环境

# 进入容器
docker exec -it --privileged=true server001 /bin/bash 

# 初始化容器环境
ping www.baidu.com -c 3 \
&& yum update -y \
&& yum install -y kde-l10n-Chinese telnet reinstall glibc-common wget ntp net-tools firewalld unzip \
&& yum clean all 

结果：
PING www.a.shifen.com (180.101.49.11) 56(84) bytes of data.
64 bytes from 180.101.49.11 (180.101.49.11): icmp_seq=1 ttl=127 time=54.7 ms
64 bytes from 180.101.49.11 (180.101.49.11): icmp_seq=2 ttl=127 time=65.2 ms
64 bytes from 180.101.49.11 (180.101.49.11): icmp_seq=3 ttl=127 time=87.8 ms...
Complete!
Loaded plugins: fastestmirror, ovl
Cleaning repos: base extras updates
Cleaning up list of fastest mirrors

3. 关闭防火墙

systemctl stop firewalld \
&& systemctl disable firewalld \
&& systemctl status firewalld

结果:
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

4. 配置中文环境

(cat <<EOF
export LC_ALL=zh_CN.utf8
export LANG=zh_CN.utf8
export LANGUAGE=zh_CN.utf8
EOF
) >> ~/.bashrc \
&& localedef -c -f UTF-8 -i zh_CN zh_CN.utf8 \
&& source ~/.bashrc \
&& echo $LANG

结果：
zh_CN.utf8

5. 配置httpd ，新增yum源

在上一篇博客中，已经在宿主机上搭建了yum本地软件库，master-server容器内需要安装httpd服务，并指定宿主机的yum源地址，就能够从宿主机上下载所需软件程序。

添加yum源参数说明：

name	yum仓库的名字
baseurl	仓库路径用的httpd的本地文件库，宿主机IP：172.16.0.2
gpgcheck	为0时不做gpg检查
enabled	为1时开启此仓库

yum -y install httpd \
&& systemctl start httpd \
&&(
cat <<EOF
[cloudera-manager]
name=Cloudera Manager 6.3.1
baseurl=http://172.16.0.2/cloudera-repos/
gpgcheck=0
enabled=1
EOF
) >> /etc/yum.repos.d/cloudera-manager.repo \
&& curl http://172.16.0.2/cloudera-repos/ \
&& yum clean all

结果：
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /cloudera-repos</title>
 </head>
 <body>
<h1>Index of /cloudera-repos</h1>
  <table>
   <tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th><th><a href="?C=D;O=A">Description</a></th></tr>
   <tr><th colspan="5"><hr></th></tr>
<tr><td valign="top"><img src="/icons/back.gif" alt="[PARENTDIR]"></td><td><a href="/">Parent Directory</a>       </td><td> </td><td align="right">  - </td><td> </td></tr>
<tr><td valign="top"><img src="/icons/text.gif" alt="[TXT]"></td><td><a href="allkeys.asc">allkeys.asc</a>            </td><td align="right">2021-07-05 00:34  </td><td align="right"> 14K</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm">cloudera-manager-age..></a></td><td align="right">2021-06-15 16:00  </td><td align="right"> 10M</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm">cloudera-manager-dae..></a></td><td align="right">2021-06-15 16:10  </td><td align="right">1.1G</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm">cloudera-manager-ser..></a></td><td align="right">2021-06-15 15:59  </td><td align="right"> 11K</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm">cloudera-manager-ser..></a></td><td align="right">2021-06-15 15:59  </td><td align="right"> 11K</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm">enterprise-debuginfo..></a></td><td align="right">2021-06-15 16:00  </td><td align="right"> 14M</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="mysql-community-client-5.7.27-1.el7.x86_64.rpm">mysql-community-clie..></a></td><td align="right">2021-07-08 17:41  </td><td align="right"> 24M</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="mysql-community-common-5.7.27-1.el7.x86_64.rpm">mysql-community-comm..></a></td><td align="right">2021-07-08 17:42  </td><td align="right">275K</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="mysql-community-devel-5.7.27-1.el7.x86_64.rpm">mysql-community-deve..></a></td><td align="right">2021-07-08 17:42  </td><td align="right">3.7M</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="mysql-community-libs-5.7.27-1.el7.x86_64.rpm">mysql-community-libs..></a></td><td align="right">2021-07-08 17:42  </td><td align="right">2.2M</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm">mysql-community-libs..></a></td><td align="right">2021-07-08 17:42  </td><td align="right">2.0M</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="mysql-community-server-5.7.27-1.el7.x86_64.rpm">mysql-community-serv..></a></td><td align="right">2021-07-08 17:43  </td><td align="right">165M</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm">oracle-j2sdk1.8-1.8...></a></td><td align="right">2021-06-15 16:05  </td><td align="right">176M</td><td> </td></tr>
<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="repodata/">repodata/</a>              </td><td align="right">2021-07-11 17:42  </td><td align="right">  - </td><td> </td></tr>
   <tr><th colspan="5"><hr></th></tr>
</table>
</body></html>

6. 安装JDK，daemons ，agent ，server

通过上述配置的yum源，安装JDK，CDH的daemons ，agent ，server服务。daemons包大小过大，第一次安装会失败，请在尝试一次

安装完JDK，并配置环境变量

yum install -y oracle-j2sdk1.8 cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server \
&& rpm -qa | grep cloudera-manager \
&& (cat <<\EOF
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export PATH=$PATH:$JAVA_HOME/bin
EOF
) >> ~/.bashrc \
&& source ~/.bashrc \
&& java -version \
&& yum clean all
   
结果：
Complete!
cloudera-manager-agent-6.3.1-1466458.el7.x86_64
cloudera-manager-server-6.3.1-1466458.el7.x86_64
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
第一次可能没有上面的结果，请再执行一次

7. 安装mysql

CDH的正常运行需要安装MySQL，该MySQL服务安装在server容器内，当然也可以将MySQL容器化单独部署，就是在初始化CDH时稍有变动。

yum install -y mysql-community-common mysql-community-libs mysql-community-devel mysql-community-libs-compat mysql-community-client mysql-community-server \
&& echo character-set-server=utf8 >> /etc/my.cnf \
&& rpm -qa | grep mysql \
&& yum clean all

结果：
Installed:
  mysql-community-client.x86_64 0:5.7.27-1.el7 mysql-community-devel.x86_64 0:5.7.27-1.el7 mysql-community-server.x86_64 0:5.7.27-1.el7

Dependency Installed:
  libaio.x86_64 0:0.3.109-13.el7                                   numactl-libs.x86_64 0:2.0.12-5.el7                                  
Complete!
mysql-community-common-5.7.27-1.el7.x86_64
mysql-community-libs-compat-5.7.27-1.el7.x86_64
mysql-community-client-5.7.27-1.el7.x86_64
mysql-community-devel-5.7.27-1.el7.x86_64
mysql-community-libs-5.7.27-1.el7.x86_64
mysql-community-server-5.7.27-1.el7.x86_64

7.1 重置密码

修改mysql登录密码，创建CDH需要的数据库，并授予相关权限

1. #启动比较久，可以另外开一个终端，去查看密码
systemctl start mysqld 

2. 查看状态显示报错：这是正常现象
systemctl status mysqld
结果：
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: activating (start) since 五 2021-07-09 08:55:25 CST; 28s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 1091 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
  Process: 1020 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
   CGroup: /docker/b4bb69b9c8884b3cc33da208eac1180c269f82cc006622df1b35b05d179cc1c0/docker/b4bb69b9c8884b3cc33da208eac1180c269f82cc006622df1b35b05d179cc1c0/system.slice/mysqld.service
           └─1093 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

7月 09 08:55:25 asrserver001 systemd[1]: Starting MySQL Server...
7月 09 08:55:30 asrserver001 systemd[1]: New main PID 1093 does not belong to service, and PID file is not owned by root. Refusing.
7月 09 08:55:30 asrserver001 systemd[1]: New main PID 1093 does not belong to service, and PID file is not owned by root. Refusing.

3. 修改密码：
# mysqld下面添加skip-grant-tables 保存退出启动服务
vim /etc/my.cnf
    [mysqld]
    skip-grant-tables

4. 免密重新登录，修改密码
systemctl restart mysqld 
mysql -u root
    use mysql
    update user set authentication_string=password('123456') where user='root' and host='localhost';
    flush privileges;
    exit;
放开忽略密码配置，重启mysqld
systemctl restart mysqld.service

5. 输入密码登录mysql
mysql -u root -p 输入自定义的密码
在执行下面的命令：
# 设置mysql密码策略，可以不设置，直接设置复杂度高的密码
set global validate_password_policy=LOW;
set global validate_password_length=6; 
set password for root@localhost = password('123456');
grant all privileges on *.* to 'root'@'%' identified by '123456';
flush privileges;

# 创建CDH运行所需的数据库
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';
flush privileges;
SHOW DATABASES;

结果:
mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| amon               |
| hue                |
| metastore          |
| mysql              |
| nav                |
| navms              |
| oozie              |
| performance_schema |
| rman               |
| scm                |
| sentry             |
| sys                |
+--------------------+

直到上述步骤，master-server容器已经安装了JDK，CDH的daemos，server，agent服务和MySQL服务，只是安装好，并没有启动也无需启动，CDH也没有初始化，到这步已经足够

systemctl status cloudera-scm-server && systemctl status cloudera-scm-agent

结果：
● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
● cloudera-scm-agent.service - Cloudera Manager Agent Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
   Active: inactive (dead)

8. 将容器提交成镜像

接下来将master-server容器打包成镜像

1. 退出server001容器：
[root@server001 ~]# exit
exit

2. 停止运行容器,将容器提交为master-server/cdh:6.3.2镜像,并打包镜像
docker stop server001 \
    && docker commit server001  master-server/cdh:6.3.2 \
    && docker images \
    && docker save -o master-server.tar.gz master-server/cdh:6.3.2 \
    && ls -l --block-size=M
结果：
server001
sha256:d4f3e4ee3f9eee71915402d8c9c5eb3bafc4f4e1eca82f11dcdb0cf3cd074493
REPOSITORY          TAG              IMAGE ID       CREATED          SIZE
master-server/cdh   6.3.2            d4f3e4ee3f9e   1 second ago     3.62GB
server/bashimage    latest           c8d140a652e0   15 minutes ago   404MB
centos              centos7.7.1908   08d05d1d5859   22 months ago    204MB
-rw------- 1 root root 3402M Sep 12 17:10 master-server.tar.gz

有问题欢迎在评论区留言，你的提问就是对我最大的支持

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。