hadoop 容灾 hdfs灾备

转载

云端小悟空 2023-10-17 20:10:31

文章标签 hadoop 容灾 java hdfs JAVA 文章分类 Hadoop 大数据

利用HDFS实现ElasticSearch7.2容灾方案

利用HDFS实现ElasticSearch7.2容灾方案
前言
快照版本兼容
备份集群

HDFS文件系统

软件下载
JDK环境
配置系统环境变量
hadoop配置

配置JAVA_HOME
配置核心组件文件
配置文件系统
配置mapred
配置 yarn-site.xml

格式化文件系统
启动hdfs
访问

ES插件安装

插件下载
插件安装

创建仓库
创建快照
恢复快照

备份恢复时间

案例快照详情
案例快照恢复详情

常见问题

启动hdfs

问题1
问题2

创建仓库

问题1

参考文档

前言

Elasticsearch 副本提供了高可靠性，它们让你可以容忍零星的节点丢失而不会中断服务。但是，副本并不提供对灾难性故障的保护。对这种情况，就需要的是对集群真正的备份（在某些东西确实出问题的时候有一个完整的拷贝）。

案例模拟ElasticSearch7.2集群环境，采用snapshot API基于快照的方式备份集群。

案例演示HDFS分布式文件系统作为仓库举例。

快照版本兼容

备份集群

HDFS文件系统

软件下载

下载地址

hadoop-3.3.0.tar.gz

JDK环境

hadoop java编写，运行需依赖jvm

jdk-8u161-linux-x64.tar.gz

配置系统环境变量

#JAVA
export JAVA_HOME=/home/hadoop/jdk1.8.0_161
export CLASSPATH=$JAVA_HOME/libdt.jar:$JAVA_HOME/tools.jar
#hadoop
export HADOOP_HOME=/home/hadoop/hadoop-3.3.0
export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

hadoop配置

hadoop-3.3.0/etc/hadoop 的目录下

配置JAVA_HOME

hadoop-env.sh

export JAVA_HOME=/home/hadoop/jdk1.8.0_161

配置核心组件文件

core-site.xml需要在和之间添加

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://172.16.176.103:9000</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/data</value>
</property>

配置文件系统

hdfs-site.xml需要在和之间添加

<!--namenode-->
<property>
        <name>dfs.namenode.name.dir</name>
        <value>/data/namenode</value>
</property>
<!--datanode-->
<property>
        <name>dfs.datanode.data.dir</name>
        <value>/data/datanode</value>
</property>
<!--副本数,默认1-->
<property>
   		<name>dfs.replication</name>
        <value>1</value>
</property>
<!--禁用权限检查,配合es-->
<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>

配置mapred

mapred-site.xml

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

配置 yarn-site.xml

yarn-site.xml

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>elasticsearch01</value>
</property>

格式化文件系统

hdfs namenode -format

启动hdfs

start-dfs.sh

$ start-dfs.sh 
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [host103]
Starting datanodes
Starting secondary namenodes [host103]

访问

http://localhost:9870/

ES插件安装

集群中每个节点都必须安装hdfs插件，安装后需重启ES

插件下载

插件版本和ES版本相对应

下载地址

repository-hdfs-7.2.0.zip

插件安装

提前下载软件包，离线安装

集群中各节点依次安装

sudo bin/elasticsearch-plugin install file:///path/to/plugin.zip

$ ./elasticsearch-plugin install file:///home/es/repository-hdfs-7.2.0.zip 
-> Downloading file:///home/es/repository-hdfs-7.2.0.zip
[=================================================] 100%   
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.sun.security.krb5
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission loadLibrary.jaas
* java.lang.RuntimePermission loadLibrary.jaas_nt
* java.lang.RuntimePermission loadLibrary.jaas_unix
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission shutdownHooks
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
* java.net.SocketPermission localhost:0 listen,resolve
* java.security.SecurityPermission insertProvider.SaslPlainServer
* java.security.SecurityPermission putProviderProperty.SaslPlainServer
* java.util.PropertyPermission * read,write
* javax.security.auth.AuthPermission doAs
* javax.security.auth.AuthPermission getSubject
* javax.security.auth.AuthPermission modifyPrincipals
* javax.security.auth.AuthPermission modifyPrivateCredentials
* javax.security.auth.AuthPermission modifyPublicCredentials
* javax.security.auth.PrivateCredentialPermission javax.security.auth.kerberos.KerberosTicket * "*" read
* javax.security.auth.PrivateCredentialPermission javax.security.auth.kerberos.KeyTab * "*" read
* javax.security.auth.PrivateCredentialPermission org.apache.hadoop.security.Credentials * "*" read
* javax.security.auth.kerberos.ServicePermission * initiate
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed repository-hdfs
$

创建仓库

创建

PUT _snapshot/my_hdfs_repository
{
  "type": "hdfs",	--类型
  "settings": {
    "uri": "hdfs://172.16.176.103:9000/",	--hdfs访问url
    "path": "/data",
    "conf.dfs.client.read.shortcircuit": "false"
  }
}

查看

GET /_snapshot
{
  "my_hdfs_repository" : {
    "type" : "hdfs",
    "settings" : {
      "path" : "/data",
      "uri" : "hdfs://172.16.176.103:9000/",
      "conf" : {
        "dfs" : {
          "client" : {
            "read" : {
              "shortcircuit" : "false"
            }
          }
        }
      }
    }
  }
}

创建快照

创建快照

不等待快照完成，即刻返回结果

PUT _snapshot/my_hdfs_repository/snapshot_i_xfjbblxt_cxfw_xfj_d12
{
    "indices": "i_xfjbblxt_cxfw_xfj_d12"
}

查看快照当前状态

GET _snapshot/my_hdfs_repository/snapshot_i_xfjbblxt_cxfw_xfj_d12
{
  "snapshots" : [
    {
      "snapshot" : "snapshot_i_xfjbblxt_cxfw_xfj_d12",
      "uuid" : "-BS9XjxvS1Sp6wW_bT02lA",
      "version_id" : 7020099,
      "version" : "7.2.0",
      "indices" : [
        "i_xfjbblxt_cxfw_xfj_d12"
      ],
      "include_global_state" : true,
      "state" : "IN_PROGRESS",	--正在做快照中
      "start_time" : "2020-10-12T14:04:49.425Z",	--开始时间
      "start_time_in_millis" : 1602511489425,
      "end_time" : "1970-01-01T00:00:00.000Z",
      "end_time_in_millis" : 0,
      "duration_in_millis" : -1602511489425,
      "failures" : [ ],
      "shards" : {
        "total" : 0,
        "failed" : 0,
        "successful" : 0
      }
    }
  ]
}

完成状态

{
  "snapshots" : [
    {
      "snapshot" : "snapshot_i_xfjbblxt_cxfw_xfj_d12",	--快照名称
      "uuid" : "-BS9XjxvS1Sp6wW_bT02lA",
      "version_id" : 7020099,
      "version" : "7.2.0",
      "indices" : [
        "i_xfjbblxt_cxfw_xfj_d12"	--索引
      ],
      "include_global_state" : true,
      "state" : "SUCCESS",	--快照成功
      "start_time" : "2020-10-12T14:04:49.425Z",	--开始时间
      "start_time_in_millis" : 1602511489425,	--开始时间戳
      "end_time" : "2020-10-12T14:24:33.942Z",	--结束时间
      "end_time_in_millis" : 1602512673942,	--结束时间戳
      "duration_in_millis" : 1184517,	--耗时（毫秒）
      "failures" : [ ],
      "shards" : {
        "total" : 5,	--总分片
        "failed" : 0,
        "successful" : 5	--成功分片
      }
    }
  ]
}

恢复快照

快照恢复如果恢复到原索引中，需要先把原索引关闭或者先删除后，在进行快照恢复

恢复快照

POST _snapshot/my_hdfs_repository/snapshot_i_xfjbblxt_cxfw_xfj_d12/_restore
{
  "indices": "i_xfjbblxt_cxfw_xfj_d12"	--快照备份索引名称
  ,"rename_pattern": "i_xfjbblxt_cxfw_xfj_d12"	--检索匹配到的索引名称
  , "rename_replacement": "restored_i_xfjbblxt_cxfw_xfj_d12"	--重命名索引
}

状态查看

{
  "restored_i_xfjbblxt_cxfw_xfj_d12" : {
    "shards" : [
      {
        "id" : 4,
        "type" : "SNAPSHOT",
        "stage" : "INDEX",
        "primary" : true,
        "start_time_in_millis" : 1602571287856,
        "total_time_in_millis" : 1249147,
        "source" : {
          "repository" : "my_hdfs_repository",
          "snapshot" : "snapshot_i_xfjbblxt_cxfw_xfj_d12",
          "version" : "7.2.0",
          "index" : "i_xfjbblxt_cxfw_xfj_d12",
          "restoreUUID" : "KM1EaKsAQkO4OxB0PwKe0Q"
        },
        "target" : {
          "id" : "DWvUrfqQRxGLIWm6SQmunA",
          "host" : "172.16.176.104",
          "transport_address" : "172.16.176.104:9300",
          "ip" : "172.16.176.104",
          "name" : "node-104"
        },
        "index" : {
          "size" : {
            "total_in_bytes" : 8312825377,
            "reused_in_bytes" : 0,
            "recovered_in_bytes" : 6781859331,
            "percent" : "81.6%"
          },
          "files" : {
            "total" : 104,
            "reused" : 0,
            "recovered" : 86,
            "percent" : "82.7%"
          },
          "total_time_in_millis" : 1249039,
          "source_throttle_time_in_millis" : 0,
          "target_throttle_time_in_millis" : 0
        },
        "translog" : {
          "recovered" : 0,
          "total" : 0,
          "percent" : "100.0%",
          "total_on_start" : 0,
          "total_time_in_millis" : 0
        },
        "verify_index" : {
          "check_index_time_in_millis" : 0,
          "total_time_in_millis" : 0
        }
      },
      --部分省略

备份恢复时间

案例快照详情

第一次快照

节点数	主分片	副本分配	数据量	大小	快照大小	耗时(快照)
3	5	1	5149535	77.4gb	40gb	19.74195分钟

案例快照恢复详情

快照恢复过程为并行恢复

分片	耗时（恢复）	恢复字节
0（主）	27.42分钟	7.75G
1（主）	27.14分钟	7.72G
2（主）	27.45分钟	7.75G
3（主）	25.89分钟	7.74G
4（主）	25.5分钟	7.74G
0（副）	18.65分钟	7.75G
1（副）	10.3分钟	7.72G
2（副）	17.21分钟	7.75G
3（副）	10.6分钟	7.74G
4（副）	18.32分钟	7.74G

常见问题

启动hdfs

问题1

$ start-dfs.sh 
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [host103]
Last login: Sun Oct 11 22:32:11 CST 2020 from 172.16.176.46 on pts/1
host103: ERROR: JAVA_HOME is not set and could not be found.
Starting datanodes
Last login: Sun Oct 11 22:32:23 CST 2020 on pts/1
localhost: ERROR: JAVA_HOME is not set and could not be found.
Starting secondary namenodes [host103]
Last login: Sun Oct 11 22:32:24 CST 2020 on pts/1
host103: ERROR: JAVA_HOME is not set and could not be found.

解决

配置java环境变量

export JAVA_HOME=/home/hadoop/jdk1.8.0_161
export CLASSPATH=$JAVA_HOME/libdt.jar:$JAVA_HOME/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

问题2

$ start-dfs.sh 
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [host103]
host103: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting datanodes
localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting secondary namenodes [host103]
host103: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

解决

hadoop用户执行

[hadoop@host103 ~]$ ssh-copy-id hadoop@host103
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@host103's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'hadoop@host103'"
and check to make sure that only the key(s) you wanted were added.

创建仓库

问题1

创建

PUT _snapshot/my_hdfs_repository
{
  "type": "hdfs",
  "settings": {
    "uri": "hdfs://172.16.176.103:9000/",
    "path": "/",
    "conf.dfs.client.read.shortcircuit": "false"
  }
}

错误

error": {
    "root_cause": [
      {
        "type": "repository_exception",
        "reason": "[my_hdfs_repository] cannot create blob store"
      }
    ],
    "type": "repository_exception",
    "reason": "[my_hdfs_repository] cannot create blob store",
    "caused_by": {
      "type": "unchecked_i_o_exception",
      "reason": "Cannot create HDFS repository for uri [hdfs://172.16.176.103:9000/]",
      "caused_by": {
        "type": "access_control_exception",
        "reason": "Permission denied: user=es, access=WRITE, inode=\"/\":hadoop:supergroup:drwxr-xr-x\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:496)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:336)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermissionWithContext(FSPermissionChecker.java:360)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:239)\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1909)\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1893)\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1852)\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:60)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3407)\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1161)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:739)\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)\n\tat org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)\n\tat org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)\n",

问题解决

新增hdfs-site.xml

<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>

参考文档

HDFS插件

https://www.elastic.co/guide/en/elasticsearch/plugins/7.2/repository-hdfs.html

HDFS SingleCluster

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。

上一篇：桌面运维部门组织架构桌面运维必备技能

下一篇：Nehalem微架构微内核架构

提问和评论都可以，用心的回复会被更多人看到评论

发布评论

相关文章

官方博客	全部文章	热门标签	班级博客
了解我们	网站地图	意见反馈

鸿蒙开发者社区	51CTO学堂
51CTO	软考资讯