任务无法提交

查看Hadoop控制台 http://172.16.0.10:8088/cluster/scheduler ，发现内存只有50%使用率； image.png

排查

登录master节点服务，yarn.scheduler.capacity.maximum-am-resource-percent 配置为 0.5 ，表示core的资源使用率不能超过50%

[root@ads-flink-online-master ~]# cat /etc/hadoop/conf.empty/capacity-scheduler.xml

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run 
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

解决

将yarn.scheduler.capacity.maximum-am-resource-percent改为0.8

[root@ads-flink-online-master ~]# vim /etc/hadoop/conf.empty/capacity-scheduler.xml

  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.8</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

Hdfs ACL授权失败

报错信息： setfacl: The ACL operation has been rejected. Support for ACLs has been disabled by setting dfs.namenode.acls.enabled to false.

解决方法： 在hdfs-site.xml文件中添加如下配置即可：

<property>
    <name>dfs.namenode.acls.enabled</name>
    <value>true</value>
</property>

重启服务：

hadoop-hdfs-namenode.service

hdfs权限问题

Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hue_product, access=WRITE, inode="/user":hdfs:hdfsadmingroup:drwxr-xr-x

解决方法：

##检查目录权限
[root@emr001 conf]# sudo -u hdfs hadoop fs -ls /user
Found 9 items
drwxr-xr-x   - hadoop   hdfsadmingroup          0 2022-08-04 04:26 /user/hadoop
drwxr-xr-x   - mapred   mapred                  0 2022-08-04 04:26 /user/history
drwxr-xr-x   - hdfs     hdfsadmingroup          0 2022-08-04 08:59 /user/hive
drwxr-xr-x   - hue      hue                     0 2022-08-04 04:26 /user/hue
drwxr-xr-x   - livy     livy                    0 2022-08-04 07:48 /user/livy
drwxr-xr-x   - oozie    oozie                   0 2022-08-04 04:27 /user/oozie
drwxr-xr-x   - root     hdfsadmingroup          0 2022-08-04 04:50 /user/root
drwxr-xr-x   - spark    spark                   0 2022-08-04 04:26 /user/spark
drwxr-xr-x   - zeppelin hdfsadmingroup          0 2022-08-04 04:26 /user/zeppelin


##将目录权限设置成777，1表示stick bit(不能删除别人创建的文件/文件夹)
sudo -u hdfs hadoop fs -chmod -R 1777 /user