Hudi编译(0.10.1版本)

原创

蹦擦擦蹦 2022-06-10 19:15:43 ©著作权

文章标签 hadoop apache spark 文章分类 OpenStack 云计算

©著作权归作者所有：来自51CTO博客作者蹦擦擦蹦的原创作品，请联系作者获取转载授权，否则将追究法律责任

一、编译

1.下载源码

https://www.apache.org/dyn/closer.lua/hudi/0.10.1/hudi-0.10.1.src.tgz

2.编译

mvn clean install -DskipTests -DskipITs -Dscala-2.11 -Dspark3 报以下错误

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-integ-test: Compilation failure: Compilation failure:
[ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[23,52] 程序包org.apache.hudi.common.testutils.minicluster不存在
[ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[39,7] 找不到符号
[ERROR] 符号:   类 ZookeeperTestService
[ERROR] 位置: 类 org.apache.hudi.integ.testsuite.helpers.ZookeeperServiceProvider
[ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[39,55] 找不到符号
[ERROR] 符号:   类 ZookeeperTestService
[ERROR] 位置: 类 org.apache.hudi.integ.testsuite.helpers.ZookeeperServiceProvider
[ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[46,7] 找不到符号
[ERROR] 符号:   类 ZookeeperTestService
[ERROR] 位置: 类 org.apache.hudi.integ.testsuite.helpers.ZookeeperServiceProvider
[ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[46,55] 找不到符号

解决：注释掉pom.xml文件中的

<module>hudi-integ-test</module>
  <module>packaging/hudi-integ-test-bundle</module>

再次编译即可。

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Hudi ............................................... SUCCESS [  4.854 s]
[INFO] hudi-common ........................................ SUCCESS [ 29.059 s]
[INFO] hudi-aws ........................................... SUCCESS [  4.438 s]
[INFO] hudi-timeline-service .............................. SUCCESS [  2.899 s]
[INFO] hudi-client ........................................ SUCCESS [  0.146 s]
[INFO] hudi-client-common ................................. SUCCESS [ 11.748 s]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [  4.333 s]
[INFO] hudi-spark-client .................................. SUCCESS [ 29.215 s]
[INFO] hudi-sync-common ................................... SUCCESS [  0.876 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [  4.108 s]
[INFO] hudi-spark-datasource .............................. SUCCESS [  0.142 s]
[INFO] hudi-spark-common_2.12 ............................. SUCCESS [ 14.039 s]
[INFO] hudi-spark3_2.12 ................................... SUCCESS [  9.784 s]
[INFO] hudi-spark_2.12 .................................... SUCCESS [ 46.734 s]
[INFO] hudi-utilities_2.12 ................................ SUCCESS [  6.831 s]
[INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [ 17.024 s]
[INFO] hudi-cli ........................................... SUCCESS [ 14.301 s]
[INFO] hudi-java-client ................................... SUCCESS [  1.872 s]
[INFO] hudi-flink-client .................................. SUCCESS [  6.390 s]
[INFO] hudi-spark2_2.12 ................................... SUCCESS [ 20.415 s]
[INFO] hudi-dla-sync ...................................... SUCCESS [  1.836 s]
[INFO] hudi-sync .......................................... SUCCESS [  0.068 s]
[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [  3.679 s]
[INFO] hudi-hive-sync-bundle .............................. SUCCESS [  1.424 s]
[INFO] hudi-spark3.1.2-bundle_2.12 ........................ SUCCESS [  7.476 s]
[INFO] hudi-presto-bundle ................................. SUCCESS [  4.860 s]
[INFO] hudi-timeline-server-bundle ........................ SUCCESS [  3.708 s]
[INFO] hudi-hadoop-docker ................................. SUCCESS [  0.693 s]
[INFO] hudi-hadoop-base-docker ............................ SUCCESS [  0.484 s]
[INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [  0.071 s]
[INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [  0.057 s]
[INFO] hudi-hadoop-history-docker ......................... SUCCESS [  0.057 s]
[INFO] hudi-hadoop-hive-docker ............................ SUCCESS [  0.694 s]
[INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [  0.130 s]
[INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [  0.062 s]
[INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [  0.072 s]
[INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [  0.067 s]
[INFO] hudi-hadoop-presto-docker .......................... SUCCESS [  0.303 s]
[INFO] hudi-examples ...................................... SUCCESS [ 10.590 s]
[INFO] hudi-flink_2.12 .................................... SUCCESS [ 34.163 s]
[INFO] hudi-kafka-connect ................................. SUCCESS [ 13.265 s]
[INFO] hudi-flink-bundle_2.12 ............................. SUCCESS [ 13.849 s]
[INFO] hudi-kafka-connect-bundle .......................... SUCCESS [ 11.594 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:39 min
[INFO] Finished at: 2022-02-08T11:31:28+08:00
[INFO] Final Memory: 340M/2823M
[INFO] ------------------------------------------------------------------------

3.测试 cd /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-cli ./hudi-cli.sh

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-cli/target/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-cli/target/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
0    [main] INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader  - Loading XML bean definitions from URL [jar:file:/Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-cli/target/hudi-cli-0.10.1.jar!/META-INF/spring/spring-shell-plugin.xml]
424  [main] INFO  org.springframework.context.support.GenericApplicationContext  - Refreshing org.springframework.context.support.GenericApplicationContext@69ea3742: startup date [Tue Feb 08 14:02:51 CST 2022]; root of context hierarchy
649  [main] INFO  org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor  - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
Table command getting loaded
HoodieSplashScreen loaded
===================================================================
*         ___                          ___                        *
*        /\__\          ___           /\  \           ___         *
*       / /  /         /\__\         /  \  \         /\  \        *
*      / /__/         / /  /        / /\ \  \        \ \  \       *
*     /  \  \ ___    / /  /        / /  \ \__\       /  \__\      *
*    / /\ \  /\__\  / /__/  ___   / /__/ \ |__|     / /\/__/      *
*    \/  \ \/ /  /  \ \  \ /\__\  \ \  \ / /  /  /\/ /  /         *
*         \  /  /    \ \  / /  /   \ \  / /  /   \  /__/          *
*         / /  /      \ \/ /  /     \ \/ /  /     \ \__\          *
*        / /  /        \  /  /       \  /  /       \/__/          *
*        \/__/          \/__/         \/__/    Apache Hudi CLI    *
*                                                                 *
===================================================================
Welcome to Apache Hudi CLI. Please type help if you are looking for help.
hudi->version
1.0