Spark-Avro学习6之Ubuntu下安装

原创

KeepLearningAI 2023-01-04 11:09:43 ©著作权

文章标签 Spark-Avro学习6之Ubuntu jar spark .net 文章分类 虚拟化云计算

©著作权归作者所有：来自51CTO博客作者KeepLearningAI的原创作品，请联系作者获取转载授权，否则将追究法律责任

1.下载：

git clone  https://github.com/databricks/spark-avro.git

2.安装：

./build/sbt package

3，记录：

hadoop@Master:~/cloud/testByXubo/soft/spark-avro$ ./build/sbt package
Using /usr/lib/jvm/jdk1.7.0 as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
Attempting to fetch sbt
Launching sbt from build/sbt-launch-0.13.9.jar
[info] Loading project definition from /home/hadoop/cloud/testByXubo/soft/spark-avro/project
[warn] Multiple resolvers having different access mechanism configured with same name 'sonatype-releases'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[info] Updating {file:/home/hadoop/cloud/testByXubo/soft/spark-avro/project/}spark-avro-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] downloading https://oss.sonatype.org/content/repositories/releases/org/scala-lang/scala-actors/2.10.2/scala-actors-2.10.2.jar ...
[info]  [SUCCESSFUL ] org.scala-lang#scala-actors;2.10.2!scala-actors.jar (8654ms)
[info] Done updating.
Missing bintray credentials /home/hadoop/.bintray/.credentials. Some bintray features depend on this.
[info] Set current project to spark-avro (in build file:/home/hadoop/cloud/testByXubo/soft/spark-avro/)
[warn] Credentials file /home/hadoop/.bintray/.credentials does not exist
[info] Updating {file:/home/hadoop/cloud/testByXubo/soft/spark-avro/}spark-avro...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] downloading http:roups/public/org/apache/avro/avro/1.7.6/avro-1.7.6.jar ...
[info]  [SUCCESSFUL ] org.apache.avro#avro;1.7.6!avro.jar(bundle) (365ms)
[info] downloading http://mublic/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar ...
[info]  [SUCCESSFUL ] com.thoughtworks.paranamer#paranamer;2.3!paranamer.jar (184ms)
[info] downloading http://mavublic/org/xerial/snappy/snappy-java/1.0.5/snappy-java-1.0.5.jar ...
[info]  [SUCCESSFUL ] org.xerial.snappy#snappy-java;1.0.5!snappy-java.jar(bundle) (402ms)
[info] downloading http://lic/org/slf4j/slf4j-api/1.6.4/slf4j-api-1.6.4.jar ...
[info]  [SUCCESSFUL ] org.slf4j#slf4j-api;1.6.4!slf4j-api.jar (182ms)
[info] downloading http://mav/public/org/apache/spark/spark-sql_2.10/1.4.1/spark-sql_2.10-1.4.1.jar ...
[info]  [SUCCESSFUL ] org.apache.spark#spark-sql_2.10;1.4.1!spark-sql_2.10.jar (619ms)
[info] downloading http://mavst_2.10.jar(bundle) (1064ms)
[info] downloading http://maroups/public/org/apache/spark/spark-core_2.10/1.4.1/spark-core_2.10-1.4.1.jar ...
[info]  [SUCCESSFUL ] org.apache.spark#spark-core_2.10;1.4.1!spark-core_2.10.jar (1335ms)
[info] downloading http://ups/public/org/apache/spark/spark-catalyst_2.10/1.4.1/spark-catalyst_2.10-1.4.1.jar ...
[info]  [SUCCESSFUL ] org.apache.spark#spark-catalyst_2.10;1.4.1!spark-catalyst_2.10.jar (706ms)
[info] downloading http://ms/public/com/twitter/parquet-column/1.6.0rc3/parquet-column-1.6.0rc3.jar ...
[info]  [SUCCESSFUL ] com.twitter#parquet-column;1.6.0rc3!parquet-column.jar (344ms)
[info] downloading http://mablic/com/twitter/parquet-hadoop/1.6.0rc3/parquet-hadoop-1.6.0rc3.jar ...
[info]  [SUCCESSFUL ] com.twitter#parquet-hadoop;1.6.0rc3!parquet-hadoop.jar (284ms)
[info] downloading http://mavc/org/jodd/jodd-core/3.6.3/jodd-core-3.6.3.jar ...
[info]  [SUCCESSFUL ] org.jodd#jodd-core;3.6.3!jodd-core.jar (217ms)
[info] downloading http://mgroups/public/org/apache/spark/spark-launcher_2.10/1.4.1/spark-launcher_2.10-1.4.1.jar ...
[info]  [SUCCESSFUL ] org.apache.spark#spark-launcher_2.10;1.4.1!spark-launcher_2.10.jar (175ms)
[info] downloading http:/groups/public/org/apache/spark/spark-network-common_2.10/1.4.1/spark-network-common_2.10-1.4.1.jar ...
[info]  [SUCCESSFUL ] org.apache.spark#spark-network-common_2.10;1.4.1!spark-network-common_2.10.jar (448ms)
[info] downloading http://mavroups/public/org/apache/spark/spark-network-shuffle_2.10/1.4.1/spark-network-shuffle_2.10-1.4.1.jar ...
[info]  [SUCCESSFUL ] org.apache.spark#spark-network-shuffle_2.10;1.4.1!spark-network-shuffle_2.10.jar (150ms)
[info] downloading http://maups/public/org/apache/spark/spark-unsafe_2.10/1.4.1/spark-unsafe_2.10-1.4.1.jar ...
[info]  [SUCCESSFUL ] org.apache.spark#spark-unsafe_2.10;1.4.1!spark-unsafe_2.10.jar (211ms)
[info] downloading http://mavennt/groups/public/net/jpountz/lz4/lz4/1.2.0/lz4-1.2.0.jar ...
[info]  [SUCCESSFUL ] net.jpountz.lz4#lz4;1.2.0!lz4.jar (226ms)
[info] downloading http:groups/public/org/spark-project/akka/akka-remote_2.10/2.3.4-spark/akka-remote_2.10-2.3.4-spark.jar ...
[info]  [SUCCESSFUL ] org.spark-project.akka#akka-remote_2.10;2.3.4-spark!akka-remote_2.10.jar (470ms)
[info] downloading hspark-project/akka/akka-slf4j_2.10/2.3.4-spark/akka-slf4j_2.10-2.3.4-spark.jar ...
[info]  [SUCCESSFUL ] org.spark-project.akka#akka-slf4j_2.10;2.3.4-spark!akka-slf4j_2.10.jar (147ms)
[info] downloading http://mgroups/public/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar ...
[info]  [SUCCESSFUL ] org.scala-lang#scala-library;2.10.4!scala-library.jar (1155ms)
[info] downloading http://groups/public/io/netty/netty-all/4.0.23.Final/netty-all-4.0.23.Final.jar ...
[info]  [SUCCESSFUL ] io.netty#netty-all;4.0.23.Final!netty-all.jar (373ms)
[info] downloading http://mps/public/io/dropwizard/metrics/metrics-core/3.1.0/metrics-core-3.1.0.jar ...
[info]  [SUCCESSFUL ] io.dropwizard.metrics#metrics-core;3.1.0!metrics-core.jar(bundle) (155ms)
[info] downloading http://1.0!metrics-jvm.jar(bundle) (216ms)
[info] downloading http://mavlic/io/dropwizard/metrics/metrics-json/3.1.0/metrics-json-3.1.0.jar ...
[info]  [SUCCESSFUL ] io.dropwizard.metrics#metrics-json;3.1.0!metrics-json.jar(bundle) (149ms)
[info] downloading http://mavic/io/dropwizard/metrics/metrics-graphite/3.1.0/metrics-graphite-3.1.0.jar ...
[info]  [SUCCESSFUL ] io.dropwizard.metrics#metrics-graphite;3.1.0!metrics-graphite.jar(bundle) (148ms)
[info] downloading http://mavoups/public/org/tachyonproject/tachyon-client/0.6.4/tachyon-client-0.6.4.jar ...
[info]  [SUCCESSFUL ] org.tachyonproject#tachyon-client;0.6.4!tachyon-client.jar (152ms)
[info] downloading http://maic/org/spark-project/akka/akka-actor_2.10/2.3.4-spark/akka-actor_2.10-2.3.4-spark.jar ...
[info]  [SUCCESSFUL ] org.spark-project.akka#akka-actor_2.10;2.3.4-spark!akka-actor_2.10.jar (689ms)
[info] downloading http://mpublic/org/spark-project/protobuf/protobuf-java/2.5.0-spark/protobuf-java-2.5.0-spark.jar ...
[info]  [SUCCESSFUL ] org.spark-project.protobuf#protobuf-java;2.5.0-spark!protobuf-java.jar(bundle) (354ms)
[info] downloading http://mavlic/org/tachyonproject/tachyon/0.6.4/tachyon-0.6.4.jar ...
[info]  [SUCCESSFUL ] org.tachyonproject#tachyon;0.6.4!tachyon.jar (585ms)
[info] downloading http://maroups/public/commons-codec/commons-codec/1.5/commons-codec-1.5.jar ...
[info]  [SUCCESSFUL ] commons-codec#commons-codec;1.5!commons-codec.jar (151ms)
[info] downloading http://maps/public/org/scala-lang/scala-compiler/2.10.4/scala-compiler-2.10.4.jar ...
[info]  [SUCCESSFUL ] org.scala-lang#scala-compiler;2.10.4!scala-compiler.jar (2113ms)
[info] downloading http://mas/public/com/twitter/parquet-common/1.6.0rc3/parquet-common-1.6.0rc3.jar ...
[info]  [SUCCESSFUL ] com.twitter#parquet-common;1.6.0rc3!parquet-common.jar (169ms)
[info] downloading http://mave/public/com/twitter/parquet-encoding/1.6.0rc3/parquet-encoding-1.6.0rc3.jar ...
[info]  [SUCCESSFUL ] com.twitter#parquet-encoding;1.6.0rc3!parquet-encoding.jar (408ms)
[info] downloading http://mavoups/public/com/twitter/parquet-generator/1.6.0rc3/parquet-generator-1.6.0rc3.jar ...
[info]  [SUCCESSFUL ] com.twitter#parquet-generator;1.6.0rc3!parquet-generator.jar (192ms)
[info] downloading http://mapublic/com/twitter/parquet-format/2.2.0-rc1/parquet-format-2.2.0-rc1.jar ...
[info]  [SUCCESSFUL ] com.twitter#parquet-format;2.2.0-rc1!parquet-format.jar (255ms)
[info] downloading http://mavublic/com/twitter/parquet-jackson/1.6.0rc3/parquet-jackson-1.6.0rc3.jar ...
[info]  [SUCCESSFUL ] com.twitter#parquet-jackson;1.6.0rc3!parquet-jackson.jar (356ms)
[info] downloading http://m/public/commons-net/commons-net/3.1/commons-net-3.1.jar ...
[info]  [SUCCESSFUL ] commons-net#commons-net;3.1!commons-net.jar (219ms)
[info] downloading https:itories/releases/org/scoverage/scalac-scoverage-runtime_2.10/1.1.0/scalac-scoverage-runtime_2.10-1.1.0.jar ...
[info]  [SUCCESSFUL ] org.scoverage#scalac-scoverage-runtime_2.10;1.1.0!scalac-scoverage-runtime_2.10.jar (4206ms)
[info] downloading http://maups/public/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar ...
[info]  [SUCCESSFUL ] org.mortbay.jetty#jetty;6.1.26!jetty.jar (380ms)
[info] downloading http://m/public/org/apache/velocity/velocity/1.7/velocity-1.7.jar ...
[info]  [SUCCESSFUL ] org.apache.velocity#velocity;1.7!velocity.jar (275ms)
[info] Done updating.
[info] Compiling 6 Scala sources to /home/hadoop/cloud/testByXubo/soft/spark-avro/target/scala-2.10/classes...
[info] Packaging /home/hadoop/cloud/testByXubo/soft/spark-avro/target/scala-2.10/spark-avro_2.10-2.0.2-SNAPSHOT.jar ...
[info] Done packaging.
[success] Total time: 183 s, completed May 2, 2016 11:19:48 AM