最近的项目用到了spark，边学边搞项目。一丢丢成就，做以记录，怕忘。
spark on yarn 模式就是将 spark 应用程序运行在 yarn 集群之上，其实并不需要集群上的 spark 运行任何进程服务，也就是说不需要在集群的每个节点上安装 spark。只需要选择一个节点安装 spark 作为客户端，甚至这个节点可以不是集群的某台机器，只要能将 spark 任务提交到 yarn 集群即可。
spark on yarn 模式遵循 yarn 的官方规范，yarn 只负责资源的管理和调度，运行那种程序由用户自己实现，因此可能在 yarn 上同时运行 MapReduce 程序和 Spark 程序，yarn 很好地对每一个程序实现了资源隔离。这使得 spark 和 MapReduce 可以运行于同一集群中，共享集群存储资源和计算资源。
spark-submit 提交任务
提交任务命令
./bin/spark-submit --class my.main.Class \
--master yarn \
--deploy-mode cluster \
--jars my-other-jar.jar,my-other-other-jar.jar \
my-main-jar.jar \
app_arg1 app_arg2部分日志：
org.apache.hadoop.yarn.client.RMProxy : Connecting to ResourceManager at sh01/172.16.98.224:8010
org.apache.spark.deploy.yarn.Client : Requesting a new application from cluster with 2 NodeManagers
org.apache.spark.deploy.yarn.Client : Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
org.apache.spark.deploy.yarn.Client : Will allocate AM container, with 1408 MB memory including 384 MB overhead
org.apache.spark.deploy.yarn.Client : Setting up container launch context for our AM
org.apache.spark.deploy.yarn.Client : Setting up the launch environment for our AM container
org.apache.spark.deploy.yarn.Client : Preparing resources for our AM container
org.apache.spark.deploy.yarn.Client : Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
org.apache.spark.deploy.yarn.Client : Uploading resource file:/private/var/folders/pc/mj2v_vln4x14q6jylbtnmvx40000gn/T/spark-95ddbd61-cb1f-4c2e-b2ce-4ddbedc07f5a/__spark_libs__1866547205795269356.zip -> hdfs://sh01:9000/user/mac/.sparkStaging/application_1656933536349_0091/__spark_libs__1866547205795269356.zip从日志中看到：因为既没有指定 spark.yarn.jars ，也没有指定 spark.yarn.archive ，spark 转向将 SPARK_HOME 下的库文件上传。实则是将 ${SPARK_HOME}/jars/*.jar 文件打包成 __spark_libs__1866547205795269356.zip 再上传到 hdfs 的指定目录下。
_spark_libs__1866547205795269356.zip 文件中总共有 226 jar包，每次提交任务就得打包上传zip文件，这个是很耗时间的，如果 jar 很多或者网络不咋地，这里肯定会砸锅。
提交优化
官方文档中有两段描述：
To make Spark runtime jars accessible from YARN side, you can specify spark.yarn.archive or spark.yarn.jars. For details please refer to Spark Properties. If neither spark.yarn.archive nor spark.yarn.jars is specified, Spark will create a zip file with all jars under $SPARK_HOME/jars and upload it to the distributed cache.为了让 spark 应用程序在 yarn 上运行时可以访问 jar包，可以使用 spark.yarn.archive 或者 spark.yarn.jars 参数指定 jar包所在位置，如果两个参数都没有指定，那 spark 会吧 $SPARK_HOME/jars 打包成 zip 包并上传到分布式缓存(HDFS)中。
List of libraries containing Spark code to distribute to YARN containers. By default, Spark on YARN will use Spark jars installed locally, but the Spark jars can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to jars on HDFS, for example, set this configuration to hdfs:///some/path. Globs are allowed.spark 代码依赖的库要分发到 yarn 容器中，默认地，spark on yarn 将使用本地 spark 的 jar。但是spark应用程序依赖的jar包可以保存到一个能被全局访问的位置–HDFS。因为 yarn 在节点上缓存了这些 jar 包，所以不必每次提交 spark 应用程序的时候再将其打包上传。只要指定 jar 包的所在位置就可以了。比如：hdfs:///some/path。
总结：可以提前将 ${SPARK_HOME}/jars/*.jar 上传到 hdfs 上，后面每次 spark-submit 时指定 jar 在 hdfs 的位置，就可以避开打包上传这个步骤了。
操作
- 在 hdfs 上创建目录并上传 jar 包。
hadoop fs -mkdir spark-jars
cd ${SPARK_HOME}/jars
hadoop fs -put ./*.jar spark-jars- 在提交任务时指定jar包位置，我用的是 java 代码，命令行自己查资料。
System.setProperty("SPARK_YARN_MODE", "true");
SparkConf sparkConf = new SparkConf();
sparkConf.setSparkHome(sparkHome);
sparkConf.setMaster("yarn");
sparkConf.setAppName("spark-yarn");
sparkConf.set("spark.submit.deployMode", "cluster");
// jar包的存放位置
String jarDir = "hdfs://sh01:9000/user/deployer/spark-jars/*.jar";
sparkConf.set("spark.yarn.jars", jarDir);
ClientArguments clientArguments = new ClientArguments(args);
Client client = new Client(clientArguments, sparkConf);
client.run();- 运行并查看日志
org.apache.hadoop.yarn.client.RMProxy : Connecting to ResourceManager at sh01/172.16.99.214:8010
org.apache.spark.deploy.yarn.Client : Requesting a new application from cluster with 2 NodeManagers
org.apache.spark.deploy.yarn.Client : Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
org.apache.spark.deploy.yarn.Client : Will allocate AM container, with 1408 MB memory including 384 MB overhead
org.apache.spark.deploy.yarn.Client : Setting up container launch context for our AM
org.apache.spark.deploy.yarn.Client : Setting up the launch environment for our AM container
org.apache.spark.deploy.yarn.Client : Preparing resources for our AM container
org.apache.spark.deploy.yarn.Client : Source and destination file systems are the same. Not copying hdfs://sh01:9000/user/deployer/spark-jars/JavaEWAH-0.3.2.jar
org.apache.spark.deploy.yarn.Client : Source and destination file systems are the same. Not copying hdfs://sh01:9000/user/deployer/spark-jars/RoaringBitmap-0.7.45.jar
org.apache.spark.deploy.yarn.Client : Source and destination file systems are the same. Not copying hdfs://sh01:9000/user/deployer/spark-jars/ST4-4.0.4.jar
org.apache.spark.deploy.yarn.Client : Source and destination file systems are the same. Not copying hdfs://sh01:9000/user/deployer/spark-jars/activation-1.1.1.jar
org.apache.spark.deploy.yarn.Client : Source and destination file systems are the same. Not copying hdfs://sh01:9000/user/deployer/spark-jars/aircompressor-0.10.jar
org.apache.spark.deploy.yarn.Client : Source and destination file systems are the same. Not copying hdfs://sh01:9000/user/deployer/spark-jars/antlr-2.7.7.jar
org.apache.spark.deploy.yarn.Client : Source and destination file systems are the same. Not copying hdfs://sh01:9000/user/deployer/spark-jars/antlr-runtime-3.4.jar
org.apache.spark.deploy.yarn.Client : Source and destination file systems are the same. Not copying hdfs://sh01:9000/user/deployer/spark-jars/antlr4-runtime-4.7.jar很明显：Source and destination file systems are the same. Not copying 文件一样，不用复制。
