tez和spark引擎比较 sparktec

Spark 源码解读（一）SparkContext的初始化之终章

1 、启动测量系统MetricsSystem

MetricsSystem 使用codahale提供的第三方测量库Metrics。MetricsSystem 中有三个概念：

• Instance:指定了谁在使用测量系统
• Source:指定了从哪里收集测量数据
• Sink:指定了往哪里输出测量数据

Spark按照Instance的不同，区分为Master,Work，Application,Driver和Executor

Spark目前使用的Sink有ConsoleSink，CsvSink，JmxSink,MetricsServlet,GraphiteSink等，

Spark中使用MetricsServlet作为默认的Sink。

MetricsSystem 的启动代码如下：

_env.metricsSystem.start()

MetricsSystem 的启动过程包括以下步骤：

• 注册Source
• 注册Sinks
• 给Sinks增加Jetty的ServletContextHandler

MetricsSystem 启动完毕之后，会遍历所有与Sinks有关的ServletContextHandler，并调用attachHandler将它们绑定到Spark UI上面。代码如下：

_env.metricsSystem.getServletHandlers.foreach(handler => ui.foreach(_.attachHandler(handler)))

1.2 注册Source

registerSource方法用于注册Source,告诉测量系统从哪里收集测量数据。

注册Source的过程分为以下步骤：

• 从metricsConfig获取Driver的Properties,默认为创建MetricsSystem的过程中解析的
• 用正则匹配Driver的Properties中以source.开头的属性。然后将属性中的Source反射得到的实例加入ArrayBuffer[Source].
• 将每个source的metricRegistry(也就是MetricsSet的子类型)注册到ConsurrentMap<String,Metrics> metrics.

代码如下：

private def registerSources() {
    val instConfig = metricsConfig.getInstance(instance)
    val sourceConfigs = metricsConfig.subProperties(instConfig, MetricsSystem.SOURCE_REGEX)

    // Register all the sources related to instance
    sourceConfigs.foreach { kv =>
      val classPath = kv._2.getProperty("class")
      try {
        val source = Utils.classForName(classPath).newInstance()
        registerSource(source.asInstanceOf[Source])
      } catch {
        case e: Exception => logError("Source class " + classPath + " cannot be instantiated", e)
      }
    }
  }

1.3 注册Sinks

registerSinks方法用于注册Sinks,即告诉测量系统往哪里输出数据。注册Sinks的步骤如下：

• 从Driver的Properties中用正则匹配以Sink.开头的属性，如{sink.servlet.class=org.apache.spark.metrics.sink.MetricsServlet, sink.servlet.path=/metrics/json}。将其转换为Map(servlet -> {class=org.apache.spark.metrics.sink.MetricsServlet, path=/metrics/json})。
• 将子属性class对应的类metricsServlet反射得到MetricsServlet实例。如果属性的key是servlet，将其设置为metricsServlet；如果是Sink，则加入到ArrayBuffer[Sink]中。
  代码如下：

private def registerSinks() {
    val instConfig = metricsConfig.getInstance(instance)
    val sinkConfigs = metricsConfig.subProperties(instConfig, MetricsSystem.SINK_REGEX)

    sinkConfigs.foreach { kv =>
      val classPath = kv._2.getProperty("class")
      if (null != classPath) {
        try {
          val sink = Utils.classForName(classPath)
            .getConstructor(classOf[Properties], classOf[MetricRegistry], classOf[SecurityManager])
            .newInstance(kv._2, registry, securityMgr)
          if (kv._1 == "servlet") {
            metricsServlet = Some(sink.asInstanceOf[MetricsServlet])
          } else {
            sinks += sink.asInstanceOf[Sink]
          }
        } catch {
          case e: Exception =>
            logError("Sink class " + classPath + " cannot be instantiated")
            throw e
        }
      }
    }
  }

1.4 给Sinks增加Jetty的ServletContextHandler

为了能够在SparkUI访问到测量数据，所以需要给Sinks增加Jetty的ServletContextHandler，MetricsSystem的getServletHandlers方法(只能在start()方法之后调用)，实现如下：

def getServletHandlers: Array[ServletContextHandler] = {
    require(running, "Can only call getServletHandlers on a running MetricsSystem")
    metricsServlet.map(_.getHandlers(conf)).getOrElse(Array())
  }

可以看到调用了getHandlers方法，其代码如下：

def getHandlers(conf: SparkConf): Array[ServletContextHandler] = {
    Array[ServletContextHandler](
      createServletHandler(servletPath,
        new ServletParams(request => getMetricsSnapshot(request), "text/json"), securityMgr, conf)
    )
  }

最终生成处理/metrics/json请求的ServletContextHandler，而请求的真正处理由getMetricsSnapshot方法，利用fastjson解析。生成的ServletContextHandler通过SparkUI的attachHandler方法，也被绑定到SparkUI。createServletHandler与attachHandler方法都已经在3.4.4节详细阐述。最终我们可以使用以下这些地址来访问测量数据。

http://localhost:4040/metrics/applications/json

http://localhost:4040/metrics/json

http://localhost:4040/metrics/master/json

2、创建和启动ExecutorAllocationManager

ExecutorAllocationManager用于对已分配的Executor进行管理，创建和启动ExecutorAllocationManager的代码如下：

val dynamicAllocationEnabled = Utils.isDynamicAllocationEnabled(_conf)
    _executorAllocationManager =
      if (dynamicAllocationEnabled) {
        schedulerBackend match {
          case b: ExecutorAllocationClient =>
            Some(new ExecutorAllocationManager(
              schedulerBackend.asInstanceOf[ExecutorAllocationClient], listenerBus, _conf))
          case _ =>
            None
        }
      } else {
        None
      }

ExecutorAllocationManager可以设置动态分配最小Executor数量、动态分配最大Executor数量、每个Executor可以运行的Task数量等配置信息，并对配置信息进行校验。默认情况下不会创建ExecutorAllocationManager，可以修改属性spark.dynamicAllocation.enabled为true来创建。start方法将ExecutorAllocationListener加入到listenerBus中，ExecutorAllocationListener通过监听listenerBus里的事件，动态添加删除executor。并且通过Thread不断的添加executor，并且遍历executor，将超时的executor杀掉并且移除。

3、ContextCleaner的创建与启动

ComtextCleaner用于清理那些超出应用范围的RDD、shuffleDependency和Broadcast对象。可以通过配置属性spark.cleaner.reference.Tracking来配置是否构造并启动ContextCleaner,默认为true.

代码如下：

_cleaner =
      if (_conf.getBoolean("spark.cleaner.referenceTracking", true)) {
        Some(new ContextCleaner(this))
      } else {
        None
      }
    _cleaner.foreach(_.start())

ContextCleaner的组成如下：

• referencQueue:缓存顶级的AnyRef引用
• referenceBuffer:缓存AnyRef的虚引用
• listeners:缓存清理工作的监听器数组
• cleaningThread:用于具体清理工作的线程。

ContextCleaner的工作原理和listenerBus一样，也采用监听器模式，由线程来处理，此线程只是调用了keepCleaning方法。keepCleaning的实现方法如下：

/** Keep cleaning RDD, shuffle, and broadcast state. */
  private def keepCleaning(): Unit = Utils.tryOrStopSparkContext(sc) {
    while (!stopped) {
      try {
        val reference = Option(referenceQueue.remove(ContextCleaner.REF_QUEUE_POLL_TIMEOUT))
          .map(_.asInstanceOf[CleanupTaskWeakReference])
        // Synchronize here to avoid being interrupted on stop()
        synchronized {
          reference.foreach { ref =>
            logDebug("Got cleaning task " + ref.task)
            referenceBuffer.remove(ref)
            ref.task match {
              case CleanRDD(rddId) =>
                doCleanupRDD(rddId, blocking = blockOnCleanupTasks)
              case CleanShuffle(shuffleId) =>
                doCleanupShuffle(shuffleId, blocking = blockOnShuffleCleanupTasks)
              case CleanBroadcast(broadcastId) =>
                doCleanupBroadcast(broadcastId, blocking = blockOnCleanupTasks)
              case CleanAccum(accId) =>
                doCleanupAccum(accId, blocking = blockOnCleanupTasks)
              case CleanCheckpoint(rddId) =>
                doCleanCheckpoint(rddId)
            }
          }
        }
      } catch {
        case ie: InterruptedException if stopped => // ignore
        case e: Exception => logError("Error in cleaning thread", e)
      }
    }
  }

4、Spark环境更新

在SparkContext的初始化过程中，可能会对其环境造成影响，所以需要更新环境，代码如下：

postEnvironmentUpdate()
 postApplicationStart()

SparkContext初始化过程中，如果设置了spark.jars属性，spark.jars指定的jar包将由addJar方法加入httpFileServer的jarDir变量指定的目录下。spark.files指定的文件将由addFile方法加入httpFileServer的fileDir变量指定的路径下，依赖文件处理如下(SparkContext.scala)：

_jars = Utils.getUserJars(_conf)
 _files = _conf.getOption("spark.files").map(_.split(",")).map(_.filter(_.nonEmpty))
      .toSeq.flatten

// Add each JAR given through the constructor
    if (jars != null) {
      jars.foreach(addJar)
    }

    if (files != null) {
      files.foreach(addFile)
    }

其中，addJar和addFile代码如下：

/**
   * Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
   * The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported
   * filesystems), an HTTP, HTTPS or FTP URI, or local:/path for a file on every worker node.
   */
  def addJar(path: String) {
    if (path == null) {
      logWarning("null specified as parameter to addJar")
    } else {
      var key = ""
      if (path.contains("\\")) {
        // For local paths with backslashes on Windows, URI throws an exception
        key = env.rpcEnv.fileServer.addJar(new File(path))
      } else {
        val uri = new URI(path)
        // SPARK-17650: Make sure this is a valid URL before adding it to the list of dependencies
        Utils.validateURL(uri)
        key = uri.getScheme match {
          // A JAR file which exists only on the driver node
          case null | "file" =>
            try {
              val file = new File(uri.getPath)
              if (!file.exists()) {
                throw new FileNotFoundException(s"Jar ${file.getAbsolutePath} not found")
              }
              if (file.isDirectory) {
                throw new IllegalArgumentException(
                  s"Directory ${file.getAbsoluteFile} is not allowed for addJar")
              }
              env.rpcEnv.fileServer.addJar(new File(uri.getPath))
            } catch {
              case NonFatal(e) =>
                logError(s"Failed to add $path to Spark environment", e)
                null
            }
          // A JAR file which exists locally on every worker node
          case "local" =>
            "file:" + uri.getPath
          case _ =>
            path
        }
      }
      if (key != null) {
        val timestamp = System.currentTimeMillis
        if (addedJars.putIfAbsent(key, timestamp).isEmpty) {
          logInfo(s"Added JAR $path at $key with timestamp $timestamp")
          postEnvironmentUpdate()
        }
      }
    }
  }

/**
   * Add a file to be downloaded with this Spark job on every node.
   * The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported
   * filesystems), or an HTTP, HTTPS or FTP URI.  To access the file in Spark jobs,
   * use `SparkFiles.get(fileName)` to find its download location.
   *
   * A directory can be given if the recursive option is set to true. Currently directories are only
   * supported for Hadoop-supported filesystems.
   */
  def addFile(path: String, recursive: Boolean): Unit = {
    val uri = new Path(path).toUri
    val schemeCorrectedPath = uri.getScheme match {
      case null | "local" => new File(path).getCanonicalFile.toURI.toString
      case _ => path
    }

    val hadoopPath = new Path(schemeCorrectedPath)
    val scheme = new URI(schemeCorrectedPath).getScheme
    if (!Array("http", "https", "ftp").contains(scheme)) {
      val fs = hadoopPath.getFileSystem(hadoopConfiguration)
      val isDir = fs.getFileStatus(hadoopPath).isDirectory
      if (!isLocal && scheme == "file" && isDir) {
        throw new SparkException(s"addFile does not support local directories when not running " +
          "local mode.")
      }
      if (!recursive && isDir) {
        throw new SparkException(s"Added file $hadoopPath is a directory and recursive is not " +
          "turned on.")
      }
    } else {
      // SPARK-17650: Make sure this is a valid URL before adding it to the list of dependencies
      Utils.validateURL(uri)
    }

    val key = if (!isLocal && scheme == "file") {
      env.rpcEnv.fileServer.addFile(new File(uri.getPath))
    } else {
      schemeCorrectedPath
    }
    val timestamp = System.currentTimeMillis
    if (addedFiles.putIfAbsent(key, timestamp).isEmpty) {
      logInfo(s"Added file $path at $key with timestamp $timestamp")
      // Fetch the file locally so that closures which are run on the driver can still use the
      // SparkFiles API to access files.
      Utils.fetchFile(uri.toString, new File(SparkFiles.getRootDirectory()), conf,
        env.securityManager, hadoopConfiguration, timestamp, useCache = false)
      postEnvironmentUpdate()
    }
  }

postEnvironmentUpdate的处理步骤如下：

• 通过调用SparkEnv的方法environmentDetails最终影响环境的JVM参数、Spark属性、系统属性、classPath等。
• 生成事件SparkListenerEnvironmentUpdate，并post到listenerBus，此事件被EnvironmentListener监听最终影响到EnviromentPage的输出。

代码如下：

/** Post the environment update event once the task scheduler is ready */
  private def postEnvironmentUpdate() {
    if (taskScheduler != null) {
      val schedulingMode = getSchedulingMode.toString
      val addedJarPaths = addedJars.keys.toSeq
      val addedFilePaths = addedFiles.keys.toSeq
      val environmentDetails = SparkEnv.environmentDetails(conf, schedulingMode, addedJarPaths,
        addedFilePaths)
      val environmentUpdate = SparkListenerEnvironmentUpdate(environmentDetails)
      listenerBus.post(environmentUpdate)
    }
  }

postApplicationStart方法很简单，只是向listener发送了SparkListenerApplicationStart:

/** Post the application start event */
  private def postApplicationStart() {
    // Note: this code assumes that the task scheduler has been initialized and has contacted
    // the cluster manager to get an application ID (in case the cluster manager provides one).
    listenerBus.post(SparkListenerApplicationStart(appName, Some(applicationId),
      startTime, sparkUser, applicationAttemptId, schedulerBackend.getDriverLogUrls))
  }

5、创建DAGSchedulerSource和BlockManagerSource

在创建DAGSchedulerSource、BlockManagerSource之前首先调用taskScheduler的postStartHook方法，其目的是为了等待backend就绪.创建DAGSchedulerSource和BlockManagerSource的过程类似于ExecutorSource，只不过DAGSchedulerSource测量的信息是stage. failedStages、stage. runningStages、stage. waiting-Stages、stage. allJobs、stage. activeJobs，BlockManagerSource测量的信息是memory. maxMem_MB、memory. remainingMem_MB、memory. memUsed_MB、memory. diskSpace-Used_MB。

创建DAGSchedulerSource和BlockManagerSource代码如下：

// Post init
    _taskScheduler.postStartHook()
    _env.metricsSystem.registerSource(_dagScheduler.metricsSource)
    _env.metricsSystem.registerSource(new BlockManagerSource(_env.blockManager))
    _executorAllocationManager.foreach { e =>
      _env.metricsSystem.registerSource(e.executorAllocationManagerSource)

postStartHook:

override def postStartHook() {
        waitBackendReady()
    }

private def waitBackendReady(): Unit = {
    if (backend.isReady) {
        return
    }
    while (!backend.isReady) {
        synchronized {
            this.wait(100)
        }
    }
}

6、将SparkContext标记为激活

SparkContext初始化的最后将当前SparkContext的状态从contextBeingConstructed（正在构建中）改为activeContext（已激活），代码如下。

// In order to prevent multiple SparkContexts from being active at the same time, mark this
  // context as having finished construction.
  // NOTE: this must be placed at the end of the SparkContext constructor.
  SparkContext.setActiveContext(this, allowMultipleContexts)

setActiveContext实现方法如下：

/**
   * Called at the end of the SparkContext constructor to ensure that no other SparkContext has
   * raced with this constructor and started.
   */
  private[spark] def setActiveContext(
      sc: SparkContext,
      allowMultipleContexts: Boolean): Unit = {
    SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
      assertNoOtherContextIsRunning(sc, allowMultipleContexts)
      contextBeingConstructed = None
      activeContext.set(sc)
    }
  }