大数据之Spark Streaming

Spark既可以使用RDD、SparkSQL做离线计算，又可以用Spark Streaming做实时计算；

Spark Streaming类似于Storm，用于流式数据的处理。根据其官方文档介绍，Spark Streaming有高吞吐量和容错能力强等特点。

DStream是Spark Streaming的基础抽象，代表持续性的数据流和经过各种Spark原语操作后的结果数据流。在内部实现上，DStream是一系列连续的RDD来表示。每个RDD含有一段时间间隔内的数据，如下图：

一、Spark例子

用户画像：就是给用户一个标签，问题：用户不登陆，如何确定用户？

可以获取用户的cookie和ip，持久化cookie信息

框架：

二、worldCount(单个批次)

模拟一个客户端写数据

StreamingWordCount 类：

package cn.itcast.spark.day5
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.streaming.{Seconds, StreamingContext}

  /**
    * Created by root on 2016/5/21.
    */
  object StreamingWordCount {

    def main(args: Array[String]) {

      LoggerLevels.setStreamingLogLevels()
      //StreamingContext
      val conf = new SparkConf().setAppName("StreamingWordCount").setMaster("local[2]")
      val sc = new SparkContext(conf)
      val ssc = new StreamingContext(sc, Seconds(5))
      //接收数据
      val ds = ssc.socketTextStream("172.16.0.11", 8888)
      //DStream是一个特殊的RDD
      //hello tom hello jerry
      val result = ds.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_)
      //打印结果
      result.print()
      ssc.start()
      ssc.awaitTermination()
    }
}

LoggerLevels类：

package cn.itcast.spark.day5
import org.apache.log4j.{Logger, Level}
import org.apache.spark.Logging

  object LoggerLevels extends Logging {

    def setStreamingLogLevels() {
      val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements
      if (!log4jInitialized) {
        logInfo("Setting log level to [WARN] for streaming example." +
          " To override add a custom log4j.properties to the classpath.")
        Logger.getRootLogger.setLevel(Level.WARN)
      }
    }
  }

三、worldCount (累加)

StateFulWordCount 类：

/**
  * Created by root on 2016/5/21.
  */
object StateFulWordCount {

    //Seq这个批次某个单词的次数
    //Option[Int]：以前的结果

    //分好组的数据
    val updateFunc = (iter: Iterator[(String, Seq[Int], Option[Int])]) => {
      //iter.flatMap(it=>Some(it._2.sum + it._3.getOrElse(0)).map(x=>(it._1,x)))
      //iter.map{case(x,y,z)=>Some(y.sum + z.getOrElse(0)).map(m=>(x, m))}
      //iter.map(t => (t._1, t._2.sum + t._3.getOrElse(0)))
      iter.map{ case(word, current_count, history_count) => (word, current_count.sum + history_count.getOrElse(0)) }
    }

    def main(args: Array[String]) {
      LoggerLevels.setStreamingLogLevels()
      //StreamingContext
      val conf = new SparkConf().setAppName("StateFulWordCount").setMaster("local[2]")
      val sc = new SparkContext(conf)
      //updateStateByKey必须设置setCheckpointDir
      sc.setCheckpointDir("c://ck")
      val ssc = new StreamingContext(sc, Seconds(5))

      val ds = ssc.socketTextStream("172.16.0.11", 8888)
      //DStream是一个特殊的RDD
      //hello tom hello jerry
      val result = ds.flatMap(_.split(" ")).map((_, 1)).updateStateByKey(updateFunc, new HashPartitioner(sc.defaultParallelism), true)

      result.print()  
      ssc.start()
      ssc.awaitTermination()
    }
}

四、Spark Streaming+flume (数据量少)

方式一： push方式（适用单个flume）

1、导入依赖

2、配置flume-push.conf

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /export/data/flume
a1.sources.r1.fileHeader = true

# Describe the sink
a1.sinks.k1.type = avro
#这是接收方
a1.sinks.k1.hostname = 192.168.31.172
a1.sinks.k1.port = 8888

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

3、启动flume

4、FlumePollWordCount 类

object FlumePollWordCount {

  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("FlumePollWordCount").setMaster("local[2]")
    val ssc = new StreamingContext(conf, Seconds(5))
    //从flume中拉取数据(flume的地址)
    val address = Seq(new InetSocketAddress("172.16.0.11", 8888))
    val flumeStream = FlumeUtils.createPollingStream(ssc, address, StorageLevel.MEMORY_AND_DISK)
    val words = flumeStream.flatMap(x => new String(x.event.getBody().array()).split(" ")).map((_,1))
    val results = words.reduceByKey(_+_)
    results.print()
    ssc.start()
    ssc.awaitTermination()
  }

}

方式二：Poll方式（适用多个flume）

1、配置flume-poll.conf

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /export/data/flume
a1.sources.r1.fileHeader = true

# Describe the sink
a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
a1.sinks.k1.hostname = master
a1.sinks.k1.port = 8888

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

2、编写FlumePollWordCount

object FlumePollWordCount {

    def main(args: Array[String]) {
      val conf = new SparkConf().setAppName("FlumePollWordCount").setMaster("local[2]")
      val ssc = new StreamingContext(conf, Seconds(5))
      //从flume中拉取数据(flume的地址)
      val address = Seq(new InetSocketAddress("172.16.0.11", 8888))
      val flumeStream = FlumeUtils.createPollingStream(ssc, address, StorageLevel.MEMORY_AND_DISK)
      val words = flumeStream.flatMap(x => new String(x.event.getBody().array()).split(" ")).map((_,1))
      val results = words.reduceByKey(_+_)
      results.print()
      ssc.start()
      ssc.awaitTermination()
    }
}

五、Spark Streaming 整合kafka

1、安装zookeeper

2、导入依赖包

方式一：使用receiver (数据量大的话容易出现溢出)

   /**
    * Created by root on 2016/5/21.
    */
   object KafkaWordCount {

     val updateFunc = (iter: Iterator[(String, Seq[Int], Option[Int])]) => {
       //iter.flatMap(it=>Some(it._2.sum + it._3.getOrElse(0)).map(x=>(it._1,x)))
       iter.flatMap { case (x, y, z) => Some(y.sum + z.getOrElse(0)).map(i => (x, i)) }
     }

     def main(args: Array[String]) {
       LoggerLevels.setStreamingLogLevels()
       val Array(zkQuorum, group, topics, numThreads) = args
       val sparkConf = new SparkConf().setAppName("KafkaWordCount").setMaster("local[2]")
       val ssc = new StreamingContext(sparkConf, Seconds(5))
       ssc.checkpoint("c://ck2")

       //"alog-2016-04-16,alog-2016-04-17,alog-2016-04-18"
       //"Array((alog-2016-04-16, 2), (alog-2016-04-17, 2), (alog-2016-04-18, 2))"

       val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
       val data = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap, StorageLevel.MEMORY_AND_DISK_SER)
       val words = data.map(_._2).flatMap(_.split(" "))
       val wordCounts = words.map((_, 1)).updateStateByKey(updateFunc, new HashPartitioner(ssc.sparkContext.defaultParallelism), true)
       ssc.start()
       ssc.awaitTermination()
     }
}

方式二：直连方式

一个kafka的partion对应一个DStream里RDD的一个分区， 可以控制每秒读多少数据，要自己控制偏移量。

package cn.itcast.spark.day5

import kafka.serializer.StringDecoder
import org.apache.log4j.{Level, Logger}
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD
import org.apache.spark.streaming.kafka.{KafkaManager, KafkaUtils}
import org.apache.spark.streaming.{Seconds, StreamingContext}


  object DirectKafkaWordCount {

    /*  def dealLine(line: String): String = {
        val list = line.split(',').toList
    //    val list = AnalysisUtil.dealString(line, ',', '"')// 把dealString函数当做split即可
        list.get(0).substring(0, 10) + "-" + list.get(26)
      }*/

    def processRdd(rdd: RDD[(String, String)]): Unit = {
      val lines = rdd.map(_._2)
      val words = lines.map(_.split(" "))
      val wordCounts = words.map(x => (x, 1L)).reduceByKey(_ + _)
      wordCounts.foreach(println)
    }

    def main(args: Array[String]) {
      if (args.length < 3) {
        System.err.println(
          s"""
             |Usage: DirectKafkaWordCount <brokers> <topics> <groupid>
             |  <brokers> is a list of one or more Kafka brokers
             |  <topics> is a list of one or more kafka topics to consume from
             |  <groupid> is a consume group
             |
          """.stripMargin)
        System.exit(1)
      }

      Logger.getLogger("org").setLevel(Level.WARN)

      val Array(brokers, topics, groupId) = args

      // Create context with 2 second batch interval
      val sparkConf = new SparkConf().setAppName("DirectKafkaWordCount")
      sparkConf.setMaster("local[*]")
      sparkConf.set("spark.streaming.kafka.maxRatePerPartition", "5")
      sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

      val ssc = new StreamingContext(sparkConf, Seconds(2))

      // Create direct kafka stream with brokers and topics
      val topicsSet = topics.split(",").toSet
      val kafkaParams = Map[String, String](
        "metadata.broker.list" -> brokers,
        "group.id" -> groupId,
        "auto.offset.reset" -> "smallest"
      )

      val km = new KafkaManager(kafkaParams)

      val messages = km.createDirectStream[String, String, StringDecoder, StringDecoder](
        ssc, kafkaParams, topicsSet)

      messages.foreachRDD(rdd => {
        if (!rdd.isEmpty()) {
          // 先处理消息
          processRdd(rdd)
          // 再更新offsets
          km.updateZKOffsets(rdd)
        }
      })

      ssc.start()
      ssc.awaitTermination()
    }
}

六、窗口函数（计算一个时间段内数据）

package cn.itcast.spark.day5
import org.apache.spark.SparkConf
mport org.apache.spark.streaming.{Milliseconds, Seconds, StreamingContext}

  /**
    * Created by ZX on 2016/4/19.
    */
  object WindowOpts {

    def main(args: Array[String]) {
      LoggerLevels.setStreamingLogLevels()
      val conf = new SparkConf().setAppName("WindowOpts").setMaster("local[2]")
      val ssc = new StreamingContext(conf, Milliseconds(5000))
      val lines = ssc.socketTextStream("172.16.0.11", 9999)
      val pairs = lines.flatMap(_.split(" ")).map((_, 1))
      val windowedWordCounts = pairs.reduceByKeyAndWindow((a:Int,b:Int) => (a + b), Seconds(15), Seconds(10))
      //Map((hello, 5), (jerry, 2), (kitty, 3))
      windowedWordCounts.print()
  //    val a = windowedWordCounts.map(_._2).reduce(_+_)
  //    a.foreachRDD(rdd => {
  //      println(rdd.take(0))
  //    })
  //    a.print()
  //    //windowedWordCounts.map(t => (t._1, t._2.toDouble / a.toD))
  //    windowedWordCounts.print()
  //    //result.print()
      ssc.start()
      ssc.awaitTermination()
    }

}

更多java、大数据学习面试资料，请扫码关注我的公众号：