spark对比两条数据是否一致

转载

编程小达 2024-10-16 22:03:14

文章标签 spark对比两条数据是否一致 scala 模式匹配 Scala 文章分类 Spark 大数据

一、模式匹配1

1、介绍

模式匹配是Scala中非常有特色，非常强大的一种功能。模式匹配，其实类似于Java中的swich case语法，即对一个值进行条件判断，然后针对不同的条件，
进行不同的处理。

但是Scala的模式匹配的功能比Java的swich case语法的功能要强大地多，Java的swich case语法只能对值进行匹配。但是Scala的模式匹配除了可以对值进行匹配之外，
还可以对类型进行匹配、对Array和List的元素情况进行匹配、对case class进行匹配、甚至对有值或没值（Option）进行匹配。

而且对于Spark来说，Scala的模式匹配功能也是极其重要的，在spark源码中大量地使用了模式匹配功能。因此为了更好地编写Scala程序，并且更加通畅地看懂Spark的
源码，学好模式匹配都是非常重要的。

2、模式匹配基础语法

// Scala是没有Java中的switch case语法的，相对应的，Scala提供了更加强大的match case语法，即模式匹配，类替代switch case，match case也被称为模式匹配
// Scala的match case与Java的switch case最大的不同点在于，Java的switch case仅能匹配变量的值，比1、2、3等；而Scala的match case可以匹配各种情况，比如变量的
类型、集合的元素、有值或无值
// match case的语法如下：变量 match { case 值 => 代码 }。如果值为下划线，则代表了不满足以上所有情况下的默认情况如何处理。此外，match case中，
只要一个case分支满足并处理了，就不会继续判断下一个case分支了。（与Java不同，java的switch case需要用break阻止）
// match case语法最基本的应用，就是对变量的值进行模式匹配


//案例：成绩评价
scala> :paste
// Entering paste mode (ctrl-D to finish)

def judgeGrade(grade: String) {
  grade match {
  case "A" => println("Excellent")
  case "B" => println("Good")
  case "C" => println("Just so so")
  case _ => println("you need to work harder")
  }
}

// Exiting paste mode, now interpreting.

judgeGrade: (grade: String)Unit

scala> judgeGrade("A")
Excellent

scala> judgeGrade("C")
Just so so

scala> judgeGrade("D")
you need to work harder

3、在模式匹配中使用if守卫

// Scala的模式匹配语法，有一个特点在于，可以在case后的条件判断中，不仅仅只是提供一个值，而是可以在值后面再加一个if守卫，进行双重过滤

// 案例：成绩评价（升级版）
scala> :paste
// Entering paste mode (ctrl-D to finish)

def judgeGrade(name: String, grade: String) {
  grade match {
    case "A" => println(name + ", your are really excellent!")
    case "B" => println(name + ", you are very good!")
    case "C" if name == "leo" => println(name + ",your parents are very hard, please come on!")
    case "C" if name == "jack" => println(name + ", your are smart, come on!")
    case "C" => println(name + ", Just so so")
    case _ => println(name + ", you need to work harder!")
  }
}

// Exiting paste mode, now interpreting.

judgeGrade: (name: String, grade: String)Unit

scala> judgeGrade("Jen","A")
Jen, your are really excellent!

scala> judgeGrade("leo","C")
leo,your parents are very hard, please come on!

scala> judgeGrade("jack","C")
jack, your are smart, come on!

scala> judgeGrade("marry","C")
marry, Just so so

4、在模式匹配中进行变量赋值

// Scala的模式匹配语法，有一个特点在于，可以将模式匹配的默认情况，下划线，替换为一个变量名，此时模式匹配语法就会将要匹配的值赋值给这个变量，
从而可以在后面的处理语句中使用要匹配的值
// 为什么有这种语法？？思考一下。因为只要使用case匹配到的值，是不是我们就知道这个值啦！！在这个case的处理语句中，是不是就直接可以使用写程序时
就已知的值！
// 但是对于下划线_这种情况，所有不满足前面的case的值，都会进入_这种默认情况进行处理，此时如果我们在处理语句中需要拿到具体的值进行处理呢？那就需要
使用这种在模式匹配中进行变量赋值的语法！！

// 案例：成绩评价（升级版）
scala> :paste
// Entering paste mode (ctrl-D to finish)

def judgeGrade(grade: String) {
  grade match {
    case "A" => println("you got A grade, excellent!")
    case "B" => println("you got B grade, good!")
    case "C" => println("you got C grade, just so so")
    case badGrade => println("you got " + badGrade + " grade, I hope that you can get C grade next time!")
  }
}

// Exiting paste mode, now interpreting.

judgeGrade: (grade: String)Unit

scala> 

scala> judgeGrade("A")
you got A grade, excellent!

scala> judgeGrade("B")
you got B grade, good!

scala> judgeGrade("D")
you got D grade, I hope that you can get C grade next time!

scala> judgeGrade("E")
you got E grade, I hope that you can get C grade next time!

二、模式匹配2

1、对类型进行模式匹配

// Scala的模式匹配一个强大之处就在于，可以直接匹配类型，而不是值！！！这点是java的switch case绝对做不到的。
// 理论知识：对类型如何进行匹配？其他语法与匹配值其实是一样的，但是匹配类型的话，就是要用“case 变量: 类型 => 代码”这种语法，而不是匹配值
的“case 值 => 代码”这种语法。

// 案例：异常处理
scala> import java.io._
import java.io._

scala> :paste
// Entering paste mode (ctrl-D to finish)

def processException(e: Exception) {
  e match {
    case e1: IllegalArgumentException => println("you passed illegal argument. exception is: " + e1)
    case e2: FileNotFoundException => println("cannot find the file. exception is " + e2)
    case e3: IOException => println("io error occurs. exception is " + e3)
    case _: Exception => println("exception occurs.")
  }
}

// Exiting paste mode, now interpreting.

processException: (e: Exception)Unit


scala> processException(new IllegalArgumentException("expect two argument, but fount one argument."))
you passed illegal argument. exception is: java.lang.IllegalArgumentException: expect two argument, but fount one argument.

scala> processException(new FileNotFoundException("test.txt not fount."))
cannot find the file. exception is java.io.FileNotFoundException: test.txt not fount.

scala> processException(new IOException("get data from socket fail."))
io error occurs. exception is java.io.IOException: get data from socket fail.

scala> processException(new ArrayIndexOutOfBoundsException("array is null,"))
exception occurs.

2、对Array和List进行模式匹配

// 对Array进行模式匹配，分别可以匹配带有指定元素的数组、带有指定个数元素的数组、以某元素打头的数组


// 案例：对朋友打招呼
scala> :paste
// Entering paste mode (ctrl-D to finish)

def greeting1(arr: Array[String]) {
  arr match {
    case Array("Leo") => println("How are you, Leo!")
    case Array(girl1, girl2, girl3) => println("Hi, girls, I'm jack, nice to meet you. " + girl1 + ", " + girl2 + ", " + girl3)
    case Array("Leo", _*) => println("Hi, Leo, why not introduce your friends to me!")
    case stranger => println(stranger + ", hey, who you are!")
  }
}

// Exiting paste mode, now interpreting.

greeting1: (arr: Array[String])Unit

scala> greeting
greeting   greeting1

scala> greeting1(Array("Leo"))
How are you, Leo!

scala> greeting1(Array("jen", "mary", "lory"))
Hi, girls, I'm jack, nice to meet you. jen, mary, lory

scala> greeting1(Array("Leo", "jack", "mike", "kitty"))
Hi, Leo, why not introduce your friends to me!

scala> greeting1(Array("mike"))
[Ljava.lang.String;@2abd838e, hey, who you are!



// 对List进行模式匹配，与Array类似，但是需要使用List特有的::操作符
scala> :paste
// Entering paste mode (ctrl-D to finish)

def greeting2(list: List[String]) {
  list match {
    case "leo" :: Nil => println("Hi, leo!")
    case girl1 :: girl2 :: girl3 :: Nil => println("Hi girls, may I know your names? " + girl1 + ", " + girl2 + ", " + girl3) 
    case "leo" :: tail => println("Hi leo, please introduce your friends to me")
    case _ => println("hei, who you are?")
  }
}

// Exiting paste mode, now interpreting.

greeting2: (list: List[String])Unit

scala> greeting2(List("leo"))
Hi, leo!

scala> greeting2(List("jen", "marry", "lory"))
Hi girls, may I know your names? jen, marry, lory

scala> greeting2(List("leo", "jack"))
Hi leo, please introduce your friends to me

scala> greeting2(List("jack"))
hei, who you are?

3、case class与模式匹配

// Scala中提供了一种特殊的类，用case class进行声明，中文也可以称作样例类。case class其实有点类似于Java中的JavaBean的概念。即只定义field，并且由Scala编译
时自动提供getter和setter方法，但是没有method。
// case class的主构造函数接收的参数通常不需要使用var或val修饰，Scala自动就会使用val修饰（但是如果你自己使用var修饰，那么还是会按照var来）
//  Scala自动为case class定义了伴生对象，也就是object，并且定义了apply()方法，该方法接收主构造函数中相同的参数，并返回case class对象

// 案例：学校门禁
scala> :paste
// Entering paste mode (ctrl-D to finish)

class Person
case class Teacher(name: String, subject: String) extends Person
case class Student(name: String, classroom: String) extends Person

def judgeIdentify(p: Person) {
  p match {
    case Teacher(name, subject) => println("Teacher, name is " + name + ", subject you teach is " + subject + "." )
    case Student(name, classroom) => println("Student, name is " + name + ", your classroom is " + classroom + ".")
    case _ => println("Illegal Access! please go out of the school, or we will call police!")
  }
}

// Exiting paste mode, now interpreting.

defined class Person
defined class Teacher
defined class Student
judgeIdentify: (p: Person)Unit


scala> val leo: Person = Student("leo", "class1")
leo: Person = Student(leo,class1)

scala> val tom: Person = Teacher("tom", "Match")
tom: Person = Teacher(tom,Match)

scala> case class Worker(name: String) extends Person
defined class Worker

scala> val jack: Person = Worker("jack")
jack: Person = Worker(jack)

scala> judgeIdentify(leo)
Student, name is leo, your classroom is class1.

scala> judgeIdentify(tom)
Teacher, name is tom, subject you teach is Match.

scala> judgeIdentify(jack)
Illegal Access! please go out of the school, or we will call police!

4、Option与模式匹配

// Scala有一种特殊的类型，叫做Option。Option有两种值，一种是Some，表示有值，一种是None，表示没有值。
// Option通常会用于模式匹配中，用于判断某个变量是有值还是没有值，这比null来的更加简洁明了
// Option的用法必须掌握，因为Spark源码中大量地使用了Option，比如Some(a)、None这种语法，因此必须看得懂Option模式匹配，才能够读懂spark源码。

// 案例：成绩查询
scala> val grades = Map("Leo" -> "A", "Jack" -> "B", "Tom" -> "C")
grades: scala.collection.immutable.Map[String,String] = Map(Leo -> A, Jack -> B, Tom -> C)

scala> :paste
// Entering paste mode (ctrl-D to finish)

def getGrade(name: String) {
  var grade = grades.get(name) 
  grade match {
    case Some(grade) => println("your grade is " + grade)
    case None => println("Sorry, your grade is not in the system. please ask your teacher!")
  }
}

// Exiting paste mode, now interpreting.

getGrade: (name: String)Unit

scala> getGrade("Leo")
your grade is A

scala> getGrade("Jack")
your grade is B

scala> getGrade("Tom")
your grade is C

scala> getGrade("Marry")
Sorry, your grade is not in the system. please ask your teacher!

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。