至少4种方式使用spark算子实现wordcout

1.使用reduceBykey

需求:读取一个存放word的文件读取这个文件使用reduceByKey算子进行wordcount演示

1.val wordrdd=sc.textFile("file:/opt/module/datas/1.txt")

2.val words=wordrdd.flatMap(_.split(" "))

3.words.map((_,1)).reduceByKey(_+_).collect

效果图:

2.使用groupbykey

1.val wordrdd=sc.textFile("file:/opt/module/datas/1.txt")

2.val words=wordrdd.flatMap(_.split(" "))

3.words.groupBy(x=>x).map(t=>(t._1,t._2.toList.size)).collect

效果图:

3.使用aggregateByKey

1.val wordrdd=sc.textFile("file:/opt/module/datas/1.txt")

2.val words=wordrdd.flatMap(_.split(" "))

3.val wordOne=words.map((_,1))

4.wordOne.aggregateByKey(0)(_+_,_+_).collect

效果图:

4.使用foldByKey

1.val wordrdd=sc.textFile("file:/opt/module/datas/1.txt")

2.val words=wordrdd.flatMap(_.split(" "))

3.val wordOne=words.map((_,1))

4.wordOne.foldByKey(0)(_+_).collect

效果图:

4.使用combineByKey

1.val wordrdd=sc.textFile("file:/opt/module/datas/1.txt")

2.val words=wordrdd.flatMap(_.split(" "))

3.val wordOne=words.map((_,1))

4.wordOne.combineByKey(x=>x,(x:Int,y:Int)=>x+y,(x:Int,y:Int)=>x+y).collect

效果图：