spark知识点

1、自定义函数UDF

import org.apache.spark.sql.functions._

val myDF = sqlContext.parquetFile("hdfs:/to/my/file.parquet")

val coder: (Int => String) = (arg: Int) => {if (arg < 100) "little" else "big"}

val sqlfunc = udf(coder)

myDF.withColumn("Code", sqlfunc(col("Amt")))

2、自定义聚合函数UDAF

http://www.jianshu.com/p/833b72adb2b6