Spark 2.3新特性

Major Features on Spark 2.3

Features

Structured Streaming

Streaming

Continuous Processing Execution Mode

相比较之前的Structured Streaming,延迟更低,At-least-once guarantees。

ML on Streaming

模型转换和预测在批处理和流处理有统一的API。

val streamOutput = transformer.transform(streamDF)

支持Image Support in Spark

val df = ImageSchema.readImages("/data/images")

PySpark

Native Spark App in K8S

新的调度

Spark on Kubernetes

Databricks Delta

Delta是一种数据管理功能，可为云数据湖带来数据可靠性和性能优化。

Standard data pipelines with Spark
Databricks Delta address these challenges

Stream-stream Joins(双流Join)

Join Type

Data Source API V2

UDF Enhancement

Java UDF and UDAF.Register Java UDF and UDAF as a SQL function and use them in PySpark.

Stable Codegen

Native ORC Support

Various SQL Features

API

History Server Using K-V Store

这里只是根据PPT大体总结了一下,2.3的新特性,还需要结合源码深入的去理解。
相关资料在https://github.com/xy2953396112/JavaPractice/issues/1