StreamingPro

作者: 祝威廉 | 来源:发表于2016-05-10 21:15 被阅读1714次

    Declarative workflows for building Spark Streaming

    Spark Streaming

    Spark Streaming is an extension of the core Spark API that enables stream processing from a variety of sources.
    Spark is a extensible and programmable framework for massive distributed processing of datasets,
    called Resilient Distributed Datasets (RDD). Spark Streaming receives input data streams and divides the data into batches, which are then processed by the Spark engine to generate the results.
    Spark Streaming data is organized into a sequence of DStreams,
    represented internally as a sequence of RDDs.

    StreamingPro

    StreamingPro is not a complete application, but rather a extensible and programmable framework for spark streaming (also include spark,storm)
    that can easily be used to build your streaming application.

    StreamingPro also make it possible that all you should do to build streaming program is assembling components(eg. SQL Component) in configuration file.

    Features

    • Pure Spark Streaming(Or normal Spark) program (Storm in future)
    • No need of coding, only declarative workflows
    • Rest API for interactive
    • SQL-Oriented workflows support
    • Data continuously streamed in & processed in near real-time
    • dynamically CURD of workflows at runtime via Rest API
    • Flexible workflows (input, output, parsers, etc...)
    • High performance
    • Scalable

    Documents

    Architecture

    Snip20160510_3.png

    Declarative workflows

    Snip20160510_4.png

    Implementation

    Snip20160510_1.png

    相关文章

      网友评论

      • 5cd97d2a74b2:想做个mysql同步hive的工具,这个支持从mysql取数据吗
      • 游戏三昧:你分享的下载链接已经不能使用了,有最新的吗?396548475@qq.com 多谢分享
        祝威廉:@游戏三昧 已经可以了。微盘过一段时间又会自动失效。。。没办法。。。
      • 牛肉圆粉不加葱:这是什么画图工具
        祝威廉:@i4late 你需要下载预编译好的包 自己编译的话 有很多依赖
        wangliang938:hi ,
        依赖的jar我直接放到了spark/lib 下,启动的时候还是报
        Caused by: java.lang.ClassNotFoundException: net.csdn.common.logging.Loggers
        net.csdn.common.jar 我已经在本地编译了。
        按照online的方式打的包,在本地测试。
        祝威廉:@牛肉圆粉不加葱 keynote

      本文标题:StreamingPro

      本文链接:https://www.haomeiwen.com/subject/qevyrttx.html