apex51/streamingpro

Build Spark Streaming Application by SQL

StreamingPro is a fast, expressive, and convenient system running on Spark with streaming, batch, interactive query, and machine learning support.

StreamingPro makes it easier to build spark applications without writing any code by means of:

• Using json file in combination with modules, which are easy to be reused. This provides users declarative configurations to build spark applications.
• Data processing is based on SQL.
• Script support.

StreamingPro is not only an out-of-box complete application, but also an extensible and programmable framework for spark since you can develop you ower compositors(a.k.a moduler).

Features

• Pure Spark Streaming (or normal Spark) program
• No need of coding, only declarative workflows
• Rest API for interactive querying
• SQL-Oriented workflows support
• Data continuously streamed in & processed in near real-time
• dynamically CURD of workflows at runtime via Rest API
• Flexible workflows (input, output, parsers, etc)
• High performance
• Scalable

Download

Download page: https://pan.baidu.com/s/1miDfTI4

streamingpro-spark-0.4.14-SNAPSHOT.jar for spark 1.6+, scala 2.10
streamingpro-spark-2.0-0.4.14-SNAPSHOT.jar for spark 2.0+, scala 2.11
streamingpro.flink-0.4.14-SNAPSHOT-online-1.2.0.jar for flink 1.2.0, scala 2.10

Documents

More Chinese articles: http://www.jianshu.com/c/759bc22b9e15

• Three steps to run your first application

• 自己编译streamingpro

• How to build streamingpro

• 三步跑起你的第一个应用

• 利用StreamingPro实现SQL-交互式查询

• 使用Spark SQL 构建流式处理程序

• 使用Spark SQL构建批处理程序

• 流式计算常见模块用法说明

• 用线性回归无编码实现文章浏览数预测

• StreamingPro添加Scala script 模块支持

• Properties

• Run your first application

• Submit application

• dynamically CURD of workflows at runtime via Rest API

• Recovery

• Useful modules introduction

• Other runtime support

Architecture

If no picture is shown, please click me.

If github is too slow to view, please click me.

Declarative workflows for building Spark Streaming

If no picture is shown, please click me.

Implementation

If no picture is shown, please click me.