hibernatexj/streamingpro

Build Spark Streaming Application by SQL

StreamingPro is a fast,expresivive,and convenient cluster system running on Spark with streaming,batch,interactive,machine learning support.

It make devlopers more easy to build spark application without writing any code by means of:

• Using json file in combination with many powerful modules which are easy to be reused provides the user declarative configurations to build your spark application.
• Data processing are SQL-Based
• Script support

StreamingPro is not only a out of box complete application, but also a extensible and programmable framework for spark since you can develop you ower compositors (moduler).

Features

• Pure Spark Streaming(Or normal Spark) program
• No need of coding, only declarative workflows
• Rest API for interactive
• SQL-Oriented workflows support
• Data continuously streamed in & processed in near real-time
• dynamically CURD of workflows at runtime via Rest API
• Flexible workflows (input, output, parsers, etc...)
• High performance
• Scalable

downloads

Precomile version based on Spark 1.6.1 ： http://share.weiyun.com/b1d38eb9436d4e05e60b44caee42ce10

Precomile version based on Spark 2.0.0: http://share.weiyun.com/62297a8023d6809efa6346370d4b39ea

Spark 2.0 based on Scala 2.10

Documents

• Three steps to run your first application
• 三步跑起你的第一个应用
• 利用StreamingPro实现SQL-交互式查询
• 使用Spark SQL 构建流式处理程序
• 使用Spark SQL构建批处理程序
• 流式计算常见模块用法说明
• 用线性回归无编码实现文章浏览数预测
• StreamingPro添加Scala script 模块支持
• Properties
• Build
• Run your first application
• Submit application
• dynamically CURD of workflows at runtime via Rest API
• Recovery
• Useful modules introduction
• Other runtime support

Architecture

If no picture show,please click me

If github is too slow to view ,please click me

Declarative workflows for building Spark Streaming

If no picture show,please click me

Implementation

If no picture show,please click me