dmitrypukhov/spark-trade

Processing stock exchange data with Apache Spark

spark-trade

Lambda architecture used: batch layer for heavy analytics and speed layer for realtime streaming.
Design:

1. Acquisition layer reads raw data to lake
2. Messaging layer listens raw streams from price providers
3. Ingestion layer listens messaging layer and transforms raw format to speed layer related
4. Ingestion layer persists streaming data from messaging layer to data lake in raw format
5. Lambda layer contains of Batch and Speed layers
6. Batch layer reads raw data from lake and prepare batch views
7. Speed layer listens ingestion layer and prepare speed views
8. Serving layer is a facade for data marts, calculated by batch and speed layers
9. Datamarts are facades and combine data from speed and batch views