Lambda architecture used: batch layer for heavy analytics and speed layer for realtime streaming.
Design:
- Acquisition layer reads raw data to lake
- Messaging layer listens raw streams from price providers
- Ingestion layer listens messaging layer and transforms raw format to speed layer related
- Ingestion layer persists streaming data from messaging layer to data lake in raw format
- Lambda layer contains of Batch and Speed layers
- Batch layer reads raw data from lake and prepare batch views
- Speed layer listens ingestion layer and prepare speed views
- Serving layer is a facade for data marts, calculated by batch and speed layers
- Datamarts are facades and combine data from speed and batch views