oguseynov/wikiflow

Wikipedia updates streaming, transformation and visualisation

Forked version of Spark flow on top of the Wikipedia SSE Stream

AnalyticsConsumer was updated:

1. Parses consumed messages.
2. Counts users and topics on a sliding time window with watermark.
3. Prints the aggregate to console and saves it to delta file.

How-to run

• Create the docker network:

make create-network

• Run the streaming appliance

make run-appliance

• To run streaming consumption of data via legacy API (DStreams), please run:

make run-legacy-consumer

• To run streaming consumption of data via structured API, please run:

make run-structured-consumer

• To run streaming consumption of data via structured API with write to delta, please run:

make run-analytics-consumer

You could also access the SparkUI for this Job at http://localhost:4040/jobs

Known issues

• Sometimes you need to increase docker memory limit for your machine (for Mac it's 2.0GB by default).
• To debug memory usage and status of the containers, please use this command:

docker stats

• Sometimes docker couldn't gracefully stop the consuming applications, please use this command in case if container hangs:

docker-compose -f <name of compose file with the job>.yaml down