searchs/transport-platform

Chicago Transit Authority Status dashboard

This is a project for Udacity Data Streaming Nanaodegree. It processes Chicago Transit Authority data providing a dashboard displaying train arrivals in real or near-real time. The goal is to consume the data using Apache Kafka, Faust and Python The dataset is publicly available via the Chicago Transit Authority.

Here's a screenshot of the expected dashboard:

Development Environment

The development requires the libraries and tool along with their dependencies:

• Python 3.7
• Apache Kafka
• Postgresql 11
• Faust
• Python libraries as specified in requirements.txt in the Producers and Consumers directories.
• Access to a computer with a minimum of 16gb+ RAM and a 4-core CPU to execute the simulation

Using the

Description

The Chicago Transit Authority (CTA) has asked us to develop a dashboard displaying system status for its commuters. We have decided to use Kafka and ecosystem tools like REST Proxy and Kafka Connect to accomplish this task.

Our architecture will look like so:

Running the system

python producers/simulation.py

cd consumers

faust -A faust_stream worker -l info

python consumers/ksql.py

python consumers/server.py

Tools/Libraries Documentation

• Confluent Python Client Documentation
• Confluent Python Client Usage and Examples
• REST Proxy API Reference
• Kafka Connect JDBC Source Connector Configuration Options