saxenam06/traffic_predictive_model_pipeline

Instructions to run:

Batch Pipeline for ML Training and writing to PostgreSQL

Make sure spark is setup on your machine. To setup Spark you can follow instructions here Spark Setup To integrate with Postgresql follow some steps mentioned here Postgresql Setup Also you must use your own username and password for PostgreSQL in pgadmin.

1. In Instance 1, make sure your postgres docker image is started.

2. Read data and Write Tranformed data to PostgreSQL.

   Read data, extract features as categories, train and save ML models.

   sudo ./streaming-pipelines/spark/run_spark_topostgresql_and_mltrain.sh
   

3. You can see now a table 'traffic' created in PostgreSQL and mappings.txt together ML models saved to ./streaming-pipelines/ML_model

Streaming Pipeline for ML inference and writing results to PostgreSQL

Make sure spark, kafka and postgreql are setup on your machine. To setup Spark, Kafka and PostgreSQL you can follow instructions here Spark Setup, Kafka Setup and PostgresqlSetup. Also you must use your own username and password for PostgreSQL in pgadmin.

1. Open two instances of your VM.
2. In Instance 1, first start zookeeper and kafka servers. Create topic with name 'trafficdata'.

   sudo ./streaming-pipelines/kafka/run_start_kafka.sh
   

3. In Instance 1, read data and stream to kafka topic 'trafficdata'

   sudo ./streaming-pipelines/kafka/run_spark_kafka_producer.sh
   

4. In Instance 2, make sure your postgres docker image is started.
5. In Instance, subscribe to streaming data being published on topic 'trafficdata', load the ML models and categorigal mappings earlier created, perform ML predictions and write the predicted results in PostgreSQL.

   sudo ./streaming-pipelines/sparkstreaming/run_streaming.sh
   

6. You can see now a table 'realtimetraffic' created in PostgreSQL with your predicted results.