This project uses Kafka, Spark Streaming, and PubNub, (and Scala) to stream localized wholesale power prices and wind data throughout Texas in realtime @ https://ian-wright.github.io/everythings-windier-in-texas/
Mainly because I wanted a project to learn some data streaming technologies, and dive into Scala programming.
But also... the Texas electricity market (ERCOT - Electricity Reliability Council of Texas) produces more wind-generated power than any other state. This massive wind capacity, combined with the effects of the Federal Production Tax Credit, make for an interesting (and often dramatic) dynamic between realtime wholesale power prices and wind resources. On several particularly windy occasions in the past, localized power prices have actually gone negative. This visualization is meant to monitor the wind - power price relationship, spatially and in realtime.
- Python scripts pull weather data from here and pricing data from here, and push the data to a local single-node two-topic Kafka broker.
- A Scala app uses Spark Streaming to connect to the Kafka broker, and cleans/filters/processes all data according to topic
- Scala app publishes the DStream to a realtime messaging cloud service, PubNub, on a singular channel
- A simple frontend client subscribes to the channel and receives all messages in realtime;
-
If you don't want to run it locally, but want to see the visualization, contact me to "turn it on".
-
Otherwise, clone this repo.
-
(tab #1) - start a local single-node zookeeper instance
kafka_2.11-0.11.0.0/bin/zookeeper-server-start.sh kafka_2.11-0.11.0.0/config/zookeeper.properties
-
(tab #2) - create the kafka topics on the broker
kafka_2.11-0.11.0.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic price
kafka_2.11-0.11.0.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic weather
-
(still tab #2) - start the single-node kafka broker
kafka_2.11-0.11.0.0/bin/kafka-server-start.sh kafka_2.11-0.11.0.0/config/server.properties
-
(tab #3) - publish weather data to kafka
python data_generators/satori_to_kafka.py
-
(tab #4) - publish price data to kafka
python data_generators/price_to_kafka.py
-
(tab #5) - compile and run
sbt compile
sbt run
-
navigate to https://ian-wright.github.io/everythings-windier-in-texas/ and watch some data pour in!