This Python script utilizes Apache Spark and Elasticsearch to monitor real-time status of Velib bike stations. It reads data from a Kafka topic, processes it with PySpark, and indexes relevant information into Elasticsearch. Kibana is then employed for data visualizations, taking advantage of the powerful analytics capabilities of the Elasticsearch-Kibana stack. Additionally, the script identifies and logs empty bike stations, providing valuable insights into station utilization and availability. The project integrates Hive LLAP (Live Long and Process), requiring dependencies on Hadoop, Hive, and Tez for optimized query performance.
Technologies Used:
Technology | Version |
---|---|
Spark | 3.3.1 |
Elasticsearch | 7.17.16 |
Kibana | 7.17.16 |
Kafka | 2.13-3.6.0 |
Hadoop | 3.3.6 |
Hive | 3.1.2 |
Tez | 0.9.2 |
For Python, ensure you have the following dependencies installed:
elasticsearch==8.10.1
pyspark==3.5.0
kafka-python==2.0.2
You can install them using the following command:
pip install -r requirements.txt
- Make sure Elasticsearch is running on
localhost:9200
. - The script creates an index named "velib-stations." If it already exists, it deletes and recreates it.
- The pyspark_code.py script reads data from a Kafka topic named "velib-stations" on
localhost:9092
. - That Kafka topic was created in the file get_stations.py
- In the file get_stations.py change the API key with yours after creating an account at https://developer.jcdecaux.com/#/opendata/vls?page=getstarted
- The Spark session is configured to connect to Elasticsearch on
localhost:9200
.
Before executing the main pyspark_consumer.py, ensure that Zookeeper and Kafka servers are running. You can start them using the following commands in different terminals and IN THAT ORDER:
-
Start Zookeeper:
./bin/zookeeper-server-start.sh ./config/zookeeper.properties
-
Start Kafka
./bin/kafka-server-start.sh ./config/server.properties
-
Run the get_stations.py file by doing
python3 Stations_producer.py
after that you can run the main script with spark-submit
spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2 pyspark_consumer.py
Description: This image displays the Kibana dashboard.
Description: This image provides an overview of the stations map in Europe.
Description: This image presents the map for bike stations and their capacity in the city of Valencia.
Description: This image illustrates the evolution of bike counts in three selected cities: Brussels, Tokyo, and Valencia.
Description: When hovering over the plot, the number of bikes in each city appears in the window.
Description: This image features the latest bike counts in different cities in the last 30 seconds.
For a comprehensive installation guide, you can refer to the Installation Guide PDF provided in this repository.
For specific and detailed installation instructions tailored to this project, please navigate to the installation folder in the repository.