This project leverages the power of Kubernetes, Apache Spark, and Cassandra to analyze airline performance and flight delays. We use .parquet
files as our primary data format.
Ensure you have kubectl
, minikube
, and helm
installed on your local machine to execute the steps outlined below.
Run the following bash script to install all required libraries:
./requirements.sh
Initialize the Minikube cluster with specified resources:
minikube start --cores=8 --memory=5120
Utilize kubectl
to interact with your cluster. Check all pods:
kubectl get po -A
For a GUI overview, launch the Minikube dashboard:
minikube dashboard
Navigate to the config folder and execute the script to create Cassandra pods and stateful set:
cd config
sh create-cassandra-pod.sh
From the config
directory, use Helm to install Spark using the custom values from sparkValues.yaml
:
helm install my-spark bitnami/spark --version 8.1.6 -f sparkValues.yaml
Copy the application JAR and .parquet
data to the Spark master pod:
kubectl cp [path-to-the-jarfile] default/my-spark-master-0:/opt/bitnami/spark
kubectl cp [path-to-the-parquet-data] default/my-spark-master-0:/opt/bitnami/spark
Submit the Spark job to process the flight data:
./bin/spark-submit \
--class org.example.FlightDelayAnalysis \
--master spark://my-spark-master-0.my-spark-headless.default.svc.cluster.local:7077 \
--num-executors 2 \
--driver-memory 1g \
--driver-cores 1 \
--executor-memory 3g \
--executor-cores 3 \
FlightDelayAnalysis-1.0-SNAPSHOT.jar "10.244.0.6" "9042"
The application performs the following steps:
- Establishes connection with Spark and Cassandra.
- Reads the
.parquet
file into a Spark DataFrame. - Conducts ETL operations on the DataFrame to prepare it for analysis.
- Performs analysis and stores the results back into Cassandra.
Use Cassandra port forwarding to access Cassandra from outside the cluster, then run a Python script to visualize the results using Matplotlib:
# Example Python script to plot data
python plot_results.py
This project sets up a robust data processing pipeline using Kubernetes, Spark, and Cassandra to analyze and visualize airline performance and delay patterns effectively. For feedback and contributions, feel free to open an issue or pull request.