This repository contains all the files for the Cloud Data Lake project of the Data Engineer Nanodegree Program by Udacity.
"A music streaming startup, Sparkify, has grown their user base and song database even more and want to move their data warehouse to a data lake. Their data resides in S3, in a directory of JSON logs on user activity on the app, as well as a directory with JSON metadata on the songs in their app."
Use the package manager pip to install the dependecies (it is recommended to create a virtual environment before doing that).
pip install -r requirements.txt
We need to update the dl.cfg file with the AWS credentials:
AWS_ACCESS_KEY=<access_key>
AWS_SECRET_KEY=<secret_key>
S3_OUTPUT_BUCKET=<s3a://<bucket_name>/>
The S3_OUTPUT_BUCKET is where we are going to store the output of the script, contaning the parquet files of the tables
python etl.py [--local]
This script will store the parquet files under the bucket . If we specify the --local option, everything will run using local directories (data and output by default).
unzip data/log-data.zip -d data
unzip data/song-data -d data
mkdir output
These 2 commands will unzip the sample data in order to be able to run the script.
After that, we are ready to run the etl.py:
python etl.py --local
Once we have the output, we could read the data with something like this:
songplays_data = os.path.join(output_data, 'songplays')
df = spark.read.parquet(songplays_data)
We could now operate with the Dataframe as usual.