sheetalbongale/big-data-using-pyspark

Demonstrate ETL processes in Big Data using PySpark and using AWS's Relational Databases to analyze and perform statistical analysis of Amazon's Vine program customer reviews.

Big Data Analysis using PySpark

Objective:

Many of Amazon's shoppers depend on product reviews to make a purchase. Amazon makes these datasets publicly available. However, they are quite large and can exceed the capacity of local machines to handle. One dataset alone contains over 1.5 million rows; with over 40 datasets, this can be quite taxing on the average local computer.

The first goal for this assignment will be to perform the ETL process completely in the cloud and upload a DataFrame to an RDS instance. The second goal will be to use PySpark or SQL to perform a statistical analysis of selected data.

1. ETL Process in Big Data using PySpark

Create DataFrames to match production-ready tables from two big Amazon customer review datasets.

2. Statistical Analysis to find bias

Analyze whether reviews from Amazon's Vine program are trustworthy and free of bias.

Technologies Used:

• AWS RDS
• PySpark
• SQL
• ZEPL Notebook