funnyleone/big-data-systems-and-data-analytics-project

This repository includes all the source codes that I used for my Big Data systems and Data Analytics final project.

Analysis on NYC Revenue

I worked on this project primarily to demonstrate my proficiency in variety of Big Data concentrating programming languages and strong in-depth understanding of Big Data Systems and Data Analytics. Rather doing some usual projects, I was much intersted in providing realtime solutions. So I wanted to implement my understanding into real life needs, which eventually led me to work on Analyzing the New York City's revenue using a large variety of Open Datasets.

Implementation available at <link> : http://namenode.abulbro.com:50070/

Software and Frameworks used

• Apache Hadoop (MapReduce programming Model)
• Apache Spark
• Apache Hive
• Apache Pig
• Apache HTTP Server
• Google BigQuery
• Google Compute Engine
• Stackdriver
• AWS EC2 Instance( To setup Node Cluster)
• AWS S3 Bucket (To store the Dataset)
• AWS VPC
• AWS Cloudwatch
• Google Cloud Storage (To store the Dataset)
• Tableau (For Data Cleaning and Data Visualization)

Why this project?

New York City has always been the framed picture in terms of quality and wealthy lifestyle. Though being the small state in United States, the Cost of living still ranks third most expensive state. Comparing to Connecticut, New York is 65% more expensive. Forty out of Fifty people would love to settle themselves in the city of New York irrespective of the expenses. But they have no idea about the real living in New York, in fact, the people who already live in NYC have very few idea about expenses in NYC. Also, NYC has a top-notch finance department for themselves and they know how to make money to run the city. To be specific, they know how to make miscellaneous money out of people to run the city. We have thoroughly worked on variety of datasets from Department of Finance and various open information that is available to public. After our deep research, we could attain at a point that NYC is making approximately $760M a year with Parking Violation.

Folder name	Description
/MapReducer	This folder contains the source code of the implementation of MapReduce module on the Hadoop Cluster.
/Other	This folder has the source code for the ticket monitoring application that was designed to send an email alert if the parking ticket has been issued.
/Pig	Pig implementation goes here.
/References	This folder includes all the reference files used to develop this project.
/Spark	Spark implementation goes here
/mysql-to-google-bigquery-master	This folder includes MySQL and BigQuery connector codes