hazunanafaru/behavior-personalization-pipeline

Build batch processing pipelines to do behavior analytics.

Behavior Personalization Pipeline with AWS

Description

Build batch processing pipelines to do behavior analytics with AWS.

Technology

Apache Airflow, AWS (S3, EMR, IAM, Redsfhit), Docker, and PySpark

Prerequisets

1. Docker
2. PostgreSQL
3. AWS Account
4. AWS CLI Configured

Usage

This code run in Windows 10 (Powershell 5.1) while the reference [1] run in linux/macOS (bash shell).

To setup the infrastructure needed, run infra_setup.ps1 with bucket_name as an argument.

Example : infra_setup.ps1 name-of-bucket

Make python environment using venv and install requirement packages

    # Make
    python -m venv bp-env
    
    # Activate
    .\bp-env\Scripts\Activate.ps1
    
    # Install
    pip install -r requirements.txt

To destroy infrastructure, run infra_teardown.ps1 with bucket_name as an argument.

Example : infra_teardown.ps1 name-of-bucket

References

[1] Data Engineering Project for Beginners - Batch edition

[2] Getting started with Amazon Redshift Spectrum

[3] Changing PowerShell's default output encoding to UTF-8

[4] authorize-security-group-ingress

[5] S3DistCp (s3-dist-cp)

[6] StepConfig

behavior-personalization-pipeline's Issues

Connection to AWS still missing region

The extra column is look like this

{region_name:$AWS_REGION}

while it should look like this

{"region_name":"$AWS_REGION"}

wait_for_movie_classification_transformation step is still have some error

wait_for_movie_classification_transformation sense EMR step in clean_movie_review.

Step Classify movie reviews cannot work properly. Need to check the EC2, hadoop, or spark logs directly in the master node.