Skittle's Projects
Safe blue/green deployment of Amazon SageMaker endpoints using AWS CodePipeline, CodeBuild and CodeDeploy.
An experimental open-source attempt to make GPT-4 fully autonomous.
A curated list of awesome Python frameworks, libraries, software and resources
Scrapes Company,CEO and other details from a Linkedin URL
Code for Data Engineer Zoomcamp course
Data warehousing date dimension and time dimension builders written in Python.
This dbt project is designed to transform Airbnb data, creating a series of models that can be used for analytical purposes. The project is organized into various directories and files, each serving a specific purpose in the ETL process..
A curated list of engineering blogs
This repository hosts a cloud-based data pipeline built on AWS. The pipeline is designed to scrape web data using a Python script, process the data, and store the results in a CSV file in an S3 bucket. The pipeline is triggered every day at midnight. It leverages several AWS services including EventBridge, Step Functions, Lambda, EC2, SSM
AWS Glue Codes (Sample Repo)
Great Expectations Data Quality Checks is a specialized repository designed to harness the capabilities of the great_expectations Python library. With a focus on ensuring data quality, this project provides robust tools and methodologies to validate and check data across various sources.
This script performs a series of data quality checks and generates a data profiling report. It utilizes Great Expectations for data validation and ydata_profiling for data profiling. The script reads data from a PostgreSQL database, applies various quality checks, and outputs validation results and a data profile HTML report.
Config files for my GitHub profile.
This repo consists of python lambdas for reading and writing an AWS kinesis stream.
This is a script to make create, update and ready AWS Lambda layers you have in your account.
Flattens a complex nested pyspark dataframe having a combination of struct and array types
This repository contains a Python script for migrating data from a PostgreSQL database to a MongoDB database. The script is designed to be robust and fault-tolerant, capable of handling large datasets.
Code snippets on how to use different features in Python 3
This repository contains python one-liners obtained from various sources.
Some of the Python One-Liners which I regularly use and feel saves a lot of time.
Python SQL Parser and Transpiler
Set of scripts and instructions for setting up a Microsoft Teams webhook pipeline with potential Azure functions integration. Includes automation scripts, database structures, and a comprehensive setup guide.