friendkak Goto Github PK

100-pandas-puzzles

100 data puzzles for pandas, ranging from short and simple to super tricky (60% complete)

adfdataflowdocs

Azure Data Factory Data Flow Documentation

airflow-etl-s3-to-redshift

ETL Pipeline to build datawarehouse from S3

airflow_aws_redshift_etl

Airflow pipeline, creating staging tables from Amazon S3, and loading into Redshift

airflow_songify_dataengineering

Use Airflow to create and update a Redshift Data Lake

amazon-redshift-data-warehouse-migration

In few hours, quickly learn how to effectively migrate oracle data warehouse workload to Amazon Redshift using AWS Schema Conversion Tool (SCT) and SCT data extractors. The lab will help them to use advance features of SCT like Virtual Partition while migrating the data warehouse to Amazon Redshift.

analytics-dailyupdate-monitoring

Monitor redshift daily job status

archive_aws-lambda-go-event

Type definitions for AWS Lambda event sources.

arthur-redshift-etl

ELT Code for your Data Warehouse

awesome-data-engineering

A curated list of data engineering tools for software developers

awesome-python

A curated list of awesome Python frameworks, libraries, software and resources

aws-alb-firehose

Application load balancer (ALB) to Kinesis Firehose lambda

aws-blog-redshift-datalake-etl-elt-patterns

Design best practices for building scalable ETL (Extract-Transform-Load) and ELT (Extract-Load-Transform) data processing pipelines using both primary and short-lived Amazon Redshift clusters and Data Lake on AWS

aws-cookbook

This repository contains the excersises and solutions from the book "O'Reilly Python and AWS Cookbook"

aws-data-wrangler

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

aws-etl-microservice-redshift-datalake

AWS Lambda based microservice built using AWS Glue API to migrate tables off of Amazon Redshift into AWS (S3) data lake.

aws-firehose-xformation-lambda

AWS Kinesis Firehose transformation Lambda

aws-glue

Glue ETL Job: MySQL to Redshift Incremental Load by Timestamp (Checkpointing via DynamoDB)

aws-glue-samples

AWS Glue code samples

aws-lambda-python-kinesis-aurora

aws-lambda-sct

aws-logging

Logging solution with Kinesis Firehose and Lambda

aws-nodejs-zillow

serverless aws nodejs zillow search

aws-redshift-etl

Extracting data from JSON files residing in AWS S3 buckets and loading them into an AWS redshift data warehouse for analysis

aws-redshift-matillion-workshop

Scripts, Instructions and Materials for AWS Redshift and Matillion ETL workshop

aws-s3-glue-redshift

automation of ETL

aws-sct-user-guide

The open source version of the AWS Schema Conversion Tool docs. You can submit feedback & requests for changes by submitting issues in this repo or by making proposed changes & submitting a pull request.

aws-serverless-architecture-using-kinesis-lambda-s3-and-twitter

Amazon Kinesis is a tool used for working with data in streams. It has a few features — Kinesis Firehose, Kinesis Analytics and Kinesis Streams and we will focus on creating and using a Kinesis firehose

aws-serverless-stream-ingest-transform-load

In this pattern, data records are ingested and then modified with simple transformations such as field level substitutions and data enrichment from relatively small and static data sets. A Lambda function is invoked by Kinesis Data Firehose as records are received by the delivery stream. the Lambda function then performs a simple processing job and returns the transformed or enriched records back to Kinesis Data Firehose. Firehose then buffers and sends the modified records to the configured destinations. A copy of the source records is saved in S3 as a backup and for future analysis.

aws-streaming-pipeline

Streaming pipeline based on AWS resources