dimoschi/squared

Where the Meltano team runs Meltano! Get it???

Squared - Meltano for Meltano

This is the project the Meltano team uses to manage their Meltano instance.

This project is used to process data for MeltanoHub and for internal metrics as defined in the Handbook.

The primary maintainer is @pnadolny13. The primary slack channel for discussion is #meltano-squared.

Why use Meltano to manage data at Meltano?

"We don't just eat our own dog food, we embrace it! Meltano supercharges every part of our day-to-day work inside the data team."

Our data team keeps on pushing the edge on use cases for Meltano, and utilizes all of the advanced features, including deployment to Kubernetes and CI/CD integrations. (All of which you can find in this repository)

We love our own Meltano because it allowed us to start simple, provides DataOps out-of-the-box, integrates with everything, is easily customizable (we do a lot of customizations), it's a mature system and has first class ELT support (our most important use case).

To get started with Meltano yourself, head over to the Installation, or if you already have Meltano installed, to the Meltano Tutorial.

---

Repo Plugins

These are all of the plugins that are currently being using in this repo.

Extractors:

• Google Analytics
• Slack
• GitLab
• GitHub
• Snowflake
• Spreadsheets Anywhere
• MeltanoHub

Mappers:

• Meltano Map Transformer

Loaders:

• Snowflake
• Yaml

Transformers:

• dbt-snowflake - Refer to the Squared dbt README for more detail on how its used in this repo.

Orchestrators:

• Apache Airflow - Refer to the Squared Airflow README for more detail on how its used in this repo. Visit MeltanoHub for installation instructions.

Analyzers:

• Apache Superset - Refer to the Squared Superset README for more detail on how its used in this repo. Visit MeltanoHub for installation instructions.

Utilities:

• AWS CLI - Refer to the Squared General README for more detail on how its used in this repo.
• SqlFluff - Refer to the Squared General README for more detail on how its used in this repo. Visit MeltanoHub for installation instructions.
• Great Expectations - Refer to the Squared Great Expectations README for more detail on how its used in this repo. Visit MeltanoHub for installation instructions.
• Permifrost - Refer to the Squared Permifrost README for more detail on how its used in this repo.

Architectural Decision Records (ADR)

This Squared project makes use of ADR's to record architectural decisions roughly as described by Michael Nygard. In a nutshell, these are used to document architectural decisions and to provide a record of the decisions made by the team and contributors in regard to the Squared project's architecture. These are held in docs/adr. To propose or add a new ADR, its simplest to create a new entry using adr-tools, and then send a long a merge request for review.

brew install adr-tools
adr new <My ADR Name>