karmaniverous/metastructure

Config-driven, enterprise-grade, open-source application infrastructure. DRY as a bone!

ARE YOU AN AWS INFRASTRUCTURE SPECIALIST? Please consider collaborating with me on Metastructure!

I'm an Engineer

Stereotypes exist for a reason.

My Story

I landed my first development contract in high school in 1985. I enlisted in the U.S. Marine Corps after high school, became a Navy Surface Warfare Officer in 1995, and hung up my uniform in 2000. Along the way I earned a degree in Weapons & Systems Engineering from the U.S. Naval Academy.

After my military service I helped found a number of technology startups, always in a product development role. In 2008 I landed my first "real" job, writing software to facilitate order entry & execution on the floor of the Chicago Board Options Exchange. In 2011 I founded a company that built data management systems for big financial institutions, including Northwestern Mutual, Invesco, and Bridgewater Associates.

In 2019 I left institutional finance and moved to Bali to chase more interesting problems, like:

• VeteranCrowd Rewards is a FinTech startp that aims to turn the entire loyalty industry upside down. I am the project's architect & lead dev, and I am VERY proud of the patterns that have come out of this work.

• metastructure is one of these patterns: a command-line tool that solves the trouble with Terraform.

• get-dotenv is the dotenv-powered automation tool that allows me to perform high-quality releases on 16 API projects, a front end, and over a dozen supporting projects, every two weeks, SINGLE-HANDEDLY. I post around 10k commits a year thanks to this tool. Check it out!

See my repos for lots more useful tools & templates!

My Rules

See Karmic Rules for Writing Pretty Good Code. They're a work in progress. That's the best kind.

My Goodies

If you want to get a process right, template it! See my pinned repos below for some great project starting points.

Don't repeat yourself! I'm growing a library of handy solutions to tricky problems. See the list here.

How Can I Serve You?

Do you have a great idea for a technology product? There's a HUGE gap between a great idea and profitable product, and it helps to have a partner who has navigated that gap before!

I'll help you work through:

CONCEPT: What is the thing you want to build? What is it NOT?

ARCHITECTURE & DESIGN: What are its parts? What do they look like? What are they made of? How do they fit together, and exactly what do they do?

IMPLEMENTATION: How will you build it? How much will it cost? How long will it take... and how early can you begin to recoup your investment?

The most efficient way to do just about anything is to do it RIGHT. From concept to implementation, strong engineering process is my superpower.

So You're A Consultant?

Nah. Consultants are glorified salesmen. I MAKE things, and I already have a job I love. But this ☝️ is exactly how I found it! I love helping people build cool things, so drop me a line and maybe I can help you move the ball forward a notch.

Contact Me

👈 Use one of the links over there!

metastructure's Issues

Optionally direct config updates to local config override file

This is useful when conducting template development to keep test values out of the repo.

Convert SSO policy to object

Right now sso.policies is a simple Record<string, string> with the policy key on once side and the policy name on the other.

This leaves no room to specify the policy description. Instead, the type should be Record<string, Record<string, unknown>>.

Validate aws_region against aws organizations supported regions

See this reference.

Extend lint & lint:fix scripts to Terraform linting & formatting

Metastructure attempts to switch workspaces before terraform init

This is often not permitted & shouldn't happen.

execa circular dependency warnings on build

src/index.ts → dist/index.mjs...
(!) Circular dependencies
node_modules/execa/lib/ipc/send.js -> node_modules/execa/lib/ipc/outgoing.js -> node_modules/execa/lib/ipc/strict.js -> node_modules/execa/lib/ipc/send.js  
node_modules/execa/lib/ipc/outgoing.js -> node_modules/execa/lib/ipc/strict.js -> node_modules/execa/lib/ipc/forward.js -> node_modules/execa/lib/ipc/incoming.js -> node_modules/execa/lib/ipc/outgoing.js
node_modules/execa/lib/ipc/strict.js -> node_modules/execa/lib/ipc/forward.js -> node_modules/execa/lib/ipc/incoming.js -> node_modules/execa/lib/ipc/strict.js
node_modules/execa/lib/ipc/graceful.js -> node_modules/execa/lib/ipc/send.js -> node_modules/execa/lib/ipc/outgoing.js -> node_modules/execa/lib/ipc/strict.js -> node_modules/execa/lib/ipc/forward.js -> node_modules/execa/lib/ipc/incoming.js -> node_modules/execa/lib/ipc/graceful.js
node_modules/execa/lib/ipc/outgoing.js -> node_modules/execa/lib/ipc/strict.js -> node_modules/execa/lib/ipc/outgoing.js
created dist/index.mjs in 4s

Make license-check-and-add optional

Unable to destroy account with attached SSO policies

Adding action: detach to an account with an attached SSO policy produces the following error:

│ Error: deleting IAM Policy (arn:aws:iam::205930623657:policy/SSOTerraformStateWriter): operation error IAM: DeletePolicy, https response error StatusCode: 409, RequestID: 1d5681bf-eb04-46d4-93f8-c18c60d014ac, DeleteConflict: Cannot delete a policy attached to entities.

On examination, the account has been removed from sso.reference.group_account_permission_set_policies, so it no longer shows up as a dependency of the related aws_ssoadmin_customer_managed_policy_attachment resource, for example here:

###############################################################################
# Add policy SSOTerraformStateWriter to permission set TerraformAdmin.
###############################################################################
resource "aws_ssoadmin_customer_managed_policy_attachment" "terraform_admin_sso_terraform_state_writer" {
  customer_managed_policy_reference {
    name = "SSOTerraformStateWriter"
  }
  instance_arn       = local.sso_arn
  permission_set_arn = aws_ssoadmin_permission_set.terraform_admin.arn
  depends_on = [
    aws_iam_policy.dev_sso_terraform_state_writer,
    aws_iam_policy.log_archive_sso_terraform_state_writer,
    aws_iam_policy.master_sso_terraform_state_writer,
    aws_iam_policy.prod_sso_terraform_state_writer,
    aws_iam_policy.test_sso_terraform_state_writer,
    aws_iam_policy.shared_services_sso_terraform_state_writer
  ]
}

It may be that excluding actioned accounts from sso.reference.group_account_permission_set_policies was the wrong choice. Investigating...

Release to prod script

Iterate through all repos & create a PR from latest release to main branch. Add specified PR attributes, including:

• tags
• reviewer
• assignee
• others?

Make SSO policies detachable

Some changes to SSO policies require detaching & reattaching all SSO policies, which is very impactful (see documentation.

Make individual policies detachable to limit the blast area.

Support local config override

Allow the creation of a local file (e.g. metastructure.local.yml that overrides selected config keys prior to processing.

Remove strict constraint from config objects

It should be easy to add new keys for consumption in templates.

Remove currently configured keys UNLESS they participate in Metastructure logic.

Add handlebars partials to config

• Create a root-level partials object where keys are partial names & values are paths to partial templates.

• Create a workspace-level partials object that overrides these.

• In generateWorkspaces.ts merge these, load files & register partials before iterating through generators.

• In template repo, encapsulate do-not-edit header into a partial & refactor templates.

• Update wiki docs.

Add support for SSO users

Need an sso.users config section. Users should have the same id & action logic at work elsewhere and a groups array.

Support driving provider & backend shared config file path from project config

Right now the template repo provider & backend templates specify the shared config file location like this:

  {{#if root.cli_params.permission_set}}shared_config_files = ["./_shared_config.local"]{{/if}}

This is necessary because the shared_config template target is specified relative to the project root but the provider & backend templates need to specify the location relative to the workspace directory.

Probably the most natural way to handle this would be to expose the path library as a Handlebars helper and then add pkgDir to the config object.