cloudposse/atmos

👽 Terraform Orchestration Tool for DevOps. Keep environment configuration DRY with hierarchical imports of configurations, inheritance, and WAY more. Native support for Terraform and Helmfile.

Automated Terraform Management & Orchestration Software (ATMOS)

Atmos simplifies complex cloud architectures and DevOps workflows into intuitive CLI commands. Its strength in managing DRY configurations at scale is supported by robust design patterns, comprehensive documentation, and a passionate community, making it a versatile tool for both startups and enterprises. Atmos is extensible to accommodate any tooling, including enterprise-scale Terraform, and includes policy controls, vendoring, and GitOps capabilities out of the box. Everything is open source and free.

Screenshots

Example of running atmos to describe infrastructure.

---

Note

This project is part of Cloud Posse's comprehensive "SweetOps" approach towards DevOps.

Learn More

It's 100% Open Source and licensed under the APACHE2.

Introduction

Atmos centralizes the DevOps chain and cloud automation/orchestration into a robust command-line tool, streamlining environments and workflows into straightforward CLI commands. Leveraging advanced hierarchical configurations, it efficiently orchestrates both local and CI/CD pipeline tasks, optimizing infrastructure management for engineers and cloud architects alike. You can then run the CLI anywhere, such as locally or in CI/CD.

The Atmos project consists of a command-line tool, a Go library, and even a terraform provider. It provides numerous conventions to help you provision, manage, and orchestrate workflows across various toolchains. You can even access the configurations natively from within terraform using our terraform-provider-utils.

Cloud Posse uses this tool extensively for automating cloud infrastructure with Terraform and Kubernetes, but it can be used to automate any complex workflow.

Tip

Did you know?

By leveraging Atmos in conjunction with Cloud Posse's expertise in AWS, terraform blueprints, and our knowledgeable community, teams can achieve operational mastery and innovation faster, transforming their infrastructure management practices into a competitive advantage.

Core Features

Atmos streamlines Terraform orchestration, environment, and configuration management, offering developers and DevOps a set of powerful tools to tackle deployment challenges. Designed to be cloud agnostic, it enables you to operate consistently across various cloud platforms. These features boost efficiency, clarity, and control across various environments, making it an indispensable asset for managing complex infrastructures with confidence.

• Terminal UI Polished interface for easier interaction with Terraform, workflows, and commands.
• Native Terraform Support: Orchestration, backend generation, varfile generation, ensuring compatibility with vanilla Terraform.
• Stacks: Powerful abstraction layer defined in YAML for orchestrating and deploying components.
• Components: A generic abstraction for deployable units, such as Terraform "root" modules.
• Vendoring: Pulls dependencies from remote sources, supporting immutable infrastructure practices.
• Custom Commands: Extends Atmos's functionality, allowing integration of any command with stack configurations.
• Workflow Orchestration: Comprehensive support for managing the lifecycle of cloud infrastructure from initiation to maintenance.

See all features of Atmos.

Use Cases

Atmos has consistently demonstrated its effectiveness in addressing these key use-cases, showcasing its adaptability and strength in the cloud infrastructure and DevOps domains:

• Managing Large Multi-Account Cloud Environments: Suitable for organizations using multiple cloud accounts to separate different projects or stages of development.
• Cross-Platform Cloud Architectures: Ideal for businesses that need to manage configuration of services across AWS, GCP, Azure, etc., to build a cohesive system.
• Multi-Tenant Systems for SaaS: Perfect for SaaS companies looking to host multiple customers within a unified infrastructure. Simply define a baseline tenant configuration once, and then seamlessly onboard new tenants by reusing this baseline through pure configuration, bypassing the need for further code development.
• Efficient Multi-Region Deployments: Atmos facilitates streamlined multi-region deployments by enabling businesses to define baseline configurations with stacks and extend them across regions with DRY principles through imports and inheritance.
• Compliant Infrastructure for Regulated Industries: Atmos empowers DevOps and SecOps teams to create vetted configurations that comply with SOC2, HIPAA, HITRUST, PCI, and other regulatory standards. These configurations can then be efficiently shared and reused across the organization via service catalogs, component libraries, vendoring, and OPA policies, simplifying the process of achieving and maintaining rigorous compliance.
• Empowering Teams with Self-Service Infrastructure: Allows teams to manage their infrastructure needs independently, using predefined templates and policies.
• Streamlining Deployment with Service Catalogs, Landing Zones, and Blueprints: Provides ready-to-use templates and guidelines for setting up cloud environments quickly and consistently.

Tip

Don't see your use-case listed? Ask us in the #atmos Slack channel, or join us for "Office Hours" every week.

Moreover, atmos is not only a command-line interface for managing clouds and clusters. It provides many useful patterns and best practices, such as:

• Enforces a project structure convention, so everybody knows where to find things.
• Provides clear separation of configuration from code, so the same code is easily deployed to different regions, environments and stages
• It can be extended to include new features, commands, and workflows
• The commands have a clean, consistent and easy to understand syntax
• The CLI code is modular and self-documenting

Documentation

Find all documentation at: atmos.tools

✨ Contributing

This project is under active development, and we encourage contributions from our community. Many thanks to our outstanding contributors:

🐛 Bug Reports & Feature Requests

Please use the issue tracker to report any bugs or file feature requests.

💻 Developing

If you are interested in being a contributor and want to get involved in developing this project or help out with Cloud Posse's other projects, we would love to hear from you! Hit us up in Slack, in the #cloudposse channel.

In general, PRs are welcome. We follow the typical "fork-and-pull" Git workflow.

1. Review our Code of Conduct and Contributor Guidelines.
2. Fork the repo on GitHub
3. Clone the project to your own machine
4. Commit changes to your own branch
5. Push your work back up to your fork
6. Submit a Pull Request so that we can review your changes

NOTE: Be sure to merge the latest changes from "upstream" before making a pull request!

🌎 Slack Community

Join our Open Source Community on Slack. It's FREE for everyone! Our "SweetOps" community is where you get to talk with others who share a similar vision for how to rollout and manage infrastructure. This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build totally sweet infrastructure.

📰 Newsletter

Sign up for our newsletter and join 3,000+ DevOps engineers, CTOs, and founders who get insider access to the latest DevOps trends, so you can always stay in the know. Dropped straight into your Inbox every week — and usually a 5-minute read.

📆 Office Hours

Join us every Wednesday via Zoom for your weekly dose of insider DevOps trends, AWS news and Terraform insights, all sourced from our SweetOps community, plus a live Q&A that you can’t find anywhere else. It's FREE for everyone!

About

This project is maintained by Cloud Posse, LLC.

We are a DevOps Accelerator for funded startups and enterprises. Use our ready-to-go terraform architecture blueprints for AWS to get up and running quickly. We build it with you. You own everything. Your team wins. Plus, we stick around until you succeed.

Your team can operate like a pro today.

Ensure that your team succeeds by using our proven process and turnkey blueprints. Plus, we stick around until you succeed.

📚 See What's Included

• Reference Architecture. You'll get everything you need from the ground up built using 100% infrastructure as code.
• Deployment Strategy. You'll have a battle-tested deployment strategy using GitHub Actions that's automated and repeatable.
• Site Reliability Engineering. You'll have total visibility into your apps and microservices.
• Security Baseline. You'll have built-in governance with accountability and audit logs for all changes.
• GitOps. You'll be able to operate your infrastructure via Pull Requests.
• Training. You'll receive hands-on training so your team can operate what we build.
• Questions. You'll have a direct line of communication between our teams via a Shared Slack channel.
• Troubleshooting. You'll get help to triage when things aren't working.
• Code Reviews. You'll receive constructive feedback on Pull Requests.
• Bug Fixes. We'll rapidly work with you to fix any bugs in our projects.

License

Preamble to the Apache License, Version 2.0

Complete license is available in the LICENSE file.

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.

Trademarks

All other trademarks referenced herein are the property of their respective owners.

Copyright © 2017-2024 Cloud Posse, LLC

atmos's Issues

`atmos describe component` not working as expected

Describe the Bug

When I run atmos describe component s3 -s dev-internal.yaml against my project, it fails with the following error:

Found ENV var ATMOS_BASE_PATH=REDACTED

Could not find config for the component 's3' in the stack 'dev-internal.yaml'.
Check that all attributes in the stack name pattern '{environment}-{stage}' are defined in stack config files.
Are the component and stack names correct? Did you forget an import?

Expected Behavior

I expect to see a YAML output of my final component configuration.

Steps to Reproduce

Steps to reproduce the behavior:

1. Go to a TF project
2. Run `atmos describe component <COMPONENT_NAME> -s
3. See error

Screenshots

N/A

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

• OS: Debian Geodesic 0.149.2
• Version: Atmos v1.3.26

Additional Context

Stack file: stacks/dev-internal.yaml

import:
  - catalog/globals
  - catalog/dev-globals # environment is set here

settings:
  spacelift:
    labels:
      - deps:config/secrets/dev-internal-secrets.yml

vars:
  stage: internal

components:
  terraform:
    network:
      vars:
        vpc_cidr: 10.111.0.0/18

    s3:
      vars: {}

    cognito:
      vars: {}

	# ... 

Atmos config: atmos.yaml

# See full configuration options @ https://github.com/cloudposse/atmos/blob/master/examples/complete/atmos.yaml

components:
  terraform:
    base_path: "./components/terraform"
stacks:
  base_path: "./stacks"
  name_pattern: "{environment}-{stage}"

Add before-plan and after-plan and similar hooks for atmos terraform workflows

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Add optional workflow commands to run e.g. between plan and apply stages

For instance, it would be nice to add an opa hook to the verify the planfile if there is a opa/ directory within the component. This would help a lot for local atmos runs or if spacelift is not used.

Unable to access values for sensitive outputs

Describe the Feature

Add the ability to supply the output name when running terraform output.

Expected Behavior

Running terraform output **name_of_output** will display the value of the supplied output

Use Case

Useful for when a user needs to access the value for a sensitive output.

Alternatives Considered

Going into the module directory and running terraform output name_of_output

Stack file imports break on secondary relative paths

Found a bug? Maybe our Slack Community can help.

Describe the Bug

Atmos v0.16.0 seems to be different from 0.10.0 of the stack-config module in TF.

Atmos requires a full path for an import based on the stack location.

stacks/ue1-corp.yaml:

import:
  - eks/eks-defaults

That's fine in a stack, but importing from eks-defaults requires a relative path from the stack file:

stacks/eks/eks-defaults.yaml:

import:
  - eks/eks-gbl-defaults

Just doing a relative fails to find the file when using Atmos. So stacks/ue1-corp.yaml imports stacks/eks/eks-defaults.yaml imports stacks/eks/eks-gbl-defaults.yaml and that could go on and on in other scenarios.

When using the TF module, it respects the import as relative from the point of the file importing it. So getting Atmos to work will break the TF module:

Error: open ../../../stacks/eks/eks/eks-gbl-defaults.yaml: no such file or directory

Directory Structure

➜ tree .
.
├── eks
│   ├── eks-defaults.yaml
│   └── eks-gbl-defaults.yaml
├── gbl-globals.yaml
├── gbl-corp.yaml
├── globals.yaml
└── ue1-corp.yaml

Expected Behavior

Imports from within an import should respect the current folder location of the file so relative paths work regardless of what parent file (stack or otherwise) did the initial import.

stacks/eks/eks-defaults.yaml:

import:
  - eks-gbl-defaults

^ This should be perfectly fine

Steps to Reproduce

Steps to reproduce the behavior:

1. Create the files/directory structure
2. Run a atmos terraform plan without a relative path from the stacks/ dir

Screenshots

N/A

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

• OS: macOs
• Version 11.2.1

Additional Context

I believe that's it

Option to sanity check stack yaml

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Option to sanity check stack yaml

$ atmos validate stacks

This would be nice and it can be configured as a local pre-commit hook to prevent stack yaml issues from being committed.

Error that brought this up

+ atmos --config-dir ../../../stacks --terraform-dir ../ terraform write varfile acm --stack=uw2-dev
Error: Error in function call

  on /modules/utils/stack-config.variant line 71:
  (source code not available)

with opt.config-dir as "../../../stacks",
     opt.stack as "uw2-dev".

Call to function "import" failed: /modules/utils/stack-config.variant:12,13-20: Error in function
call; Call to function "import" failed: /modules/utils/stack-config.variant:10,43-50: Invalid
function argument; Invalid value for "path" parameter: no file exists at
../../../stacks/catalog/elasticache-redis/xyz.yaml;

Inspiration section

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Since this is a universal tool of tools, we have learned (and evolved) from a lot of tools like the following

• variant
• ahoy
• choria-io

And I think this should be shown in an inspiration section. We agreed to this in a previous PR #168 and I wrote this up so we do not forget about it.

Opt in or opt out analytics for how atmos is used

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Would be nice to set the following in atmos.yaml

analytics:
  # component names used
  components_enabled: true
  # if an error is found, hit an api endpoint that creates a ticket
  auto_crash_log_send_enabled: true
  # what commands are used the most often ?
  commands_enabled: true
  # etc

This may help mold atmos in the future to make it easier and extendable for everyone.

Option to log the output of all atmos commands

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

I'd love to see previous atmos command output (stdout and stderr) in some kind of ~/.atmos/logs/ folder.

$ atmos terraform plan eks --stack ue2-auto
... atmos output ...
$ atmos terraform apply eks --stack ue2-auto
... atmos output ...
$ atmos terraform output eks --stack ue2-auto
... atmos output ...

Then

$ ls -la ~/.atmos/logs/
YYMMDD-HHMMSS-terraform-plan-ue2-auto-eks.log
YYMMDD-HHMMSS-terraform-apply-ue2-auto-eks.log
YYMMDD-HHMMSS-terraform-output-ue2-auto-eks.log
etc

Filter out empty stacks from `describe stacks`

Describe the Feature

I would like describe stacks to show me only stacks that match the remaining conditions. Currently it always lists all the configured stacks, with stacks not matching the configuration simply being listed as empty. For example, I want

atmos describe stacks --components=vpc 

to show me only the stacks that have the vpc component configured.

Use Case

I want to know where a component is being used for numerous reasons, such as

• updating the configurations when a breaking change is made to the component
• finding a valid stack name to use as input to atmos for commands requiring a stack name (such as describe component)
• comparing currently configured stacks to the list of Terraform workspaces for the component to discover orphaned resources

I am not interested in the stacks which have no configuration for the component.

Include derived components when filtering on a base component

Describe the Feature

When running a command like

atmos describe stacks --components=vpc 

I want to see stacks which refer to the vpc component by deriving from it. For example, if I have

components:
  terraform:
    vpc-ipv6:
      metadata:
        component: vpc
      vars: {}

I want to see that in the output.

Use Case

Similar to #141, I want to know what stacks are using a component, for example to update them when I make a breaking change to the component.

Filter spacelift stacks based on account directories like `folder:stage/qa`, `folder:stage/dev`, etc

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Problem: I'd like to see all the stacks associated for a specific account

Solution: Filter spacelift stacks based on account directories like folder:qa, folder:dev, etc

For instance, my eks component is deployed in auto, corp, dev, qa, staging, and prod in both use2 and usw2 regions. The spacelift stacks for eks all have a label on it like this component/eks so I can search for this label and I can see all the eks components. Success!

However, there isn't a stage specific label. If I wanted to search for all spacelift stacks for the qa account, I could search qa but this would not be helpful since the string is too short and there is no way to have the cardinality that I want.

I suggest we have a new folder called stage and in it we can have the accounts that we desire.

README out of date now that golang version is live

Describe the Bug

README.md is fairly out of date as it still mentions variant2, developing your own CLI (I'm assuming this is no longer supported), and doesn't mention the now required configuration (.atmos.yaml and similar).

Expected Behavior

As a new engineer on-boarding to use atmos or an existing atmos user, I would expect that I can find out about the configuration options so I can avoid the error when the atmos configuration file is not found.

Steps to Reproduce

1. Read README.md

Screenshots

N/A

Environment (please complete the following information):

N/A

Additional Context

N/A

Terraform workspace is left behind after an `atmos terraform destroy`

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

When an atmos terraform destroy is run without a target, once all the resources are successfully destroyed, the workspace should also be removed.

https://www.terraform.io/cli/commands/workspace/delete

# switch to the default workspace
terraform workspace select default
# finally delete the old workspace
terraform workspace delete <stack>

Perhaps this can also be another input configuration var in atmos.yaml

Terraform import fails on missing option

Found a bug? Maybe our Slack Community can help.

Describe the Bug

The following error is observed when attempting a terraform import:

atmos terraform import account aws_organizations_organizational_unit.this["mgmt"] ou-k3gn-1no08klw --stack gbl-root 
{"account_email_format":"devops_%[email protected]","account_iam_user_access_to_billing":"DENY","aws_service_access_principals":["cloudtrail.amazonaws.com","ram.amazonaws.com"],"enabled_policy_types":["SERVICE_CONTROL_POLICY","TAG_POLICY"],"namespace":"crl","organization_config":{"accounts":[],"organization":{"service_control_policies":[]},"organizational_units":[{"accounts":[{"name":"dev","tags":{"eks":true}}],"name":"platform","service_control_policies":["DenyLeavingOrganization"]},{"accounts":[{"name":"audit","tags":{"eks":false}}],"name":"mgmt","service_control_policies":["DenyLeavingOrganization"]}],"root_account_stage_name":"root"},"organization_enabled":false,"region":"us-east-2"}
Initializing modules...
Downloading git::https://github.com/cloudposse/terraform-aws-service-control-policies.git?ref=tags/0.4.0 for accounts_service_control_policies...
- accounts_service_control_policies in .terraform/modules/accounts_service_control_policies
Downloading git::https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.21.0 for accounts_service_control_policies.this...
- accounts_service_control_policies.this in .terraform/modules/accounts_service_control_policies.this
Downloading git::https://github.com/cloudposse/terraform-aws-service-control-policies.git?ref=tags/0.4.0 for organization_service_control_policies...
- organization_service_control_policies in .terraform/modules/organization_service_control_policies
Downloading git::https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.21.0 for organization_service_control_policies.this...
- organization_service_control_policies.this in .terraform/modules/organization_service_control_policies.this
Downloading git::https://github.com/cloudposse/terraform-aws-service-control-policies.git?ref=tags/0.4.0 for organizational_units_service_control_policies...
- organizational_units_service_control_policies in .terraform/modules/organizational_units_service_control_policies
Downloading git::https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.21.0 for organizational_units_service_control_policies.this...
- organizational_units_service_control_policies.this in .terraform/modules/organizational_units_service_control_policies.this
Downloading git::https://github.com/cloudposse/terraform-yaml-config.git?ref=tags/0.1.0 for service_control_policy_statements_yaml_config...
- service_control_policy_statements_yaml_config in .terraform/modules/service_control_policy_statements_yaml_config
Downloading git::https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.21.0 for service_control_policy_statements_yaml_config.this...
- service_control_policy_statements_yaml_config.this in .terraform/modules/service_control_policy_statements_yaml_config.this
Downloading git::https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.21.0 for this...
- this in .terraform/modules/this

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/http versions matching ">= 2.0.0"...
- Finding hashicorp/local versions matching ">= 1.3.0"...
- Finding hashicorp/aws versions matching ">= 3.0.0"...
- Finding hashicorp/template versions matching ">= 2.0.0, >= 2.2.0"...
- Using hashicorp/template v2.2.0 from the shared cache directory
- Using hashicorp/http v2.1.0 from the shared cache directory
- Using hashicorp/local v2.1.0 from the shared cache directory
- Using hashicorp/aws v3.37.0 from the shared cache directory

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Workspace "gbl-root" doesn't exist.

You can create this workspace with the "new" subcommand.
Created and switched to workspace "gbl-root"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.
Error: Unsupported attribute

  on /modules/terraform/terraform-core.variant line 223:
  (source code not available)

This object does not have an attribute named "region".

Error: Unsupported attribute

  on /modules/terraform/terraform-core.variant line 223:
  (source code not available)

This object does not have an attribute named "region".

Error: 1 error occurred:
	* step "write provider override": job "terraform write override": config "override-contents": source 0: job "terraform provider override": /modules/terraform/terraform-core.variant:223,27-34: Unsupported attribute; This object does not have an attribute named "region"., and 1 other diagnostic(s)

This appears to be due to the missing option referenced here, which is also not available on the caller.

The provider override is deprecated, so perhaps this entire step is unneeded?

Expected Behavior

The import should apply cleanly.

Steps to Reproduce

1. Attempt to import an existing OU into the new account module
2. See error

Screenshots

N/A

Environment (please complete the following information):

N/A

Additional Context

N/A

Add support for `terraform refresh`

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

terraform refresh is missing from the available commands.

Expected Behavior

Allow use of terraform refresh.

Use Case

The `terraform refresh~ command is used to reconcile the state Terraform knows about (via its state file) with the real-world infrastructure. This can be used to detect any drift from the last-known state, and to update the state file.

https://www.terraform.io/docs/cli/commands/refresh.html

Describe Ideal Solution

It supports the same flow as terraform output.

Alternatives Considered

N/A

Additional Context

N/A

Panic when trying to get command help

Found a bug? Maybe our Slack Community can help.

Describe the Bug

When attempting to run atmos terraform --help I get a panic instead of the help.

Expected Behavior

I was expecting to see the command help

Steps to Reproduce

Steps to reproduce the behavior:

1. run atmos terraform --help
2. See error - panic: runtime error: index out of range [1] with length 1

Screenshots

If applicable, add screenshots or logs to help explain your problem.

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

• OS: Linux Mint
• Version: 20

Additional Context

The same issue happens with the helmfile command.

Extend Component to include Catalog

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

component only looks for components at the atmos components terraform base path, for example /atmos_root/components/terraform. It would be nice to be able to extend component to include components defined in the catalog.

Use Case

For example, say you have 1 component named s3-bucket and 2 implementations of that bucket named s3-logs and s3-scripts. You could have a catalog with 3 files:
catalog/s3-bucket/default.yaml:

components:
  terraform:
    s3-bucket:
      backend:
        s3:
          workspace_key_prefix: s3-bucket
      vars:
           ...

catalog/s3-bucket/s3-logs.yaml:

import:
  - catalog/s3-bucket/default

components:
  terraform:
    s3-logs:
      component: s3-bucket
      vars:
        example-variable: foo

catalog/s3-bucket/s3-scripts.yaml:

import:
  - catalog/s3-bucket/default

components:
  terraform:
    s3-scripts:
      component: s3-bucket
      vars:
        example-variable: bar

Then in a stack, you could create these different components as such:

import:
- catalog/s3-bucket/default
- catalog/s3-bucket/s3-logs
- catalog/s3-bucket/s3-scripts

components:
  terraform:
    s3-example:
      component: s3-bucket
      vars:
        bucket_name: "example-default"
    s3-logs-example:
      component: s3-logs
      vars:
        bucket_name: "example-logs"
    s3-scripts-example:
      component: s3-scripts
      vars:
        bucket_name: "example-scripts"

For now, that would return this error:
Component 's3-logs' does not exist in /atmos_root/components/terraform
and
Component 's3-scripts' does not exist in /atmos_root/components/terraform

Schema validation: For unrecognized keys, show warnings or throw an error

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

To the untrained eye, this may look good

components:
  terraform:
    eks:
      enabled: true
      availability_zones: [ "us-east-2a" ]

However, atmos will gladly use this config and pass it along to terraform but will omit the vars because the vars key is not used above.

To prevent the above, atmos could look for keys like settings, backend, and vars after components.terraform.xyz. Since enabled doesn't match any of those, it would be nice if atmos threw an error or showed a warning to describe the unrecognized inputs.

Describe Ideal Solution

Warning: unrecognized input `enabled` in `components.terraform.eks` in `stacks/catalog/eks.yaml`. Current location allows for the following keys: `settings`, `backend`, or `vars`.

or

Error: unrecognized input `enabled` in `components.terraform.eks` in `stacks/catalog/eks.yaml`. Current location allows for the following keys: `settings`, `backend`, or `vars`.

Atmos `version` not being embedded in `darwin_amd64` binary

Found a bug? Maybe our Slack Community can help.

Describe the Bug

The Atmos version is not being properly embedded on Darwin AMD64 releases.

Expected Behavior

The Atmos version should be embedded properly just as it is currently embedded in Linux.

Steps to Reproduce

Steps to reproduce the behavior:

1. Download latest darwin_amd64 release
2. Run atmos version
3. See 0.0.1

Screenshots

$ atmos version
0.0.1

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

• OS: Darwin (macOS Big Sur)
• Version 1.4.0

Additional Context

This implementation works fine on linux_amd64

$ atmos version
1.4.0

The implementation itself is quite clear. We use ldflags to override cmd.Version using the environment variable exposed by goreleaser:

atmos/.goreleaser.yml

Lines 7 to 27 in c9c63b0

	builds:
	- env:
	# goreleaser does not work with CGO, it could also complicate
	# usage by users in CI/CD systems like Terraform Cloud where
	# they are unable to install libraries.
	- CGO_ENABLED=0
	mod_timestamp: '{{ .CommitTimestamp }}'
	goos:
	- darwin
	- freebsd
	- windows
	- linux
	goarch:
	- amd64
	- '386'
	- arm
	- arm64
	binary: atmos
	ldflags:
	# Set `atmos` version to the GitHub release tag using Go `ldflags`
	- '-s -w -X "github.com/cloudposse/atmos/cmd.Version={{.Env.GORELEASER_CURRENT_TAG}}"'

Not sure why it's not working as expected.

components folder missing from examples

Found a bug? Maybe our Slack Community can help.

Describe the Bug

Getting started guide mentions a components folder in the example file. That folder does not exist. Running atmos terraform apply eks -s ue2-dev returns the following error

* step "terraform workspace": job "terraform workspace": job "terraform shell": job "shell": command "bash -c /usr/bin/terraform-0.13 workspace select ue2-dev || /usr/bin/terraform-0.13 workspace new ue2-dev" in "./components/terraform/eks": chdir ./components/terraform/eks: no such file or directory

Expected Behavior

A clear and concise description of what you expected to happen.

Steps to Reproduce

Steps to reproduce the behavior:

1. git clone [email protected]:cloudposse/atmos.git
2. cd atmos
3. cd example
4. make all
5. assume-role
6. atmos terraform plan eks --stack=ue2-dev
7. atmos terraform apply eks --stack=ue2-dev

Screenshots

If applicable, add screenshots or logs to help explain your problem.

Environment:

Anything that will help us triage the bug will help. Here are some ideas:

• OS: [e.g. Linux, OSX, WSL, etc]
• Version [e.g. 10.15]

Additional Context

Add any other context about the problem here.

Stale providers in a `.terraform.lock.hcl` file can cause issues

Found a bug? Maybe our Slack Community can help.

Describe the Bug

Sometimes old providers are cached in a .terraform.lock.hcl file and need to be manually removed to allow upgrading modules.

It would be nice to have an atmos.yaml setting to remove a component's .terraform.lock.hcl (without running a full clean) prior to running a terraform init. So if this setting was enabled, and a plan/deploy/apply was run, it would first remove the lock file, then run the init, and then the desired command.

or just always run terraform init -upgrade

Trim whitespace of input variable values

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

If the trailing whitespace is inputted, atmos will include that into terraform

components:
  terraform:
    elasticache-redis:
      vars:
        redis_clusters:
          my-redis:
            parameters:
              - name: activedefrag
                # there is a bunch of whitespace after "no"
                value: no        

It would be nice if all whitespace from input values could be trimmed when creating the tfvars file

Add a `version` command

Describe the Feature

atmos version should print out the release version of the atmos binary. If it could also print out the versions of included folders, that would be nice.

Use Case

Without a version command, it is virtually impossible to determine which version of atmos is installed.

Allow component.yaml arguments for `atmos vendor pull`

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Can we make the atmos vendor pull -c ecr take a path to put the component ?

e.g. for component

atmos vendor pull ecr \
  --path components/terraform \
  --repo-uri "github.com/cloudposse/terraform-aws-components.git//modules/ecr"

e.g. for stacks

atmos vendor pull ecr \
  --path stacks/catalog/ecr \
  --repo-uri "github.com/cloudposse/terraform-aws-components.git//stacks/catalog/ecr"

Official homebrew formula

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Allow installing from the official homebrew repo

brew install atmos

References

• Homebrew/homebrew-core#89482

Shell / Tab completion for atmos stacks and components

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Tab completion for atmos commands

Expected Behavior

$ atmos<tab>
helm helmfile help istioctl stack terraform vendor version workflow

$ atmos terraform<tab>
apply backend clean console deploy destroy import init output plan show workspace

$ atmos terraform plan<tab>
tfstate-backend aws-sso github-runners # etc

$ atmos terraform plan github-runners --stack<tab>
gbl-root gbl-prod gbl-staging

Use Case

I want the command and stacks to complete using tab instead of having to type it in each time

Describe Ideal Solution

It's implemented

Alternatives Considered

None

Additional Context

• https://stackoverflow.com/questions/5570795/how-does-bash-tab-completion-work
• https://github.com/perlpunk/shell-completions

A way to prefix component names using an inheritable metadata

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

# stacks/catalog/eks/defaults.yaml
components:
  terraform:
    eks/defaults:
      metadata:
        type: abstract
      settings:
        spacelift:
          workspace_enabled: true
      vars:
        enabled: true

# stacks/catalog/eks/blue/defaults.yaml
import:
  - stacks/catalog/eks/defaults

components:
  terraform:
    eks/blue/defaults:
      metadata:
        type: abstract
        component_prefix: eks/blue
        inherits:
          - eks/defaults

# stacks/catalog/eks/blue/defaults.yaml
import:
  - stacks/catalog/eks/blue/defaults

components:
  terraform:
    # resolve to eks/blue/cluster
    cluster:
      metadata:
        inherits:
          - eks/blue/defaults

    # resolve to eks/blue/reloader
    reloader:
      metadata:
        inherits:
          - eks/blue/defaults

Stack file import stops after any given file name

Found a bug? Maybe our Slack Community can help.

Describe the Bug

Stack file import stops looking for additional config files with similar names after one is found. Meaning that files cannot have the same name even if they are within different paths

Expected Behavior

All stack files should be imported

Steps to Reproduce

Steps to reproduce the behavior:

1. Create a stack file with the default stack directory, stacks/example-stack.yaml
2. Create a resource folder within the catalog of stacks, stacks/catalog/example-resource/
3. Create a stack file for that example resource with the same name as the first stack file, stacks/catalog/example-resource/example-stack.yaml
4. Run atmos for that stack, atmos terraform plan example-component -s example-stack

Whichever stack file is referenced first will be imported. The latter will be skipped

To resolve the issue, simply change the name of either stack file.

Screenshots

sanitized output, but this is what is returned for the bug:

Found the config file for the provided stack:
- catalog/example-stack/example-resource.yaml

Stack 'example-resource' does not exist

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

• OS: Geodesic
• Version 0.147.3

Improve import file extension handling

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

import:
  - ue1-globals
  - catalog/eks.yaml

Note the incidental .yaml here.

Error:

Error: Error in function call

  on /modules/utils/stack-config.variant line 71:
  (source code not available)

with opt.config-dir as "./stacks",
     opt.stack as "ue1-auto".

Call to function "import" failed: /modules/utils/stack-config.variant:12,13-20: Error in function
call; Call to function "import" failed: /modules/utils/stack-config.variant:10,43-50: Invalid
function argument; Invalid value for "path" parameter: no file exists at
stacks/catalog/eks.yaml.yaml; this function works only with files that are distributed as part of
the configuration source code, so if this file will be created by a resource in this configuration
you must instead obtain this result from an attribute of that resource...

Error: Error in function call

  on /modules/utils/stack-config.variant line 71:
  (source code not available)

with opt.config-dir as "./stacks",
     opt.stack as "ue1-auto".

Call to function "import" failed: /modules/utils/stack-config.variant:12,13-20: Error in function
call; Call to function "import" failed: /modules/utils/stack-config.variant:10,43-50: Invalid
function argument; Invalid value for "path" parameter: no file exists at
stacks/catalog/eks.yaml.yaml; this function works only with files that are distributed as part of
the configuration source code, so if this file will be created by a resource in this configuration
you must instead obtain this result from an attribute of that resource...

Error: 1 error occurred:
        * step "write varfile": job "terraform write varfile": config "config-component": source 0: job "stack config": /modules/utils/stack-config.variant:71,13-20: Error in function call; Call to function "import" failed: /modules/utils/stack-config.variant:12,13-20: Error in function call; Call to function "import" failed: /modules/utils/stack-config.variant:10,43-50: Invalid function argument; Invalid value for "path" parameter: no file exists at stacks/catalog/eks.yaml.yaml; this function works only with files that are distributed as part of the configuration source code, so if this file will be created by a resource in this configuration you must instead obtain this result from an attribute of that resource...

Expected Behavior

A clean error/warning or just allow the extension.

Use Case

It seems reasonable some people may choose to add an extension; ex: yaml vs yml.

Describe Ideal Solution

Allow an extension or default to yaml.

Alternatives Considered

Require no extension, but provide a clean error.

Additional Context

Throw an error if missing terraform backend

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

It would be nice if atmos would throw an error if a backend is missing from the terraform component. This would only apply for atmos 1.x if autogenerating the backend is set to false.

If atmos detects no backend, it could even suggest enabling this or at the very least show a warning that a backend is missing

fyi the existing setting in atmos.yaml

    # Can also be set using `ATMOS_COMPONENTS_TERRAFORM_AUTO_GENERATE_BACKEND_FILE` ENV var, or `--auto-generate-backend-file` command-line argument
    auto_generate_backend_file: false

Support command and yaml aliases

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

atmos terraform output
atmos terraform outputs

in the stack yaml

import:
imports:

Increase Parsing Error Verbosity to Include YAML Filename

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Provide additional details on atmos errors resulting from stack config parsing issues.

Currently if any stack config yaml is malformed you will get an error indicating a failing line number but no indication which file is causing issues. The only current method I have found to get this is to run a script or loop through something like jq/yq until you get an error.

atmos terraform plan config-bucket --stack core-use2-audit
Found ENV var ATMOS_BASE ...
...
yaml: line 68: did not find expected key

Expected Behavior

It would be desirable to have additional verbosity in the error that indicated the failing file.

atmos terraform plan config-bucket --stack core-use2-audit
Found ENV var ATMOS_BASE ...
...
iam-primary-roles.yaml: line 68: did not find expected key

Use Case

Decrease troubleshooting time

Describe Ideal Solution

A clear and concise description of what you want to happen. If you don't know, that's okay.

Alternatives Considered

Explain what alternative solutions or features you've considered.

Script, jq, yq, etc

Additional Context

Add any other context or screenshots about the feature request here.

Improve `terraform import` functionality

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

In order to import resources, the var file needs to be supplied.

atmos terraform import eks -s use2-corp \
  -var-file /components/terraform/eks/use2-corp-eks.terraform.tfvars.json \
  "module.eks_cluster.aws_eks_cluster.default[0]" \
  acme-use2-corp-eks-cluster

It would be nice if the var file can be automatically generated and supplied by atmos like this

atmos terraform import eks -s use2-corp \
  "module.eks_cluster.aws_eks_cluster.default[0]" \
  acme-use2-corp-eks-cluster

otherwise atmos isn't adding anything since I could do the above import by cd'ing directly into the component directory and run the terraform import manually with the supplied import var.

cd components/terraform/eks
terraform import \
  -var-file /components/terraform/eks/use2-corp-eks.terraform.tfvars.json \
  "module.eks_cluster.aws_eks_cluster.default[0]" \
  acme-use2-corp-eks-cluster

Atmos fails when using the s3 backend if not logged in

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Problem: Atmos fails when using the s3 backend if not logged in

It would be nice if atmos could fail early if aws sts get-caller-identity returns nothing... or whatever golang sdk can quickly see if we currently have a logged-in session within aws.

This could also be a new input param in atmos.yaml.

`terraform clean` should remove the lock file

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

atmos terraform clean [project] does not remove the .terraform.lock.hcl file.

Expected Behavior

atmos terraform clean [project] should remove the .terraform.lock.hcl file.

Use Case

Cleaning a project

Describe Ideal Solution

Just works

Alternatives Considered

Manually deleting the file

Additional Context

TF 0.14 introduced the lock file.

Improve error handling for invalid components

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Running Atmos with an invalid component throws an error that is unclear to users.

atmos terraform plan accounts -s gbl-root

panic: writing ./components/terraform/accounts/gbl-root-accounts.terraform.tfvars.json: open ./components/terraform/accounts/gbl-root-accounts.terraform.tfvars.json: no such file or directory

Expected Behavior

A clear error showing the problem.

Use Case

Running a component via the CLI.

Describe Ideal Solution

Show a clean error like "Missing component: accounts" or something much more readable.

Alternatives Considered

N/A

Additional Context

N/A

Expand Documentation on Use of (optional) `tenant` Label

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

The README currently does not explain the optional tenant label and how stacks.name_pattern can be configured to either use it or not use it.

Expected Behavior

The README should clearly and concisely explain how to configure atmos to use the optional tenant label, and how to write create the stack YAML configs accordingly.

Use Case

Users attempting to use atmos.yaml in its default form will be forced to use the tenant label, and no documentation indicates that this is an optional label and how to configure atmos to not use this label.

Describe Ideal Solution

Expand the documentation as described in Expected Behaviour.

Alternatives Considered

N/A

Additional Context

https://sweetops.slack.com/archives/CQCDCLA1M/p1643728355414279

Add `--args` support to `terraform init` command

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

--args isn't supported for atmos terraform init which prevents an -upgrade arg to update providers in Terraform 0.14.x+.

atmos terraform init --help

Run `terraform init`

Usage:
  atmos terraform init [COMPONENT] [flags]

Flags:
      --command string   Command to execute, e.g. 'terraform', or path to the command, e.g. '/usr/local/terraform/0.13/bin/terraform'
  -h, --help             help for init
  -i, --interactive      Interactive
  -s, --stack string     Stack

Global Flags:
      --cluster-name-pattern string       Cluster name pattern
      --config-dir string                 Stacks config directory
      --dry-run                           Disable execution of any commands and echo the command instead
      --helm-aws-profile-pattern string   AWS profile pattern for helm and helmfile
      --helmfile-dir string               Helmfile components directory
      --kubeconfig-path string            folder to save kubeconfig
      --terraform-dir string              Terraform components directory
      --vendor-config-path string         Path to the vendor configuration file

atmos terraform plan --help

Run 'terraform plan'

Usage:
  atmos terraform plan [COMPONENT] [flags]

Flags:
      --args terraform plan   A string of arguments to supply to terraform plan
      --command string        Command to execute, e.g. 'terraform', or path to the command, e.g. '/usr/local/terraform/0.13/bin/terraform'
  -h, --help                  help for plan
  -i, --interactive           Interactive
  -s, --stack string          Stack

Global Flags:
      --cluster-name-pattern string       Cluster name pattern
      --config-dir string                 Stacks config directory
      --dry-run                           Disable execution of any commands and echo the command instead
      --helm-aws-profile-pattern string   AWS profile pattern for helm and helmfile
      --helmfile-dir string               Helmfile components directory
      --kubeconfig-path string            folder to save kubeconfig
      --terraform-dir string              Terraform components directory
      --vendor-config-path string         Path to the vendor configuration file

Expected Behavior

atmos terraform init component -s stack --args='-upgrade' works to allow component upgrades.

Use Case

.terraform.lock.hcl file exists with a lower version provided in terraform > required_providers (hcl).

terraform {
  required_version = "~> 0.14.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.32"
    }
    external = {
      source  = "hashicorp/external"
      version = "~> 2.0"
    }
    http = {
      source  = "hashicorp/http"
      version = "~> 2.0"
    }
    local = {
      source  = "hashicorp/local"
      version = "~> 2.0"
    }
    template = {
      source  = "hashicorp/template"
      version = "~> 2.2"
    }
    utils = {
      source  = "cloudposse/utils"
      version = "~> 0.3"
    }
  }
}

Running init without telling TF to upgrade, will trigger this error:

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
locked provider registry.terraform.io/hashicorp/aws 3.27.0 does not match
configured version constraint ~> 3.32; must use terraform init -upgrade to
allow selection of new versions

Describe Ideal Solution

init supports --args.

Alternatives Considered

cd to the dir and run the tf cli manually

Additional Context

https://www.terraform.io/docs/cli/commands/init.html#plugin-installation

Logging/debug information should be sent to stderr, not stdout

Describe the Bug

When running a command like

atmos describe stacks --format=json | jq 

diagnostic/logging information such as

Found ENV var ATMOS_BASE_PATH=/localhost/devsrc

is output to stdout, breaking the pipeline by sending non-JSON to jq.

Although this is a particularly damaging issue for describe stacks --format=json, it is a general problem throughout atmos that prevents being able to parse the output of atoms commands.

Expected Behavior

Logging and diagnostic information should go to stderr.

Alternatively, you can provide a flag or atmos.yaml setting to suppress such output altogether, or a command-line option to send it to a separate file (e.g. /dev/fd/2 or /dev/null).

Environment:

atmos version 1.4.7

"can't find a shell to execute" when executing `atmos terraform shell [...]`

Found a bug? Maybe our Slack Community can help.

Describe the Bug

Executing the atmos terraform shell command fails with

can't find a shell to execute

Expected Behavior

A shell would open

Steps to Reproduce

Steps to reproduce the behavior:

1. Run Geodesic
2. Install Atmos v1.3.26
3. Go to a terraform directory
4. Execute a shell for a component: atmos terraform shell system -s gbl-root
5. See that atmos command results in can't find a shell to execute

Screenshots

N/A

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

• OS: Linux
• Version: Geodesic 0.149.1 based on Debian GNU/Linux 10 (buster) (10.11)

Additional Context

 ✗ . [none] (HOST) oc-terraform ⨠ atmos terraform shell system -s gbl-root

Variables for the component 'system' in the stack 'gbl-root':

REDACTED

Writing the variables to file:
components/terraform/system/gbl-root-system.terraform.tfvars.json

Executing command:
/usr/bin/terraform init
Initializing modules...

Initializing the backend...

Initializing provider plugins...
REDACTED

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Command info:
Terraform binary: terraform
Terraform command: shell
Arguments and flags: []
Component: system
Stack: gbl-root
Working dir: components/terraform/system

Executing command:
/usr/bin/terraform workspace select gbl-root
Switched to workspace "gbl-root".

Starting a new interactive shell where you can execute all native Terraform commands (type 'exit' to go back)
Component: system
Stack: gbl-root
Working directory: components/terraform/system
Terraform workspace: gbl-root

Setting the ENV vars in the shell:
TF_CLI_ARGS_plan=-var-file=gbl-root-system.terraform.tfvars.json
TF_CLI_ARGS_apply=-var-file=gbl-root-system.terraform.tfvars.json
TF_CLI_ARGS_refresh=-var-file=gbl-root-system.terraform.tfvars.json
TF_CLI_ARGS_import=-var-file=gbl-root-system.terraform.tfvars.json
TF_CLI_ARGS_destroy=-var-file=gbl-root-system.terraform.tfvars.json

can't find a shell to execute

-> Run 'assume-role' to login to AWS with aws-vault
 ⧉  OC Toolbox
 ✗ . [none] (HOST) oc-terraform ⨠ echo $SHELL
/bin/bash

Terraform version:

 ✗ . [none] (HOST) oc-terraform ⨠ terraform version
Terraform v1.0.0

Here is the error code in question that is getting executed, but I'm not sure why... I'll continue to debug and see if I can post a fix, but if not then I wanted to surface for others.

atmos/internal/exec/utils.go

Lines 655 to 657 in 4ba3e8c

	if len(executableName) == 0 {
	return errors.New("can't find a shell to execute")
	}

Copying context.tf and other mixins is too much copy paste

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Copying context.tf and other mixins is too much copy paste

I'd like a way to setup a directory of common terraform files (mixins) that should be copied into components when running an atmos terraform command.

mixins/context.tf
mixins/introspection.tf

If this was implemented, it would have to copy to components/terraform/**/* due to also requiring mixins in submodules

This will prevent having to copy these files in manually and will allow easier updating of these files.

Note that these files should still be committed in each component so that vanilla terraform command can still run without atmos

Cannot reference terraform backend generate command in workflow

Hello,

When using terraform backend generate command in workflow it's not working due the code above. Any command with more than 2 arguments will not work in workflow

atmos/atmos/modules/workflow/workflow.variant

Lines 76 to 79 in f440764

	variable "job-name" {
	type = string
	value = format("%s %s", var.job-split[0], var.job-split[1])
	}

Our workflow config file

region-deploy-all:
    description: Deploy terraform project and helmfiles for a stack (provided as a command-line argument)
    steps:
      - job: terraform backend generate
      - job: terraform deploy dns-delegated
      - job: terraform deploy vpc

Perhaps need to add a specific option for commands who don't require any arguments

Thanks

Atmos errors are written to stdout instead of stderr

Found a bug? Maybe our Slack Community can help.

Describe the Bug

atmos terraform write varfile reloader --stack=mdev-usw2-sbx01 -f spacelift.auto.tfvars.json > stdout 2> stderr

⨠ cat stderr
⨠ cat stdout

Could not find config for the component 'reloader' in the stack 'mdev-usw2-sbx01'.
Check that all attributes in the stack name pattern '{tenant}-{environment}-{stage}' are defined in the stack config files.
Are the component and stack names correct? Did you forget an import?

Switching between atmos versions (asdf plugin)

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Sometimes I need to switch between atmos versions in order to debug something or perhaps a client uses version X and another client uses version Y. I usually have to run another go install of a specific atmos release or use apt install atmos="<version>-*" from geodesic.

It would be nice to use a tool like asdf with an atmos plugin

https://asdf-vm.com/
https://github.com/asdf-vm/asdf
https://github.com/asdf-vm/asdf-plugins

Another advantage of supporting asdf is that we can pin versions of atmos using the asdf .tool-versions file for users that want to use atmos within geodesic and from outside.

`atmos terraform shell` should be a top level command and not a subcommand

Describe the Feature

atmos terraform shell should be a top level command parsed by Cobra and not a pass-through so it can have proper help documentation.

Expected Behavior

When I run atmos terraform shell --help, I get help docs for the shell command.

Use Case

Allow the shell command to better documented and useful to atmos users.

Describe Ideal Solution

Similar to atmos terraform generate, shell will be documentable via cobra docs and allow for better usage from atmos users.

Alternatives Considered

N/A

Additional Context

I would like to start contributing, so I can implement this if we agree to the approach, but figured I should bring it up for discussion before throwing up a PR.

Dryrun: See commands before running them

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

At the moment, if I ran something like this

atmos terraform plan eks --stack use2-sbx01

It would run something like this

cd components/terraform/eks
atmos terraform write varfile eks-example/eks --stack mdev-use2-sbx01
terraform init
terraform workspace select use2-sbx01
terraform plan

It would be ideal if we could see the above printed out when we run something like this

atmos --dryrun terraform plan eks --stack use2-sbx01
atmos --dry-run terraform plan eks --stack use2-sbx01
atmos -d terraform plan eks --stack use2-sbx01

Add support for `terraform graph`

Describe the Feature

Implement the graph command in atmos

https://www.terraform.io/docs/cli/commands/graph.html

Expected Behavior

Should do exactly what the terraform command does

Use Case

For small terraform stacks, terraform graph could be a nice way to make sure your architecture diagrams stay in sync with your infrastructure.

Unmarshalling of atmos.yaml not working as expected

Found a bug? Maybe our Slack Community can help.

Describe the Bug

I have been trying to get the stack.name_pattern to read from the atmos.yaml file to no avail. The pattern only works if I use an environment variable.

Expected Behavior

I expect that the value I set in atmos.yaml under the components.stacks.name_pattern key to be used to find the stack

Steps to Reproduce

Steps to reproduce the behavior:

1. Set stacks.name_pattern to anything
2. Run atmos terraform plan <component> -s <stack> I get the following message must be provided in 'stacks.name_pattern' config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable
3. Set environment variable export ATMOS_STACKS_NAME_PATTERN="{tenant}-{environment}-{stage}". This seem to work.

Screenshots

If applicable, add screenshots or logs to help explain your problem.

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

• OS: Linux Mint 20.2 & Docker (debian bullseye)

Additional context

atmos.yaml

components:
  terraform:
    base_path: "./components/terraform"
    apply_auto_approve: false
    deploy_run_init: true
    auto_generate_backend_file: false
  helmfile:
    base_path: "./components/helmfile"
    kubeconfig_path: "/dev/shm"
    helm_aws_profile_pattern: "{namespace}-{tenant}-gbl-{stage}-helm"
    cluster_name_pattern: "{namespace}-{tenant}-{environment}-{stage}-eks-cluster"

  stacks:
    base_path: "./stacks"
    included_paths:
      - "**/*"
    excluded_paths:
      - "globals/**/*"
      - "catalog/**/*"
      - "**/*globals*"
    name_pattern: "{tenant}-{environment}-{stage}"

  logs:
    verbose: true
    colors: true

Show components deployed per stack

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

It would be nice to see what components are in the stack

Perhaps something like describe component but for stacks.

atmos describe stack --stack ue2-corp

Spacelift triggers on every change resulting in many no-change plans

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Describe the Feature

Problem: Spacelift triggers on every root stack change

Option 1: commit all the spacelift stacks using stacks_generated/ and keep rego proposed/tracked policy

I'd like a way for atmos to read the atmos.yaml configuration's stacks.base_path e.g. ./stacks and generate a new file for each component of its full stack yaml in a ./stacks_generated directory which is then committed into the repo but never modified manually. The files could be generated on a pre-commit hook and a github action.

If Spacelift's rego policy then looked at ./stacks_generated instead of ./stacks, it would then only trigger on component stacks that were added/modified instead of their root stacks. This would substantially reduce the number of Spacelift runs.

This functionality already exists via the describe subcommand

atmos describe component vpc --stack mdev-use2-sbx01

Pros

• It would work

Cons

• Nobody would enjoy dealing with auto committed files
• The bot that auto committed would require write access
• The repo settings would need to be loosened

Option 2: Use spacelift_run instead of rego proposed/tracked policy ?

Throw out the projects_affected rego block and use the spacelift_run resource to trigger a stack if the tfvars (mounted file)'s hash has changed.

Pros

• It would work

Cons

• May lose out on some nice features with rego

References

• https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation/blob/d08afd415a1814d1d64429371633640e60fe523d/catalog/policies/git_push.tracked-run.rego#L75-L85