wayfair-incubator/pyterraformer

pyterraformer's Issues

Generate .tf files when necessary during write back

Problem Statement

HumanSerializer.render_object assumes that the file exists within a Terraform namespace. For example, attempting to add an AWS bucket in the README example will try to write to the resources.tf file. This will fail with a FileNotFoundError if the workspace does not contain that file already.

Proposed Solution

Create the file during the "write back" phases of modifications

Alternatives Considered

Throw a more helpful error to describe what path was attempted, and what file should be created (even if blank).

Additional Context

When using a vanilla terraform workspace during the use of the README example, the process fails at the hs.render_object method due to the resources.tf file not existing.

🧹 Add Renovate

Description

👋 This repository is not currently configured for Renovate. This issue proposes the steps necessary to add Renovate to this project!

💡 Not familiar with Renovate, or are confused about what advantages it holds over GitHub's Dependabot? Learn more here!

Steps to Add

1. Review the guide for Adding Renovate to Existing Projects.
2. Assign yourself to this issue to signal to others that you intend to work on it. If you ultimately decide not to pursue this, please remember to un-assign yourself so that others may participate!
3. If the renovate[bot] account has already auto-filed a Configure Renovate PR against this repository, feel free to reference the proposed changes in your own Pull Request. If you are contributing to this project as a Hacktoberfest participant, you must file your own PR in order to get credit for your contribution!
4. You may find that the CI build for this project is failing for unrelated reasons. If you are not already a contributor to this project and don't feel comfortable attempting to fix the build, that's okay! There's plenty of other ways you can contribute to Wayfair's open source projects :) Feel free to consult the list of our other participating repositories here!
5. In order to catch potential JSON syntax errors or other mis-configurations, please add Renovate linting to this project's existing GitHub Workflow CI pipeline, or create a new one (eg. .github/workflows/lint.yml). See here for an example.
6. If this repository is currently configured to use GitHub's Dependabot, you must also deprecate support for Dependabot in order to avoid conflicts with Renovate. This is typically as simple as removing the .github/dependabot.yml file. See here for an example.

Checklist

• I have read the Adding Renovate to Existing Projects guide.
• I have assigned this issue to myself avoid duplicating efforts with other potential contributors.
• I have verified this repository does not already have Renovate configured (or proposed in an open PR by another contributor).
• If the renovate[bot] account has already auto-filed a Configure Renovate PR in this repository, I confirm that I will create a separate PR under my own GitHub account, using the initial PR as inspiration.
• I confirm that I have added Renovate linting to this project's existing CI pipeline, or have created a new linting workflow if one doesn't already exist.
• If this repository is currently configured to use GitHub's Dependabot, my PR will also deprecate support for Dependabot in order to avoid conflicts with Renovate.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update actions/checkout action to v4
• Update suzuki-shunsuke/github-action-renovate-config-validator action to v1

Detected dependencies

github-actions

.github/workflows/lint.yml

• actions/checkout v3
• suzuki-shunsuke/github-action-renovate-config-validator v0.1.3

.github/workflows/pythonpackage.yml

• actions/checkout v3
• actions/setup-python v5

.github/workflows/pythonpublish.yml

• actions/checkout v3
• actions/setup-python v5

pip_requirements

docs/requirements.txt

requirements-test.txt

requirements.txt

---

• Check this box to trigger a request for Renovate to run again on this repository

Unexpected Object Type when parsing files

Why does the comment parser return a list instead of a TerraformObject?

pyterraformer/pyterraformer/serializer/human_resources/engine.py

Line 327 in bd8b314

return [f"comment-{metadata.start_pos}", out]

When parsing through resources of a given file, we are expecting there to be a _type which is present on the Comment class; but that is found on the second element in the array. I feel like the purpose of this was to be able to identify a comment by it's row (likely for modification); but I think we should instead check for _type and then check metadata for start_pos (or something like that). This would keep us consistent with modifying actual ResourceObjects (where we check _type and then check tf_id).

That way, we could reliably return a list of TerraformObjects and ensure that we're properly handling any downstream checks / assignments. Thoughts?

For example, here is a list from an existing resource.tf file I have (each object is displaying by printing the type() and then the value)

['comment-0', comment(text="//")]
<class 'list'>
['comment-3', comment(text="//   Any of the Terraform modules can be utilized here as well as Terraform")]
<class 'list'>
['comment-79', comment(text="//   `resource` and `data` statements.")]
<class 'pyterraformer.core.resources.resource_object.ResourceObject'>
google_service_account(account_id="var.dataproc_service_account_name", display_name="var.dataproc_service_account_name", project="var.gcp_project_us")
<class 'pyterraformer.core.modules.ModuleObject'>
module(source=""https://artifactory.service.bo1.csnzoo.com/artifactory/terraform/modules/tf-mod-google-dataproc-cluster/tf-mod-google-dataproc-cluster_v3.0.2.tar.gz"", name=""tmori-test-cluster"", project="var.gcp_project_us", wayfair_vpc="var.dataproc_subnet["subnet_vpc_name"]", subnetwork="var.dataproc_subnet["subnet_name"]", region="local.region", service_account_email=""${var.dataproc_service_account_name}@${var.gcp_project_us}.iam.gserviceaccount.com"", comment-831="comment(text="#metastore_connection is utilizing the main dataproc_prod connection")", hive_metastore_connection_name="var.dataproc_metastore_connection", hive_warehouse_bucket="var.dataproc_warehouse_bucket", comment-1033="comment(text="#kms_key taken from ds_catalog_dataproc; will obviously need to be replaced")", hive_credential_kms_key=""projects/wf-gcp-us-ae-dataproc-prod/locations/us-central1/keyRings/hadoop-key-ring/cryptoKeys/hive-key"", labels="{}", history_bucket="var.dataproc_history_bucket", staging_bucket="var.dataproc_staging_bucket", autoscaling_enable="false")
<class 'list'>
['comment-1392', comment(text="# granting this custom SA ephemeral dataproc worker role; as it is the only custom role with the correct perms")]
<class 'list'>
['comment-1503', comment(text="# this should be changed to dataproc.worker when possible")]
<class 'pyterraformer.core.resources.resource_object.ResourceObject'>
google_project_iam_member(project="var.gcp_project_us", role=""organizations/825417849120/roles/tf_wf_ae_svc_ephemeral_dataproc_worker"", member=""serviceAccount:${var.dataproc_service_account_name}@${var.gcp_project_us}.iam.gserviceaccount.com"")
<class 'pyterraformer.core.resources.resource_object.ResourceObject'>

The error I receive is:

  File "C:\Users\tmori\AppData\Roaming\JetBrains\PyCharmCE2021.1\scratches\tmori_test_scratch.py", line 31, in <module>
    namespace.add_object(GoogleStorageBucket(
  File "C:\Users\tmori\wayfair-incubator\pyterraformer\pyterraformer\core\namespace.py", line 151, in add_object
    indexes = [
  File "C:\Users\tmori\wayfair-incubator\pyterraformer\pyterraformer\core\namespace.py", line 152, in <listcomp>
    idx for idx, val in enumerate(self.objects) if val._type == object._type
AttributeError: 'list' object has no attribute '_type'