Error: Invalid for_each argument" because local.azurerm_policy_definition_external_lookup will be known only after apply

[krmohansingh]

Error: Invalid for_each argument" because local.azurerm_policy_definition_external_lookup will be known only after apply

[krmohansingh]

Even tried to compare the state file for the time stamp when the pipeline start throwing error with the pipeline got succeeded, not able to find any difference.

Tried to restore the state file for the previous day when pipeline was running fine , still unable to fix the problem.

Could you please suggest what else we could do as a troubleshooting steps.

[jtracey93]

This generally indicates a policy definition ID in one of the lib files is incorrect or has an ID to a policy that is incorrect.

Have you checked all these for any changes?

Have you also tried disabling the custom lib and running a plan to see if it completes? As I take it this happens at plan stage?

[krmohansingh]

@jtracey93 Here is the scenario, we have two branches

1. Develop branch ------ > we use feature/xxxx to deploy anything to development environment
2. Main branch --------> we use release from develop branch to deploy to production environment via main branch

We have tried to disable custom lib and run terraform plan, it succeeds. But when we create any feature branch out of main branch or develop branch terraform plan fails with the above error.

Although if there is an issue with main branch, it should throw the same error, but it's not throwing any error in the main branch. Only develop or feature branches are getting failed.

Could you please suggest something else we could try to troubleshoot this?

However, I am verifying all the policy definition ID again and see if something is going wrong in the ID of them.

[jtracey93]

The fact it succeeds without a custom lib suggests this is something wrong in the lib...

[krmohansingh]

may be and we are trying to isolate the problem here. Now we were able to run the pipeline without any issue, but with no policy assignment defined in the archtype.

And as soon as we keep one policy assignment, we are getting below error related to role_assignment.

We checked state file and remove all the role assignment reference and still we are getting this. Please find below is our policy assignment .json file

Another thing which i observed is whenever we are referencing the custom policies defined within the initiatives, we are not seeing any issues . But, when we are referencing built-in policies within the policy set definition we started getting below error . Seems like data source is not working.
Please suggest how can we resolve this, as this was working before without any issues.

[matt-FFFFFF]

Hi @krmohansingh would you consider a workaround to assign the built-in policies directly?

I can then help triage and repro

[thejrayasam]

Hi @krmohansingh, as we are facing the error only for the built-in policies, could you please share the sample tf files that you used for referring the built-in policies? It will help us in replicating the issue

Furthermore, Terraform is not able to get the list of Role Definitions per scope as I understand the error.
https://github.com/Azure/terraform-azurerm-caf-enterprise-scale/blob/main/modules/role_assignments_for_policy/locals.tf

may be and we are trying to isolate the problem here. Now we were able to run the pipeline without any issue, but with no policy assignment defined in the archtype.

And as soon as we keep one policy assignment, we are getting below error related to role_assignment.

We checked state file and remove all the role assignment reference and still we are getting this. Please find below is our policy assignment .json file

Another thing which i observed is whenever we are referencing the custom policies defined within the initiatives, we are not seeing any issues . But, when we are referencing built-in policies within the policy set definition we started getting below error . Seems like data source is not working. Please suggest how can we resolve this, as this was working before without any issues.

[matt-FFFFFF]

I cannot see the attachment from the last post. Please reattach using GitHub's website.

[krmohansingh]

@thejrayasam Please find the policy set definition and assignment we tried to use.

Policy_set_definition.json

Policy_assignement.json

Archetype definition where we called this assignment

@matt-FFFFFF As we have discussed over call, please find the screenshot below like as soon as we assign that assignment that has reference to built-in policy, we start getting the below error related to role assignement.

How we managed to resolve the issue temporary basis we ran terraform apply -var-file terraform.tfvars and then tried to add the policy reference which had built-in reference, terraform plan started working with policy with even built-in reference.

After that there were some subscriptions defined in the terraform.tfvars file which our customer found that those subscriptions are already decommissioned, after removing that we are not seeing the error anymore. But, question is if removing the stale subscription fixed the issues then why it started working temporarily when we ran terraform apply -ver-file terraform.tfvars file
. Please check and let us know if you were able to replicate.

[matt-FFFFFF]

Hi

please make sure that inputs to this module are literals, and do not come from the resource graph.

using data sources or other resources can cause issues like this.

[krmohansingh]

@matt-FFFFFF thank you for your inputs, that can be the possibility.
@thejrayasam Issue seems to be fixed now.

Based on my analysis which I have done, I feel below could be the root cause of the issue which I would your help to confirm my analysis.

Analysis
The pipeline was working till now because subscription was decommissioned. Based on the documents if the subscription is in disabled mode, it will take 90 days to get permanently deleted.
https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/subscription-states

The root cause could be that a manual Terraform apply fixed the pipeline issues temporarily. Subsequently, another Terraform apply from the develop pipeline tried to download all dependencies for the subscriptions mentioned in the dev/terraform.tfvars file. However, these subscription IDs reference were missing from the prod/terraform.tfvars file. As a result, the develop pipeline was failing while the release pipeline was working fine. Essentially, after one change that executed a policy to the subscriptions, all subscriptions defined in the tfvars files were read for execution. Due to some subscriptions being missing, the pipeline encountered an error. When we manually ran "terraform plan" and removed the deleted subscriptions, the issue was resolved.

To ensure this issue does not recur, please make sure we remove all references to deleted subscriptions from the terraform.tfvars file.

[matt-FFFFFF]

Can this be closed now?