microsoft/azure-pipelines-terraform

Azure Pipelines tasks for installing Terraform and running Terraform commands in a build or release pipeline.

Terraform Extension for Azure DevOps

Overview:

This repo contains the Azure DevOps Pipeline tasks for installing Terraform and running Terraform commands in a build or release pipeline. The goal of this extension is to guide the user in the process of using Terraform to deploy infrastructure within Azure, Amazon Web Services(AWS) and Google Cloud Platform(GCP).

This extension contains the following contributions:

• Terraform tool installer - for installing Terraform if not installed on the build agent
• Terraform - for executing the core Terraform commands
• Amazon Web Services(AWS) service connection - for creating a service connection for AWS to provide AWS credentials
• Google Cloud Platform(GCP) service connection - for creating a service connection for GCP to provide GCP credentials

The tasks are capable of running on the following build agent operating systems:

• Windows
• MacOS
• Linux

For more detailed information about the tasks, see the README for each from the below links:

• Terraform tool installer
• Terraform

Contact Information

This extension is authored by Microsoft DevLabs. It is not supported by Microsoft.

To report a problem with this extension, create an issue in this repository. The maintainers of this repository will review and respond to the issue.

You can also report problems or share feedback about this extension on Developer Community Forum.

azure-pipelines-terraform's Issues

Don't output sensitive values

I'm using now the great TerraformTaskV3 task and the output command.
It's working fine, however the whole json is outputed to the logs, including all sensitive values.
Is it possible to hide the output, or at least the sensitive values from the output command?

Example log extract:

/__t/terraform/1.2.5/x64/terraform output -json
{
  "password": {
    "sensitive": true,
    "type": "string",
    "value": "ujYU9/R2"
  }
}

Thank you

Need to be able to use secret variables

In the definition of my pipeline, I'm setting a variable which contains an API access token. I consider this to be a secret and have thus set the variable to secret.

Because of this, Azure DevOps will NOT automatically export the secret variable to a task. In order to use a secret variable in a task, it has to be defined in the "environment" block of a task.

See https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=classic%2Cbatch#secret-variables → "Classic".

Each task that needs to use the secret as an environment variable does remapping. If you want to use a secret variable called mySecret from a script, use the Environment section of the scripting task's input variables. Set the environment variable name to MYSECRET, and set the value to $(mySecret).

The TerraformTaskV2 doesn't have an environment block.

There needs to be a way to be able to use secret variables in TerraformTaskV2.

Terraform tool installer does not install latest version on hosted agent

The terraform tool installer does not install the latest version on a self hosted agent, output is:

(...)
2022-02-01T09:10:13.3133701Z Getting latest Terraform version.
2022-02-01T09:10:14.3695807Z Unable to get latest version: %s using 1.0.8
2022-02-01T09:10:14.3746273Z Downloading: https://releases.hashicorp.com/terraform/1.0.8/terraform_1.0.8_windows_amd64.zip
(...)
2022-02-01T09:10:27.7927297Z Your version of Terraform is out of date! The latest version
2022-02-01T09:10:27.7928079Z is 1.1.4. You can update by downloading from https://www.terraform.io/downloads.html

Input in version field ist "latest"

YAML:

steps:
- task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
  displayName: 'Install Terraform latest'

worked as expected when a specific version is inserted:

YAML:

steps:
- task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
  displayName: 'Install Terraform 1.1.4'
  inputs:
    terraformVersion: 1.1.4

terraform plan: Save human readable plan to file

It doesn't seem possible to write the human readable output of terraform plan to a file using this Task. For example, this does not work:

- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV1@0
  inputs:
   command: plan
   commandOptions: '-out=tfplan &> $(Build.StagingDirectory)/TFPlan.txt'

It would be super helpful if we could output the human readable plan to a file.

This code removes a temp file that contains the human readable plan. I propose 1. a Task property that, when true, does not remove the file and 2. an output variable that contains the path to the file.

For example:

- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV1@0
  inputs:
   command: plan
   commandOptions: '-out=tfplan'
   saveHumanReadablePlan: true

[TerraformV1] multiple providers not supported

I have provider configuration like this:

provider "azurerm" {
  subscription_id = "7fc00504-cef0-40d1-9923-0cb98ea33037"
  features {}
}

provider "azurerm" {
  alias           = "devops"
  subscription_id = "95dj317e-9222-4250-ba79-8a82384e3889"
  features {}
}

The second provider is also azurerm but in different subscription. I use this provider to create data source for querying. Current task only supports one environmentServiceNameAzureRM which is a service connection for terraform provider. So how to configure a second provider?

Terraform plan shows error with extra command

Upon the error condition below, the task did not fail the task nor the pipeline, and produced a jsonPlanFilePath of size 0.
It seems an extra "terraform show -json" command is being run, but the command line is not output to the logs. This command does not take into account my -var parameters so ends up failing when using the var to access a key vault entry.

Code is at https://github.com/algattik/terraform-azure-pipelines-starter/tree/github.com/microsoft/azure-pipelines-extensions/issues/747

    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV1@0
      name: 'terraform_plan'
      displayName: 'Terraform plan'
      inputs:
        command: plan
        workingDirectory: '$(System.DefaultWorkingDirectory)/infrastructure/terraform'
        commandOptions: -out tfplan -var environment=${{ parameters.environment}} ${{ parameters.TerraformVariables }}
        environmentServiceNameAzureRM: '${{ parameters.TerraformEnvironmentServiceConnection }}'
        backendServiceArm: '${{ parameters.TerraformBackendServiceConnection }}'

2019-12-12T06:28:43.8040581Z ##[section]Starting: Terraform plan
2019-12-12T06:28:43.8045948Z ==============================================================================
2019-12-12T06:28:43.8046383Z Task         : Terraform
2019-12-12T06:28:43.8046621Z Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
2019-12-12T06:28:43.8046841Z Version      : 0.0.137
2019-12-12T06:28:43.8047032Z Author       : Microsoft Corporation
2019-12-12T06:28:43.8047286Z Help         : [Learn more about this task](https://aka.ms/AA5j5pf)
2019-12-12T06:28:43.8047561Z ==============================================================================
2019-12-12T06:28:43.9546191Z [command]/opt/hostedtoolcache/terraform/0.12.18/x64/terraform providers
2019-12-12T06:28:44.2264902Z .
2019-12-12T06:28:44.2266821Z ├── provider.azurerm =1.38.0
2019-12-12T06:28:44.2267798Z └── module.sqlserver
2019-12-12T06:28:44.2268733Z     └── provider.azurerm (inherited)
2019-12-12T06:28:44.2269282Z 
2019-12-12T06:28:44.2328246Z ##[warning]Multiple provider blocks specified in the .tf files in the current working drectory.
2019-12-12T06:28:44.2342819Z [command]/opt/hostedtoolcache/terraform/0.12.18/x64/terraform plan -out tfplan -var environment=stage -var department=Engineering
2019-12-12T06:28:45.5400834Z �[0m�[1mRefreshing Terraform state in-memory prior to plan...�[0m
2019-12-12T06:28:45.5402427Z The refreshed state will be used to calculate this plan, but will not be
2019-12-12T06:28:45.5403137Z persisted to local or remote state storage.
2019-12-12T06:28:45.5405546Z �[0m
2019-12-12T06:28:47.7062647Z �[0m�[1mazurerm_resource_group.main: Refreshing state... [id=/subscriptions/a4ed7b9a-b128-49b4-a6ee-fd07ff6e296d/resourceGroups/rg-starterterraform-stage-main]�[0m
2019-12-12T06:28:47.7063754Z �[0m�[1mdata.azurerm_client_config.current: Refreshing state...�[0m
2019-12-12T06:28:47.7072014Z �[0m�[1mdata.azurerm_key_vault.keyvault: Refreshing state...�[0m
2019-12-12T06:28:47.7594208Z �[0m�[1mmodule.sqlserver.data.azurerm_key_vault_secret.sql_password: Refreshing state...�[0m
2019-12-12T06:28:47.9504198Z �[0m�[1mmodule.sqlserver.azurerm_sql_server.example: Refreshing state... [id=/subscriptions/a4ed7b9a-b128-49b4-a6ee-fd07ff6e296d/resourceGroups/rg-starterterraform-stage-main/providers/Microsoft.Sql/servers/algattik01sqlserver]�[0m
2019-12-12T06:28:48.0025319Z 
2019-12-12T06:28:48.0027927Z ------------------------------------------------------------------------
2019-12-12T06:28:50.2890823Z 
2019-12-12T06:28:50.2892856Z �[0m�[1m�[32mNo changes. Infrastructure is up-to-date.�[0m�[32m
2019-12-12T06:28:50.2893194Z 
2019-12-12T06:28:50.2893376Z This means that Terraform did not detect any differences between your
2019-12-12T06:28:50.2893635Z configuration and real physical resources that exist. As a result, no
2019-12-12T06:28:50.2895874Z actions need to be performed.�[0m
2019-12-12T06:28:50.3229687Z [command]/opt/hostedtoolcache/terraform/0.12.18/x64/terraform version
2019-12-12T06:28:50.8207298Z Terraform v0.12.18
2019-12-12T06:28:50.8209431Z + provider.azurerm v1.38.0
2019-12-12T06:28:54.7166180Z �[31m
2019-12-12T06:28:54.7168517Z �[1m�[31mError: �[0m�[0m�[1mKeyVault Secret "dev-sql-password" (KeyVault URI "https://terraformstarter.vault.azure.net/") does not exist�[0m
2019-12-12T06:28:54.7170425Z 
2019-12-12T06:28:54.7172146Z �[0m  on sqlserver/main.tf line 1, in data "azurerm_key_vault_secret" "sql_password":
2019-12-12T06:28:54.7173621Z    1: data "azurerm_key_vault_secret" "sql_password" �[4m{�[0m
2019-12-12T06:28:54.7177543Z �[0m
2019-12-12T06:28:54.7179193Z �[0m�[0m
2019-12-12T06:28:55.7041120Z �[31mTerraform couldn't read the given file as a state or plan file.
2019-12-12T06:28:55.7042289Z The errors while attempting to read the file as each format are
2019-12-12T06:28:55.7042705Z shown below.
2019-12-12T06:28:55.7042933Z 
2019-12-12T06:28:55.7043727Z State read error: Error loading statefile: open /home/vsts/work/1/s/plan-binary-54b477c5-fc04-4727-9cab-f98aa11a7d87.tfplan: no such file or directory
2019-12-12T06:28:55.7044407Z 
2019-12-12T06:28:55.7045166Z Plan read error: open /home/vsts/work/1/s/plan-binary-54b477c5-fc04-4727-9cab-f98aa11a7d87.tfplan: no such file or directory�[0m�[0m
2019-12-12T06:28:55.7109502Z ##[section]Finishing: Terraform plan

Terraform Init does not expose AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY as environment variables

Hi,

It looks like there is a difference how the Terraform init command works in regards to AWS credentials.

The credentials from the service connection are only passed as CLI arguments -backend-config=access_key=xxx and -backend-config=secret_key=xxx and not as env variables too.

As you can see from the log below, we are unable to download modules hosted in a bucket as Terraform cannot locate credentials to make that request.

Is it possible to expose the access key id and access key as environment variables for the init command?

Starting: Terraform : init
==============================================================================
Task         : Terraform
Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
Version      : 3.201.18
Author       : Microsoft Corporation
Help         : [Learn more about this task](https://aka.ms/AAf0uqr)
==============================================================================
/opt/hostedtoolcache/terraform/1.2.5/x64/terraform init -backend-config=bucket=xxx -backend-config=key=terraform.tfstate -backend-config=region=*** -backend-config=access_key=*** -backend-config=secret_key=***
Initializing modules...
Downloading s3::[https://xxx.s3.***.amazonaws.com/xxx/module.zip](https://xxxxx.amazonaws.com/xxxx/module.zip) for xxxx...
Downloading s3::[https://xxxx.s3.***.amazonaws.com/xxxxx/module.zip](https://xxxx.amazonaws.com/xxxx/module.zip) for xxxx...
╷
│ Error: Failed to download module
│ 
│ Could not download module "xxxxxx" (main.tf:4) source code
│ from
│ "s3::[https://xxxxx.s3.***.amazonaws.com/xxxxx/module.zip"](https://xxxxx.amazonaws.com/xxxx/module.zip%22):
│ NoCredentialProviders: no valid providers in chain
│ caused by: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not
│ found in environment
│ SharedCredsLoad: failed to load shared credentials file
│ caused by: FailedRead: unable to open file
│ caused by: open /home/vsts/.aws/credentials: no such file or directory
│ EC2RoleRequestError: no EC2 instance role found
│ caused by: 400 Bad Request: Bad Request
│ 	status code: 400, request id: 
│ caused by: EC2MetadataError: failed to make EC2Metadata request
│ <?xml version="1.0" encoding="utf-8"?>
│ <Error xmlns:xsi="[http://www.w3.org/2001/XMLSchema-instance"](http://www.w3.org/2001/XMLSchema-instance%22)
│ xmlns:xsd="[http://www.w3.org/2001/XMLSchema">](http://www.w3.org/2001/XMLSchema%22%3E)
│     <Code>InvalidHttpVerb</Code>
│     <Message>The HTTP verb specified was not recognized by the server.</Message>
│     <Details>'PUT' is not a supported verb.</Details>
│ </Error>
│ 	status code: 400, request id: 
╵

service connection with managed identity not working

When we are using service connection with managed identity in the terraform, init is failed because client_id and client_secret sent as undefined

terraform init -backend-config=storage_account_name=devopsterraformstatefile -backend-config=container_name=terraformstatefile -backend-config=key=iotdevsubscription.tfstate -backend-config=resource_group_name=AzureDevOps -backend-config=subscription_id= -backend-config=tenant_id= -backend-config=client_id=undefined -backend-config=client_secret=undefined

Support terraform import command.

Currently import isn't a valid option in the terraform extension.

Due to a bug in our marketplace image, we need to complete an apply, wait for it to fail, then import the finished item.

We don't need anything complicated for import commands, just let me provide a command line set of arguments.

Thanks

Terraform Plan/Apply unable to read state file from another subscription

Hi,

We have created central storage account in separated subscription to store state files in secure location. In single stage we have terraform init which is using service connection A to subscription A (for backend setup) and in next step we are having terraform plan or apply with service connection B to subscription B. Additionally SPN under B has rights to storage account from A.

Such setup was working for couple of months but after 25th of May (more or less) something has changed and we have started to get following errors on plan/apply.

Error: Error loading state: Error retrieving keys for Storage Account "<storage_account_name>": storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group '<resource_group_name>' could not be found

Init itself was successful all the time. There was no code change on our side. We even extended permissions for tests and we set Contributor for B SPN to entire A subscription but issue was the same.

The strange thing is that successful (prior issue) and failed steps have the same Terraform extension version: 0.0.142. We install Terraform version 0.12.3.

We have some kind of workaround but it would require reconfiguration in many release pipelines and tens of commits. If you could revert change in extension it would be great. Thanks!

"Failed to Download task" errors

Hi,

After the Terraform extension was updated to 0.1.16 on our on-prem Azure DevOps server, we've been seeing the attached error in the pipelines leveraging Terraform tasks:

##[warning]Failed to download task 'TerraformInstaller'. Error No task definition found matching ID a4789e5d-f6e8-431f-add9-379d640a883c and version 0.203.0. You must register the task definition before uploading the package.
##[warning]Back off 29.59 seconds before retry.
##[error]No task definition found matching ID a4789e5d-f6e8-431f-add9-379d640a883c and version 0.203.0. You must register the task definition before uploading the package.

Add support for state command

I have sensitive state stored in an Azure storage account. The only process that has access to that state are the agents in a vm scale set via a service connection that is used in an AzDO pipeline run. We need to take actions against that state such as list, mv and rm.

document for yaml users

yaml pipelines are best practice over gui. it would help with adoption if you documented the variables, keywords, etc. for example: how can i set the version to download?

below installs terraform but the default 0.12.3 version

- task: TerraformInstaller@0
  inputs:
    version: "0.12.18"

Changes Present Usage

Hey sorry for the potentially silly question

But how do you use the changes present output variable ? i dont see any way to give my task a reference in release pipelines ?

TerraformTaskV1@0 init without backendServiceArm using -backend=false

I am looking to have a pipeline which only runs a Terraform install, Terraform init then Terraform validate. This works normally when I specify my backendServiceArm, backendAzureRmResourceGroupName, backendAzureRmStorageAccountName, backendAzureRmContainerName and backendAzureRmKey inputs.

- task: TerraformTaskV1@0
  displayName: 'Terraform init'
  inputs:
    provider: 'azurerm'
    command: 'init'
    backendServiceArm: '$(backendServiceArm)'
    backendAzureRmResourceGroupName: '$(backendAzureRmResourceGroupName)'
    backendAzureRmStorageAccountName: '$(backendAzureRmStorageAccountName)'
    backendAzureRmContainerName: '$(backendAzureRmContainerName)'
    backendAzureRmKey: '$(backendAzureRmKey)'

I'm aiming to remove the backend configuration so that I dont need to grant this pipeline access into Azure, and any validation can be done only within Azure DevOps.

I have tried combinations of the following but it always errors with "##[error]Error: Input required: backendServiceArm"

- task: TerraformTaskV1@0
  displayName: 'Terraform init'
  inputs:
    provider: 'azurerm'
    command: 'init'
    commandOptions: -backend=false

Is there a way to accomplish this using the extension?

changesPresent is missing in V3

azure-pipelines-terraform/Tasks/TerraformTask/TerraformTaskV3/task.json

Line 297 in 79290bf

"outputVariables": [

Cancelling Step Does Not Release State Lock

Cancelling a step in mid operation does not gracefully exit which means there is a hanging state lock and you get this error:

Error: Error acquiring the state lock
...

Is there a way to ensure a graceful exit on cancel?

Logs are shown with weird symbols in Azure Release

When running Azure Releases the log shows weird symbols in the agent log.

2022-08-22T10:32:48.9498187Z �[33m│�[0m �[0m
2022-08-22T10:32:48.9498710Z �[33m│�[0m �[0m�[0mThe root module does not declare a variable named "web_linuxvm_admin_user"
2022-08-22T10:32:48.9499362Z �[33m│�[0m �[0mbut a value was found in file "dev.tfvars". If you meant to use this value,
2022-08-22T10:32:48.9500262Z �[33m│�[0m �[0madd a "variable" block to the configuration.
2022-08-22T10:32:48.9500848Z �[33m│�[0m �[0m
2022-08-22T10:32:48.9501542Z �[33m│�[0m �[0mTo silence these warnings, use TF_VAR_... environment variables to provide
2022-08-22T10:32:48.9502181Z �[33m│�[0m �[0mcertain "global" settings to all configurations in your organization. To
2022-08-22T10:32:48.9502859Z �[33m│�[0m �[0mreduce the verbosity of these warnings, use the -compact-warnings option.
2022-08-22T10:32:48.9503313Z �[33m╵�[0m�[0m
2022-08-22T10:32:48.9503665Z �[33m╷�[0m�[0m
2022-08-22T10:32:48.9504139Z �[33m│�[0m �[0m�[1m�[33mWarning: �[0m�[0m�[1mValue for undeclared variable�[0m
2022-08-22T10:32:48.9504579Z �[33m│�[0m �[0m
2022-08-22T10:32:48.9505096Z �[33m│�[0m �[0m�[0mThe root module does not declare a variable named "web_linuxvm_size" but a
2022-08-22T10:32:48.9505743Z �[33m│�[0m �[0mvalue was found in file "dev.tfvars". If you meant to use this value, add a
2022-08-22T10:32:48.9506307Z �[33m│�[0m �[0m"variable" block to the configuration.
2022-08-22T10:32:48.9506723Z �[33m│�[0m �[0m
2022-08-22T10:32:48.9507255Z �[33m│�[0m �[0mTo silence these warnings, use TF_VAR_... environment variables to provide
2022-08-22T10:32:48.9507887Z �[33m│�[0m �[0mcertain "global" settings to all configurations in your organization. To
2022-08-22T10:32:48.9508933Z �[33m│�[0m �[0mreduce the verbosity of these warnings, use the -compact-warnings option.

how to use Hetzner hcloud provider

I am not able to use hetzner hcloud provider using this Terraform plug in for azure devops.

Do you have any suggestions or work around.?

Release pipeline is failing with terraform project template for azure devops -TerrafForm-DevOps

I have implemented Build pipeline and release pipeline using following lab link
https://azuredevopslabs.com/labs/vstsextend/terraform/ However it is failing at the release pipeline due to the following error

and below is the error captured from the logs:
m�[31mError: �[0m�[0m�[1mFailed to get existing workspaces: Error creating storage client for storage account "terraformstoraged7d37c12": azure: malformed storage account key: illegal base64 data at input byte 0�[0m
Please assist.

Backend Config: Option to NOT inject -backend-config credentials

Feel like I am missing something simple.
We want to be able in our CI pipeline to basically init & plan, then zip the directory for release in a CD pipeline, but not have to re-init on the CD pipeline. The issue is the init is correctly using the service connection and sets the ARM_* environment variables, BUT it also passes these into the backend as command line options, which causes the secret to be written/stored directly into the local state file. This exposes the secret in the artifact archive. Is there a simple way to avoid this, it seems redundant since at runtime the SC is already logged in (via the ARM_* env.)

it would be very nice to have an option to just pick if you do or don't want the SC credentials injected into the backend.

Terraform state file not being created in Azure

I created a pipeline with Terraform init, plan and apply actions. The main.tf file being processed is below.
In the init task, I specified the subscription, resource group (one that is not managed by the Terraform file), storage account and container.

When I run the pipeline the first time, it succeeds in creating the resources in the main.tf file, but it doesn't create a state file in my blob container.
So the second time I run the pipeline, it fails, because Terraform thinks it has to create the resources again, and since they already exist, it crashes.

What could be the issue? I've also copied the init task log below the main.tf file content.
Thanks

# Configure the Azure provider
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0.2"
    }
  }

  required_version = ">= 1.1.0"
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "example-rg"
  location = "eastus"
}

2022-08-11T16:07:11.3114500Z ##[section]Starting: Terraform : init
2022-08-11T16:07:11.3123609Z ==============================================================================
2022-08-11T16:07:11.3123892Z Task         : Terraform
2022-08-11T16:07:11.3124247Z Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
2022-08-11T16:07:11.3124564Z Version      : 3.201.18
2022-08-11T16:07:11.3124772Z Author       : Microsoft Corporation
2022-08-11T16:07:11.3125019Z Help         : [Learn more about this task](https://aka.ms/AAf0uqr)
2022-08-11T16:07:11.3125330Z ==============================================================================
2022-08-11T16:07:11.4783128Z [command]/opt/hostedtoolcache/terraform/1.2.7/x64/terraform init -backend-config=storage_account_name=<storageAccountName>-backend-config=container_name=terraformstate -backend-config=key=terraform.tfstate -backend-config=resource_group_name=<terraformResourceGroup> -backend-config=subscription_id=<azureSubscription> -backend-config=tenant_id=<tenantId> -backend-config=client_id=*** -backend-config=client_secret=***
2022-08-11T16:07:11.5120904Z 
2022-08-11T16:07:11.5123208Z �[0m�[1mInitializing the backend...�[0m
2022-08-11T16:07:11.5123683Z 
2022-08-11T16:07:11.5124386Z �[0m�[1mInitializing provider plugins...�[0m
2022-08-11T16:07:11.5125784Z - Finding hashicorp/azurerm versions matching "~> 3.0.2"...
2022-08-11T16:07:11.6708607Z - Installing hashicorp/azurerm v3.0.2...
2022-08-11T16:07:13.3194252Z - Installed hashicorp/azurerm v3.0.2 (signed by HashiCorp)
2022-08-11T16:07:13.3194605Z 
2022-08-11T16:07:13.3195295Z Terraform has created a lock file �[1m.terraform.lock.hcl�[0m to record the provider
2022-08-11T16:07:13.3195869Z selections it made above. Include this file in your version control repository
2022-08-11T16:07:13.3196380Z so that Terraform can guarantee to make the same selections by default when
2022-08-11T16:07:13.3197011Z you run "terraform init" in the future.�[0m
2022-08-11T16:07:13.3197253Z 
2022-08-11T16:07:13.3197679Z �[33m�[33m╷�[0m�[0m
2022-08-11T16:07:13.3198250Z �[33m│�[0m �[0m�[1m�[33mWarning: �[0m�[0m�[1mMissing backend configuration�[0m
2022-08-11T16:07:13.3198776Z �[33m│�[0m �[0m
2022-08-11T16:07:13.3199409Z �[33m│�[0m �[0m�[0m-backend-config was used without a "backend" block in the configuration.
2022-08-11T16:07:13.3199935Z �[33m│�[0m �[0m
2022-08-11T16:07:13.3200534Z �[33m│�[0m �[0mIf you intended to override the default local backend configuration,
2022-08-11T16:07:13.3201275Z �[33m│�[0m �[0mno action is required, but you may add an explicit backend block to your
2022-08-11T16:07:13.3201894Z �[33m│�[0m �[0mconfiguration to clear this warning:
2022-08-11T16:07:13.3202372Z �[33m│�[0m �[0m
2022-08-11T16:07:13.3202781Z �[33m│�[0m �[0mterraform {
2022-08-11T16:07:13.3203263Z �[33m│�[0m �[0m  backend "local" {}
2022-08-11T16:07:13.3203709Z �[33m│�[0m �[0m}
2022-08-11T16:07:13.3204090Z �[33m│�[0m �[0m
2022-08-11T16:07:13.3204709Z �[33m│�[0m �[0mHowever, if you intended to override a defined backend, please verify that
2022-08-11T16:07:13.3205475Z �[33m│�[0m �[0mthe backend configuration is present and valid.
2022-08-11T16:07:13.3205952Z �[33m│�[0m �[0m
2022-08-11T16:07:13.3206355Z �[33m╵�[0m�[0m
2022-08-11T16:07:13.3206718Z �[0m�[0m
2022-08-11T16:07:13.3207283Z �[0m�[1m�[32mTerraform has been successfully initialized!�[0m�[32m�[0m
2022-08-11T16:07:13.3207790Z �[0m�[32m
2022-08-11T16:07:13.3208170Z You may now begin working with Terraform. Try running "terraform plan" to see
2022-08-11T16:07:13.3208703Z any changes that are required for your infrastructure. All Terraform commands
2022-08-11T16:07:13.3209089Z should now work.
2022-08-11T16:07:13.3209235Z 
2022-08-11T16:07:13.3209585Z If you ever set or change modules or backend configuration for Terraform,
2022-08-11T16:07:13.3210124Z rerun this command to reinitialize your working directory. If you forget, other
2022-08-11T16:07:13.3210813Z commands will detect it and remind you to do so if necessary.�[0m
2022-08-11T16:07:13.3293653Z ##[section]Finishing: Terraform : init

Feature Request: Option to set Terraform Outputs as variables

Hi there,

It would be great to have an option on the apply task to have the task set pipeline variables for all outputs returned by Terraform. These can be used for subsequent steps/jobs/stages to then deploy code into the terraform provisioned infrastructure.

I have found [this blog post][https://www.natmarchand.fr/using-terraform-output-values-in-azure-devops-pipeline/] detailing how to achieve this with a PowerShell command, but it would be nice to have it integrated into the apply task.

Terraform plan and possibly apply strips double quotes from commandOptions

Hello :)
I'm trying to use this task to deploy and plan my terraform code. The issue i'm running into is supplying a -var-file to the commandOptions.

The command option i'm running is
commandOptions: "-input=false -var-file='./${{ parameters.varFileLocation }}' -out=${{ parameters.workingDirectory }}/plan.tfplan"

The error i'm facing is this:

│ Given variables file './terraform.tfvars'
│ -out=/agent/_work/1/s/Kallidus.Infrastructure.Terraform///plan.tfplan
│ -detailed-exitcode does not exist.

I have verified that the file actually exists on the server in the path.
I think the issue has to with the fact that if you pass in -var-file="./terraform.tfvars" the called command removes the quotes arround it.

Terraform tool installer does not allow to specify specific download url

Hi team,

in some protected environments/networks it is not possible to access url like: https://releases.hashicorp.com/terraform/${version}/terraform_${version}_${platform}_${architecture}.zip

It would be very beneficial to enable users to specify url to their own Nexus/File Share/Storage Account.
IMHO, most simple way to achieve it would be to have environment variable or if-statement that will check if version starts with 'http'

Here is where code in question is located: https://github.com/microsoft/azure-pipelines-extensions/blob/5dd1dba15c474153c8b8d971c77959112f0b12a3/Extensions/Terraform/Src/Tasks/TerraformInstaller/src/terraform-installer.ts#L81

Regards,
Yury

Terraform : Init error building ARM Config: obtain subscription() from Azure CLI

Hello,
I'm trying to use the Terraform : Init task from an Azure DevOps release pipeline. The task is using an azure subscription (via a service connection that uses a Service Principal account) and uses a storage account to store the tfstate file. Due to some changes, we are trying to use the -migrate-state additional argument. However, when we do this, the task produces an error message:

Error building ARM Config: obtain subscription() from Azure CLI: parsing json result from the Azure CLI: waiting for the Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.

At this point, I'm stuck on how to resolve the issue. I don't know if this as an error or if I missed a step in my Release [NOTE: I tried to add an Azure CLI task to perform the AZ login action using the SP credentials but this didn't solve the issue].

Any assistance would be appreciated.
Thanks
Ken

TerraformTaskV2 has a dependancy on subscription ID for backend config

AzureDevops now has Subscription and Management Group scoped Service Connections but the TerraformTask is enforcing the Subscription ID on the Service Connection object in handleBackend

I suggest making it possible to have the ARM_SUBSCRIPTION_ID used for the subscriptionid backendConfig set when the backendServiceName doesn't have it available in the azure terraform command handler similar to how handleProvider can.

When a Management Group scoped service connection is currently used you can see that the terraform init command has a blank entry for the parameter.
e.g "-backend-config=subscription_id="

This results in the error.

• A Subscription ID must be configured when authenticating as a Service Principal using a Client Secret.

Affects TerraformTaskV2, TerraformTaskV1

Has this been upgraded to force TLS 1.2 or higher?

now that Azure Devops is forcing TLS 1.2 this task is filing. We think it is trying to do a lower version.

Terraform not showing on tasks after installed to Azure devops organization

Hello,
I was doing a CD and wanted to add Terraform task and yet it keeps showing as a grey button that means it’s already installed but I can’t use it .

So far I have tried to fix this issue.
Re-install the terraform.
Created another Project in the same organization, but still the same issue.
Created another organization, now i can use it.
Signed up with another email and i can use it as well.

my conclusion is that i have a bug in my organization, what can i do to fix this issue?

TerraformTaskV1: Terraform plan executed twice

Hello,

I recently stumbled upon this Terraform extension to Azure Pipelines and thought I would try implementing it in my deployments. I was surprised to find that the terraform plan command is executed twice instead of just once, which increases the time it takes to run the pipeline.

Would it be possible to modify the task so the plan command is executed once? For example if the -out parameter is present in the first plan command, the existing terraform plan can be used to produce the JSON output and therefore eliminating the need for a second plan step?

Allow for terraform tool installer to pull "required_version" from configuration files

Hi, I'm wondering if there's a way to have the Terraform Tool Installer task pull the "required_version" field from configuration, instead of having to explicitly set it in the task?

Release pipeline is failing with terraform project template for azure devops -TerrafForm-DevOps

I have implemented Build pipeline and release pipeline using following lab link
https://azuredevopslabs.com/labs/vstsextend/terraform/ However it is failing at the release pipeline due to the following error

m�[31mError: �[0m�[0m�[1mFailed to get existing workspaces: Error creating storage client for storage account "terraformstoraged7d37c12": azure: malformed storage account key: illegal base64 data at input byte 0�[0m
Please assist.

terraform show is not a command

i Have this yaml template using the terraform task extension

- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2
  name: terraform_${{ parameters.terraformCommand }}
  displayName: 'Terraform ${{ parameters.terraformCommand }}'
  inputs:
    command: ${{ parameters.terraformCommand }}
    workingDirectory: '${{ parameters.pathToTerraformRootModule }}'
    commandOptions: ${{ parameters.terraformCommandOptions }}
    environmentServiceNameAzureRM: '${{ parameters.serviceConnection }}'
    backendServiceArm: '${{ parameters.serviceConnection }}'
    backendAzureRmResourceGroupName: '${{ parameters.storageAccountResourceGroup }}'
    backendAzureRmStorageAccountName: '${{ parameters.storageAccount }}'
    backendAzureRmContainerName: '${{ parameters.stateBlobContainer }}'
    backendAzureRmKey: '${{ parameters.stateContainer }}'

I use it to -out=tfplan a terraform plan, and wanted to show the tfplan as json file to compare different plans between terraform versions.

               - template: ../templates/terraform.yaml
                  parameters:
                    terraformCommand: "plan"
                    terraformCommandOptions: "-lock=false -out=$(System.DefaultWorkingDirectory)/tfplan"
                    pathToTerraformRootModule: "$(pathToTerraformRootModule)"
                    serviceConnection: "$(serviceConnection)"
                    storageAccountResourceGroup: "$(storageAccountResourceGroup)"
                    storageAccount: "$(storageAccount)"
                    stateContainer: "$(MODULE)-$(TF_VAR_APPLICATION)-${{ parameters.Environment }}.tfstate"
                    stateBlobContainer: "$(stateBlobContainer)"

            - template: ../templates/terraform.yaml
              parameters:
                terraformCommand: "show"
                terraformCommandOptions: "-json $(System.DefaultWorkingDirectory)/tfplan > $(Build.ArtifactStagingDirector)/tfplan.json"
                pathToTerraformRootModule: "$(pathToTerraformRootModule)"
                serviceConnection: "$(serviceConnection)"
                storageAccountResourceGroup: "$(storageAccountResourceGroup)"
                storageAccount: "$(storageAccount)"
                stateContainer: "$(MODULE)-$(TF_VAR_APPLICATION)-${{ parameters.Environment }}.tfstate"
                stateBlobContainer: "$(stateBlobContainer)"

Expected result is the tfplan.json is created for further use.

Actual result is bellow :

Starting: Terraform show
==============================================================================
Task         : Terraform
Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
Version      : 2.188.1
Author       : Microsoft Corporation
Help         : [Learn more about this task](https://aka.ms/AA5j5pf)
==============================================================================
peError: provider[command] is not a function
Finishing: Terraform show

Does the terraform show command taken in account ?

In V3 output variables are no longer there.

In V3 output variables are no longer there.

example:

          - task: TerraformTaskV1@3
            displayName: 'Terraform Apply'
            name: 'TerraformApply'
            inputs:
              provider: 'azurerm'
              command: 'apply'
              workingDirectory: '$(System.DefaultWorkingDirectory)/Infrastructure/'
              environmentServiceNameAzureRM: '${{ parameters.azureSubscription }}'


          - task: PowerShell@2
            name: VarDeclaration
            displayName: Declare Terraform outputs as variables
            inputs:
              targetType: 'inline'
              script: |
                Write-Host "Declaring all Terraform outputs as variables"
                Write-Host "$(TerraformApply.jsonOutputVariablesPath)"

This fails with

+ Write-Host "$(TerraformApply.jsonOutputVariablesPath)"
+               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (TerraformApply.jsonOutputVariablesPath:String) [], ParentContainsErrorR 
   ecordException
    + FullyQualifiedErrorId : CommandNotFoundException

I've tested with V2 of the apply task and that worked

AWS Federated Account usage

Hello,

I am trying to make Terraform tasks in pipeline to create resources in AWS. Our organization is limited to use of federated accounts.

Federated accounts have temporary Access key ID, Access key secret along with Access key token. AFAIK this kind of access id has default life span of 4 hours.

Is there any possibility to specify access key token, so the tasks will reuse it? We are okay to parametrize these 3 values and enter them at the start of pipeline execution.

Too many command line arguments. Configuration path expected - While running simple TF plan

We started seeing this error for the terraform plan within our orgs pipelines. It seems to have started happening with this PR was merged

#36

/opt/hostedtoolcache/terraform/0.12.31/x64/terraform plan undefined -detailed-exitcode
Too many command line arguments. Configuration path expected.
##[error]Error: Terraform Plan failed with exit code: 1
Finishing: Terraform : plan

TerraformTaskV3 support for workspace?

In trying to help with this question, I noticed in the V3 README that the word workspace does not exist. Does that mean tasks like

          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV3@3
            inputs:
              provider: aws
              command: custom
              customCommand: 'workspace select devstage'

should expect to see an exit code 127 (job's command can not be found or executed)?

Please add assume-role support to the service connection or let us use AWS service connection.

Please add assume role support to this service connection itself just like the AWS service connection so that we do not have to pass assume role everywhere in our configuration or at least allow us to use AWS service Connection which supports assume role.

TerraformTaskV1: Support taint command

I want terraform to perform a slot swap with a azure web app. To perform this i have to taint a resource.

Currently the taint command is not supported

Add Support for fmt command

Want to be able to utilise these tasks and Terraform fmt -check - Currently not supported - Could this potentially be implemented?

Error: Endpoint data parameter not present: subscriptionid using management group scoped spn

We are using the Terraform@V3 task to deploy infrastructure at the management group level (Tenant Root Group). The following is the pipeline yaml.

trigger:
- master

pool:
  vmImage: windows-latest

steps:
- task: TerraformInstaller@0
  inputs:
    terraformVersion: 'latest'
- task: TerraformTaskV3@3
  inputs:
    provider: 'azurerm'
    command: 'init'
    workingDirectory: '$(System.DefaultWorkingDirectory)\caf'
    backendServiceArm: 'devops (73e0699d-f086-4960-8b97-abad78a39916)'
    environmentServiceNameAzureRM: 'tf-connect-manual'
    backendAzureRmResourceGroupName: 'rg-storage-terraform'
    backendAzureRmStorageAccountName: 'terraform2319'
    backendAzureRmContainerName: 'terraform-state'
    backendAzureRmKey: '<secret removed>'

- task: TerraformTaskV3@3
  inputs:
    provider: 'azurerm'
    command: 'plan'
    workingDirectory: '$(System.DefaultWorkingDirectory)\caf'
    environmentServiceNameAzureRM: 'tf-connect-manual'

• Running the tf code locally using az login (works fine).
• environmentServiceNameAzureRM: a service connection that is subscription scoped (works fine)
• environmentServiceNameAzureRM: a service connection that is management group scoped (doesn't work - error below)

This is where we are setting the different service connection scopes

How do we get past this error?

Terraform import command support

We are planning to use terraform import command in Azure devops but unable to find import command under Microsoft terraform task :https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks

Please let us know the ETA for the import command to be available on terraform task or any alternative option for using the import command in azure devops.

Also, Please confirm if it is safe in security point of view to use the 3rd party extension in the azure DevOps like: https://marketplace.visualstudio.com/items?itemName=charleszipp.azure-pipelines-tasks-terraform.

Terraform workspace support

I am looking to have a pipeline which selects a Terraform workspace, similar like the instructions below.

- task: TerraformTaskV1@0
        displayName: 'apply terraform'
        inputs:
          provider: 'azurerm'
          command: 'apply'
          workspace: <SELECTED_WORKSPACE>
          customCommand: 'workspace'
          commandOptions: '-input=false -var-file="$(System.DefaultWorkingDirectory)/infrastructure/azure/vars/default.tfvars"'
          environmentServiceNameAzureRM: '${{ variables.backendServiceArm }}'
          workingDirectory: '$(System.DefaultWorkingDirectory)/infrastructure/azure'

Could this functionality please be added to the terraform extension.

Pipeline not parsing quotes correctly

When passing a quoted string to TerraformTaskV2@2 the first " becomes just a \ instead of a double quote. This only happen with the first " passed to the task. Any additional once seem to be replaced correctly.

I found a work around, if you pass "" where you want the first double quote to be it works.

The first attachment shows the arguments being passed in correctly then transformed without the leading double quote, second one shows it being sent in with the ""

save Terraform plan changes as pull request comment

We have a process where pull request changes are validated using a "terraform plan" release stage (pull request deployment). It would be great if the plan output would be added as a comment to the PR.

Today we need to navigate to the right deployment and scroll through the logs. It would be a nice improvement to the usability and workflow.

While it is not as straightforward to obtain the pull request ID in a release pipeline it seems it can be looked up through the API using the artifacts' sourceversion and repoID.

V3 Not Available As a Task

Hi,

I was super keen to start using V3 since the PR was merged - I've reinstalled the marketplace extension but still have no way to reference the V3 task - Wondering if this was on purpose or I'm missing something obvious

Thanks

Cancel TF Apply breaks state

When running terraform from the CLI, and I hit Ctrl-C, is gracefully exists the process and still updates the state based on the progress of the update.

However, if I cancel the TF Apply task from DevOps while it's running, it kills the process immediately and doesn't let it exit out on it's own, breaking state.

I'm using AWS and an S3 bucket for state.

AWS backend configuration - static region code :/

Currently the setup of AWS backendConfig region is only possible via ServiceConnection :/

Based on TerraformCommandHandlerAWS.ts :

    private setupBackend(backendServiceName: string) {
        this.backendConfig.set('bucket', tasks.getInput("backendAWSBucketName", true));
        this.backendConfig.set('key', tasks.getInput("backendAWSKey", true));
        this.backendConfig.set('region', tasks.getEndpointAuthorizationParameter(backendServiceName, "region", true));
        this.backendConfig.set('access_key', tasks.getEndpointAuthorizationParameter(backendServiceName, "username", true));
        this.backendConfig.set('secret_key', tasks.getEndpointAuthorizationParameter(backendServiceName, "password", true));
    }

Would be possible to include a dynamic configuration via task input, such as "backendAWSRegion"?

Feature Request: Terraform Refresh Command

I'd like a support for the terraform refresh command.

This would allow for separation of refresh logs from the terraform plan and terraform apply outputs.

Having the plan output separate from the refresh will remove significant noise on larger projects.

A typical workflow might look like this:

terraform init
terraform validate
terraform refresh
terraform plan -refresh=false -out=tfplan -no-color
terraform apply -auto-approve tfplan -no-color

Control characters in logs

Generates by actions: Plan, Apply