Failure sending request: StatusCode=429 -- Original Error: context deadline exceeded

[azure]

Failure sending request: StatusCode=429 -- Original Error: context deadline exceeded

[krowlandson]

Proposed approach:

• Gather further evidence of the issue (how widespread is this problem)
• Investigate retry and backoff behaviour in Terraform / AzureRM provider / autorest (why is Terraform not backing off to allow the API to return to normal operation)
• Are customers being impacted by this?
• Can we tune the test framework down to reduce the impact of the API throttling without unreasonably increasing the test duration or reducing test coverage?
• Enable trace logs (TF_LOG=1) and raise an issue with the provider

[dan-leanix]

Don't know if this is helpful but I left the following comment on the issue with the DNS resources.

Whatever it is it was introduced with the Terraform Azure provider 2.x versions from the very beginning. We could reproduce it with version 2.1.0. So, I assume that this exists since version 2.0.0.

Terraform Azure provider version 1.44.0 just works.

hashicorp/terraform-provider-azurerm#9816

[krowlandson]

Interesting observation @dan-leanix... I've not had a chance to pinpoint the exact point where it became the primary method of controlling requests to the AzureRM API, but the autorest client was introduced from v2.3.0 so I suspect there was a general effort from v2.0.0 to change the way the provider interacts with the API.

In our particular case, there has clearly been a change on the platform side which is resulting in a provider throttling requests when it didn't previously. However 429 responses are common across most others, including the DNS service as mentioned in Creating Over 2000 DNS Records Gives me StatusCode=429 hashicorp/terraform#27266.

Unfortunately it seems that when running high numbers of parallel tasks against the same API (on an individual Resource Provider / Service), Terraform isn't correlating related HTTP/429 responses, so the backoff isn't aggressive enough to let the API rate limiting to be lifted.

[krowlandson]

Workaround implemented by #80 but leaving issue open to collect additional feedback and in case additional work is needed.

Not ideal having to reduce parallelism setting from 256 to 50.

[krowlandson]

Closing issue as this has now been linked to an Azure platform update designed to limit API requests for this provider.