terraform apply randomly gets stuck with "Still creating"

[timohirt]

terraform apply randomly gets stuck with "Still creating"

[tagirb]

@hetznercloud @LKaemmerling Dear Hetzner Cloud team, could you probably consider maintaining this Terraform provider? Hetzner DNS is probably an important part of many Hetzner infrastructures and having a maintained up-to-date Terraform provider for this would be real cool!

[kimdre]

We forked this project to germanbrew/terraform-provider-hetznerdns since we depend heavily on it but it is no longer maintained and has many issues and missing features, which we fixed in our fork.
We also wrote an migration guide https://registry.terraform.io/providers/germanbrew/hetznerdns/latest/docs/guides/migration-from-timohirt-hetznerdns

[kimdre]

I just ran it again with Log Level set to TRACE.
This is the output right before all remaining entries get stuck in "Still Creating...":

I replaced some domains with domain_com, domain2_com, etc.

2022-12-15T23:23:35.448+0100 [INFO]  provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:35 [DEBUG] Creating resource zone: timestamp=2022-12-15T23:23:35.447+0100
2022-12-15T23:23:35.448+0100 [INFO]  provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:35 [DEBUG] Creating resource zone: timestamp=2022-12-15T23:23:35.447+0100
2022-12-15T23:23:35.448+0100 [INFO]  provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:35 [DEBUG] Creating resource zone: timestamp=2022-12-15T23:23:35.447+0100
2022-12-15T23:23:35.448+0100 [INFO]  provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:35 [DEBUG] HTTP request to API POST https://dns.hetzner.com/api/v1/zones: timestamp=2022-12-15T23:23:35.447+0100
2022-12-15T23:23:35.448+0100 [INFO]  provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:35 [DEBUG] Creating resource zone: timestamp=2022-12-15T23:23:35.447+0100
2022-12-15T23:23:35.448+0100 [INFO]  provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:35 [DEBUG] Creating resource zone: timestamp=2022-12-15T23:23:35.447+0100
2022-12-15T23:23:35.448+0100 [INFO]  provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:35 [DEBUG] Creating resource zone: timestamp=2022-12-15T23:23:35.447+0100
2022-12-15T23:23:35.448+0100 [DEBUG] provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:35 [DEBUG] POST https://dns.hetzner.com/api/v1/zones
2022-12-15T23:23:35.567+0100 [DEBUG] provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:35 [DEBUG] POST https://dns.hetzner.com/api/v1/zones (status: 422): retrying in 1s (10 left)
2022-12-15T23:23:36.589+0100 [DEBUG] provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:36 [DEBUG] POST https://dns.hetzner.com/api/v1/zones (status: 422): retrying in 2s (9 left)
2022-12-15T23:23:38.611+0100 [DEBUG] provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:38 [DEBUG] POST https://dns.hetzner.com/api/v1/zones (status: 422): retrying in 4s (8 left)
2022-12-15T23:23:40.229+0100 [TRACE] dag/walk: vertex "hetznerdns_record.mydomain_com-MX" is waiting for "hetznerdns_zone.domain_com"

The only output after that is just the same with different dns records:

2022-12-15T23:23:40.229+0100 [TRACE] dag/walk: vertex "hetznerdns_record.wildcard-domain_com-AAAA" is waiting for "hetznerdns_zone.domain_com"
2022-12-15T23:23:40.229+0100 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/timohirt/hetznerdns\"] (close)" is waiting for "hetznerdns_record.mail_domain_com-CNAME"
2022-12-15T23:23:40.229+0100 [TRACE] dag/walk: vertex "hetznerdns_record.autodiscover-drechsel_xyz-CNAME" is waiting for "hetznerdns_zone.drechsel_xyz"
...
2022-12-15T23:23:40.229+0100 [TRACE] dag/walk: vertex "hetznerdns_record.ns3-domain2_com-NS" is waiting for "hetznerdns_zone.domain2_com"
2022-12-15T23:23:40.229+0100 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/timohirt/hetznerdns\"] (close)"
2022-12-15T23:23:40.229+0100 [TRACE] dag/walk: vertex "hetznerdns_record.ns2-drechsel_xyz-NS" is waiting for "hetznerdns_zone.drechsel_xyz"
...
2022-12-15T23:23:42.642+0100 [DEBUG] provider.terraform-provider-hetznerdns_v2.2.0.exe: 2022/12/15 23:23:42 [DEBUG] POST https://dns.hetzner.com/api/v1/zones (status: 422): retrying in 8s (7 left)

and so on.

[timohirt]

Sad to hear @kimdre. Currently, I don't have time to look into it. Maybe over the weekend, maybe after Christmas. Sorry. Did you try to debug into it?

[orefalo]

I have the same issue - seems like a timeout problem, 11 retries or 3m is not enough

[yoo]

If a DNS record already exists the Hetzner API returns 422 unprocessable entity. Which leads to retries because of:

terraform-provider-hetznerdns/hetznerdns/api/client.go

Line 38 in f5310c0

if !ok && resp.StatusCode == http.StatusUnprocessableEntity {

Either change the retry policy or the create call has to lookup up the record first and error.

[orefalo]

correct, upon retry threshold, the process should

• refetch the list of DNS from the source
• identify if the record is in the new fetch

• -> DELETE IT
  

• then proceed with the creation attempt

[yoo]

Terraform shouldn't touch existing resources. That's what importing is for. It should just be an error.

[kimdre]

I experienced this problem with completely new DNS zones aswell as existing ones. Deleting records when nothing exists before apply is not a solution.

[yoo]

@kimdre From your post last year, my guess is the DNS Zone already exists and the API returns 422 on the POST request to the Zone.

This could be the case when forcing Terraform to stop the execution, skipping the cleanup. Now the DNS Zone exists at Hetzner but not in the Terraform state.

Edit: Okay, I did not see the line for the successful request: POST https://dns.hetzner.com/api/v1/zones, and then it gets stuck on 422 errors.
Maybe it creates the zone successfully and then runs into the retry loop because the zone already exists.

[tagirb]

If a DNS record already exists the Hetzner API returns 422 unprocessable entity. Which leads to retries because of:

terraform-provider-hetznerdns/hetznerdns/api/client.go

Line 38 in f5310c0

if !ok && resp.StatusCode == http.StatusUnprocessableEntity {

Either change the retry policy or the create call has to lookup up the record first and error.

Also having this issue. I wonder if retrying on 422 is the correct behaviour. Wouldn't it be better to fail with something like 'Record already exists'?

[micheljung]

I figured out why this happened in my case, I hope it applies to yours as well.

My resource is specified like this:

resource "hetznerdns_record" "a" {
  zone_id = hetznerdns_zone.main.id
  name    = "@"
  value   = var.ingress_ipv4
  type    = "A"
}

This lead to 422, even though the record didn't exist. The issue was that the source of my variable was specified like so:

output "ingress_ipv4" {
  value = kubernetes_ingress_v1.demo_ingress.status.0.load_balancer.0.ingress.2.ip
}

But the IPv4 address isn't guaranteed to be at a certain index (I didn't trust it to be when I wrote it but I thought "let's try"), so the request sent to hetznerdns was:

{"zone_id":"oDgLe5RVwtofvkKNLuuJ66","type":"A","name":"@","value":"2a01:4f7:1c1d:a24::1"}

Which resulted in:

{"record":{"id":"","type":"","name":"","value":"","zone_id":"","created":"","modified":""},"error":{"message":"invalid A record","code":422}}