Add Exponential Backoff When adding Heroku Domain

[heroku]

Add Exponential Backoff When adding Heroku Domain

[davidji99]

Proposed solution will be add some exponential backoff potentially.

[talbright]

I ran into a form of this the other week as well. In my case the app already existed in the statefile, but I was adding a heroku_domain. Of course re-running apply can get one past the problem, but that's far from ideal and not expected.

The aws provider client has some built-in retry/backoff mechanism, we might look to see if that can be extracted. I've also use https://github.com/cenkalti/backoff before, but I would like to get some additional 👀 on it to see if that looks like a dependency we want to bring in.

Ideally, I'd like to see retry logic built into the client, that way we get this for free across all our API calls and don't have to retrofit retry logic everywhere. I'll open a discussion in https://github.com/heroku/heroku-go.

[talbright]

Actually perhaps the right thing to do here is use this: https://godoc.org/github.com/hashicorp/terraform/helper/resource#StateRefreshFunc

[bernerdschaefer]

I think using StateRefreshFunc is the more appropriate solution for this case, rather than baking retries into the client.

[talbright]

Agreed, the more I think about it the more that seems like the right thing to do.

[bernerdschaefer]

@talbright Thinking about this more, I think the refresh should be done on the app after creation. By waiting there for the app to be ready, we would also fix https://github.com/terraform-providers/terraform-provider-heroku/issues/37 and anything else which depends on the app being provisioned for resources to be added. It would be a smarter version of the work-around that we've been using of introducing a sleep after app creation (https://github.com/terraform-providers/terraform-provider-heroku/issues/37#issuecomment-356378747).

[talbright]

That makes sense to me as well. There's an issue with a couple of tests, I'm waiting to hear back from @davidji99 on and get those cleared out. If this hasn't been worked on by the time I get that cleaned up, I will implement the fix suggested here on the app resource.

[talbright]

Hitting this up next.

[talbright]

@bernerdschaefer I don't see anything in the production API for checking app status. I could hit the webhook API and look for past events, but that seems dodgy. There's a status field for the domain endpoint (https://devcenter.heroku.com/articles/platform-api-reference#domain) but that wouldn't fix the original issue here:

heroku_domain.app: Post https://api.heroku.com/apps/meh/domains: App meh is being provisioned. Please try again later.

Now I can fix this in the heroku resource by greping for this error message. That would not however fix if for the broader scenerios we have discussed here.

[talbright]

Addressed by #142