You should be able to bootstrap and use create resources in the same run.

[rancher]

You should be able to bootstrap and use create resources in the same run.

[rawmind0]

@jgreat, bootstrap resource is a special resource. You couldn't mix any other provider resource with bootstrap.

Anyway, to make your pattern work in one .tf file, take a look to https://github.com/terraform-providers/terraform-provider-rancher2/issues/13#issuecomment-496581493

[jgreat]

The two provider option is ok with the limitation of TF, but the piece that's missing here is allowing for the url and token to be set by generated values. The Helm and Kubernetes providers allow this if you populate the kubernetes block with host, username, certs, etc... This way you can use the bootstrap resource and pass the resulting values into another rancher2 provider.

Example of values generated by rke_cluster resource and then used with the helm provider.

provider "helm" {
  install_tiller  = true
  namespace       = "kube-system"
  service_account = "tiller"

  kubernetes {
    host                   = "${rke_cluster.rancher_server.api_server_url}"
    username               = "${rke_cluster.rancher_server.kube_admin_user}"
    client_certificate     = "${rke_cluster.rancher_server.client_cert}"
    client_key             = "${rke_cluster.rancher_server.client_key}"
    cluster_ca_certificate = "${rke_cluster.rancher_server.ca_crt}"
  }
}

[cloudnautique]

@jgreat I don't follow, in the example block posted in @rawmind0 s comment, the second provider (non bootstrap) is being populated with the URL and API token generated from the bootstrap resource. Is there missing data that we should be exporting?

[drpebcak]

When you try to do this all in one run like this:

provider "rancher2" {
  alias     = "bootstrap"
  api_url   = "https://${local.name}.${local.domain}"
  bootstrap = true
}

resource "rancher2_bootstrap" "admin" {
  provider = "rancher2.bootstrap"
  password = "${local.rancher_password}"
}

provider "rancher2" {
  alias     = "authenticated"
  api_url   = "${rancher2_bootstrap.admin.url}"
  token_key = "${rancher2_bootstrap.admin.token}"
}

data "rancher2_setting" "server-url" {
  provider = "rancher2.authenticated"
  name     = "server-url"
}

Terraform complains:

Error: Error refreshing state: 1 error occurred:
	* provider.rancher2.authenticated: [ERROR] No api_url provided

If you hardcode api_url it still complains about the keys:

Error: Error refreshing state: 1 error occurred:
	* provider.rancher2.authenticated: [ERROR] No token_key nor access_key and secret_key are provided

[stepman0]

I have a similar issue here. I created two terraform modules, one provisioning the server and the second is adding a cluster. The second script (cluster) needs the token_key generated by the first script (server).

Passing the token_key as a variable to the second script results in:
Error: [ERROR] No token_key nor access_key and secret_key are provided

[rawmind0]

@drpebcak , as explained at https://github.com/terraform-providers/terraform-provider-rancher2/issues/13#issuecomment-496581493, you'd need to run terraform apply -target=rancher2_bootstrap.admin at least once, to just generate rancher2_bootstrap resource, then next terraform apply will work fine.

@stepman0, if you are passing token_key variable properly, it should work. Could you please check it??

[SinisterMinister]

@rawmind0 Only needing to apply once is the desired behavior. Currently, you cannot build, bootstrap, and then use the cluster in a single apply. You should be able to take the stuff generated from the bootstrapping process an use it to handle your cluster in the same run.

[rawmind0]

@SinisterMinister, yes i get the point but it's a chicken egg problem. May be bootstrap should be another tf provider, but btm it is what it is.

More than happy to merge if you have any proposal to make it work. :)

[SinisterMinister]

@rawmind0 I've never done any provider development, but from what I can tell, the only thing stopping this is the check within provider.go that's trying to make sure either the token_key or the access_key/secret_key combo has been set and the call to config.ManagementClient() as it calls the Rancher API as part of the client instantiation.

https://github.com/terraform-providers/terraform-provider-rancher2/blob/40199a8a59c3eb04e594f9f5f03a53687e8ca2e0/rancher2/provider.go#L173-L180

I've tried to find how other providers handled such constraints, but AFAIK, they seem to be handling it within the actions and not in the provider instantiation. I've removed the offending code and tested it locally and it works as desired. The trade-off I'm seeing is that there will not be an error thrown on plan if there are no credentials provided and you'll likely get a 401/403 error on apply. There may be others as I'm not sure what the desired effect of calling config.ManagementClient() is, other than maybe validating the connection of the provider on provider instantiation.

[SinisterMinister]

I also ran the test suite against the code and everything passed. The acceptance tests seem to be broken as they error out for me midway with a fresh clone. Building in Ubuntu 18.04 LTS.

If you want, I can make a PR with the changes. I'd just prefer to understand the caveats beforehand as this seemingly tiny change could cascade in ways I can't expect without more domain knowledge. It's definitely a contract change to the provider, albeit backwards compatible.

[rawmind0]

@SinisterMinister , i think this isn't a solution. For sure you'll got 401/403 errors if no token key nor access and secret key are provided, basically you can't connect with rancher API.
These lines, https://github.com/terraform-providers/terraform-provider-rancher2/blob/40199a8a59c3eb04e594f9f5f03a53687e8ca2e0/rancher2/provider.go#L173-L180 are indeed to checking that you provide 1 of 2 options to authenticate. These checks are needed if you are not bootstrapping.

config.ManagementClient() is the function that connect with rancher management API, and is tested on provider initialization to check that it's able to connect with rancher API, as majority of providers do.

AFAIK, the acceptance tests are working fine. Every PR that is submitted/merged all the acceptance tests are passed. The only bug that we know about AT is an eventual fail if the vm hasn't enough resources, making the cluster got inactive, and some tests fail. Restarting should work again

[SinisterMinister]

@rawmind0 I'm not sure I agree. I just dug through the AWS provider and the Kubernetes provider and while they build their clients as part of instantiation, they are connecting lazily. There seems to be a mixing of concerns here within instantiating the provider: validating the config input and verifying its correctness. It shouldn't be doing the latter as the inputs to the provider may not exist yet and causes issues like we're discussing here. And since the provider code is doing out-of-band validation to work around the lack of dynamic schemas, it has no way to know if values are TBD or just empty strings. Seems like this is just how Terraform is designed to deal with providers in general. My bad, I'm an idiot.

The real question here is where do we raise the error, on instantiation, or at run-time. I do not think you can do the former and enable the requested functionality. Personally, I'm fine with run-time errors as long as they are descriptive or the docs are well defined. At the very least, I'd rather deal with run-time errors if it means I have a provider that can be configured to work without out of process workarounds.

[SinisterMinister]

As for the acceptance tests, I get this error and it bails:

++ /usr/bin/docker cp 166158374a2af09bdb7424187629f6f9edd2186f61314658cc9837a02808065b:/tmp/testacc_kubeconfig.yaml /home/codey/go/src/github.com/terraform-providers/terraform-provider-rancher2/scripts/tmp/testacc_kubeconfig.yaml
Error: No such container:path: 166158374a2af09bdb7424187629f6f9edd2186f61314658cc9837a02808065b:/tmp/testacc_kubeconfig.yaml
+ cleanup
( . . . )
GNUmakefile:26: recipe for target 'testacc' failed
make: *** [testacc] Error 1

[SinisterMinister]

From the docs:

However, since provider configurations must be evaluated in order to perform any resource type action, provider configurations may refer only to values that are known before the configuration is applied. In particular, avoid referring to attributes exported by other resources unless their values are specified directly in the configuration.

Now I understand the hesitation to implement this functionality. Even if you were to split this into two separate providers, this guidance still advises avoiding such a pattern. So I guess given the guidance from Terraform, maybe this should be implemented with something like Terragrunt instead?

[drpebcak]

I guess it might be worth trying to get some guidance from Hashicorp on this, because it's a pretty common pattern. I'd hate to require some outside tooling like terragrunt just for this.

[SinisterMinister]

I’ve created a PR that solves the issue with this provider, but I’m not sure if it will get merged or not given the official guidance. It doesn’t solve the api_url field not working with computed values though as that’s handled by Terraform internally and would need to be set to optional to function in this manner.

[rawmind0]

@SinisterMinister, thanks for the PR but i think that at least we must assure that the auth parameters (token_key or access_key and secret_key) are set. Other point is check that the parameters are valid or not to make management client to connect on init time. This could be moved to run time.

I've made a quite similar PR to yours #67 but checking that auth parameters are at least set in the provider.

[SinisterMinister]

Hey, as long as it lets me create the cluster in one module and use it in another module in the same run, I'm happy with whatever implementation ships.

[rawmind0]

Merged PR. Please reopen issue if needed.