crossplane-contrib/terraform-provider-gen

This project is deprecated. See https://github.com/crossplane-contrib/terrajet.

terraform-provider-gen's Issues

Translate snake case fields to camel case

Terraform convention is to use snake case field names, while Kubernetes (and Crossplane) is to use camel case. It seems like we may need to convert the JSON struct tags of our generated types (and be able to map them back to the original snake case field names). There are modules that can help with this, e.g. https://github.com/iancoleman/strcase.

incorrect variable name in containerCollectionSingletonTypeDecodeTemplate

angryjet: error: error loading packages using pattern ./...: /Users/kasey/src/crossplane-contrib/provider-terraform-vsphere/generated/resources/vnic/v1alpha1/decode.go:98:21: undeclared name: f

I think this just needs to get changed to pass p to the submethods instead of f.

De-duplicate nested struct names

Within the same parent resource, multiple resources may exist with the same name. The attributes and nested blocks of these conflicting sub-resources may be the same, similar, or different. The resulting go code is invalid owing to multiple structs having the same name.

We need to build the capability to post-process these resources and rename the conflicts. A flexible way to do this would be a recursive visitor on the generator.ManagedResource type which is executed after the generator.Fields graphs have been created. This callback would hold a map[string]Field internally to track instances of a particular name so that the Field.Name of each duplicate instance for a given name can be changed before go code is generated.

For code clarity and readability, it could be preferable to detect identical resources structures and prefer these over divergent ones. eg if there are 2 conflicts with the same structure, and 1 which is different, allow the 2 identical fields to point at the same struct, and only rename the one that is different. For now we will simply rename all fields that conflict with one previous seen.

We need a reasonable scheme for renaming these fields. Because they are never surfaced to users, it should be sufficient to rename them something like <ConflictingResource>0...<ConflictingResource>N. Whenever a duplicate is detected, the rename function can increment the int following the base name in a loop until a free slot is found.

Deterministic Field order traversal

In solving #2 we add a new complication to the fact that the ordering of fields in generated code is somewhat non-deterministic. Deterministic field ordering would ensure that every subsequent rerun of code gen assigned the same duplicate number to the same instance of a field name.

More importantly, this would also produce cleaner diffs when code is regenerated in the future, streamlining review of generated code changes and enabling integration tests to use fixtures.

Generic ProviderConfig generation

Terraform gives us the schema of provider configuration fields which we can use to code generate a provider with initialization methods. Currently we are hard-coding these types and initialization methods. Since the pattern of having a username and password or token is fairly common, we could also have boilerplate config secret generation. The config yaml for the provider in /provider-configs/ could specify the names of fields in the secret which would be automatically mapped to the corresponding fields in the ProviderConfig -- these fields would also then be filtered out of the providerconfig spec.

Ordering of code generation stages

Currently there is a circular dependency in the code generation process. Before the controller-gen and angryjet tools are run, a generated type asserted as a resource.Managed or metav1.Object will be invalid, because the methodsets that satisfy these interfaces are not yet present.

controller-gen and angryjet can't run on invalid code, so the source files that rely on these type assertions or inferences (compare.go, encode.go, decode.go) can't be put in place until after these tools have successfully run.

The code generation tool needs to break up the process of generating source files into a couple distinct stages for this to flow cleanly from a single execution:

• generated/generate.go (which contains the go generate tags to run controller-gen and angryjet), and for each resource types.go and doc.go
• invoke go generate on the target source tree
• generate compare.go, configure.go, decode.go, encode.go, index.go`
• generate the index over all resource types (which is how main.go discovers them to be registered as controllers)

Note that all these files are currently generated, this issue is simply about refactoring the code that drives code generation to reorder them and take a break to run go generate after the first step.

decode.go generation does not work for slice types

go generate ./...
angryjet: error: error loading packages using pattern ./...: /Users/kasey/src/crossplane-contrib/provider-terraform-vsphere/generated/resources/host_port_group/v1alpha1/decode.go:165:20: cannot range over p (variable of type *[]Ports)
exit status 1
generated/generate.go:29: running "go": exit status 1

This was one of the cases marked as a TODO from the version of the generator demo'd at community day. The current state of this particular case is basically copy/paste from encode, so this task entails reversing the logic.

build tool automation to obtain terraform plugin

Currently the terraform plugin location is specified as a cli argument to the code generator or runtime. Obtaining the plugin by its name and version at codegen and build time would be a significant improvement to build automation.

Match the package layout of contemporary providers?

Currently the package layout of the providers we generate from Terraform is fairly different from our hand curated (or ACK generated) providers - e.g. type definitions and logic for each managed resource are under generated/resources/, while contemporary providers follow a fairly typical convention (perhaps from kubebuilder boilerplate?) of keeping types under apis/ and logic elsewhere (we've been gravitating toward putting it all under internal/ rather than cargo-culting the pkg/ convention).

Is there rationale behind this divergence, or is it mostly subjective?

Generate vmware vsphere provider

What problem are you facing?

In order to push terraform-provider-gen to generalize for other providers, and to test the value of it for long-tail providers, generate a provider for a provider that isn't aws/gcp/azure.

How could Crossplane help solve your problem?

We should consider how widely used terraform providers are in this decision. In looking at the ranking of terraform providers in the registry by downloads, the top provider that covers a cloud-like use case is vmware vsphere at about 2 million downloads.

A vsphere simulator called vcsim is available to aid in testing. As we do not have expertise in vsphere or an active environment, this project will initially be used for development and testing.

compare.go generation error: List/Singleton mismatch in some nested structs

in vsphere generated/resources/host_port_group/v1alpha1/compare.go
ie MergeHostPortGroup_Ports(k *Ports, p *Ports

where AtProvider.Ports is a list of Ports.

Improve error reporting

All Terraform operations return a structure[1] that is used to inspect errors that occurred processing the requested operation. There can be multiple errors in this collection. We should surface these -- maybe as k8s events -- to help with debugging.

[1] https://github.com/hashicorp/terraform/blob/master/tfdiags/diagnostics.go#L22

angryjet stopped generating methodsets for generated resources

As I attempted uncomment resources from the exclusion list to work on some of the other codegen issues, I saw that angryjet wasn't outputing zz_generated.managed.go or zz_generated.managedlist.go for any generated resources.