natikgadzhi/protoc-gen-tfschema

Protoc plugin for terraform providers

This repo is a spike, and is not maintained

Check out Teleport Plugins if you need Teleport's Terraform plugin.

protoc-gen-tfschema

protoc-gen-tfschema is a protoc plugin that generates Terraform-compatible schemas that you can use to build your own Terraform providers for a service that already has a GRPC API and defines it's with Protobuf.

protoc-gen-schema is in active development. Your contributions are very welcome. Maybe don't use it in production yet, as the schema output format may change.

Usage

You'd need several components to build a Terraform provider:

• your API client and settings to authenticate it
• resource schema definitions
• funcitons to create, update, delete, and read the data from your API
• as a part of those funcitons, marshalling and unmarshalling data between terraform and your API formats.

If your target app provides .proto description of it's API data types, you can use protoc-gen-tfschema to generate the resource definitions and marshalling/unmarshalling components automatically.

You'll still need to provide an API client initialization code, and call the API to execute CRUD features.

We assume that protobuf & protobuf-gen-go are installed on your system.

Here's how to generate resource schemas:

go install github.com/nategadzhi/protoc-gen-tfschema

protoc --proto_path ./proto/teleport --proto_path=. --proto_path=/Users/xnutsive/go/src/github.com/gravitational/teleport/vendor/github.com/gogo/protobuf --proto_path=/Users/xnutsive/go/src types.proto --tfschema_out=./out/teleport --go_out=./out/teleport --tfschema_opt="types=Metadata,UserSpecV2,UserV2"

You also could clone this repo and run:

make example # Builds example schema for Terraform

Command line options

--tfschema_opt= accepts several options:

• types defines what Protobuf Messages (types) to generate resource schemas for.
• pkgname defines what package name to use for the resulting files. Defaults to tfschema.

Features

• Resource Schemas: generate schemas for your resources with a single CLI command.
• Unmarshalling from Terraform data: generates functions that unmarshall data form Terraform to Go types generated from the Protobuf messages.

Things that are NOT supported yet

• The plugin can't generate Computed flag for schema fields. There's no way to know that the field is computed from the proto.

Next steps

• [*] An example project to test the generated schemas
• Provide unmarshalling generators
• Provide terraform data source schema generators
• Provide terraform schema importers, versions, and updaters

protoc-gen-tfschema's Issues

Regression: generated inline list Elem has Optional / Required field when it shouldn't

In the generated schema:

		// types.UserSpecV2.Roles
		"roles": {
			Type:     schema.TypeList,
			Optional: true,
			Elem: &schema.Schema{
				Type:     schema.TypeString,
				Optional: true,
			},
		},

When running terraform plan:

Error: InternalValidate
Internal validation of the provider failed! This is always a bug
with the provider itself, and not a user issue. Please report
this bug:
2 errors occurred:
        * resource teleport_user: roles: Elem must have only Type set
        * resource teleport_role: values: Elem must have only Type set

Wrapped protobuf values result in messy schemas

Teleport's proto uses wrapped values like these:

    // NodeLabels is a map of node labels (used to dynamically grant access to
    // nodes).
    wrappers.LabelValues NodeLabels = 3 [
        (gogoproto.nullable) = false,
        (gogoproto.jsontag) = "node_labels,omitempty",
        (gogoproto.customtype) = "Labels"
    ];

These tend to result in a recursive schema.Resource, but essentially, we could unwrap them and just declare the underlying type. Two lists of wrappers are interesting:

• Right now, we want to support teleport-specific wrappers. Supporting these will be great for now.
• There's also the full gogoproto wrappers proto here.

I don't see an obvious way for the schema generator to understand that some nested resource is essentially a wrapper and should be omitted, and other is a legit important nested structure that we don't want to squash.

One way to work with this would be to have a list of wrapper-types (like we already do with Timestamp) and a special function that would check if the field type is a wrapper and should be unwrapped, and if so — it would skip one level and insert the wrapped type's value type instead of the wrapper.

Regression: `types` flag not parsed in the builder template version

@gzigzigzeo,

types option stopped working correctly after the merge. If I put comma-separated list of types, as in the readme (I believe that's a legacy readme that I've written before), it errors out like this, and no schema is generated:

nate-mbp:~/g/s/g/g/t/terraform protoc --proto_path=. \                                                                                                            (base) 
                                 --proto_path=/Users/xnutsive/go/src/github.com/gravitational/teleport/vendor/github.com/gogo/protobuf \
                                 --proto_path=/Users/xnutsive/go/src/github.com/gravitational/teleport/api/types \
                                 --proto_path=/Users/xnutsive/go/src \
                                 types.proto \
                                 --tfschema_out=./tfschema/ \
                                 --go_out=./tfschema/ \
                                 --tfschema_opt="types=UserV2,RoleV3"
INFO[0000] Reading CodeGeneratorRequest from stdin.     
INFO[0000] Command-line arguments: types=UserV2,RoleV3  
FATA[0000] no such flag -RoleV3                         
--tfschema_out: protoc-gen-tfschema: Plugin failed with status code 1.
nate-mbp:~/g/s/g/g/t/terraform 

If I list types separated by a space (this way it worked before), it ignores provided types and schema file is generated, but it doesn't contain any schemas, just the package name and imports:

nate-mbp:~/g/s/g/g/t/terraform protoc --proto_path=. \                                                                                                            (base) 
                                 --proto_path=/Users/xnutsive/go/src/github.com/gravitational/teleport/vendor/github.com/gogo/protobuf \
                                 --proto_path=/Users/xnutsive/go/src/github.com/gravitational/teleport/api/types \
                                 --proto_path=/Users/xnutsive/go/src \
                                 types.proto \
                                 --tfschema_out=./tfschema/ \
                                 --go_out=./tfschema/ \
                                 --tfschema_opt="types=UserV2 RoleV3"
INFO[0000] Reading CodeGeneratorRequest from stdin.     
INFO[0000] Command-line arguments: types=UserV2 RoleV3  
2021/02/02 10:06:28 WARNING: Missing 'go_package' option in "github.com/gravitational/teleport/api/types/wrappers/wrappers.proto", please specify:
	option go_package = "github.com/gravitational/teleport/api/types/wrappers";
A future release of protoc-gen-go will require this be specified.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

2021/02/02 10:06:28 WARNING: Missing 'go_package' option in "types.proto",
please specify it with the full Go package path as
a future release of protoc-gen-go will require this be specified.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

INFO[0000] Package name: tfschema                       
INFO[0000] Types to export: [UserV2 RoleV3]             
INFO[0000] Protoc version: v3.14.0                      
INFO[0000] Generating schema files...                   
INFO[0000] types.tfschema.go prepared                   
INFO[0000] 1 files generated                            
2021/02/02 10:06:28 WARNING: Missing 'go_package' option in "github.com/gravitational/teleport/api/types/wrappers/wrappers.proto", please specify:
	option go_package = "github.com/gravitational/teleport/api/types/wrappers";
A future release of protoc-gen-go will require this be specified.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

2021/02/02 10:06:28 WARNING: Missing 'go_package' option in "types.proto",
please specify it with the full Go package path as
a future release of protoc-gen-go will require this be specified.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

schema:

// DO NOT EDIT.
// Code is generated by protoc-gen-tfschema 0.0.2
// Using protoc v3.14.0

package tfschema

import (
	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/validation"
)

And if I set no types flag at all — all types are generated. I'm fine with that for now ;)

I think the problem is that protoc plugin flags are not parsed as regular CLI flags, and the parser code from my version is required to do that correctly. It's a bummer and it looks like crap, but it works.