crossplane-contrib / provider-jet-template Goto Github PK
View Code? Open in Web Editor NEWA template for building terrajet based @crossplane providers
License: Apache License 2.0
A template for building terrajet based @crossplane providers
License: Apache License 2.0
the make file appears to not work, im not sure if there is dependencies im expected to do but they arent in the guide
make generate
make: *** No rule to make target 'generate'. Stop.
even just make does
make
make: installing: Command not found
Makefile:64: recipe for target '/terraform-1.1.6' failed
make: *** [/terraform-1.1.6] Error 127
follow the guide on ubuntu wsl
panic error during make generate
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x1294e56]
goroutine 1 [running]:
github.com/crossplane/terrajet/pkg/types.init.0()
/home/rspazzol/git/crossplane-github-provider/.work/pkg/pkg/mod/github.com/crossplane/[email protected]/pkg/types/reference.go:73 +0x116
exit status 2
apis/generate.go:33: running "go": exit status 1
06:50:29 [FAIL]
make[1]: *** [build/makelib/golang.mk:271: go.generate] Error 1
make: *** [build/makelib/common.mk:419: generate] Error 2
this error does not happen with go 1.17, it seems to happen with following versions of go 1.18 and 1.19.
template provider should be tested with those versions of go, or upgraded to them.
There is an error when trying to generate an azuread resource
1.6457344224525993e+09 DEBUG events Warning {"object": {"kind":"Application","name":"azuread-application","uid":"9b8ccec1-484b-4faf-9e01-92f1b5ab2ad3","apiVersion":"azuread.azuread.jet.crossplane.io/v1alpha1","resourceVersion":"7502087"}, "reason": "CannotObserveExternalResource", "message": "cannot run refresh: refresh failed: Extraneous JSON object property: No argument or block type is named \"name\".: File name: main.tf.json"}
1.6457344824875672e+09 DEBUG provider-jet-azuread Reconciling {"controller": "managed/azuread.azuread.jet.crossplane.io/v1alpha1, kind=application", "request": "/azuread-application"}
1.6457344826753385e+09 DEBUG provider-jet-azuread refresh ended {"workspace": "/tmp/9b8ccec1-484b-4faf-9e01-92f1b5ab2ad3", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.1.6\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-02-24T20:28:02.519063Z\",\"terraform\":\"1.1.6\",\"type\":\"version\",\"ui\":\"1.0\"}\n{\"@level\":\"error\",\"@message\":\"Error: Extraneous JSON object property\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-02-24T20:28:02.672277Z\",\"diagnostic\":{\"severity\":\"error\",\"summary\":\"Extraneous JSON object property\",\"detail\":\"No argument or block type is named \\\"name\\\".\",\"range\":{\"filename\":\"main.tf.json\",\"start\":{\"line\":1,\"column\":311,\"byte\":310},\"end\":{\"line\":1,\"column\":317,\"byte\":316}},\"snippet\":{\"context\":\"resource.azuread_application.azuread-application\",\"code\":\"{\\\"provider\\\":{\\\"azuread\\\":{\\\"client_id\\\":\\\"XXXXXXXXXXXXXXXXXXXXXXXXX\\\",\\\"client_secret\\\":\\\"XXXXXXXXXXXXXXXXXXXXXXXXX\\\",\\\"tenant_id\\\":\\\"XXXXXXXXXXXXXXXXXXXXXXXXX\\\"}},\\\"resource\\\":{\\\"azuread_application\\\":{\\\"azuread-application\\\":{\\\"display_name\\\":\\\"test_application\\\",\\\"lifecycle\\\":{\\\"prevent_destroy\\\":true},\\\"name\\\":\\\"azuread-application\\\"}}},\\\"terraform\\\":{\\\"required_providers\\\":{\\\"azuread\\\":{\\\"source\\\":\\\"hashicorp/azuread\\\",\\\"version\\\":\\\"2.18.0\\\"}}}}\",\"start_line\":1,\"highlight_start_offset\":310,\"highlight_end_offset\":316,\"values\":[]}},\"type\":\"diagnostic\"}\n"}
1.645734482675425e+09 DEBUG provider-jet-azuread Cannot observe external resource {"controller": "managed/azuread.azuread.jet.crossplane.io/v1alpha1, kind=application", "request": "/azuread-application", "uid": "9b8ccec1-484b-4faf-9e01-92f1b5ab2ad3", "version": "7502087", "external-name": "azuread-application", "error": "cannot run refresh: refresh failed: Extraneous JSON object property: No argument or block type is named \"name\".: File name: main.tf.json", "errorVerbose": "refresh failed: Extraneous JSON object property: No argument or block type is named \"name\".: File name: main.tf.json\ncannot run refresh\ngithub.com/crossplane/terrajet/pkg/controller.(*external).Observe\n\t/mnt/c/dev/git/provider-jet-azuread/.work/pkg/pkg/mod/github.com/crossplane/[email protected]/pkg/controller/external.go:125\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\t/mnt/c/dev/git/provider-jet-azuread/.work/pkg/pkg/mod/github.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:758\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\t/mnt/c/dev/git/provider-jet-azuread/.work/pkg/pkg/mod/github.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/mnt/c/dev/git/provider-jet-azuread/.work/pkg/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/mnt/c/dev/git/provider-jet-azuread/.work/pkg/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/mnt/c/dev/git/provider-jet-azuread/.work/pkg/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/mnt/c/dev/git/provider-jet-azuread/.work/pkg/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/mnt/c/Users/Mike.Bolland/go/src/runtime/asm_amd64.s:1581"}
1.6457344826756868e+09 DEBUG events Warning {"object": {"kind":"Application","name":"azuread-application","uid":"9b8ccec1-484b-4faf-9e01-92f1b5ab2ad3","apiVersion":"azuread.azuread.jet.crossplane.io/v1alpha1","resourceVersion":"7502087"}, "reason": "CannotObserveExternalResource", "message": "cannot run refresh: refresh failed: Extraneous JSON object property: No argument or block type is named \"name\".: File name: main.tf.json"}
1.6457345090279298e+09 DEBUG provider-jet-azuread Reconciling {"controller": "managed/azuread.azuread.jet.crossplane.io/v1alpha1, kind=application", "request": "/azuread-application"}
1.645734509027983e+09 DEBUG provider-jet-azuread Cannot get managed resource {"controller": "managed/azuread.azuread.jet.crossplane.io/v1alpha1, kind=application", "request": "/azuread-application", "error": "Application.azuread.azuread.jet.crossplane.io \"azuread-application\" not found"}
1.6457345093279011e+09 DEBUG provider-jet-azuread Reconciling {"controller": "providerconfig/providerconfig.azuread.jet.crossplane.io", "request": "/azuread-creds-next"}
1.6457345094354095e+09 DEBUG provider-jet-azuread Reconciling {"controller": "providerconfig/providerconfig.azuread.jet.crossplane.io", "request": "/azuread-creds-next"}
1.6457345431087437e+09 DEBUG provider-jet-azuread Reconciling {"controller": "managed/azuread.azuread.jet.crossplane.io/v1alpha1, kind=application", "request": "/azuread-application"}
1.645734543108836e+09 DEBUG provider-jet-azuread Cannot get managed resource {"controller": "managed/azuread.azuread.jet.crossplane.io/v1alpha1, kind=application", "request": "/azuread-application", "error": "Application.azuread.azuread.jet.crossplane.io \"azuread-application\" not found"}
Follow the generating a provider guide on the website and use azuread as your terraform provider
Crossplane version:
crossplane/provider-aws:v0.24.1
k8s: v1.21.5-eks-bc4871b
The Readme is referring to make image
and make push
but when using them an error is returned:
make: *** No rule to make target `image'. Stop.
make: *** No rule to make target `push'. Stop.
make image
Get error. File to remove does not exist
make build or make publish
NA
apiVersion: branch.github.jet.crossplane.io/v1alpha1
kind: Branch
metadata:
labels:
app.kubernetes.io/instance: github-managed-repos-poc
name: hello-terrajet
spec:
forProvider:
branch: hello-terrajet
repositoryRef:
name: repository-hello-crossplane-pre-existing-repo
providerConfigRef:
name: github-provider
The error is:
CannotConnectToProvider
cannot get a terraform workspace for resource: cannot create directory for workspace: mkdir /tmp: permission denied
If I roll back the changes in provider-jet-github-controller/Dockerfile to initial commit then everything works.
Note the work around in issue #25 was required to get make build and make publish to work
Use provider-jet-github-controller/Dockerfile in branch main to build the controller image then deploy a provider resource
Crossplane version:
Provider version:
I'm creating a github jet provider following the instruction and startigw tih this povider template. When I create a git resource , I get the following error:
1.6608241939409325e+09 DEBUG provider-jet-github Reconciling {"controller": "managed/organizationwebhook.github.jet.crossplane.io/v1alpha1, kind=webhook", "request": "/backstage"}
1.660824194043038e+09 DEBUG provider-jet-github init ended {"workspace": "/tmp/549720f8-c01b-4c47-bed8-99348d5bdeb1", "out": "\n\u001b[0m\u001b[1mInitializing provider plugins...\u001b[0m\n\nThe following providers do not have any version constraints in configuration,\nso the latest version was installed.\n\nTo prevent automatic upgrades to new major versions that may contain breaking\nchanges, it is recommended to add version = \"...\" constraints to the\ncorresponding provider blocks in configuration, with the constraint strings\nsuggested below.\n\n* provider.github: version = \"~> 4.9\"\n\n\u001b[33m\n\u001b[1m\u001b[33mWarning: \u001b[0m\u001b[0m\u001b[1mSkipping backend initialization pending configuration upgrade\u001b[0m\n\n\u001b[0mThe root module configuration contains errors that may be fixed by running the\nconfiguration upgrade tool, so Terraform is skipping backend initialization.\nSee below for more information.\n\u001b[0m\u001b[0m\n\u001b[0m\u001b[32m\n\u001b[0m\u001b[1mTerraform has initialized, but configuration upgrades may be needed.\u001b[0m\n\nTerraform found syntax errors in the configuration that prevented full\ninitialization. If you've recently upgraded to Terraform v0.12, this may be\nbecause your configuration uses syntax constructs that are no longer valid,\nand so must be updated before full initialization is possible.\n\nTerraform has installed the required providers to support the configuration\nupgrade process. To begin upgrading your configuration, run the following:\n terraform 0.12upgrade\n\nTo see the full set of errors that led to this message, run:\n terraform validate\u001b[0m\n"}
1.6608241940600905e+09 DEBUG provider-jet-github refresh ended {"workspace": "/tmp/549720f8-c01b-4c47-bed8-99348d5bdeb1", "out": "\u001b[31mUsage: terraform apply [options] [DIR-OR-PLAN]\n\n Builds or changes infrastructure according to Terraform configuration\n files in DIR.\n\n By default, apply scans the current directory for the configuration\n and applies the changes appropriately. However, a path to another\n configuration or an execution plan can be provided. Execution plans can be\n used to only execute a pre-determined set of actions.\n\nOptions:\n\n -backup=path Path to backup the existing state file before\n modifying. Defaults to the \"-state-out\" path with\n \".backup\" extension. Set to \"-\" to disable backup.\n\n -auto-approve Skip interactive approval of plan before applying.\n\n -lock=true Lock the state file when locking is supported.\n\n -lock-timeout=0s Duration to retry a state lock.\n\n -input=true Ask for input for variables if not directly set.\n\n -no-color If specified, output won't contain any color.\n\n -parallelism=n Limit the number of parallel resource operations.\n Defaults to 10.\n\n -refresh=true Update state prior to checking for differences. This\n has no effect if a plan file is given to apply.\n\n -state=path Path to read and save state (unless state-out\n is specified). Defaults to \"terraform.tfstate\".\n\n -state-out=path Path to write state to that is different than\n \"-state\". This can be used to preserve the old\n state.\n\n -target=resource Resource to target. Operation will be limited to this\n resource and its dependencies. This flag can be used\n multiple times.\n\n -var 'foo=bar' Set a variable in the Terraform configuration. This\n flag can be set multiple times.\n\n -var-file=foo Set variables in the Terraform configuration from\n a file. If \"terraform.tfvars\" or any \".auto.tfvars\"\n files are present, they will be automatically loaded.\u001b[0m\u001b[0m\n"}
1.6608241339366581e+09 DEBUG provider-jet-github Cannot observe external resource {"controller": "managed/organizationwebhook.github.jet.crossplane.io/v1alpha1, kind=webhook", "request": "/backstage", "uid": "549720f8-c01b-4c47-bed8-99348d5bdeb1", "version": "11680", "external-name": "backstage", "error": "cannot run refresh: readObjectStart: expect { or n, but found \u001b, error found in #1 byte of ...|\u001b[31mUsage:|..., bigger context ...|\u001b[31mUsage: terraform apply [options] [DIR-OR-PLAN]|...: refresh failed", "errorVerbose": "refresh failed\nreadObjectStart: expect { or n, but found \u001b, error found in #1 byte of ...|\u001b[31mUsage:|..., bigger context ...|\u001b[31mUsage: terraform apply [options] [DIR-OR-PLAN]|...\ncannot run refresh\ngithub.com/crossplane/terrajet/pkg/controller.(*external).Observe\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/github.com/crossplane/[email protected]/pkg/controller/external.go:125\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/github.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:767\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/github.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1571"}
1.6608241339367838e+09 DEBUG events Warning {"object": {"kind":"Webhook","name":"backstage","uid":"549720f8-c01b-4c47-bed8-99348d5bdeb1","apiVersion":"organizationwebhook.github.jet.crossplane.io/v1alpha1","resourceVersion":"11680"}, "reason": "CannotObserveExternalResource", "message": "cannot run refresh: readObjectStart: expect { or n, but found \u001b, error found in #1 byte of ...|\u001b[31mUsage:|..., bigger context ...|\u001b[31mUsage: terraform apply [options] [DIR-OR-PLAN]|...: refresh failed"}
It looks like a parsing issue due to this unexpected character \u001b
.
This is my provider config:
apiVersion: v1
stringData:
credentials: |
{
"base_url": "http://github.com/",
"owner": "raf-backstage-demo",
"token": "redacted"
}
kind: Secret
metadata:
name: example-creds
namespace: crossplane-system
type: Opaque
and this is the code to read it:
package clients
import (
"context"
"encoding/json"
"github.com/crossplane/crossplane-runtime/pkg/resource"
"github.com/pkg/errors"
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/client"
"github.com/crossplane/terrajet/pkg/terraform"
"github.com/crossplane-contrib/provider-jet-github/apis/v1alpha1"
)
const (
keyBaseURL = "base_url"
keyOwner = "owner"
keyToken = "token"
)
const (
// error messages
errNoProviderConfig = "no providerConfigRef provided"
errGetProviderConfig = "cannot get referenced ProviderConfig"
errTrackUsage = "cannot track ProviderConfig usage"
errExtractCredentials = "cannot extract credentials"
errUnmarshalCredentials = "cannot unmarshal github credentials as JSON"
)
// TerraformSetupBuilder builds Terraform a terraform.SetupFn function which
// returns Terraform provider setup configuration
func TerraformSetupBuilder(version, providerSource, providerVersion string) terraform.SetupFn {
return func(ctx context.Context, client client.Client, mg resource.Managed) (terraform.Setup, error) {
ps := terraform.Setup{
Version: version,
Requirement: terraform.ProviderRequirement{
Source: providerSource,
Version: providerVersion,
},
}
configRef := mg.GetProviderConfigReference()
if configRef == nil {
return ps, errors.New(errNoProviderConfig)
}
pc := &v1alpha1.ProviderConfig{}
if err := client.Get(ctx, types.NamespacedName{Name: configRef.Name}, pc); err != nil {
return ps, errors.Wrap(err, errGetProviderConfig)
}
t := resource.NewProviderConfigUsageTracker(client, &v1alpha1.ProviderConfigUsage{})
if err := t.Track(ctx, mg); err != nil {
return ps, errors.Wrap(err, errTrackUsage)
}
data, err := resource.CommonCredentialExtractor(ctx, pc.Spec.Credentials.Source, client, pc.Spec.Credentials.CommonCredentialSelectors)
if err != nil {
return ps, errors.Wrap(err, errExtractCredentials)
}
githubCreds := map[string]string{}
if err := json.Unmarshal(data, &githubCreds); err != nil {
return ps, errors.Wrap(err, errUnmarshalCredentials)
}
// set environment variables for sensitive provider configuration
// Deprecated: In shared gRPC mode we do not support injecting
// credentials via the environment variables. You should specify
// credentials via the Terraform main.tf.json instead.
/*ps.Env = []string{
fmt.Sprintf("%s=%s", "HASHICUPS_USERNAME", githubCreds["username"]),
fmt.Sprintf("%s=%s", "HASHICUPS_PASSWORD", githubCreds["password"]),
}*/
// set credentials in Terraform provider configuration
ps.Configuration = map[string]interface{}{
keyBaseURL: githubCreds[keyBaseURL],
keyOwner: githubCreds[keyOwner],
keyToken: githubCreds[keyToken],
}
return ps, nil
}
}
Can someone help me troubleshoot this issue?
None of the error messages in the provider initialization code seems to be triggered and yet the error.
What is the code trying to do when the error happens? Is it already trying to connect to github?
What is returning that unexpected character?
my code is here: https://github.com/raffaelespazzoli/provider-jet-github
running the code with make run
should allow to reproduce the issue.
Crossplane version: latest
Provider version: N/A
running on local minikube
running with this version 4.28.0
of the terraform-provider-github
The first thing users will want to do with their new provider is open PRs (make reviewable
?) and publish the images to the Upbound registry or others.
Make it easy to get started by wiring up common settings. The Terrajet guide should include any search/replace needed to get .github
updated for their environment.
Hi, why terrajet version is not v0.4.2 at provider-jet-template config?
github.com/crossplane/terrajet v0.4.0-rc.0.0.20220510203225-5e7094f2ea5c
https://github.com/crossplane-contrib/provider-jet-template/blob/main/go.mod#L8
https://github.com/crossplane-contrib/provider-jet-template/blob/main/go.sum#L159
Maybe v4.0.2 is not supported?
https://github.com/crossplane/terrajet/releases/tag/v0.4.2
Thanks.
Currently the template hard-codes github.com/crossplane-contrib/
without ability to update the source prefix with environment variables like ProviderNameUpper
| ProviderNameLower
. Also why not use env vars for the following as well in the docs instead of editing Makefile?
export TERRAFORM_PROVIDER_SOURCE := hashicorp/null
export TERRAFORM_PROVIDER_VERSION := 3.1.0
export TERRAFORM_PROVIDER_DOWNLOAD_NAME := terraform-provider-null
export TERRAFORM_PROVIDER_DOWNLOAD_URL_PREFIX
clone template and follow guide
Crossplane version: 1.7
Provider version: 3.1.0
Terrajet version: 0.4.2
Terraform version: 1.1.6
Hi. I have some issue with the terrajet version because some features need the terrajet 0.4.2. How can i upgrade the terrajet version in my own repo ?
The change i need :
crossplane/terrajet@4238a8c
But i can not edit it manually because it is in a generated file with no write rights.
The steps i followed to genreate a crossplane provider :
https://github.com/crossplane/terrajet/blob/main/docs/generating-a-provider.md
Terraform v1.2.8
go version go1.19.1
linux/amd64
It appears like the Terrajet documentation (and related provider-jet-template Makefile) for generating a provider, only works with Terraform providers that are downloadable from a Terraform registry.
If an organization has a custom/internal Terraform provider, or even one that is public like todo-for-terraform
it is not clear how to point at this, so that a crossplane provider can be created.
Update the documentation and provider-jet-template to cover this (and similar) use cases.
Looks like as we consume sigs.k8s.io/[email protected]
as we update the terrajet and crossplane-runtime dependencies, we need to remove the crds.clean
make target.
Please see crossplane-contrib/provider-jet-azure#143 and https://github.com/crossplane-contrib/provider-jet-azure/pull/143/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.