kubernetes-sigs/kubebuilder

Kubebuilder - SDK for building Kubernetes APIs using CRDs

Kubebuilder

Kubebuilder is a framework for building Kubernetes APIs using custom resource definitions (CRDs).

Similar to web development frameworks such as Ruby on Rails and SpringBoot, Kubebuilder increases velocity and reduces the complexity managed by developers for rapidly building and publishing Kubernetes APIs in Go. It builds on top of the canonical techniques used to build the core Kubernetes APIs to provide simple abstractions that reduce boilerplate and toil.

Kubebuilder does not exist as an example to copy-paste, but instead provides powerful libraries and tools to simplify building and publishing Kubernetes APIs from scratch. It provides a plugin architecture allowing users to take advantage of optional helpers and features. To learn more about this see the Plugin section.

Kubebuilder is developed on top of the controller-runtime and controller-tools libraries.

Kubebuilder is also a library

Kubebuilder is extensible and can be used as a library in other projects. Operator-SDK is a good example of a project that uses Kubebuilder as a library. Operator-SDK uses the plugin feature to include non-Go operators e.g. operator-sdk's Ansible and Helm-based language Operators.

To learn more see how to create your own plugins.

Installation

It is strongly recommended that you use a released version. Release binaries are available on the releases page. Follow the instructions to install Kubebuilder.

Getting Started

See the Getting Started documentation.

Also, ensure that you check out the Deploy Image Plugin. This plugin allows users to scaffold API/Controllers to deploy and manage an Operand (image) on the cluster following the guidelines and best practices. It abstracts the complexities of achieving this goal while allowing users to customize the generated code.

Documentation

Check out the Kubebuilder book.

Resources

• Kubebuilder Book: book.kubebuilder.io
• GitHub Repo: kubernetes-sigs/kubebuilder
• Slack channel: #kubebuilder
• Google Group: [email protected]
• Design Documents: designs
• Plugin: plugins

Motivation

Building Kubernetes tools and APIs involves making a lot of decisions and writing a lot of boilerplate.

In order to facilitate easily building Kubernetes APIs and tools using the canonical approach, this framework provides a collection of Kubernetes development tools to minimize toil.

Kubebuilder attempts to facilitate the following developer workflow for building APIs

1. Create a new project directory
2. Create one or more resource APIs as CRDs and then add fields to the resources
3. Implement reconcile loops in controllers and watch additional resources
4. Test by running against a cluster (self-installs CRDs and starts controllers automatically)
5. Update bootstrapped integration tests to test new fields and business logic
6. Build and publish a container from the provided Dockerfile

Scope

Building APIs using CRDs, Controllers and Admission Webhooks.

Philosophy

See DESIGN.md for the guiding principles of the various Kubebuilder projects.

TL;DR:

Provide clean library abstractions with clear and well exampled godocs.

• Prefer using go interfaces and libraries over relying on code generation
• Prefer using code generation over 1 time init of stubs
• Prefer 1 time init of stubs over forked and modified boilerplate
• Never fork and modify boilerplate

Techniques

• Provide higher level libraries on top of low level client libraries
  • Protect developers from breaking changes in low level libraries
  • Start minimal and provide progressive discovery of functionality
  • Provide sane defaults and allow users to override when they exist
• Provide code generators to maintain common boilerplate that can't be addressed by interfaces
  • Driven off of //+ comments
• Provide bootstrapping commands to initialize new packages

Versioning and Releasing

See VERSIONING.md.

Troubleshooting

• Bugs and Feature Requests:

  If you have what looks like a bug, or you would like to make a feature request, please use the Github issue tracking system. Before you file an issue, please search existing issues to see if your issue is already covered.

• Slack

  For realtime discussion, you can join the #kubebuilder slack channel. Slack requires registration, but the Kubernetes team is open invitation to anyone to register here. Feel free to come and ask any questions.

Contributing

Contributions are greatly appreciated. The maintainers actively manage the issues list, and try to highlight issues suitable for newcomers. The project follows the typical GitHub pull request model. See CONTRIBUTING.md for more details. Before starting any work, please either comment on an existing issue, or file a new one.

Supportability

Currently, Kubebuilder officially supports OSX and Linux platforms. So, if you are using a Windows OS you may find issues. Contributions towards supporting Windows are welcome.

Apple Silicon

Apple Silicon (darwin/arm64) support begins with the go/v4 plugin.

Community Meetings

The following meetings happen biweekly:

• Kubebuilder Meeting

You are more than welcome to attend. For further info join to [email protected]. Every month, our team meets on the first Thursday at 11:00 PT (Pacific Time) to discuss our progress and plan for the upcoming weeks.

kubebuilder's Issues

README is outdated wrt supported version of kubernetes

...and we should find a way to clarify which version of kubernetes are supported. 1.10 definitely is.

Some generated files are not well-formated

When walking through the files generated through kubebuilder, I found codes in some files are not well-aligned, for example in cmd/controller-manager/main.go:

import (
        "flag"
        "log"
    
    // Import auth/gcp to connect to GKE clusters remotely
    _ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
    
    configlib "github.com/kubernetes-sigs/kubebuilder/pkg/config"
        "github.com/kubernetes-sigs/kubebuilder/pkg/inject/run"
        "github.com/kubernetes-sigs/kubebuilder/pkg/install"
    "github.com/kubernetes-sigs/kubebuilder/pkg/signals"
        extensionsv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
    
    "example.com/pkg/inject"
    "example.com/pkg/inject/args"
)

var installCRDs = flag.Bool("install-crds", true, "install the CRDs used by the controller as part of startup")

// Controller-manager main.
func main() {
        flag.Parse()

    stopCh := signals.SetupSignalHandler()

Of course a user can easily format the files via go fmt, but for better experience, I think we'd better

1. modify existing templates and the place to generate files, to ensure the generated codes are well-formated
2. or maybe we can exec go format for output files automatically

@pwittrock WDYT?

Make it so we don't need to specify the environment variables when running integration tests

The test framework requires the following environment variables to be set:

  export TEST_ASSET_KUBECTL=/usr/local/kubebuilder/bin/kubectl
  export TEST_ASSET_KUBE_APISERVER=/usr/local/kubebuilder/bin/kube-apiserver
  export TEST_ASSET_ETCD=/usr/local/kubebuilder/bin/etcd

Can we do something in the test harness (e.g. _suite.go) to make it so these are defaulted to this location?

(M) - Rebase code on top of 1.10 kubernetes libraries

Standardize kubebuilder codegeneration labels to have +kubebuilder: prefix

Many of the codegen labels don't have this prefix

https://godoc.org/github.com/kubernetes-sigs/kubebuilder/pkg/gen/apis#example-package
https://godoc.org/github.com/kubernetes-sigs/kubebuilder/pkg/gen/controller

Must be done in a backward compatible way - add support for the new label and change the defaults created when bootstrapping.l

Create e2e testing for `kubebuilder docs`

Right an e2e test that runs kubebuilder docs and verifies the output.

• Run and make sure the command exists successfully producing and index.html
• Verify the contents of the docs/reference/includes/*, docs/reference/config.yaml and docs/reference/manifest.json files since they are relatively stable.

Add support for kubectl categories

Categories are groups of resources (example: all is a category). CRDs can be part of categories:

https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#categories

It would be useful to drive the categories of a resource via a kubebuilder:categories=foo,bar,zab annotation.

Running "kubebuilder docs" fails with 0.1.7

Steps to repro, in an empty directory in $GOPATH/src:

$ kubebuilder init --domain example.com
$ kubebuilder create resource --group example --version v1alpha1 --kind Example --controller=false
$ kubebuilder docs

Failure:

$ kubebuilder docs
2018/05/01 14:46:37 docker run -v $GOPATH/src/example.com/example:/host/repo -e DOMAIN=example.com -e DIR=src/example.com/example -e OUTPUT=docs/reference gcr.io/kubebuilder/gendocs
+ export D=src/example.com/example/docs/reference
+ D=src/example.com/example/docs/reference
+ mkdir -p src/example.com/example/docs/reference
+ cp -r /host/repo/pkg src/example.com/example/pkg
+ '[' -d /host/repo/docs/reference ']'
+ cp -r /host/repo/docs/reference/config.yaml /host/repo/docs/reference/examples /host/repo/docs/reference/openapi-spec /host/repo/docs/reference/static_includes src/example.com/exampledocs/reference
+ '[' '!' -d src/example.com/example/boilerplate.go.txt ']'
+ touch src/example.com/example/boilerplate.go.txt
+ cd src/example.com/example
+ apiserver-boot init repo --domain example.com
2018/05/01 21:46:38 installing vendor/ directory.  To disable this, run with --install-deps=false.
+ apiserver-boot build generated clean
+ apiserver-boot build generated
/usr/local/apiserver-builder/bin/apiregister-gen --input-dirs src/example.com/exampler/pkg/apis/... --input-dirs src/example.com/example/pkg/controller/...
2018/05/01 21:46:40 failed to run apiregister-gen panic: Could not find string matching // +domain=.+ in apis/doc.go
 
goroutine 1 [running]:
github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators.(*APIsBuilder).ParseDomain(0xc427e3a7e0)
        /Users/pwittroc/apiserver-builder/src/github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators/parser.go:566 +0x12d
github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators.NewAPIsBuilder(0xc427e13f20, 0xc42001a780, 0x4cba11)
        /Users/pwittroc/apiserver-builder/src/github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators/parser.go:186 +0x81
github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators.(*Gen).Packages(0xc42000a800, 0xc427e13f20, 0xc42001a780, 0x6000104, 0x0, 0xffffffffffffffff)
        /Users/pwittroc/apiserver-builder/src/github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators/package.go:89 +0xa1
github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators.(*Gen).Packages-fm(0xc427e13f20, 0xc42001a780, 0x765a6e, 0x6, 0xc427e13f20)
        /Users/pwittroc/apiserver-builder/src/github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators/package.go:45 +0x3e
github.com/kubernetes-incubator/apiserver-builder/cmd/vendor/k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc42001a780, 0xc4200f83f0, 0x765a6e, 0x6, 0xc426711ee8, 0x20, 0x718e60)
        /Users/pwittroc/apiserver-builder/src/github.com/kubernetes-incubator/apiserver-builder/cmd/vendor/k8s.io/gengo/args/args.go:165 +0x1e6
github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators.(*Gen).Execute(0xc42000a800, 0xc42001a780, 0x0, 0x0)
        /Users/pwittroc/apiserver-builder/src/github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/generators/package.go:45 +0x85
main.main()
        /Users/pwittroc/apiserver-builder/src/github.com/kubernetes-incubator/apiserver-builder/cmd/apiregister-gen/main.go:47 +0x136
 exit status 2
2018/05/01 14:46:41 error: exit status 1

Imports lowercased

Same issue as code-generator:

kubernetes/code-generator#20

I can work around it with a simple script that does a search/replace

kubebuilder resource creation fails when resource name is already its own plural

When I try to create a resource with a name that's already it's own plural, the resource creation fails:

$ kubebuilder create resource --group foo --version v1alpha1 --kind Metadata
2018/05/14 14:22:16 Client code generation expects singular --kind.

It looks like the code here is triggering this. Perhaps it would be useful to add a flag to bypass this check?

Update cloudbuild.yaml to automatically push GitHub releases on builds

Add more support for CRD validation

We parse the following tags, but don't actually set them on the CRDs:

• Enum
• Format
• MinLength
• MaxLength
• UniqueItems
• MinItems
• MaxItems

Figure out canonical story for CRD generation from goidl

The team working on apply is running into this same issue. We should work together to make sure we solve it once in a canonical fashion. Lets work with the folks building CRDs to come up with a design doc for the canonical way to generate CRDs from goidl.

• How to reference core API types without inlining / generating them
• How to deal with recursive types / self-referencing types (e.g. type has a field which is a pointer to itself)
• How to declare validation and format (already exists, but make sure this is the canonical way)
• How to declare openapi extensions and which extensions exist
• ???

Support pluralized Kinds

Generated files such as zz_generated.kubebuilder.go do not handle plural form kind name consistently. For instance, some informers call to AppReplicas kind via .AppReplicases(), but others via .AppReplicass().

Furthermore, k get redisreplicases failed to list the objects.

Perhaps we can conform all kind names to singular form.

Getting started link is broken

The following getting started link in the main README is broken.

http://book.kubebuilder.io/quick_start.html

Have Dockerfile.controller use CMD and ENTRYPOINT

Test against multiple versions of Kubernetes.

• Use the environment variables to test against N-2 versions of Kubernetes.

Check that the go version is 1.10+ in kubebuilder init

Kubebuilder doesn't work with go 1.8. We should check the go version is sufficient when creating a new project.

Write test to verify the go struct CRD and yaml config CRD are identical

The go struct CRD generated in the versioned resource package should match the install.yaml CRDs from create config. We should test this against some cases. Got a report the map[string][string] fields are represented differently.

(S) - Fix kubebuilder versioning

• version is missing from kubebuilder version
• version name is inconsistent - sometimes the v is dropped
• change versioning to be semantic - e.g. 0.1.5

Have Dockerfile.controller use from scratch instead of from ubuntu.

Make sure it actually works

Missing struct fields and import issues

Some generated structs have missing fields.
For instance, InjectArgs struct does not have some necessary fields such as Clientset:

arguments.Clientset undefined (type args.InjectArgs has no field or method Clientset)

There are several import issues after generating the codes too.
For instance, in /cmd/controller-manager/main.go, there is a conflicting import for k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1 due to 2 vendor packages in the repo (one is <pkg>/vendor, another is <pkg>/cmd/vendor).

Same errors happened after re-downloading kubebuilder and re-generating the files.

Consider copying over history from apiserver-builder

Figure out if there is anything in the commit history from the apiserver-builder repo we should copy over.

Generate checksums for builds

In the cloudbuild.yaml also create checksums for the binaries and upload them to gcs alongside the binaries. Look at how other projects do this such as etcd

Support for different reconcile events

Currently, Reconcile loop will be triggered periodically or by any event. It would be great if the controller provides the functionality to perform specific reconcile effort based on event type. For example, there are 3 loops for add, update, and delete events (or one reconcile loop that performs differently based on event type).

@pwittrock
/cc @liyinan926

Use Kubernetes Godep.json from 1.9 and `dep init` to make sure all the dependencies are the same

Release CHANGELOG publishes all changes since the first commit

The change log for the most recent release includes the very first commit ever :P

CRD Schema generation to support custom formats

To fix #62.

Allow users to annotate fields with the format so they can specify "email" or "date" formats for strings.

generate e2e testing framework

Kubebuilder generates integration testing framework in pkg/controller/<some-controller-name>/controller_test.go. It would be great if kubebuilder also generates e2e testing framework too.

/cc @droot @pwittrock

Double check that CRD spec uses the right names

Not sure if this is still an issue.

CRD spec lists bool, when it should be boolean, int32 should be integer

AdditionalProperties CRD validation bug

When running kubebuilder create config to generate a controller image, I encountered a CRD validation
bug that specifies AdditionalConfiguration field cannot be set to false.

For instance, I have the bug for options field in my types.go file:

// Options is additional configuration options that will be passed to the pods.
// +optional
Options map[string]string `json:"options,omitempty"`

In zz_generated.kubebuilder.go file, the AdditionalConfiguration field is initialized as follow:

AdditionalProperties: &v1beta1.JSONSchemaPropsOrBool{
        Allows: true,
	//Schema: &,
},

@Liujingfang1

Allow resources to be annotated with warnings and notes with `// +kubebuilder:docs`

kubebuilder docs generates a config.yaml file that is used to control the table of contents for docs. The config.yaml can also be used to set warnings and description notes on specific APIs, however since the file is generated, users shouldn't edit it directly.

Instead we should support the following annotations on resource structs:

• // +kubebuilder:docs:warning
• // +kubebuilder:docs:note

These should be added to the generated config.yaml after group here as is shown here and here

Support for old object access

Kubebuilder should support access to old objects. Currently, the Reconcile loop only provides a key. There is no easy way to access the old objects after update operations. A workaround is to save the object state periodically via global variables, but there should be a better way to access the old objects.

@pwittrock
/cc @liyinan926

Add section to book about creating events for CRDs.

(M) - Make `dep ensure` idempotent

AAUI can run dep ensure to install the dependencies for building a project with Kubebuilder and I can add new dependencies to the Gopkg.toml file and run dep ensure to additional pkgs.

The Gopkg.toml could use some cleanup

• Replace the kubebuilder libraries version in the Gopkg.toml with the current tag when building
• Add kubebuilder libraries to the Gopkg.lock
• Figure out if the ignore lines are required
• Make sure dep runs cleanly on clients. Fix errors:

Warning: the following project(s) have [[constraint]] stanzas in Gopkg.toml:

  ✗  github.com/pkg/errors
  ✗  github.com/spf13/cobra
  ✗  k8s.io/apiserver
  ✗  k8s.io/kube-aggregator

Improve test coverage of WatchControllerOf method of GenericController

We realized we have less than adequate test coverage of Generic Controller's watchControllerOf API. For ex. this fix uncovered a bug where we end up an infinite loop be1a28b

Access denied when downloading a release

When trying to download the release under https://storage.googleapis.com/kubebuilder-release/kubebuilder-release-v1.9-alpha.30-darwin-amd64.tar.gz I get an access denied message:

wget https://storage.googleapis.com/kubebuilder-release/kubebuilder-release-v1.9-alpha.30-darwin-amd64.tar.gz  
--2018-03-15 10:55:18-- https://storage.googleapis.com/kubebuilder-release/kubebuilder-release-v1.9-alpha.30-darwin-amd64.tar.gz
Resolving storage.googleapis.com... 172.217.22.240
Connecting to storage.googleapis.com|172.217.22.240|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2018-03-15 10:55:20 ERROR 403: Forbidden.

Add integration test for backwards compatibility

Check the following are backwards compatible in a test:

• Libraries
• Creation commands

Make `create resource` also create a Recorder.

The generated code from create resource should create a Recorder

Add support for admission webhooks

• Clean packages similar to go core http package
• Auto wire the handlers using the inject code generator from generate
• Install the webhook (no certs) in the config from create config
• Add an RBAC rule for create config that allows the controller to update the WebHook (and only that webhook) if installing a webhook.
• Have the controller write the certs in the WH when it starts

e2e test framework for kubebuilder

We need to be able to run e2e tests that

New project

• create a new project by running kubebuilder commands
• copy over kubebuilder pkgs into vendor
• create new resources from kubebuilder commands
• modify some files (maybe by copying other files to overwrite)
• run code generators and build
• standup test controlplane using the framework
• run kubectl commands
• look at output from controller logs

Define how to express uri and email forms (Renamed)

Is there a way to specify validation for things just as email and URL/URI?

Indicate in generated CRDs that they were built by kube-builder

add annotation to the CRD itself like kubebuilder.k8s.io/version: v1.2.3

Version helps with debugging reports.

(S) Define list of tests needed (e2e and unit)

Incorrect pluralize when Kind name ends with an 's'

I am getting following error when trying to create a resource with kind as 'Postgres'.

$ kubebuilder create resource --group group1 --version v1alpha1 --kind Postgres

2018/05/14 16:18:00 Client code generation expects singular --kind.

I think line 47 in https://github.com/kubernetes-sigs/kubebuilder/blob/f6d054862aeb05016b0b5c1c39d4f867d93dc944/cmd/kubebuilder/create/util/util.go needs to accommodate Kind names which are singular and ending with a 's'.

(L) Support for creating controllers for core types

Watching and syncing resources created to support CRD objects

I have a use case that as part of handling a newly added CRD object, a StatefulSet gets created and owned by the CRD object (through an OwnerReference added on the StatefulSet). The generated controller code for the CRD resource allows me to add a watch on StatefulSets. But how can I register/add functions that react to events on the StatefulSets and reconcile them?

Also it appears to me that watches with custom label selectors are not yet supported.

Make create resource copy create a controller struct that has InjectArguments instead of individual fields

The created code would be simpler if it just uses the InjectArguments directly

Keep descriptions and warnings in docs/reference config.yaml when regenerating

We generate docs/reference/config.yaml from the resource types we find, however users may add notes and warnings to the config.yaml. We should look for these when regenerating, and keep them.

"kubebuilder update vendor" should update the Gopkg files as well

Currently, kubebuilder update vendor only updates the vendor directory, not the Gopkg.lock and Gopkg.toml files. This leaves them out-of-sync, which is not ideal for understanding the versions of dependencies in the repo.

Alternatively, kubebuilder could not initialize a project with the Gopkg files, and leave that to the project's discretion.