kptdev/kpt

Automate Kubernetes Configuration Editing

kpt: Automate Kubernetes Configuration Editing

kpt is a package-centric toolchain that enables a WYSIWYG configuration authoring, automation, and delivery experience, which simplifies managing Kubernetes platforms and KRM-driven infrastructure (e.g., Config Connector, Crossplane) at scale by manipulating declarative Configuration as Data.

Configuration as Data is an approach to management of configuration which:

• makes configuration data the source of truth, stored separately from the live state
• uses a uniform, serializable data model to represent configuration
• separates code that acts on the configuration from the data and from packages / bundles of the data
• abstracts configuration file structure and storage from operations that act upon the configuration data; clients manipulating configuration data don’t need to directly interact with storage (git, container images).

See the FAQ for more details about how kpt is different from alternatives.

Why kpt?

kpt enables WYSIWYG editing and interoperable automation applied to declarative configuration data, similar to how the live state can be modified with imperative tools.

See the rationale for more background.

The best place to get started and learn about specific features of kpt is to visit the kpt website.

Install kpt

kpt installation instructions can be found on kpt.dev/installation

kpt components

The kpt toolchain includes the following components:

• kpt CLI: The kpt CLI supports package and function operations, and also deployment, via either direct apply or GitOps. By keeping an inventory of deployed resources, kpt enables resource pruning, aggregated status and observability, and an improved preview experience.

• Function SDKs: Any general-purpose or domain-specific language can be used to create functions to transform and/or validate the YAML KRM input/output format, but we provide SDKs to simplify the function authoring process, in Go, Typescript, and Starlark, a Python-like embedded language.

• Function catalog: A catalog of off-the-shelf, tested functions. kpt makes configuration easy to create and transform, via reusable functions. Because they are expected to be used for in-place transformation, the functions need to be idempotent.

• Package orchestrator: The package orchestrator enables the magic behind the unique WYSIWYG experience. It provides a control plane for creating, modifying, updating, and deleting packages, and evaluating functions on package data. This enables operations on packaged resources similar to operations directly on the live state through the Kubernetes API.

• Config Sync: While the package orchestrator can be used with any GitOps tool, Config Sync provides a reference GitOps implementation to complete the WYSIWYG management experience and enable end-to-end development of new features, such as OCI-based packages. Config Sync is also helping to drive improvements in upstream Kubernetes. For instance, Config Sync is built on top of git-sync and leverages Kustomize to automatically render manifests on the fly when needed. It uses the same apply logic as the kpt CLI.

• Backstage UI plugin: We've created a proof-of-concept UI in the form of a Backstage UI plugin to demonstrate the WYSIWYG experience that's possible on top of the package orchestrator. More scenarios can be supported by implementing form-based editors for additional Kubernetes resource types.

Roadmap

You can read about the big upcoming features in the roadmap doc.

Contributing

If you are interested in contributing please start with contribution guidelines.

Contact

We would love to keep in touch:

1. Join our Slack channel. You'll need to join Kubernetes on Slack first.
2. Join our email list

kpt's Issues

Don't introduce CRD support as first thinking

Defer explaining CRD support until the basics are clear.

npm init should install the Kubernetes 1.17 types by default without the need for a cluster. generating from a cluster should be opt-in

e2e tutorial for `anthos cli`

Create timeout for reconcile containers

• If you kill kpt reconcile it should kill the containers
• kpt reconcile should timeout

clean up documentation

Support [email protected]: syntax

parse : as separator

In comprehensive guide -- inline examples

The guide links to a set of examples, but its unclear which ones are good examples to start with. Also just show me the example next to the interface so I can compare them.

e2e tutorial for acm

test Kpt tree and grep commands

• invert matches
• multiple level list matches

Enable GitHub pages

Add `istioctl analyze`

Cat, Grep, Tree commands

Support Cat, Grep, Tree Commands

Add tests for custom commands

e2e tutorial for `app delivery`

Move code to GitHub repo

Generate documentation from built in command help

Similar to how we generate .md docs from the built-in command help, we should generate a nicer docs page. Consider using: https://mmistakes.github.io/minimal-mistakes/

Documentation usability feedback for run.md

https://github.com/GoogleContainerTools/kpt/blob/master/docs/fn/run.md

• Caveats section is very long since it's explaining multiple concepts. It's probably worth creating subsection for each
• Need to use verbatim code style in many instances. e.g. metadata.annotations.[config.kubernetes.io/function]. Otherwise, it's hard to read.
• Config Functions -> KPT Functions ?
• We should simplify number of ways to invoke functions declarativey. Or at least, emphasize one approach in docs so user doesn't get overwhelmed.
• It's ambiguous whether functions directory is required or just a convention used (functions dir is in fact a special dir recognized by kpt).
• Also note that the implicit pattern is currently broken: #188

Follow up on yaml decoding / encoding issue for copyright header comments

There is a bug where the yaml libraries decode

old: resource
---
# Copyright
new: resource

as

old: resource
# Copyright
---
new: resource

To work around this, the ByteWriter replaces --- with \n---\n, which fixes the issue so that the decoded properly recognizes the resource / comment boundaries.

Follow up on this to see if it is an issue in the underlying libraries, or just how we use them.

setup CI for the repo

Support KRM merge for updates

`kpt svr status fetch` messes up the shell color scheme

Note how after running the command, it didn't restore the font color:

[doc] `kpt svr apply` doc needs to be more detailed

I thought it would be mostly similar to kubectl apply, but I got an error that didn't make sense to me:

$ kpt svr apply .
error: unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/Users/davidebbo/tmp/tmp/helloworld'

The doc doesn't mention needing a kustomization file, so I'm not sure what I did wrong.

Support adding Reconcile image to Kptfile

Broken links in docs

in docs/fn/README.md:

[kpt-functions-sdk]
[kustomize/kyaml]

Add ACM source, sink, vet functions

source: Hydrate ACM repo
vet: Run validations rules
sink: Write configs into ACM repo format

make all binaries and container images public

Link to KPT Functions Catalog

Either in:

https://github.com/GoogleContainerTools/kpt/blob/master/docs/fn/README.md

or one of the child commands, it'd be good to have a link to KPT Functions Catalog:

https://github.com/GoogleContainerTools/kpt-functions-catalog

test Kptfile update behaviors

consider renaming insert to create or replace

insert implies update and create, but there is no update, only create and replace

Support merging Kptfiles on update

Sample hello world images don't seem to be accessible

Trying to deploy the sample to a test GKE cluster failed with:

Failed to pull image "gcr.io/kpt-dev/helloworld-gke:0.1.0": rpc error: code = Unknown desc = Error response from daemon: pull access denied for gcr.io/kpt-dev/helloworld-gke, repository does not exist or may require 'docker login'

Do the images need to be made public?

[doc] Binary installation should be more details

Right now, it's just a link to some binary, and you're on your own. Some people like more guidance. e.g. on my Mac, there were a bunch of extra step after clicking the link:

• First I had to `chmod +x'
• Then it was still blocked by the OS and I had to go in the system settings to allow it
• The file needed to be renamed to kpt
• It also needed to be moved to some folder on the path

Those may seem trivial, but I think it will lose some readers.

Things to Test umbrella issue

kpt:

• get with tag including the path -- e.g. @v01.0 --> @package-examples/[email protected]
• sync
• environment variable controls

kustomize:

• Merge -- Invert order
• Cat Styles
• Cat Strip comments
• Fmt override file annotations
• RC kind=false
• ElementSetter
• IsAssociative
• ElementValues
• MapSlice
• filters.merge
• filters.merg3
• filters.IsReconciler

Return error on 3-way merge conflicts and support explicit deletion through `~`

Setup CI for the repo

Setup domain to work with the go module alias'

Need to be able to go get kpt.dev and go get lib.kpt.dev -- requires configuring the domain to respond with the correct go headers

See: https://golang.org/cmd/go/#hdr-Remote_import_paths

Also see:
https://github.com/GoogleCloudPlatform/govanityurls

Setup docs page

Possibly using GitHub pages.

`kpt diff` doesn't diff against the right version

to reproduce:

kpt pkg get [email protected]:GoogleContainerTools/kpt.git/package-examples/[email protected] helloworld
kpt cfg set helloworld replicas 3
git add . && git commit -m 'fetched helloworld'
kpt pkg diff [email protected]

This should show the replicas diff AND the labels change. It only shows the replicas diff.

If you update the package to v0.2.0, then it will change the labels, but this isn't shown in the diff.

[doc] Sample `kpt pkg get` command doesn't work

Doc has this example:

$ kpt pkg get https://github.com/GoogleContainerTools/\
  kpt.git/package-examples/[email protected] helloworld

But pasting it as is fails because it ends up with an extra space in the middle of the url:

Error: accepts 2 arg(s), received 3
Usage:
  kpt pkg get REPO_URI[.git]/PKG_PATH[@VERSION] LOCAL_DEST_DIRECTORY [flags]

Write api documentation as `kpt help` topic

Generate documentation for Kptfile, Filters, etc. Use the go-struct comments to generate both .md docs and built-in kpt help api Kptfile docs.

fish food kpt

create gifs for README

[doc] `kpt svr status fetch` doc refers to a `resource` tool that doesn't seem to exist

Sample has:

# Fetch all resources in the cluster and wait up to 5 minutes for all of them to become Current
kubectl get all --all-namespaces -o yaml | resource status fetch

Maybe resource status fetch is an obsolete name for kpt svr status fetch?

add realistic simple examples in the comprehensive guide

Example Set 1:

• Generate: Create a Deployment + Service from an image (as data on ConfigMap input)
• Validate: Make sure Deployments have Resources reservations set
• Inject: Set Resource reservations to a default if unset

Example Set 2:

• Inject: a side-car into a Deployment
• Inject: an init container into a Deployment
• Validate: Deployment and Service labels match if names match

[doc] Walkthroughs would be helpful to get started

Spending some time in the docs and playing with some of it, what I feel is missing the most is a good walkthrough that takes the user through an entire scenario command by command.

As it stands, there is a lot of great information, but you have to go to the doc for each command one by one and try to figure out how it all fits together.

The walkthrough doesn't have to cover everything, but could include e.g.:

• Starting from nothing
• Pulling down a demo package at some older revision using pkg
• Applying the config to a cluster using svr
• Making local changes to the package, e.g. by using cfg set
• Pulling from the package source and showing how things get can get merged
• Showing off functions

A nice screencast following the same walkthrough would be nice as well.

Make Repo public

Write tutorial help topics built-in to `kpt help`

Add help topics for various tutorials

• fetch and update a package
  • various merge strategies
• use tree, grep, fmt, man
• using subpackages
• writing and publishing packages

Support Diff

• between local and upstream source
• between local and upstream source at some version

Increase coverage for kio and yaml packages

Type generation from a cluster openAPI spec should be optional

orgs will probably want to explicitly choose the types that are used, rather than have them generated ad-hoc per user.

Support mapping APIs to images in the Kptfile

Allow the GVK -> image mapping to be pulled from the Kptfile for Reconcile