tektoncd/resolution

Tekton Resolution

NOTE: The separate Tekton Resolution project has been deprecated. Its code has been moved into https://github.com/tektoncd/pipeline, and will be included in Pipeline release v0.40.0.

Pluggable resolution for Tekton resources (like Tasks and Pipelines). Store and utilize Tekton resources from git, from oci registries, from the tekton hub, or from anywhere else.

Tekton Resolution is aiming for the following near-term goals:

• Pluggable. Allow integrations with Tekton Pipeline's resolution machinery without having to upstream changes to Tekton Pipelines.
• Configurable. Allow operators to choose which remote locations resources can be fetched from in their CI/CD clusters.

Getting Started

Requirements

• A cluster running the Tekton Pipelines from its main branch with the alpha feature gate enabled.
• ko installed.

Install

Out of the box Tekton Resolution provides a simple Git resolver that can fetch files from public git repositories.

1. Create the tekton-remote-resolution namespace and install the ResolutionRequest controller from the root of this repo:

$ ko apply -f ./config

2. Install a resolver or get started writing your own.

Resolvers

Resolvers do the heavy lifting fetching tekton resources from remote places (like repos, registries, etc...). These are the resolvers that are currently implemented. Once a Resolver is installed in your Tekton cluster all users in that cluster can start making use of it.

Name	Description	Status
Bundle	Returns entries from oci bundles	Alpha
Git	Returns files from git repos	Alpha
Hub	Uses the Tekton Hub API to fetch tasks and pipelines	Alpha
ClusterScoped	Shares a single set of tasks and pipelines across all namespaces in your cluster	Alpha

Want to integrate with a remote location that isn't listed here? Write a new resolver or post an issue requesting one.

---

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License.

resolution's Issues

Seed documentation for the resolution project

Feature request

There are essentially no docs for resolution currently. Introduce an initial set that covers:

• Tutorial on how to get setup and running from scratch with tekton pipelines + resolution + gitresolver
• Developer-focused how-to guide on writing a new resolver from scratch
• Technical explanation on the current implementation of resolution and reasons for it being the way it is
• Technical reference on the types and behaviours that Tekton Resolution has introduced

Use case

Help end-users get started with tekton resolution, help developers attempting to write their own resolver, and capture as much background context as possible for the reasoning behind resolution's current state. I'm using https://documentation.divio.com/ as a guide to the different kinds of docs that it would be worth having initially.

Setup Continuous Integration with tektoncd/plumbing

Feature request

Setup whatever scripts, yamls, objects etc are needed for continuous integration to start on pull requests. See https://github.com/tektoncd/plumbing for the infra and https://github.com/tektoncd/results for a recent example of a project setting up CI.

Benchmark `ResourceRequest` objects in cluster

etcd publishes guidance on the approximate max amount of data it can handle given cluster size / hardware. https://etcd.io/docs/v3.5/op-guide/hardware/#example-hardware-configurations

Benchmark the numbers of supported resolved ResourceRequest objects before cluster performance is negatively impacted and document here. We are specifically interested in the size of total data in the cluster so make sure to benchmark with many 1.5MB ResourceRequests.

Enable E2E testing

Feature request

Add the boilerplate and setup code necessary for e2e testing of the resolution project.

Use case

When new code is added it should be coverable with some kind of e2e testing.

[gitresolver] Switch from git-go git-clone to use providers API

Feature request

On large (mono)?repository, a git clone may take times a lot of time, if we need to just checkin a few files.

It would be nice if we could instead use Git providers (i.e: github/gitlab) APIs instead of checking out the full tree just to grab a few files.

We may along the way will be able to support private repository in gitresolver.

Use case

Large repository checking,

Note

• We can just use go-scm to abstract a lot of the naughty different api implementation but the devil may be in the details of each provider.
• User may be able to provider access "token" for auth and ratelimit.
• We can still use anonymous for at least github and maybe other git providers.

Core Reconciler tests and easy-to-reuse Resolver test harnesses

Feature request

Testing reconcilers requires quite a bit of setup code which we don't have yet for this project. This is an important short-term problem for three different classes of test:

1. The core ResolutionRequest reconciler, though relatively small, needs thorough test coverage
2. The "resolver framework" needs to have the contracts (interfaces) it exposes tested to ensure they work consistently for all resolvers.
3. Resolver developers need a pain-free way of spinning up a test harness so that they can quickly add coverage to new resolvers.

Use case

Primarily concerned here with supporting developers of new resolvers by giving them a pain-free-as-possible way for them to exercise their resolver code quickly and easily.

Since a resolver is just a reconciler under the hood hopefully we can find a way to provide a Resolver Test Harness that is general purpose for anyone's resolver development.

Inspiration

The Tekton Pipelines unit test suite includes a plethora of setup code that performs all the necessary injection gubbins to support testing their Reconcilers. This might be a good first place to look for inspiration: https://github.com/tektoncd/pipeline/blob/main/test/controller.go

Setup nightly builds for resolution

Feature request

Nightly builds for resolution

Use case

Nightly builds enable installing the project from a known / signed version so we can be used in environments beyond dev - like the robocat and dogfooding cluster, or users who'd like to experiment with this.

Specifically I would like to do nightly installs on robocat like for other projects, to catch update issues, and I would like to run the service in dogfooding and start using it to refactor some of the logic used for nightly builds (today we rely on kustomize's ability to clone remote git repos on the fly).

Add ConfigMap support to resolvers so they can support admin configuration

Feature request

Right now any resolver based on pkg/resolver/framework does not have an easy way to pull in admin configuration options. In any realistic production-ready resolver it's highly likely that some amount of configuration will be required:

• timeouts <- this is a big one
  • right now the core ResolutionRequest reconciler has a hard-coded global timeout of 60 seconds
  • independent of the 60 second global timeout, every resolver based on framework.Resolver interface has a hard-coded 30 second timeout
• an allow-list of repositories that a git resolver is able to fetch from
• the names of secrets to use for auth
• namespaces of tenants allowed to use the resolver
• the API endpoint to hit for a private git server
• ... or any of a thousand different parameters that a resolver developer might want to support

So the feature request is to add a new optional interface for Resolvers to implement. Something like:

// ResolverConfigMap is an optional interface for a resolver to implement that allows it
// to register the name of a configmap that holds its admin settings.
type ResolverConfigMap interface {
  // OnConfigMapUpdate is called whenever the watched configmap changes
  OnConfigMapUpdated(ctx, v1.ConfigMap)
}

If a resolver implements this interface the framework would start monitoring the resolver's ConfigMap using knative's ConfigMap helpers. For inspiration on exact implementation see Tekton Pipeline's Config store codebase.

Use case

• Allow a resolver developer to support a configmap for storing its admin-level settings.
• Allow an admin deploying a resolver to configure its settings.

Deliverables

Alongside support in code it would also be useful to add:

• instructions to the developer howto
• an update to the copy/pasteable resolver template
• A ConfigMap with timeout setting to the git resolver and bundles resolver

Allow global `ResolutionRequest` timeout to be configured

Feature request

Right now every ResolutionRequest is timed out after 1 minute.

This feature request is to allow an admin to configure this global timeout to be longer or shorter depending on their cluster's needs. Individual resolvers can still set their own timeout for requests (so long as its less than the global one) but this feature is specifically for the global maximum enforced across all requests.

The reason for this global maximum is that it prevents "zombie" requests with a misconfigured resolution.tekton.dev/type label from sticking around in incomplete state forever. It also means that a resolver crashing or hitting errors doesn't put a resolution request in permanent limbo.

Use case

Admins should be able to control the maximum amount of time any resolution request is allowed to take before being marked as "timed out" and failed, thereby also failing the PipelineRun depending on it.

Update docs for every resolver to indicate the `type` it responds to

Description

Each resolver's docs should clearly indicate the type label string that it responds to.

Right now we document the params that each resolver expects but one oversight is that we don't document the type. e.g. git for git resolution, bundles for bundle resolution, etc...

Create test helpers for downstream projects to use

Feature request

Add a set of test helpers to resolution that projects like Pipelines can use to easily stub / mock ResourceRequest objects, Requester and ResolvedResource implementations, and errors.

Use case

Currently Pipelines maintains its own set of test helpers for stubbing resolution types and also code for mocking the base64 encoded response to a ResourceRequest object: https://github.com/sbwsg/pipeline/blob/pipelineref-remote-res/pkg%2Freconciler%2Fpipelinerun%2Fpipelinerun_test.go#L8770-L8791

It would be great if Pipelines did not have to host this code and instead this kind of test support was available from resolution test helpers. Once these helpers are written Pipelines should be updated to use them.

Expose Resolver params as openapi spec

Feature request

Add a method to the framework.Resolver interface that lets a resolver specify the params it accepts, maybe as openapi? Then return that param spec to Pipelines somehow for it to use when validating taskRef.resource and pipelineRef.resource.

Use case

When Tekton Pipelines validates the resolver and resource fields of a pipelineRef or taskRef Pipelines' validating webhook should be able to check that those are the params that the resolver of that type expects. e.g. the git resolver expects params url, commit or branch, path. This should be something Pipelines can do before it submits those params in a ResourceRequest so that errors can be very quickly bubbled back to the user.

Figure out a way for git branch/tag to resolve to the same commit across multiple references in a Pipeline

If a Pipeline references multiple Tasks that all utilize the same branch or tag then we need to make sure that all those references resolve to the same commit across that entire PipelineRun. Figure out a way to do this.

Document example PipelineRuns for each resolver type

Description

Right now each resolver includes an example ResolutionRequest in its README that shows how it can be used. However, this is not very user-centred since users will most likely be coming to tekton resolution from tekton pipelines.

To better help users understand how resolution is used each resolver should document an example PipelineRun that shows exactly how the resolver is used from Tekton Pipelines.

Upgrade to latest version of knative.dev/pkg

The version of knative.dev/pkg that tekton resolution is built on is now quite old. Upgrade to latest version.