vmware-tanzu/concourse-kpack-resource

Use a kpack image in a concourse pipeline naturally.

Concourse kpack Resource

Use a kpack image in a concourse pipeline naturally.

Contributing

Please read the contributing doc to begin contributing.

Installing

The recommended method to use this resource is with resource_types in the pipeline config as follows:

---
resource_types:
- name: kpack-image
  type: registry-image
  source:
    repository: gcr.io/cf-build-service-public/concourse-kpack-resource
    tag: "1.0"

Note: If you are using an older kpack version use the 0.0.9 tag.

Source configuration

Each resource corresponds to a kpack image resource in an kubernetes cluster

resources:
- name: order-service-image
  type: kpack-image
  source:
    image: "some-existing-image-name"
    namespace: "some-namespace"
    
    ## configuration to access cluster. Described below.

• image: Required string.

  The name of a kpack image resource.

• namespace: Required string.

  The namespace of the kpack image resource.

Connecting to a cluster using a kubeconfig

resources:
- name: order-service-image
  type: kpack-image
  source:
    image: "some-existing-image-name"
    namespace: "some-namespace"
    
    kubeconfig: ((kubeconfig))

Connecting to a gke cluster

resources:
- name: order-service-image
  type: kpack-image
  source:
    image: "some-existing-image-name"
    namespace: "some-namespace"
    
    gke:
      json_key: ((service-account-key))
      kubeconfig: ((kubeconfig))

• json_key: Required string.

  The contents of a service account json key. The service account key must have access to the configured image and namespace.

• kubeconfig: Required string.

  The kubeconfig for the GKE cluster generated using the method described here.

Connecting to a tkgi cluster

resources:
- name: order-service-image
  type: kpack-image
  source:
    image: "some-existing-image-name"
    namespace: "some-namespace"
    
    tkgi:
      api: https://api.tkgi.my-foundation.com
      cluster: example.tkgi.cluster.com
      insecure: false
      username: ((username))
      password: ((password))

• api: Required string.

  The pks api to connect to.

• cluster: Required string.

  The pks cluster to connect to.

• insecure: Optional boolean.

  Allow insecure server connections when communicating with pks api.

• username: Required string.

  The username to authenticate with.

• password: Required string.

  The username to authenticate with.

Behavior

check: check for new images built by kpack

Discovers all images produced by kpack builds. Will ignore new builds that produce images with the same digest as the previous build.

in: fetch the fully qualifed built image reference

Files created by the resource

• ./image: A file containing the fully qualied image reference, e.g. my-registry.com/my-image@sha256:...

out: update image with updated git revision

This will update the exisiting image with the provided source revision. It will wait for kpack to build a new image (if needed) and stream relevent build logs.

Parameters

• commitish: Optional string

  Relative path to a file containing a git revision.

  If you are using the git resource, this path will be: source-code/.git/ref

• blob_url_path: Optional string

  Relative path to a file containing a remote blob url.

Sample Pipeline

resource_types:
- name: kpack-image
  type: registry-image
  source:
    repository: gcr.io/cf-build-service-public/concourse-kpack-resource
    tag: "1.0"

resources:
- name: source-code
  type: git
  source:
    uri: https://github.com/my-app.git
    branch: master

- name: app-image
  type: kpack-image
  source:
    image: "app-image"
    namespace: "kpack-app-namespace"

    gke:
      json_key: ((service-account-key))
      kubeconfig: ((kubeconfig))

jobs:    
- name: unit
  plan:
  - get: source-code
    trigger: true
  - put: app-image
    params:
      commitish: source-code/.git/ref

- name: deploy-dev
  plan:
  - get: app-image
    trigger: true
  - task: deploy
    ...

Why does the app-image not have a passed constraint on unit tests?

kpack will automatically rebuild images on stack and buildpack updates. The deploy-dev job will be triggered on all new built images. A passed constraint on app-image would exclude images that were not the direct result of a source update.

Gotchas

• The kpack image must already exist to be used with this resource.

• At this time you cannot push local source code. Please let us know if that would be helpful.

concourse-kpack-resource's Issues

Reference Architecture Repo

It would be useful to have an official reference architecture repo that demonstrate this flow end to end.

Allow source to uploaded on `put`

Applications that need preprocessing before integrating with kpack (Java jars) will need to upload application source on a "PUT" step or provide an external blob storage. This would be expected if users are transitioning from existing CF resources.

• A user of our concourse resource should be able to add a path to application source or an application jar that our concourse pipeline will push as application source.

- put: some-image
  params:
     path: <some-tag>
     docker_username: 
     docker_password:

Notes:

• This will require a DOCKER_USERNAME & DOCKER_PASSWORD

Allow putting to an image with environment variable updates

Often pipelines contain information that will get updated with and needs to be applied as an environment variable.

Some examples of this are:

• A pipeline rebuilding tomcat configuration and updating the env var to point at the newest version: BP_TOMCAT_EXT_CONF_URI
• Providing a version to the build to the build as environment variable that comes from the semantic version resource. (kpack itself would use this as we currently hard-code the value.)

- put: some-image
  params:
     environment_variables:
          BP_GO_TARGET: "./cmd/controller"
     environment_variables_file:
          CUSTOM_VERSION: "version-resource/version" 

Then: the build-service image is updated with the provided env variables.

• Is there a better name then environment_variables_file?

Please Add Topics and Give `All-Pivotal` Team Read Access

Hey @matthewmcnew ,

Thanks for creating a new repo in /pivotal!

One of the goals in the new org is to make code as internally accessible as possible. We hope this spirit of inner-sourcing will help inspire ideas and collaboration outside of sometimes silo'd teams/pairs.

To help with this (if it is possible), can you please grant the 'All-Pivotal' team read access and tag your repo with topics?

For All-Pivotal Access:

Since the owner of a repo will understand the content of that repo - and any secrecy needs around it - much better than we can, we are leaving this decision up to you. More on this can be found here: https://github.com/pivotal/read-me-first/blob/master/adding-collaborators.md

For Tags:

We ask that you tag your repo with the following information:

• Team
• Product
• Coding Language(s)
• Subject Area

You can find more information on this here: https://github.com/pivotal/read-me-first/blob/master/organization-convention.md

If you have any questions or concerns, please don't hesitate to reach out to [email protected] and a member of our admin team will be happy to assist.

Thank you!

/cc

Open Source Modifications For Concourse Resource Repo

Add support to access to generic kubernetes cluster

Currently, we only support gke & pks clusters. This excludes kpack running on other clusters like TKG.

We should add the ability to provide access with a generic kubeconfig.

Allow resource to create images

Currently, the concourse resource does not support creating image resources. This is expected as other resources such as the "CF" resource already require this. It will also require less toil switching between kpack and concourse to start using build service.

- put: some-image 
params: 
  tag: <some-tag> 
  git: <some-git> 
  revision: <some-revision> 

- put: some-image
params
  file: <path-to-yaml> 

We need to decide how to provide configuration for tag/secrets/builders/etc.