zapier/kubechecks

Check your Kubernetes changes before they hit the cluster

kubechecks - Fearless Kubernetes App Updates

kubechecks allows users of Github and Gitlab to see exactly what their changes will affect on their current ArgoCD deployments, as well as automatically run various conformance test suites prior to merge.

Pull/Merge Request driven checks

When using ArgoCD, it can be difficult to tell just how your Pull/Merge Request (PR/MR) will impact your live deployment. kubechecks was designed to address this problem; every time a new PR/MR is created, kubechecks will automatically determine what's changed and how it will impact your main/default branchs state, informing you of the details directly on the PR/MR. As a bonus, it also lints and checks your Kubernetes manifests to let you know ahead of time if something is outdated, invalid, or otherwise not good practice.

How it works

This tool provides a server function that processes webhooks from Gitlab/Github, clones the repository at the HEAD SHA of the PR/MR, and runs various check suites, commenting the output of each check in a single comment on your PR/MR. kubechecks talks directly to ArgoCD to get the live state of your deployments to ensure that you have the most accurate information about how your changes will affect your production code.

Architecture

kubechecks consists of a high level structures for communicating with your VCS provider of choice, representing a PR/MR internally, and running checks for that code; read more in the docs

Installation

Helm

See Installation Docs

Contributing

The contributing has everything you need to start working on kubechecks.

Documentation

To learn more about kubechecks go to the complete documentation.

Contributors

• mplatcher
• sl1pm4t
• polyrain
• jhouser
• davidwin93
• sontek
• MeNsaaH

---

Made by SRE Team @

kubechecks's Issues

Add ability to force kubechecks to re-run on an MR without new changes

We should add a command like kubechecks again to force kubechecks to run again. This is useful in certain edge cases when external dependencies change.

Fix gitlab comments cases where content is more than 1000000 characters

Create/update gitlab external build

Similar to #83 but for gitlab

App detection isn't working

Hi again,

Still having troubles with app detection, even with KUBECHECKS_MONITOR_ALL_APPLICATIONS: true.
I added some extra debug statements to help me map out the flow in my head a bit easier. I'm def making changes to ArgoCD apps and appsets and the repo is def getting cloned to the pod.

4:45PM DBG Debug level logging enabled.
4:45PM INF Initialized logger.
4:45PM INF setting logrus log level log_level=debug
4:45PM INF Initializing telemetry
4:45PM INF otel disabled
4:45PM INF Server Configuration: 
4:45PM INF Webhook URL Base: https://kubechecks.mycompany.io
4:45PM INF Webhook URL Prefix: 
4:45PM INF VCS Type: github
Starting KubeChecks:  
4:45PM DBG building VCS to Application Map
4:45PM DBG NewVcsToArgoMap, are we here?
4:45PM DBG Token Length - 40
4:45PM INF setting up hook handler path=/hooks
4:45PM INF hook handler setup complete path=/hooks projectPath=/github/project
4:45PM ??? ⇨ http server started on [::]:8080
4:46PM DBG Received hook request
4:46PM INF handling Github open, sync event from PR action=opened
4:46PM ERR could not load Github user email
4:46PM DBG git credentials set
4:46PM INF Repository cloned to: /tmp/kubechecks-mr-clone3735614283 project=argo-staging ref=test-check-3
4:46PM DBG Merging MR commit 8962bf82edfa08a3d08c7c8b1164dd90bc42f96d into a tmp branch off of main for manifest generation...
4:46PM DBG Changed files: tools/argo/cd/clusters/bootstrap/apps/access.yaml event_id=24 repo=argo-staging
4:46PM DBG using an argocd matcher
4:46PM DBG looking for https://github.com/myrepo/argo-staging.git repos
4:46PM DBG GetAppsInRepo is trying to find apps in https://github.com/myrepo/argo-staging.git repo URL
4:46PM DBG oops not found in https://github.com/myrepo/argo-staging.git
4:46PM DBG found no apps
4:46PM DBG app count 0
4:46PM DBG Affected apps: [] event_id=24 repo=argo-staging
4:46PM DBG Affected appSets: [] event_id=24 repo=argo-staging
4:46PM DBG Hiding kubecheck messages in PR 24 in repo myrepo/argo-staging
4:46PM INF No affected apps or appsets, skipping event_id=24 repo=argo-staging
4:46PM DBG Posting message to PR 24 in repo myrepo/argo-staging
4:46PM DBG Received hook request

The changed file in this case is an ApplicationSet. If feels like the vcstoargomap isn't working for me?

HorizontalPodAutoscaler v2beta1 is deprecated in v1.22+ and removed in v1.25

It's not possible to enable autoscaling of kubechecks on a v1.25+ Kubernetes clusters, because the chart uses the HorizontalPodAutoscaler v2beta1 API which has been removed.

Kubechecks cannot find the correct Argo CD application

Hey there, me again!

I got Kubechecks up and running, Webhooks are working fine and it is able to post comments. However, it is searching for the wrong application name in ArgoCD and thus, cannot find it:

rpc error: code = NotFound desc = error getting application: applications.argoproj.io "gitlab-runner-gitlab" not found

This leads me to the question, how does Kubechecks determine the application name? Is this something I can configure to match our GitOps structure or is there some logic baked in to find a matching application?

Our scenario is the following: We have a Repo (GitLab project) called gitlab that defines several helm charts, like gitlab-runner. Now Kubechecks looks for an application called gitlab-runner-gitlab. However, our repo consists of two branches staging and production, where the applications are deployed to a different cluster (by the same ArgoCD instance) for each. The correct application name would be gitlab-runner-staging-gitlab, so the MR's target branch name would be required to get there.

Cheers and thanks in advance!

EDIT: Accoring to spec.source, Kubechecks might be able to find the correct application name by querying all apps (again, the MR's target branch name staging is important here):

kind: Application
metadata:
  name: gitlab-runner-staging-gitlab
spec:
  source:
    repoURL: https://git.mycompany.com/k8s/namespaces/gitlab.git
    targetRevision: staging
    path: charts/gitlab-runner

Monitor applications automatically

We should have all the information necessary to find all the argocd applications and ensure that they have correctly configured webhooks: connection to argocd, and tokens to github/gitlab. It would be super convenient if you could turn on kubechecks, pass it the right tokens, and be done.

Github rejects comments with more than 64k characters

Show diff of new apps

App selection doesn't use the branch

For example, if you have two apps, both pointed at the same folder in a repo, but one is using a production branch and the other is using staging, both apps will get diff'd when a commit is pushed to the staging branch, when it should only diff the staging app.

When logging is in debug mode, github access tokens are displayed

If logging is set to debug, the github username and access token during pod startup.

10:36PM INF Initializing git settings
10:36PM DBG building command args=["config","--global","user.email","[email protected]"]
10:36PM DBG building command args=["config","--global","user.name","kubechecks-companyname"]
10:36PM DBG building command args=["https://kubechecks-companyname:ghp_<rest-of-token>@github.com"]
10:36PM DBG building command args=["config","--global","credential.helper","store"]
10:36PM DBG git credentials set

Diffing: RPC PermissionDenied

Using a docker image >= v1.5.3 I get following issue during the argo diff operation

10:28AM INF check result error="rpc error: code = PermissionDenied desc = permission denied" app=myapp app_name=myapp app_path=application/myapp/manifests check="generating diff for app" event_id=59 repo=kubechecks-test result=

The token used to connect to my argocd installation has admin permissions and the debug log doesn't provide any more info on that topic.

PS: Images older than v1.5.3 can run the diff but always detect "created" instead of "modified". For example when changing replica counts in plain manifests.

Change ':heavy_exclamation_mark:' emoji

The emoji string we use for issues (:heavy_exclamation_mark:) isn't a valid emoji for use on either GitHub or GitLab. We should replace it with either :exclamation: (❗) or remove it from the messages on errors.

Allow option to run kubechecks in namespaced scope instead of clusterscope always

Diffing applications doesn't work when generated from an app-of-app-sets

We're starting to use Helm directly within ArgoCD more often vs rendering them via a CMP. Are Helm rendering diffs supported?

For example

argocd app diff <cluster>-grafana-agent-operator --revision fix-name

===== /ConfigMap <cluster>-monitoring/grafana-agent-traces ======
13c13
<                 value: <cluster>1
---
>                 value: <cluster>2

Produces no changes in the kubechecks output

I tried with both the argocd matcher and the best effort matcher

Add better support for apps in different repositories

Currently, Kubechecks has issues when adding a new application that has a different repoURL than the repo a CheckEvent is generated from. We should improve this functionality.

Create/update commit status check in github, rather than relying entirely on the comments for progress

This way we can do things like:

• require the check before merging
• show "in progress" while we're building
• show commit success/failure

ArgoCD authentication

Hello again,

I am not yet sure, how to get a long-lived API token from ArgoCD. I managed fetching a token this way but it seems like it is only valid for 24h. Wouldn't it make more sense to provide Kubechecks a dedicated user account so it can fetch its own API tokens?

New resources show up as modified, rather than added

Option to disable failing builds during kubechecks failures

consider adding some variable that configures what should make kubechecks fail in pipeline. KUBECHECKS_FAIL_PIPELINE_ON_KUBEPUG, KUBECHECKS_FAIL_PIPELINE_ON_ERROR , KUBECHECKS_FAIL_ON_KUBECONFORM_REPORT

Can't Pull most of the image tags from ghcr.io

Hi,

when attempting to pull the image I get the following error:

I have the same errors with other tags.

The only one I have successfully pulled was this one:

VCS error?

Or Argo error?

vcsToArgoMap.GetAppsInRepo seems to be the culprit. is it failing trying to compare?

10:40PM INF Server Configuration: 
10:40PM INF Webhook URL Base: http://kubechecks.myorg.io
10:40PM INF Webhook URL Prefix: 
10:40PM INF VCS Type: github
Starting KubeChecks: v1.0.8-linux-amd64 a9adf64
10:40PM DBG building VCS to Application Map
10:40PM DBG Token Length - 40
10:40PM INF setting up hook handler path=/hooks
10:40PM INF hook handler setup complete path=/hooks projectPath=/github/project
10:40PM ??? ⇨ http server started on [::]:8080
10:40PM DBG Received hook request
10:40PM INF handling Github open, sync event from PR action=opened
10:40PM ERR could not load Github user email
10:40PM DBG git credentials set
10:40PM INF Repository cloned to: /tmp/kubechecks-mr-clone2133604070 project=argo-staging ref=test-kubechecker
10:40PM DBG Merging MR commit 0ed97d159c4268e7e3b98a04b61ac01047c2d94a into a tmp branch off of main for manifest generation...
10:40PM DBG Changed files: tools/argo/cd/apps/thirdparty/reloader.yaml event_id=21 repo=argo-staging
10:40PM DBG using an argocd matcher
10:40PM DBG looking for https://github.com/myorg/argo-staging.git repos
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1f19a3e]

goroutine 102 [running]:
github.com/zapier/kubechecks/pkg/app_directory.(*AppDirectory).Count(...)
	/src/pkg/app_directory/app_directory.go:31
github.com/zapier/kubechecks/pkg/affected_apps.NewArgocdMatcher({0xc000658240}, 0xc000ac22a0)
	/src/pkg/affected_apps/argocd_matcher.go:19 +0xbe
github.com/zapier/kubechecks/pkg/events.(*CheckEvent).GenerateListOfAffectedApps(0xc00067a500, {0x3aa0568, 0xc000cf8b10})
	/src/pkg/events/check.go:173 +0x218
github.com/zapier/kubechecks/pkg/server.(*VCSHookHandler).processCheckEvent(0xc000ff6880, {0x3aa04f8, 0xc000056040}, 0xc000ac22a0)
	/src/pkg/server/hook_handler.go:178 +0xc66
created by github.com/zapier/kubechecks/pkg/server.(*VCSHookHandler).groupHandler
	/src/pkg/server/hook_handler.go:113 +0x358

POSTs result in 404s

Hi
I'm trying this out but not getting very far, as POSTs always result in a 404.

Pod log:

11:20PM DBG Debug level logging enabled.
11:20PM INF Initialized logger.
11:20PM INF setting logrus log level log_level=debug
11:20PM INF Initializing telemetry
11:20PM INF otel disabled
11:20PM INF Server Configuration: 
11:20PM INF Webhook URL Base: https://kubechecks.mycompany.io
11:20PM INF Webhook URL Prefix: 
11:20PM INF VCS Type: github
Starting KubeChecks: v1.0.7-linux-amd64 7e4bc21
11:20PM DBG building VCS to Application Map
11:20PM DBG Token Length - 40
11:20PM INF setting up hook handler path=/hooks
11:20PM INF hook handler setup complete path=/hooks projectPath=/github/project
11:20PM ??? ⇨ http server started on [::]:8080

Configmap:

  KUBECHECKS_ARGOCD_API_SERVER_ADDR: https://argo.mycompany.io
  KUBECHECKS_LOG_LEVEL: debug
  KUBECHECKS_PERSIST_LOG_LEVEL: 'true'
  KUBECHECKS_VCS_BASE_URL: https://github.com/company/repo
  KUBECHECKS_VCS_TYPE: github
  KUBECHECKS_WEBHOOK_URL_BASE: https://kubechecks.mycompany.io

Ingress:

  rules:
    - host: kubechecks.mycompany.io
      http:
        paths:
          - path: /hooks
            pathType: Prefix
            backend:
              service:
                name: kubechecks
                port:
                  number: 8080

I thought maybe my Ingress config was wrong so i tried execing into the pod and curling localhost:8080/hooks and that results in a 404 too.

Pod isn't spitting out any other info, so what did i screw up?

"Ensuring webhooks" feature panics

If I understand correctly, setting the flag KUBECHECKS_ENSURE_WEBHOOKS should make Kubecheck autoconfigure its webhook for all known repositories in the VCS via API calls, right?

I wanted to give that option a try so I don't have to configure all repositories manually, but it is failing with the following error:

10:53AM DBG Debug level logging enabled.
10:53AM TRC Trace level logging enabled.
10:53AM INF Initialized logger.
10:53AM INF setting logrus log level log_level=error
10:53AM INF Initializing telemetry
10:53AM INF otel disabled
10:53AM INF Server Configuration:
10:53AM INF Webhook URL Base: https://kubechecks.mycompany.com
10:53AM INF Webhook URL Prefix:
10:53AM INF VCS Type: gitlab
Starting KubeChecks: 24h b94b428
10:53AM DBG building VCS to Application Map
10:53AM INF ensuring all webhooks are created correctly
10:53AM DBG Token Length - 26
10:53AM INF webhook URL for this kubechecks instance webhookUrl=https://kubechecks.mycompany.com/hooks/gitlab/project
10:53AM ERR could not parse GitLab URL error="repository host 'git.mycompany.com' not supported" [email protected]:k8s/namespaces/security
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x2a3718f]

goroutine 98 [running]:
github.com/zapier/kubechecks/pkg/gitlab_client.parseRepoName({0xc0010a7000, 0x3a}
	/src/pkg/gitlab_client/client.go:126 +0xa
github.com/zapier/kubechecks/pkg/gitlab_client.(*Client).GetHookByUrl(0xc00063d110, {0x33bd147?, 0x28?}, {0xc0010a7000?, 0x37?}, {0xc00016bc20, 0x37}
	/src/pkg/gitlab_client/client.go:130 +0x4
github.com/zapier/kubechecks/pkg/server.(*Server).ensureWebhooks(0xc00063d140
	/src/pkg/server/server.go:107 +0x42
github.com/zapier/kubechecks/pkg/server.(*Server).Start(0xc00063d140
	/src/pkg/server/server.go:43 +0x9
created by github.com/zapier/kubechecks/cmd.glob..func
	/src/cmd/controller_cmd.go:32 +0x205

I am using the temporary build from #55 (with the fix for custom vcs URLs) and the following settings:

KUBECHECKS_ARGOCD_API_INSECURE: true
KUBECHECKS_ARGOCD_API_SERVER_ADDR: argocd.mycompany.com
KUBECHECKS_ARGOCD_WEBHOOK_URL: https://argocd.mycompany.com/api/webhook
KUBECHECKS_ENSURE_WEBHOOKS: true
KUBECHECKS_LOG_LEVEL: trace
KUBECHECKS_MONITOR_ALL_APPLICATIONS: true
KUBECHECKS_NAMESPACE: gitlab
KUBECHECKS_WEBHOOK_URL_BASE: https://kubechecks.mycompany.com
KUBECHECKS_VCS_BASE_URL: https://git.mycompany.com
KUBECHECKS_VCS_TYPE: gitlab

Thanks again and cheers

OpenAI likes to add ```, but it screws up the following diff

We should just get rid of "```" in the open ai summary

Schema validation always fails for CRDs

Hey there,
unfortunately, I couldn't get the schema validation for CRDs to work.

• If I stick to the default configuration, the /app/schemas directory in the container is empty and Kubechecks does not know about any CRDs. If there was any content from the container image, it is superseded by a volume mount, maybe due to this?
• If I tell Kubechecks to use, e.g.,

  KUBECHECKS_SCHEMAS_LOCATION: https://github.com/datreeio/CRDs-catalog.git

  it successfully pulls from the repository, but still fails validation with

  Error: external-secrets.io/v1beta1 ExternalSecret kubechecks - could not find schema for ExternalSecret
  

  (the schema for external-secrets.io/v1beta1/ExternalSecret is present in this repository)
• I also tried using a custom git repository with all schema files in a flat structure but it still won't find anything
• Mounting schemas from a volume to /app/schemas has no effect, either

Looking at the logs it seems to me that Kubechecks always searches for these schemas in the application source repos, no matter if I provide an absolute or relative path for KUBECHECKS_SCHEMAS_LOCATION:

WRN failed to find in-repo path error="stat /tmp/kubechecks-mr-clone3517385009/app/schemas: no such file or directory" location=/app/schemas

How can I make use of this feature? Is there an option to disable schema validation altogether?

EDIT: We're running kubechecks:v1.3.3

Read conftest policies from an external source

Currently the conftest policies are baked into the image, which is not very useful.

401 Unauthorized - using private repositories as a git chart dependency (Aws Ecr)

kubechecks/pkg/argo_client/manifests.go

Line 76 in 14e5941

return nil, errors.Wrap(err, "failed to generate manifests")

Hey, getting the bellow exception:

ERR Unable to get manifests for in error="failed to generate manifests: helm dependency build failed exit status 1: Error: could not download oci://dkr.ecr..amazonaws.com/: pulling from host dkr.ecr..amazonaws.com failed with status code [manifests ]: 401 Unauthorized" event_id=595 repo=

Aws Ecr requires helm login before downloading the charts.
and today I don't see any support for it.
All of my charts are packaged and stored directly in aws Ecr.
I can't use this tool without it. only the values files are stored in github.

Investigate using ArgoCD "Core" authentication to simplify deployments

It doesn't seem super well documented, but it might let us bypass creating users and tokens if kubechecks is deployed to the same cluster as ArgoCD.

Add a CLI so that it can be used in github actions

Feature request: ArgoCD dry run

It would be neat if kubechecks ran a dry sync with the argocd CLI.

https://argo-cd.readthedocs.io/en/stable/user-guide/commands/argocd_app_sync/

Flags of interest:
--dry-run
--local
--server-side

If this works how I think it does, this would allow users to identify things like validating webhook violations at PR time, rather than deploy time

ARGOCD_API_INSECURE not being respected

I have a secret that contains the environment variable

KUBECHECKS_ARGOCD_API_INSECURE: "true"

But the app tries to talk to a https endpoint, regardless of the value. Oddly, I don't see any argocd client information being posted to the pod log

Version: 1.3.3
Pod log:

3:17PM DBG Debug level logging enabled.
3:17PM TRC Trace level logging enabled.
3:17PM INF Initialized logger.
3:17PM INF setting logrus log level log_level=error
3:17PM INF Initializing telemetry
3:17PM INF otel disabled
3:17PM INF Server Configuration: 
3:17PM INF Webhook URL Base: https://kubechecks-webhook.prod.foo.com
3:17PM INF Webhook URL Prefix: 
3:17PM INF VCS Type: github
3:17PM DBG Token Length - 40
3:17PM INF Initializing git settings
3:17PM DBG building command args=["config","--global","user.email","[email protected]"]
3:17PM DBG building command args=["config","--global","user.name","deploy"]
3:17PM DBG building command args=["https://deploy:********@github.com"]
3:17PM DBG building command args=["config","--global","credential.helper","store"]
3:17PM DBG git credentials set
Starting KubeChecks: v1.3.3 197a3c0
3:17PM DBG building VCS to Application Map
3:17PM WRN failed to build vcs app map from argo error="failed to list applications: failed to applications: rpc error: code = Unknown desc = Post \"https://argocd-server.argocd.svc.cluster.local:443/application.ApplicationService/List\": EOF"
Method	Path
GET	/ready
GET	/live
POST	/hooks/github/project
GET	/metrics
3:17PM ??? ⇨ http server started on [::]:8080
3:18PM DBG Received hook request
3:18PM INF ignoring Github pull request event due to non commit based action action=closed
3:18PM DBG Ignoring event, not a merge request

Move from `whilp/git-urls` to `chainguard-dev/git-urls` to address CVE-2023-46402

whilp/git-urls appears to be unmaintained so we should move over to chainguard-dev/git-urls to address CVE-2023-46402

From the README:

This repository is a fork of git-urls. The fork was created to fix an existing vulnerability GHSA-3f2q-6294-fmq5 in the upstream repository, which looked unmaintained.

Chainguard plans to keep maintaining this repository under our organization.

---

GitHub Advisory - GHSA-3f2q-6294-fmq5

Add support for multiple ArgoCD servers.

Some folks deploy an ArgoCD server per Kubernetes cluster. We should add support for multiple ArgoCD deploys that is configurable:

• per repository
• per application

Cluster should not be marked with a green checkmark if there's an error

If kubechecks runs into an error when generating a diff it will print the error but mark the cluster with a green checkmark. This masks simple errors such as malformed yaml.

Links to abandoned repos

The org that hosts the CRD catalog has ceased trading. Their org and repo is referenced twice in this repo. Is there any plan to deprecate those links, either in favour of a repo hosted under the zapier org, or perhaps standardising on the infinityflow fork?

Github client panic v1.3.0

Hey guys

Just pulled v1.3.0 (new app capability, yay!) - hitting this during pod startup.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x29a4a69]

goroutine 1 [running]:
github.com/zapier/kubechecks/pkg/vcs/github_client.CreateGithubClient()
	/src/pkg/vcs/github_client/client.go:76 +0x409
github.com/zapier/kubechecks/cmd.createVCSClient({0xc000058074?, 0x334550f?})
	/src/cmd/vcs.go:16 +0x45
github.com/zapier/kubechecks/cmd.glob..func1(0x58a4ae0?, {0x333c6b4?, 0x0?, 0x0?})
	/src/cmd/controller_cmd.go:30 +0x4a
github.com/spf13/cobra.(*Command).execute(0x58a4ae0, {0x593e390, 0x0, 0x0})
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x58a50a0)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
github.com/zapier/kubechecks/cmd.Execute()
	/src/cmd/root.go:37 +0xc5
main.main()
	/src/main.go:8 +0x17

sorry for being so annoying :-)

Kubechecks calls gitlab.com instead of privately hosted VCS

Hey there,

Kubechecks looks like a really nice tool to me and I'd love to get it to work. I tried deploying Kubechecks via Helm, and not all of the Kubecheck's ENVs are clear to me. The container is still failing with the error message

FTL could not create Gitlab token user error="GET https://gitlab.com/api/v4/user: 401 {message: 401 Unauthorized}"

I am not sure what Kubecheck tries to do at this point but I need it to work with our local GitLab installation, whose base URL I have configured in KUBECHECKS_WEBHOOK_URL_BASE ("The URL where KubeChecks receives webhooks from"). Does Kubecheck support local GitLab instances at this point?

Cheers

Unable to set location for remote repository

It is assumed that every repository will follow the file structure that's defined here. However, if I want to use something like https://github.com/datreeio/CRDs-catalog/ which uses a different layout {{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json, I'm unable to define both the repository and the file structure.