redhat-certification / chart-verifier Goto Github PK

A rules-based tool to certify Helm Charts.

License: Apache License 2.0

Dockerfile 0.07% Go 85.21% PowerShell 0.01% Shell 0.17% Makefile 1.21% Smarty 1.50% Python 11.05% Gherkin 0.78%

chart-verifier's Introduction

chart-verifier: Rules based tool to certify Helm charts

The chart-verifier CLI tool allows you to validate the Helm chart against a configurable list of checks. The tool ensures that the Helm charts include the associated metadata and formatting, and are distribution ready.

The tool allows users to validate a Helm chart URL and provides a report where each check has a positive or negative result. A negative result from a check indicates a problem with the chart, which needs correction. It ensures that the Helm chart works seamlessly on Red Hat OpenShift and can be submitted as a certified Helm chart in the OpenShift Helm Repository.

The input is provided through the command-line interface, with the only required input parameter being the uri option. The output is represented through a YAML format with descriptions added in a human-readable format. The report should be submitted with a full set of checks thus validating the Helm chart.

The tool provides the following features:

Helm chart verification: Verifies if a Helm chart is compliant with a certain set of independent checks with no particular execution order.
Red Hat OpenShift Certified chart validation: Verifies the Helm chart's readiness for being certified and submitted in the OpenShift Helm Repository.
Report generation: Generates a verification report in a YAML format.
Customizable checks: Defines the checks you wish to execute during the verification process.

For more information see:

For developer specific information, see:

chart-verifier's People

Contributors

Stargazers

Watchers

Forkers

baijum avni-sharma mmulholla isutton subhayuroy srivaralakshmi harshcasper dperaza4dustbit zonggen iamsenorespana sebrandon1 mmorency2021 tisutisu tetchel itsmitul9 tonyskapunk wying3 pcaruana growi jfrancin kartikey-star imdad000 danielchristod jsm84 pataknight ethinx acornett21 itroyano komish nbycomp bcrochet skattoju vikasmulaje dcurran90 tumido acmenezes atiwary91 lokanandaprabhu eco-exp-eng mgoerens ochienged desmax74 zvigrinberg rocrisp trgeiger andrerodriggs mrizzi

chart-verifier's Issues

Chart.yaml includes keywords mapped to OpenShift categories

ref. https://issues.redhat.com/browse/HELM-107

Non zero code should be returned for failed verifications

Currently, as long as a chart undergoes verification, regardless of the outcome, a return code of zero is recorded.

Failed verifications should return a non-zero code so that the tool can be more easily integrated into an automation process (eg: CI/CD)

Add code coverage using codecov.io

asciinema screencast and embed in README

Ref. https://asciinema.org/

Chart verifier does not report error correctly

I ran this command:

out/chart-verifier verify https://github.com/redhat-certification/chart-verifier/blob/main/pkg/chartverifier/checks/chart-0.1.0-v3.valid.notest.tgz

and the entire output was ""

Need to do a thorough review of potential errors and make sure the tool outputs a suitable error message which is useful to the user.

Disallow running containers as privileged

"Alert on deployments with containers running in privileged mode"

This check is available in kube-linter.

Readme should have docs on how to use the container image

Since we do not publish binaries, and rather encourage usage of the container image the README should highlight usage of the container image instead of the following:

chart-verifier certify ./chart.tgz
``

Enable integration with other chart verification system

Enable the ability for the chart-verifier to integrate with other linting tools. ie. StackRox

chart-verifier tool does not work for online charts

If you try using the tool to download a chart from the web you get an error from the helm loader:

gzip: invalid header

This is because we are using loadArchive which expects a gzip file.

I suspect we need to download the chart first and then run the verifier, so for a client who wants to keep their chart private the only way to generate a report will be in premise.

Create report schema

Reference to #65

Create a schema for this report to define expected values

Enable users to run checks against the charts

Enable partners to run checks against a Helm Chart

Checks:

Helm Chart Checks #47

Features:

Ability to integrate with other Helm Chart Check providers #36 (ie. Stackrox, IBM cv)
Ability to define which checks are mandatory, optional, experimental

Outcome of the check:

Digest of Helm Chart content (not sha of tarball) #42
OCP version used for performing the checks #43
Outcome of all performed checks (passed/failed)

Add a link to the latest stable image in the readme.

Feature: Verify existence of external resources

A Chart.yaml may contain external referenceable material. Examples include:

home
sources
maintainers.url
icon

Verify fields are properly formatted and/or confirm that they are reachable

Execute checks against Helm Chart

Returns Passed or Failed

Verify chart does not install infra plugins and/or drivers

Ref. https://issues.redhat.com/browse/HELM-61

http/https tests sometimes fail due to internal http server not being available

Some tests rely on an internal http server to be started, and sometimes they fail (see https://github.com/redhat-certification/chart-verifier/runs/1879928916?check_suite_focus=true).

I believe this is a race condition where we assume in the test the server will be available but it can't be accessed yet.

README update for GA

readme needs updating for GA, including (in no particular order):

purpose of the tool
tool errors : cause and actions,
what to do with the report (editing, signing, including in a PR).
details of report content.
how to download and run the docker container.
Verification with an OCP cluster.
How to submit issues and suggestions

Helm Chart Checks

Note rough sizings in person days are included

Required High (GA 5/10 mandatory)

Required manual checks

~~Chart README.md includes pull secret instructions for private registries if required - will be a manual review of the PR at least for GA.~~

Required Medium (GA 5/10 nice to have)

(5 pd) Check to make sure app is not using deprecated APIs (https://issues.redhat.com/browse/HELM-64)
(10 pd) Chart.yaml includes keywords mapped to OpenShift Categories #39 (https://issues.redhat.com/browse/HELM-107)
(5 pd) Chart includes appropriate license file

Required not prioritized (pots GA 5/10)

(10 pd) Integrate with other Helm Chart Check providers #36 (ie. Stackrox, IBM cv)
Chart does not install infra plugins and drivers (network, storage, hardware etc) #37 ((https://issues.redhat.com/browse/HELM-61)
Compare minKubeVersion against specific OpenShift release #40 (https://issues.redhat.com/browse/HELM-108)
Chart README.md includes prerequisites for installing the chart if it exists

Recommended

(Recommended) Chart can be installed without manual prerequisites #41 (https://issues.redhat.com/browse/HELM-62)
(Recommended) Chart can be installed without admin privileges (https://issues.redhat.com/browse/HELM-63)

Add annotations to the report

See: https://docs.google.com/spreadsheets/d/1WoRZrIJU9UiOwA9hhJxLwMrL4x8Pv81eSHk_2npyAqU/edit#gid=0

Generated by report:

timetsamp
ocp version used for test (see #43)
digest of chart (see #42)

From the chart.yml, pick helm-chart.openshift.io annotations

Have a user focused top-level README

The top-level readme focuses a lot on how to clone repo, build binary and execute the same on helm charts. We must move the dev docs into a different folder and link from the top-level readme.

How will new rules be versioned?

This is kind of in-line with my other issue #32 in that this project doesn't have any tags or releases yet. The Docker images are also only tagged with "latest" and "main" which will obviously keep rolling forward with new changes.

I fear that if I add this as a check to my chart's CI workflow and fix all the issues, then my workflow may suddenly start failing if a new default check is added to this project.

How will the verification rules be versioned to prevent this from happening? Are they built into the binary, or fetched at runtime?

Provide report to std out instead of a file

As main use of the tool is as a docker container, the report should be output to the console.
User will need to capture the output into a file for submission.

Generate output in SARIF format

Generate output in SARIF format instead of a custom JSON format. This allows the tool to be integrated easier with git providers like GitHub or vscode

There are libraries like go-sarif that can help with the implementation of it.

Check if chart is compatible with latest OpenShift

Check whether a Helm chart defining OpenShift specific resources are supported in the latest OpenShift version, using the available resource information (such as group, version and kind).

Verify that images used in chart are Red Hat certified

Ref. https://issues.redhat.com/browse/HELM-53

Current planned approach: Look for standard k8s resources (Deployments, ReplicaSet, Jobs, DeamonSets,...) in templates.

Propagate user options to checks

It is a fair assumption that users might want to parametrize checks in different situations; one particular example is a check asserting whether the Helm chart is compatible with a given version of OpenShift, which needs the user's input to specify which one.

The --set flag should be added to the verify command; its value has the KEY=VALUE format, where KEY is the path to a particular configuration field, and VALUE is its value. For example verify --set compat.version=openshift-4.6 would overwrite the field compat.version if defined in the configuration file. The flag should also be accepted multiple times.

readme-contains-values-schema check not implemented

Documentation via the README.md indicate that the readme-contains-values-schema has been implemented, however this is not the case.

In addition, how was this check planned to validate the README of the chart?

Generate Chart index entry

Includes:
Mandatory:

Chart Url

Optional:

Authentication mechanism (eg Client certificate)

Feature: Verify Chart Version in Sem Ver 2 Format

When performing verification, confirm the chart version is in sem ver 2 format

Verify chart can be installed without manual prerequisites

Ref. https://issues.redhat.com/browse/HELM-62

Add a downloadable CLI binary in "Releases"

Compare kubeVersion against specific OpenShift Release

Ref. https://issues.redhat.com/browse/HELM-108

Update chart verifier to use verify instead of certify

Use verify instead of certify to align with tooling name

chart-verifier certify --disable is-helm-v3 https://www.example.com/chart.tgz

Readme update required.

Verify that containers aren't running with elevated scc

Ref https://www.openshift.com/blog/managing-sccs-in-openshift

The default is "restricted". User workloads ideally should work with "restricted". If not, we must discuss.

I'm not sure if there's a way to verify this with kube-linter.

Add code to generate a report from execution of the tool

Include

tool version
location of chart verified
content of chart.yaml

Generate a digest of Helm Chart

Generate the digest of Helm Chart content

Execute 'hack/test.sh' when code is pushed to main

Review cli options

Review and adjust cli options, considering oc and kubectl options prior art.

-o and --output should be used to specify the output format.
A mandatory argument instead of an option should be used to specify the chart URL.
--only and --except should work as expected: only a subset of all checks or all checks except the informed list.

Another option could be to introduce the --checks option expecting a []string representing all checks to be executed, being all registered checks set by default.

Change container base image from Fedora to something more appropriate

Publish docker image

As part of the release process we need to publish the verifier tool docker image and ensure the readme provides instruction of where to get it form and how to use it.

For docs see: #59

Usage examples in README are missing "certify" command

Commands fail because "certify" is needed before adding other parameters

Upgrade to Go 1.16

Go 1.16 has the embed module which will be useful to implement #19

https://golang.org/pkg/embed/

Capture OCP version

Capture OCP version used for performing the checks.

Provide a schema for the report

Schema would be useful for verifying a submitted report is a valid report and make it easier to read. Also maybe useful is vendor could run the check toos o they can validate prior to submission.

Create schema and check for Partner annotations

To capture partner annotations details create a schema to define expected values and a check to verify that the chart contains the requested data.

As of Apr 9, 2021:
helm-chart.openshift.io/provider | name of chart provider (e.g. Red Hat), ready to be displayed in UI
helm-chart.openshift.io/name | human readable chart name, ready to be displayed in UI
helm-chart.openshift.io/supportURL | how users can contact support
helm-chart.openshift.io/archs | comma separated list of supported architectures (e.g. x86_64, s390x,...)

Container CPU and memory limits

Verify that all containers have memory and cpu limits set. Without this, a misbehaving container may bring down a node.

Implementation notes:

Use the kube-linter check: https://github.com/stackrox/kube-linter

Chart version using appVersion instead of version from Chart.yaml

Tool makes use of Chart.yaml appVersion field. This is an optional field and instead should use the version field instead

Alternate ways to install and run

Hi, I would love to see a more fleshed out Install section for this project. Right now the instructions only seem to mention running through the Docker image.

I would like to see instructions for installing with go get -u so I can install the CLI directly.

Likewise, a tagged GitHub release with binaries for each supported platform and a changelog would be great.

Looking forward to seeing where this goes.

Verify if containers have set readOnlyRootFilesystem=true

Containers should set the security context such that the root filesystem is not writeable by the process

Implementation note: leverage kube-linter checks.