aquasecurity/chain-bench

An open-source tool for auditing your software supply chain stack for security compliance based on a new CIS Software Supply Chain benchmark.

📖 Documentation

Chain-bench is an open-source tool for auditing your software supply chain stack for security compliance based on a new CIS Software Supply Chain benchmark. The auditing focuses on the entire SDLC process, where it can reveal risks from code time into deploy time. To win the race against hackers and protect your sensitive data and customer trust, you need to ensure your code is compliant with your organization’s policies.

Read more in the Chain-bench Documentation

Contents

• Contents
• Introduction
• Quick start
  • Installation
  • Usage
    • Using docker
    • Using GitHub Actions
    • Using Gitlab CI (beta)
• Requirements
• Supported Providers
• Please Note
• Contributing
• Roadmap

Introduction

Chain-bench is an open-source tool for auditing your software supply chain stack for security compliance based on a new CIS Software Supply Chain benchmark. The auditing focuses on the entire SDLC process, where it can reveal risks from code time into deploy time.

Quick start

The primary way to run chain-bench is as a standalone cli. It requires an access token for your account and the repository url in order to access your SCM.

Installation

Get Chain-bench via your favorite installation method. See installation section in the documentation for details. For example:

• brew install chain-bench
• nix-env --install -A nixpkgs.chain-bench
• docker run aquasec/chain-bench
• Download binary from https://github.com/aquasecurity/chain-bench/releases/latest/

Usage

chain-bench scan --repository-url <REPOSITORY_URL> --access-token <TOKEN> -o <OUTPUT_PATH>

Using Self-hosted or Dedicated SCM Platforms (with custom domains)

chain-bench scan --repository-url <REPOSITORY_URL> --scm-platform <SCM_PLATFORM> --access-token <TOKEN> -o <OUTPUT_PATH>

Supported options for scm-platform are "github" and "gitlab" (beta)

Using docker

docker run aquasec/chain-bench scan --repository-url <REPOSITORY_URL> --access-token <TOKEN>

Using GitHub Actions

See the repository at https://github.com/aquasecurity/chain-bench-action

Example output

2022-06-13 15:22:18 INF 🚩	Fetch Starting
2022-06-13 15:22:19 INF 🏢	Fetching Organization Settings Finished
2022-06-13 15:22:29 INF 🛢️	Fetching Repository Settings Finished
2022-06-13 15:22:29 INF 🌱	Fetching Branch Protection Settings Finished
2022-06-13 15:22:29 INF 👫	Fetching Members Finished
2022-06-13 15:22:31 INF 🔧	Fetching Pipelines Finished
2022-06-13 15:22:31 INF 🏁	Fetch succeeded
   ID                                                 Name                                                Result                  Reason
-------- ----------------------------------------------------------------------------------------------- -------- ---------------------------------------
 1.1.3    Ensure any change to code receives approval of two strongly authenticated users                 Passed
 1.1.4    Ensure previous approvals are dismissed when updates are introduced to a code change proposal   Failed
 1.1.5    Ensure that there are restrictions on who can dismiss code change reviews                       Failed
 1.1.6    Ensure code owners are set for extra sensitive code or configuration                            Failed
 1.1.8    Ensure inactive branches are reviewed and removed periodically                                  Failed   20 inactive branches
 1.1.9    Ensure all checks have passed before the merge of new code                                      Passed
 1.1.10   Ensure open git branches are up to date before they can be merged into codebase                 Passed
 1.1.11   Ensure all open comments are resolved before allowing to merge code changes                     Passed
 1.1.12   Ensure verifying signed commits of new changes before merging                                   Failed
 1.1.13   Ensure linear history is required                                                               Passed
 1.1.14   Ensure branch protection rules are enforced on administrators                                   Failed
 1.1.15   Ensure pushing of new code is restricted to specific individuals or teams                       Passed
 1.1.16   Ensure force pushes code to branches is denied                                                  Failed
 1.1.17   Ensure branch deletions are denied                                                              Failed
 1.2.1    Ensure all public repositories contain a SECURITY.md file                                       Failed
 1.2.2    Ensure repository creation is limited to specific members                                       Failed
 1.2.3    Ensure repository deletion is limited to specific members                                       Passed
 1.2.4    Ensure issue deletion is limited to specific members                                            Passed
 1.3.1    Ensure inactive users are reviewed and removed periodically                                     Failed   22 inactive users
 1.3.3    Ensure minimum admins are set for the organization                                              Passed
 1.3.5    Ensure the organization is requiring members to use MFA                                         Passed
 1.3.7    Ensure 2 admins are set for each repository                                                     Failed
 1.3.8    Ensure strict base permissions are set for repositories                                         Passed
 1.3.9    Ensure an organization's identity is confirmed with a Verified badge                            Failed
 2.3.1    Ensure all build steps are defined as code                                                      Failed   No build job was found in pipelines
 2.3.5    Ensure access to the build process's triggering is minimized                                    Passed
 2.3.7    Ensure pipelines are automatically scanned for vulnerabilities                                  Passed
 2.3.8    Ensure scanners are in place to identify and prevent sensitive data in pipeline files           Failed   Repository is not scanned for secrets
 2.4.2    Ensure all external dependencies used in the build process are locked                           Failed   16 task(s) are not pinned
 2.4.6    Ensure pipeline steps produce an SBOM                                                           Passed
 3.1.7    Ensure dependencies are pinned to a specific, verified version                                  Failed   16 dependencies are not pinned
 3.2.2    Ensure packages are automatically scanned for known vulnerabilities                             Passed
 3.2.3    Ensure packages are automatically scanned for license implications                              Passed
 4.2.3    Ensure user's access to the package registry utilizes MFA                                       Passed
 4.2.5    Ensure anonymous access to artifacts is revoked                                                 Passed
 4.3.4    Ensure webhooks of the package registry are secured                                             Passed
-------- ----------------------------------------------------------------------------------------------- -------- ---------------------------------------
 Total Passed Rules: 19 out of 36
2022-06-13 15:22:31 INF Scan completed: 13.108s

Using Gitlab CI (beta)

You can integrated chain-bench results into Gitlab Vulnrability Report by adding a new step within your CI defintion:

chain-bench-scanning:
  stage: test
  image:
    name: docker.io/aquasec/chain-bench
    entrypoint: [""]
  script:
    - chain-bench scan --repository-url $CI_PROJECT_URL --access-token $CHAIN_BENCH_TOKEN --scm-platform gitlab -o results.json --template @/templates/gitlab_security_scanner.tpl
  artifacts:
    reports:
      container_scanning: results.json

• You have to create new token with Maintainer role that has read_api & read_repository permission and use it as environment variables (eg. $CHAIN_BENCH_TOKEN)

Requirements

It is required to provide an access token with permission to these scopes: repo(all), read:repo_hook, admin:org_hook, read:org

Supported Providers

We currently support Github and Gitlab SCMs, with PAT authentication.

Please Note

Chain-bench implements the CIS Software Supply Chain Benchmark as closely as possible. You can find the current implemented checks under AVD - Software Supply Chain CIS - 1.0 that update every night based chain-bench metadata.json files Please raise issues here if chain-bench is not correctly implementing the test as described in the Benchmark. To report issues in the Benchmark itself (for example, tests that you believe are inappropriate), please join the CIS community.

Contributing

Kindly read Contributing before contributing. We welcome PRs and issue reports.

Roadmap

Going forward we plan to release updates to chain-bench to increase the benchmark coverage with more checks and support more platforms. chain-bench is an Aqua Security open source project part of Trivy Family.

chain-bench's Issues

Remediation instructions should be permalinks

On the aquasec website listing the issues and their remediation, it is currently not possible to give a link pointing to the exact subsection.

For example, it is only possible to link the "Code Changes" (https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1) page, but not to point directly to the "1.1.3 Ensure any change to code receives approval of two strongly authenticated users" item.

Then this permalink should be used in the "Url" value of the report file generated by chain-bench.

Using the json output is missing information about the repository

Using the json output: chain-bench scan -r $repo --access-token ${GITHUB_AUTH_TOKEN} -o my.json

The result don't have any information about the repository URL.

{
  "metadata": {
    "date": "2022-09-14T22:49:49-03:00",
    "statistics": {
      "passed": 5,
      "failed": 3,
      "unknown": 28,
      "total": 36
    }
  },
  "results": [
    {
      "id": "1.1.3",
      "name": "Ensure any change to code receives approval of two strongly authenticated users",
      "description": "Ensure that every code change is reviewed and approved by 

Add a Markdown version of the 'CIS Software Supply Chain Security Guide'

From the PDF:

The hope with the publication of this Guide is to elicit feedback from the global community that will help ensure the future platform-specific guidance (CIS Benchmarks) is even more accurate and relevant.

To facilitate feedback (comments, issues, PRs, etc), it would be great if the recommendations were available in a format like Markdown.

Add the ability to get branch name as parameter

Get the branch name as parameter in case the repository has another branch needed to be checked.

Non-conventional JSON format

Description

When saving the results as JSON, the keys are formatted in a non-conventional nor consistent manner. This makes deserializing the file with other tools unnecessary complicated.

What did you expect to happen?

I expected all the keys to use a conventional encoding, like lowercase or camelCase.

For instance, looking at the JSON API recommendations (https://jsonapi.org/recommendations/), they recommend the keys to use camelCase (which seems to be the most commonly accepted one):

Member names SHOULD be camel-cased (i.e., wordWordWord)

Looking at several libraries, the more common encodings are:

• "lowercase"
• "UPPERCASE"
• "PascalCase"
• "camelCase"
• "snake_case"
• "SCREAMING_SNAKE_CASE"
• "kebab-case"
• "SCREAMING-KEBAB-CASE"

What happened instead?

Instead the id key is UPPERCASE, and the other ones are Capitalized.

[
  {
    "ID": "1.1.3",
    "Name": "Ensure any change to code receives approval of two strongly authenticated users",
    "Descrition": "Ensure that every code change is reviewed and approved by two authorized contributors who are strongly authenticated.",
    "Remediation": "An organization can protect specific code branches — for example, the \"main\" branch which often is the version deployed to production — by setting protection rules. These rules secure your code repository from unwanted or unauthorized changes. You may set requirements for any code change to that branch, and thus specify a minimum number of reviewers required to approve a change.",
    "Result": "Failed",
    "Reason": "",
    "Url": "https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1"
  }
]

Additional info

Tools like gomodifytags can simplify the process of mass editing the struct tags.

chain-bench with gitlab

Hi! I'm using the SaaS Community Gitlab version, I have the following error.

chain-bench -v scan --repository-url https://gitlab.com/krol1/go-cowsay --access-token glpat-xxxxxx
2022-11-22 18:26:07 INF 🚩	Fetch Starting
2022-11-22 18:26:10 INF 🛢️	Fetching Repository Settings Finished
2022-11-22 18:26:10 INF 🌱	Fetching Branch Protection Settings Finished
2022-11-22 18:26:11 WRN failed to fetch approval configuration
2022-11-22 18:26:11 DBG failed to fetch approval configuration error="GET https://gitlab.com/api/v4/projects/41277134/push_rule: 404 {message: 404 Not Found}"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x30 pc=0x10151aa04]

It seems that the push_rule it's a premium feature: https://docs.gitlab.com/ee/user/project/repository/push_rules.html

Issue when the sub group has the same name as the repository

Description

Issue when the sub group has the same name as the repository.
Example: https://my-gitlab-instance.com/top-group/sub-group/repo-name/repo-name

getRepoInfo returns:

host: my-gitlab-instance.com
namespace: top-group/sub-group
repo: repo-name
err: %!s(<nil>)

What did you expect to happen?

should return:

host: my-gitlab-instance.com
namespace: top-group/sub-group/repo-name
repo: repo-name
err: %!s(<nil>)

What happened instead?

getRepoInfo returns:

host: my-gitlab-instance.com
namespace: top-group/sub-group
repo: repo-name
err: %!s(<nil>)

Made a PR for this but due to the contribution guidelines I created this issue as well.

output error while running the chain-bench scan

Description

Executing this simple command: docker run aquasec/chain-bench scan --repository-url [git repo] --access-token [git token]

Produce these errors:
2022-07-28 03:44:50 INF 🚩 Fetch Starting
2022-07-28 03:44:50 ERR error in authenticated user data
2022-07-28 03:44:50 ERR error in fetching repository data
2022-07-28 03:44:50 INF 🛢️ Fetching Repository Settings Finished
2022-07-28 03:44:50 ERR error in fetching branch protection
2022-07-28 03:44:50 INF 🌱 Fetching Branch Protection Settings Finished
2022-07-28 03:44:50 ERR error in fetching workflows
2022-07-28 03:44:50 INF 🔧 Fetching Pipelines Finished
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xb3f265]

goroutine 1 [running]:
github.com/aquasecurity/chain-bench/internal/scm-clients/clients.FetchClientData({0x0, 0x0}, {0x7fffb403df52?, 0x1?})
/home/runner/work/chain-bench/chain-bench/internal/scm-clients/clients/clients.go:48 +0x3c5
github.com/aquasecurity/chain-bench/internal/commands.NewScanCommand.func1(0xc000242280?, {0xcfc484?, 0x2?, 0x2?})
/home/runner/work/chain-bench/chain-bench/internal/commands/scan.go:22 +0xac
github.com/spf13/cobra.(*Command).execute(0xc000242280, {0xc00024c040, 0x2, 0x2})
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:872 +0x694
github.com/spf13/cobra.(*Command).ExecuteC(0xc000242000)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:918
github.com/aquasecurity/chain-bench/internal/commands.Execute({0xe4c9a8?, 0xc0000021a0?})
/home/runner/work/chain-bench/chain-bench/internal/commands/root.go:21 +0x32
main.main()
/home/runner/work/chain-bench/chain-bench/cmd/chain-bench/main.go:12 +0x27

CIS-Software-Supply-Chain-Security-Guide-v1.0.pdf link is broken

Description

Clicking on https://github.com/aquasecurity/chain-bench/blob/main/docs/CIS-Software-Supply-Chain-Security-Guide-v1.0.pdf provide the following error

chain bench config file and output assertion rules

My idea here is creating a .chain-benchrc | chain-bench.config.json file that the repository will hold
And upon running the cli in the context of that repository it will read that config file for any configuration for chain-bench

Leveraging that config file, add rules key into the json with sub keys pass and fail those will hold assertions over the json output of chain-bench that will decide what will be the exit code.

For now when I want to assert over the chain bench output (JSON format), I am doing it with jq or rego.

How many checks are in GitLab scan

Hello!
First of all I would like to say that I really appreciate your work to add support for GitLab. I know that currently chain-bench GitLab are in beta but how many checks are in GitLab scan now?

Remove the needs for write permissions, and/or use fine grained permission tokens

At present the documentation states this needs full repo access, it would be advantageous to make this only require read permission scopes.

If this isn't possible - it should use the new Github fine grained tokens which provide improved permission scopes.

https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/

---

"It is required to provide an access token with permission to these scopes: repo(all), read:repo_hook, admin:org_hook, read:org"

scan locally a repository

It will be nice to use the chain-bench in local mode, for example

chain-bench --local .

Adjust CLI logging level

By default the CLI shows a bunch of detailed errors:

./chain-bench scan --repository-url https://github.com/buildsec/frsca --access-token $GITHUB_TOKEN -o ./results
2022-06-17 09:10:07 INF 🚩 Fetch Starting
2022-06-17 09:10:08 ERR error in fetching organization hooks error="GET https://api.github.com/orgs/buildsec/hooks: 404 Not Found []"
2022-06-17 09:10:08 INF 🏢 Fetching Organization Settings Finished
2022-06-17 09:10:08 ERR error in fetching org packages error="GET https://api.github.com/orgs/buildsec/packages?package_type=npm&state=active: 403 You need at least read:packages scope to list packages. []"
2022-06-17 09:10:13 ERR error in fetching hooks data error="GET https://api.github.com/repos/buildsec/frsca/hooks: 404 Not Found []"
2022-06-17 09:10:13 INF 🛢️ Fetching Repository Settings Finished
2022-06-17 09:10:13 ERR error in fetching branch protection error="GET https://api.github.com/repos/buildsec/frsca/branches/main/protection: 404 Not Found []"
2022-06-17 09:10:13 INF 🌱 Fetching Branch Protection Settings Finished
2022-06-17 09:10:13 INF 👫 Fetching Members Finished
2022-06-17 09:10:13 WRN file .github/workflows/ci.yaml not found
2022-06-17 09:10:14 WRN file dynamic/pages/pages-build-deployment not found
2022-06-17 09:10:14 INF 🔧 Fetching Pipelines Finished
2022-06-17 09:10:14 INF 🏁 Fetch succeeded

But these errors just clutter the output, and are not very useful unless debugging information is needed. These types of details should be displayed for debug or tracing log level.

As a user I should be able to adjust the log level from the CLI, for example by supplying -v flags (1 for info, 2 for debug, 3 for trace).

GitLab CI/CD failed

Description

Hey team!
I saw that you added support for GitLab (beta). I tried to run scan against dummy repo hosted on GitLab, but unfortunately it failed. I created a token with the appropriate role and permissions.

What did you expect to happen?

The scan has run successfully

What happened instead?

The scan immediately failed

Out from gitlab ci/cd runner:

$ chain-bench scan --repository-url $CI_PROJECT_URL --access-token $CHAIN_BENCH_TOKEN -o results.json --template @/templates/gitlab_security_scanner.tpl
2022-10-26 11:31:13 INF 🚩	Fetch Starting
2022-10-26 11:31:14 ERR error in fetching repository data
2022-10-26 11:[31](https://gitlab.com/XYZ/security/dummy-repo/-/jobs/3229947555#L31):14 INF 🛢️	Fetching Repository Settings Finished
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xbd97a5]
goroutine 1 [running]:
github.com/aquasecurity/chain-bench/internal/scm-clients/clients.FetchClientData({0x7ffd195cb03b, 0x1a}, {0x7ffd195caff6?, 0x1?}, {0x0, 0x0})
	/home/runner/work/chain-bench/chain-bench/internal/scm-clients/clients/clients.go:40 +0x1a5
github.com/aquasecurity/chain-bench/internal/commands.NewScanCommand.func1(0xc000252a00?, {0xe3962a?, 0x8?, 0x8?})
	/home/runner/work/chain-bench/chain-bench/internal/commands/scan.go:22 +0xcc
github.com/spf13/cobra.(*Command).execute(0xc000252a00, {0xc00024e800, 0x8, 0x8})
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:872 +0x694
github.com/spf13/cobra.(*Command).ExecuteC(0xc000252780)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:918
github.com/aquasecurity/chain-bench/internal/commands.Execute({0xf93cc8?, 0xc0000021a0?})
	/home/runner/work/chain-bench/chain-bench/internal/commands/root.go:21 +0x[32](https://gitlab.com/tidio/security/automated-aws-audit/-/jobs/3229947555#L32)
main.main()
	/home/runner/work/chain-bench/chain-bench/cmd/chain-bench/main.go:12 +0x27
Uploading artifacts for failed job

chain-bench only works for organizations

Description

When using chain-bench to audit repositories which do not belong to an organization, the process failed.

What did you expect to happen?

As a user I expected the tool to work in a similar way for repositories that do not belong to an organization.

What happened instead?

The audit did not start due to 404 errors being returned since the repository was not part of an organization.

./chain-bench scan --repository-url https://github.com/rgreinho/trauma --access-token $GITHUB_TOKEN -o ./results
2022-06-17 09:08:59 INF 🚩 Fetch Starting
2022-06-17 09:09:00 ERR error in fetching organization error="GET https://api.github.com/orgs/rgreinho: 404 Not Found []"
2022-06-17 09:09:00 INF 🏢 Fetching Organization Settings Finished
2022-06-17 09:09:01 INF 🛢️ Fetching Repository Settings Finished
2022-06-17 09:09:01 INF 🌱 Fetching Branch Protection Settings Finished
2022-06-17 09:09:01 ERR error in fetching members error="GET https://api.github.com/orgs/rgreinho/members: 404 Not Found []"
2022-06-17 09:09:01 ERR Failed to fetch client data error="GET https://api.github.com/orgs/rgreinho/members: 404 Not Found []"
Error: GET https://api.github.com/orgs/rgreinho/members: 404 Not Found []

Additional details (base image name, container registry info...)

Implementation ideas

The type of repository (e.g.: User or Organization) could be determined by querying the repo with the
Get A Repository endpoint and used to adjust next requests:

$ curl -sL https://api.github.com/repos/rgreinho/trauma| jq .owner.type
"User"

Not implemented: "3.2.3: Ensure packages are automatically scanned for license implications"

Description

The check "3.2.3: Ensure packages are automatically scanned for license implications" does not seem to be implemented. At https://github.com/aquasecurity/chain-bench/blob/main/internal/checks/dependencies/validate_packages/rules.rego#L16, it appears to be checking the same thing as 3.2.2, whether there are vulnerability scan tasks.

What did you expect to happen?

It checks for license scan tasks.

What happened instead?

It checks for vuln scan tasks

are_pipelines_dependencies_scanned_for_licenses {
	count({job | job := input.Pipelines[_].jobs[_]; does_job_contain_one_of_tasks(job, constsLib.pipeline_vulnerability_scan_tasks)}) == 0
}

Additional details (base image name, container registry info...):

The remediation does not really explain how to fix this. It would be more helpful if it specifically said which types of pipeline tasks it is looking for, in both the vuln scan and license scan checks.

Feature: GitHub Action

Would be nice to be able to run this as a GitHub Action.

odd tags

Description

Is there a reason for the non-release tags like 0.1.4+1? Do they serve any purpose or can they be removed?

overview Risk

Hi! I can see in the results that it containers the total of controls with status passed/failed and unknown.
How can I know what of the controls represent a critical risk ?

Thank you

Chain Bench score

Similar to go report, add the option to add the Chain Bench score to a repository's README.md

missing community standard checks

GitHub provides a list of community standard checks that help improving the quality of a repository.

For instance for the frsca project:

As a user I would like chain-bench to report this missing community standards so that I can improve the overall quality of my repositories.

1.1.16 and 1.1.17 producing false positives

Description

1.1.16 states that for each repository in use, we must validate that no one can “force push” code.

1.1.17 states that for each repository that is being used, we must verify that protected branches cannot be deleted.

The rule logic for these two benchmarks appears to be written in such a way that it produces false positives. When Allow force pushes and Allow deletions are checked, thus permitting the ability to force pushes and/or delete branches, Chain-Bench outputs a Passed where a Failed would be expected.

The opposite will happen if you have them unchecked - you'll get a Failed result.

Looking at the rule logic in question

#Looking for default branch protection that restrict force push to branch
CbPolicy[msg] {
	not is_no_branch_protection
	is_branch_protection_restrict_force_push
	msg := {"ids": ["1.1.16"], "status": constsLib.status.Failed}
}

#Looking for default branch protection that restrict who can delete protected branch
CbPolicy[msg] {
	not is_no_branch_protection
	is_branch_protection_restrict_delete_branch
	msg := {"ids": ["1.1.17"], "status": constsLib.status.Failed}
}

this reads to say "when the branch is protected and disallows force pushes or deletions (in other words, if AllowForcePushes and AllowDeletions == false), produce a Failed result. In my mind, this should read as "when the branch is protected and allows force pushes or deletes, produce a Failed result.

Prepending not to both L226 and L233 causes Chain-Bench to produce an expected result.

Sarif report for chain-bench

Hi! could be possible the output of chain-bench result deliver a sarif report to integrate with Github?

https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/uploading-a-sarif-file-to-github

Code signing

Does chain-bench recognize code signing tools like sigstore (cosign, fulcio, rekor)?

Duplicate section heading in PDF

Description

I was reviewing the pdf of checks: https://github.com/aquasecurity/chain-bench/blob/main/docs/CIS-Software-Supply-Chain-Security-Guide-v1.0.pdf and noticed that for section 1.2, there's a duplication of the 1.2.2 value, so it reads as 1.2.2: 1.2.2 Name Of Check

What did you expect to happen?

n/a

What happened instead?

n/a

Additional details (base image name, container registry info...):

This is just for your information if you want to fix it, it causes no actual problems that I can see.

Self-hosted SCM support

Chain-bench currently only supports the SaaS version of GitHub and Gitlab, it would be useful to have support for other self-hosted or private SCMs.

Add support to SLSA compliance

the chain-bench could support SLSA requirements. https://slsa.dev/spec/v0.1/requirements

Support Bitbucket server SCM

Description

Support Bitbucket server SCM in addition of Github and Gitlab.

Provide a directly actionable solution whenever possible as part of the remediation

When a check fails, it is most of the time possible to fix it using the GitHub API.

For example, the number of reviewers required can be updated with one REST request:

curl -s \
  -X PATCH \
  -H "Accept: application/vnd.github.v3+json" \
  -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/repos/buildsec/frsca/branches/main/protection/required_pull_request_reviews \
  -d '{"required_approving_review_count":2}'

Attaching this snippet to the remediation explanation (https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1) would help the end users to resolve the issue.

New release?

Is there any plan to cut a new release? We would need for PR #71 to be available to move forward with our work.

false positive when the endpoint is not accessible (e.g. not enough permissions)

Description

Running ./chain-bench scan --repository-url https://github.com/buildsec/frsca --access-token $GITHUB_TOKEN -o ./results, some checks where marked as failed due to not having enough permissions associated with the GitHub token.

For instance:

1.1.3 Ensure any change to code receives approval of two strongly authenticated
users Failed

The problem was that the GITHUB_TOKEN did not have permissions to read the branch protection settings, therefore marking this check as failed.

However it should in this case show another status, like Unknown or not evaluated for instance, as the check was not able to read the results (the endpoint returned a 404).

The failed status implies that the requirement was not met, and should be reserved for cases where the number of required reviewers was strictly less than 2.

The same problem applies to all the checks that returned a 404.

Additional details in case it does not expose sensitive data (scanned pipeline files, PR, etc):

scan: segmentation fault while fetching authorized user

Description

scan throws segmentation fault.

What did you expect to happen?

$ chain-bench scan --repository-url github.com/Dentrax/cocert --access-token $TOKEN

What happened instead?

Error line:

chain-bench/internal/scm-clients/clients/clients.go

Line 32 in 25fe233

authorizedUser, _ := adapter.GetAuthorizedUser()

2022-07-07 17:12:19 INF 🚩      Fetch Starting
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x17370bb]

goroutine 1 [running]:
github.com/aquasecurity/chain-bench/internal/scm-clients/clients.FetchClientData({0x7ff7bfeff380, 0x1}, {0x7ff7bfeff357?, 0x1?})
        /home/runner/work/chain-bench/chain-bench/internal/scm-clients/clients/clients.go:32 +0x9b
github.com/aquasecurity/chain-bench/internal/commands.NewScanCommand.func1(0xc00020e280?, {0x18f4738?, 0x4?, 0x4?})
        /home/runner/work/chain-bench/chain-bench/internal/commands/scan.go:22 +0xac
github.com/spf13/cobra.(*Command).execute(0xc00020e280, {0xc000175f80, 0x4, 0x4})
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc00020e000)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/aquasecurity/chain-bench/internal/commands.Execute({0x1a41c1c?, 0xc0000021a0?})
        /home/runner/work/chain-bench/chain-bench/internal/commands/root.go:21 +0x32
main.main()
        /home/runner/work/chain-bench/chain-bench/cmd/chain-bench/main.go:12 +0x27

Additional details (base image name, container registry info...):

Does not work with corporative repository

It's OK with github.com for instance. But "panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb3ef3b]" with corporative gitlab repository even for admin with all token permissions.

chain-bench does not work with gitlab if user id 1 does not exist

Description

I tried to use chain-bench with our private GitLab instance.
Unfortunately, it fails with error in authenticated user data.
Starting in with -v reveals that the cause is:
error in authenticated user data error="GET https://myinstance/api/v4/1: 404 {message: 404 User not Found}"

This makes sense because we deleted the default root user with the id 1.
The lowest User ID in our Instance is 2.
I'm unsure why chain-bench requires the default user to work.
Imho it is quite common to delete it for security reasons after you created different admin accounts.

What did you expect to happen?

I expected chain-bench to scan my repository

What happened instead?

error in authenticated user data error="GET https://myinstance/api/v4/1: 404 {message: 404 User not Found}"

Additional details (base image name, container registry info...):

Tested with Version 0.1.7

Show showing all columns in the CLI table

The CLI format table has static columns, but the JSON results file has more data that is worth showing in the table, particularly "Remediation". Presumably the goal is for someone to see the results and try to improve them, and it's easier to see the remediation suggestions in table form than in JSON.

scan: ability to local directory

For example:

$ chain-bench scan .

I think currently passing --repository-url is a must.

Give a final rating

Chain-bench only provides the number of tests that passed e.g.: Total Passed Rules: 5 out of 36, more like unit test frameworks do.

While this provides a good overview, not all the checks are equal and should be weighted accordingly to generate a score for the repository. For example, using MFA should be worth more points, than defining a SECURITY.md file.

As a user I would like to get a score (e.g. 87%) associated to my repositories instead of simply displaying the number of tests that passed.

Rego schema support

OPA version 0.41.0 supports schema enforcement - https://www.openpolicyagent.org/docs/v0.41.0/schemas.
Please add schema for the object that passes to the Rego files and define the schema of the returned value.

link to compliance rules missing trailing slash

Description

In JSON output, links to compliance details are broken because they're missing a trailing slash

What did you expect to happen?

Links work

What happened instead?

Links are broken

(paste your output here)

Additional details (base image name, container registry info...):

Output from github action:
"url":"https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1"

working link: https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1/

A GitHub Action support

A GitHub Action would be very nice to be able to run it.

Originally posted by @hazcod in #26

False positives in control `1.2.3` and control `1.2.4`

Description

For controls 1.2.3 and 1.2.4 it always shows PASSED irrespective of the setting in Github

 1.2.3    Ensure repository deletion is limited to specific members                                       Passed                                                                           
 1.2.4    Ensure issue deletion is limited to specific members                                            Passed                                                                           

Additional details in case it does not expose sensitive data (scanned pipeline files, PR, etc):

Here's the snippet from mapper.go which denotes the value has been hard-coded for the setting.

Reference GitHub setting snapshot which allows users to restrict/allow repository deletion and issue deletion -

The result file has no date/identifier

As a user, I would like to be able to compare scan results to ensure we've improved and make sure we did not regress over time. This could also help capturing human mistakes (e.g. an admin changed a setting by mistake).

Add metadata

An idea could be to add a metadata section at the top of the file, which would contain at least the following information:

• Date
• Result score (related to #37)

Improve the output - help message

Current options in the chain-bench version 0.1.3

Flags:
  -c, --config-file string   the path to a local configuration file
  -h, --help                 help for chain-bench
  -l, --log-file string      set to print logs into a file
      --log-format string    sets the format of the logs (normal, json)
      --no-color             disables output color
  -o, --output-file string   the path to a file that will contain the results of the scanning
  -q, --quiet                silence logs, prints only error messages
  -v, --verbose count        set the verbosity level (-v: debug, -vv: trace), default: info
      --version              version for chain-bench

The message could be improved to explain that only support json. Similar like to

-o, --output-file string Export in json the results of the scanning