interlynk-io/sbomqs

SBOM quality score - Quality metrics for your sboms

sbomqs's Issues

Fix ShareLink to point to sbombenchmark.dev

Change to official link

sbomqs version has unnecessary detail

The attached result of sbomqs version shows GitCommit and GitTreeStatus, which are unnecessary for the official release and can be omitted.

CycloneDX XML SBOM incorrectly checks for bomFormat

Scoring for the two SBOMs at

• https://sbomlc.s3.amazonaws.com/syft-0.73.0_busybox-1-glibc.cdx.xml?AWSAccessKeyId=AKIA2ZBFUJ4NNQGYD5OF&Signature=u7AiVh34yhTcPM59TJkvr9WAClc%3D&Expires=1709195074
• https://sbomlc.s3.amazonaws.com/syft-0.73.0_busybox-1-glibc.cdx.json?AWSAccessKeyId=AKIA2ZBFUJ4NNQGYD5OF&Signature=QbISBlxFIfeq0XKSnw4p7pQwLXA%3D&Expires=1709195074

even though both are prepared for the same image/tag with the same tool (syft-0.73.00). The root cause is bomFormat and version fields which are required in JSON and omitted from XML : https://cyclonedx.org/docs/1.4/xml/#element_bom

Expected Result:
Both sboms should produce identical scores.

Unique SPDXID comply with NTIA Minimume Elements requirements

sbomqs uses the presence of CPE/PURL to check for 'Other unique identifiers' requirements. However, after discussing it with Kate Stewart here, it is clear that SPDXID alone can satisfy that criterion for the SDPX.

Let us change sbomqs towards the original intent.

Align rules with OWASP SCVS initiative

OWASP SCVS is formalizing verification requirements for SBOMs.
sbomqs rules to test SBOM against as well output should be aligned to meet the proposed checks in SCVS.

Add user_guide

Add a user guide to help users of the tool, to understand the various use cases

New rule for checking of "vulnerability lookup identifier" present

Issue #48 requires checking for the presence of SPDXID to meet NTIA Minimum Elements requirement. However, the underlying original issue will leave SPDX SBOMs without a vulnerability lookup key (CPE/PURL) with the same score as the ones with such a key present.

I propose a new rule under Quality category called.

Components can be checked for vulnerabilities with a key with values listing all components with a unique CPE/PURL present.

The score for a file like the one attached should be 8/15 components normalized to 10.0
bom-alpine-3.15.spdx.txt

Integrate quality scoring on Known Unknowns from NTIA Minimum Elements

NTIA minimum elements recommend SBOM to include known unknowns. A declaration of unknowns must be used to calculate quality - TBD>

Automate homebrew publishing via goreleaser

Using brew directive, automate homebrew release via goreleaser.

Make file names as default for scoring

Tool currently requires --filepath for input SBOM
--filepath string sbom file path
Let's make that default so that we can get this done

build/sbomqs score ~/sbom-samples/repos/trivy-bom.spdx

without --filepath

Build sbomqs with goreleaser

Lets use goreleaser to cross-compile binaries for various platform, this will help users to download binaries for their platform, without the need to compile this code.

https://goreleaser.com/
https://github.com/goreleaser/goreleaser

Integrate Dependabot

Integrate Dependabot to improve alerting and updates of underlying packages

configurable scoring criteria

As of version 0.0.4, sbomqs tests against a set of rules with equal weights and a max score of 10 for each, however, only some rules apply to some organizations (e.g., an organization using SBOMS for internal monitoring might not generate checksum).
The proposed enhancement is to list rules and weights for the rule in a configurable file and apply only the rules from that configuration. A sample configuration file could look like:

Rule 1: Components have versions

• Weight: 1.0
• Max Score: 10.0

Rule 2: Components have uniq ids

• Weight: 0.0 (Disables the rule)
• Max Score: 10.0

Rule 3: Doc has authors

• Weight: 0.5
• Max Score: 10.0

Final score = SUM(wiScore(Ri))

run is
sbomqs score --config scoring_config.yaml --filepath bom.cdx.json

Invalid reportFormat must result in help message #23

Acual:

Expected:

1. error in reportFormat value ('custom') should be reported with a help message
2. Exit code should not be 0

Well structure the README.md file

The current readme is just a data dump. We need to structure it well for consumption.

Improve help message by listing supported category and reportFormat values

Actual:

Expected:

1. List reportFormats options: ['basic'|'detailed']
2. List category options: ['NTIA-minimum-elements'|'Quality'|'Semantic'|'Sharing'|'Structural'|'All']
3. Indicate defaults : detailed, all

Integrate tests into CI before merge

Reference: https://github.com/ossf/scorecard/blob/93900aca36f0f11a82efb7a3cc64341061ce5658/docs/checks.md#ci-tests

Improvements in reportFormat JSON for errors during processing

When --reportFormat is JSON, we must ensure:

1. Output is a valid JSON
2. Score is set to 0 if there are any internal or processing errors.
3. An error string is populated with any necessary error strings.

I recommend following changes:

  "run_id": "e6468750-1506-42ff-b635-16068fdcdb27",
  "timestamp": "2023-02-27T20:37:11Z",
  "creation_info": {
    "name": "sbomqs",
    "version": "v0.0.7",
    "scoring_engine_version": "2"
  },
  "files": [
    {
      "file_name": "sboms/trivy-0.36.1_alpine-3.17.2.spdx.json",
      "spec": "", # Cannot be filled
      "spec_version": "", # Cannot be filled
      "file_format": "json",
      "avg_score": 0,
      "error": "Error processing file: Invalid SBOM"
   }```

Add consistent creationInformation as a SBOM Quality Check

Creator tool information is critical to tracking the creator's capabilities and maturity.

Rule Proposal Creator Tool information must be valid and include the tool's name and version.
Category Quality
Notes

• SPDX requires the tool and version to be in this format:
  Tool: toolidentifier-version
• CycloneDX requires tool and version to be present within metadata

name: The name of the tool
version: The version of the tool

Score
0 if either tool name or version is missing
10 if both tool name and version are present.

[Scoring] Files Analyzed

A component's metadata in an SBOM is probably more accurate if the generator tool has analyzed the files for the repo. We should consider using this metric for scoring. We know this field exists for SPDX however for cyclonedx we are still re-searching how this is encoded.

@surendrapathak

PURL as a type

Similar to #9

We need to follow the PURL spec.

Create signed releases

Reference: Signing releases on GitHub
https://wiki.debian.org/Creating%20signed%20GitHub%20releases

SPDX License NOASSERTION values should be counted towards "absent" values

Actual
Quality score for the test.spdx.txt

| Components have no deprecated | 10.0/10.0 | 0/331 components have | | | licenses | | deprecated licenses |

| Components have no restricted | 10.0/10.0 | 0/331 components have | | | licenses | | restricted licenses |

Expected
Given that all licenses are NOASSERTION (Line: 4686), which is not a useful value, the score should be 0/10.0

Add goreleaser supply chain template

https://github.com/goreleaser/goreleaser-example-supply-chain

Add component count to json output

To be consistent with other output formats, added component counts to the json format.

Add scoring for restrictive licenses

This scoring method should fall under the "Quality" Category.

Restricted Licenses are licenses that require mandatory source distribution of the included application or library in which they are included.

Lets add the following licenses to this scoring method

[Scoring] Quality - Describes a main document

We need a scoring check in quality if the sbom, describes a main component. I'm not sure what does multiple document describe in SPDX means, but we can discuss that.
@surendrapathak
e.g

SPDX has a documentDescribes
CycloneDX has metadata->component

Improve error reporting from processing of the file

As of v0.0.7, any error from processFile(

sbomqs/cmd/score.go

Line 81 in 1d31e60

func processFile(ctx context.Context, filePath string) (sbom.Document, scorer.Scores, error) {

) ends up with identical error message.

sbomqs score --filepath flux_sbom.spdx.json --reportFormat basic
failed to parse flux_sbom.spdx.json
Error: error processing file
Usage:
  sbomqs score [flags]

Flags:
      --category string       scoring category
      --dirpath string        sbom dir path
      --filepath string       sbom file path
  -h, --help                  help for score
      --reportFormat string   reporting format basic/detailed/json

The tool should provide actionable error messages e.g.

1. Stats failure should be reported as "SBOM not found at location {loc}".
2. Open failure should be reported as "SBOM open failure at {loc}."
3. Parsing failure should be reported as "SBOM fails specification checks at {loc}."

In addition:

• remove showing "usage" for any of those three condition (presuming user has already run it and there is no problem with the command itself).
• always report back score (0.0) in case of error for downstream tool consumption

Result ordering is not fixed across SBOM types

Consider two SBOMs (CDX-JSON/SPDX-TV) for the same image/tag below.

sbomqs score --filepath /Users/spathak/interlynk/sbomlc/trivy-0.36.1_busybox-1-glibc.cdx.json --category NTIA-minimum-elements
SBOM Quality Score:2.9	components:0	/Users/spathak/interlynk/sbomlc/trivy-0.36.1_busybox-1-glibc.cdx.json
+-----------------------+--------------------------------+-----------+--------------------------------+
|       CATEGORY        |            FEATURE             |   SCORE   |              DESC              |
+-----------------------+--------------------------------+-----------+--------------------------------+
| NTIA-minimum-elements | Components have supplier names | 0.0/10.0  | 0/0 have supplier names        |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have names          | 0.0/10.0  | 0/0 have names                 |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Doc has authors                | 10.0/10.0 | doc has 1 authors              |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Doc has creation timestamp     | 10.0/10.0 | doc has creation timestamp     |
|                       |                                |           | 2023-02-28T07:16:03+00:00      |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Doc has relationships          | 0.0/10.0  | doc has 0 relationships        |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have uniq ids       | 0.0/10.0  | 0/0 have unique ID's           |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have versions       | 0.0/10.0  | 0/0 have versions              |
+-----------------------+--------------------------------+-----------+--------------------------------+
sbomqs score --filepath /Users/spathak/interlynk/sbomlc/trivy-0.36.1_busybox-1-glibc.spdx.tv --category NTIA-minimum-elements
SBOM Quality Score:7.1	components:1	/Users/spathak/interlynk/sbomlc/trivy-0.36.1_busybox-1-glibc.spdx.tv
+-----------------------+--------------------------------+-----------+--------------------------------+
|       CATEGORY        |            FEATURE             |   SCORE   |              DESC              |
+-----------------------+--------------------------------+-----------+--------------------------------+
| NTIA-minimum-elements | Components have supplier names | 0.0/10.0  | 0/1 have supplier names        |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have names          | 10.0/10.0 | 1/1 have names                 |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Doc has relationships          | 10.0/10.0 | doc has 1 relationships        |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Doc has creation timestamp     | 10.0/10.0 | doc has creation timestamp     |
|                       |                                |           | 2023-01-30T02:05:22Z           |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have versions       | 0.0/10.0  | 0/1 have versions              |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have uniq ids       | 10.0/10.0 | 1/1 have unique ID's           |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Doc has authors                | 10.0/10.0 | doc has 2 authors              |
+-----------------------+--------------------------------+-----------+--------------------------------+

The resulting ordering needs to be more consistent. For example, check Doc has authors is 2nd check with CycloneDX but the last check is with SPDX.

Expected behavior
The result ordering is consistent across SBOM types.

trivy-0.36.1_busybox-1-glibc.cdx.json.txt

trivy-0.36.1_busybox-1-glibc.spdx.tv.txt

Make sbomqs accessible with homebrew

Goal
To achieve the following method of installing sbomqs

$brew new-tap interlynk-io/sbomqs
$brew install sbomqs
$brew update sbomqs

[Feature Request] Output JSON

Great tool!

An ability via a command-line flag to output the results in JSON would be much appreciated. I know I would benefit. Perhaps other users would benefit too.

[Scoring] Copyright

Currently we check a components metadata for presence of licenses, however we do not check for copyright information. The presence of this information, helps consumer tooling to better consume this SBOM.

ChatGPT

Copyright and license are related but distinct concepts in the realm of intellectual property law.

Copyright is a legal right granted to the creator of an original work, such as a literary, artistic, or musical work, which gives the creator the exclusive right to use, reproduce, distribute, display, and perform the work. Copyright protection is automatic and does not require registration, although registration can provide additional benefits, such as the ability to sue for infringement and recover damages.

A license, on the other hand, is a legal agreement between the copyright holder and another party that grants permission to use the copyrighted work in certain ways. Licenses can be exclusive or non-exclusive, and can specify the scope, duration, and terms of use of the copyrighted work. In other words, a license is a formal permission granted by the copyright holder to another party to use the work under certain conditions.

In summary, copyright is a legal right that protects the original works of creators, while a license is a legal agreement that grants permission to use a copyrighted work in certain ways. A license is a means by which the copyright holder can permit others to use their copyrighted work while still retaining ownership and control over it.

@surendrapathak

On readme - Goals - 1, Assess is misspelled

It appears that on Goals; Goal 1, assess is misspelled on readme.md.

[SCVS] Implement tests from SCVS Bom Maturity Model V2 Control Family

Placeholder for : https://docs.google.com/spreadsheets/d/1wu6KbgwuokC5357ikrhFN-QkwQ7Pyb6z0zE80sTNNus/edit#gid=0&range=D17

On readme.md #Goals: 1: Assess is misspelled

It appears that on Goals; Goal 1, assess is misspelled.

Report format should not include full-path

sbomqs score --filepath ~/Downloads/bomber.cyclonedx.json --reportFormat json
{
  "run_id": "e2c649b0-b33a-4e60-b973-d98bc9b5182c",
  "timestamp": "2023-03-16T22:22:07Z",
  "creation_info": {
    "name": "sbomqs",
    "version": "v0.0.10",
    "scoring_engine_version": "4"
  },
  "files": [
    {
      "file_name": "/Users/spathak/Downloads/bomber.cyclonedx.json",
      "spec": "spdx",
      "spec_version": "SPDX-2.3",

Expected Behavior
The file_name should only include name of the file

Actual Behavior
The file_name includes an entire path to the file (/Users/spathak/ etc).

Add sbomqs to oss scorecard github action

https://github.com/ossf/scorecard-action#installation

SPDX PackageSupplierName NOASSERTION values should be counted towards "absent" values

Actual
Quality score for the test.spdx.txt
file results in 10.0/10.0 score for supplier names:
| | Components have supplier names | 10.0/10.0 | 331/331 have supplier names |

Expected
Given that all supplier names are NOASSERTION (Line: 4682), which is not a useful value, the score should be 0/10.0

Tool Roadmap

Adding creationInfo details for each sbom being processed

when --reportFormat is JSON, we want to improve certain parts:

1. In case of any error, wrap the error in JSON and set score to 0
2. Include the "tool creator information with each file to help eliminate second parsing for downstream tools. e.g for Syft-0.73.0

      "file_name": "/Users/spathak/Downloads/bomber.cyclonedx.json",
      "spec": "spdx",
      "spec_version": "SPDX-2.3",
      "file_format": "json",
      "avg_score": 7.945727482678983,
      "creation_info": {
        "name": "syft",
        "version": "v0.73.0"
      }

SWID is a recognized SBOM format by NTIA

Add support to score SBOM generated in SWID format

SWID tags can be used as an SBOM, since they provide identifying information for a software
component, a listing of files and cryptographic hashes for the constituent artifacts that make up
a software component, and provenance information about the SBOM (tag) creator and software
component creator. Tags can explicitly link to other tags, enabling a representation of a
dependency tree

CPE as a type

Currently we implicitly trust the content in a CPE field to be a valid CPE. Its quite possible
the CPE does not follow the CPE specification.

To ensure that in fact these URL match spec, we should introduce cpe as a type, with a Valid method.

Invalid category must result in help message

Same as #22 for category

Actual:

Expected:
error in category value ('NTIA-minimum-element') should be reported with a help message
The exit code should not be 0

Share your score with others.

Users would like to be able to share their sbom scores with sbom generators or consumers. We can enable this workflow by providing a share command in sbomqs.

1. The SBOM should never leave the users machine.
2. We should only post the score json for the file.

Host: https://sbombenchmark.dev
Api: /user/score
Req: Post
Body: json request

sbomqs share <sbom-file> 

SBOM Quality Score:6.3  components:23   cdxgen-bom.json
ShareLink: https://sbombenchmark.dev/user/score?id=f0c3a36a-25e5-4da5-af3c-fe350489b85d

Empty component list should not be treated as low quality

The attached sbom has a score of 0 for all component checks.

Example:

• Doc has relationships - 0.0/10.0
• Components have uniq ids 0.0/10.0
• Components have names 0.0/10.0

sbomqs score --filepath /Users/spathak/interlynk/sbomlc/trivy-0.36.1_busybox-1-glibc.cdx.json --category NTIA-minimum-elements
SBOM Quality Score:2.9	components:0	/Users/spathak/interlynk/sbomlc/trivy-0.36.1_busybox-1-glibc.cdx.json
+-----------------------+--------------------------------+-----------+--------------------------------+
|       CATEGORY        |            FEATURE             |   SCORE   |              DESC              |
+-----------------------+--------------------------------+-----------+--------------------------------+
| NTIA-minimum-elements | Doc has creation timestamp     | 10.0/10.0 | doc has creation timestamp     |
|                       |                                |           | 2023-02-28T07:16:03+00:00      |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have supplier names | 0.0/10.0  | 0/0 have supplier names        |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Doc has relationships          | 0.0/10.0  | doc has 0 relationships        |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have uniq ids       | 0.0/10.0  | 0/0 have unique ID's           |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have names          | 0.0/10.0  | 0/0 have names                 |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Components have versions       | 0.0/10.0  | 0/0 have versions              |
+                       +--------------------------------+-----------+--------------------------------+
|                       | Doc has authors                | 10.0/10.0 | doc has 1 authors              |
+-----------------------+--------------------------------+-----------+--------------------------------+

However, the sbom itself has no components. Therefore, it should be not be given low score for missing names/ids.

Expected Behavior
If the component list is empty, assume SBOM is accurate and default to providing a full score (10.0) for checks requiring component details

trivy-0.36.1_busybox-1-glibc.cdx.json.txt

Multi Vuln lookup rule re-evaluate

In the multi vuln lookup rule, we check to see if a component has both CPE & PURL. Our thinking was the more the merrier for looking up the vuln in NVD. CPE have been known to be error prone and manual. This scoring rule could influence sbom generators to just put in a CPE, even if they dont have the correct data.

We should re-evaluate if this is a legit concern or not.

Auto release when new tag is created.

We already make use of goreleaser. Now using github actions we can auto-release when a new tag is made.

[Feature] Add detection if new version of the tool is available

Check github if new versions of the tools are avaliable and inform the user.

[Scoring] Add depth of sbom components to metadata

Surface component depth in all reporting formats.

Add sbomqs sbom to our release output

Add sbom to sbomqs release output