tweag/genealogos

Genealogos, a Nix sbom generator

Genealogos

About The Project

The Genealogos project is a tool that takes output from Nix evaluation tools and produces BOM files. Currently, it takes input from nixtract and produces json output compliant with the CycloneDX 1.3 or 1.4 specification. Output from Genealogos can be used by various other tools to perform further analysis.

Note Nix is mainly just suitable for Software, and so the BOM output by Genealogos is nearly always an SBOM. However, for consistency, we will refer to the output as a BOM.

The project is still very early stages, so the output may as of yet be of little use.

Installing Genealogos-cli

Right now, Genealogos is only distributed via our Nix Flake. Ensure you have the experimental features flakes and nix-command enabled, and then run:

# To run once
nix run github:tweag/genealogos -- --help

# To temporarily add to $PATH
nix shell github:tweag/genealogos

You can also forgo Nix and use cargo directly instead. However, in this case you are sponsible for ensuring the dependencies are available.

cargo install --git https://github.com/tweag/genealogos.git genealogos-cli

Hacking

Prerequisites

Development of Genealogos requires Cargo, and some other dependencies. We additionally recommend the use of rust-analyzer, cargo-hack, and nixtract. The easiest way to get these tools it through the Nix development environment.

nix develop

Alternatively you can install the dependencies manually, but in that case you are on your own.

Building

Due to a bug in Cargo, running cargo [build, check, clippy, etc] will fail. There are two ways to solve this, either by specifying a specific package cargo COMMAND -p [genealogos, genealogos-cli, genealogos-api] or by running cargo hack COMMAND.

Usage

genealogos-cli

Analyzing a local flake:

genealogos /path/to/your/local/flake

Analyzing hello from nixpkgs:

genealogos nixpkgs#hello

Using a trace file: This section assumes you are using the latest main version version of [nixtract][nixtract].

nixtract --target-attribute-path hello /tmp/out && genealogos -f /tmp/out

For more nixtract arguments, see nixtract --help.

Setting backend options: Any type that implements the Backend must have a way to include nar info and only include runtime options. Genealogos will forward --include-narinfo and --runtime-only to the backend.

genealogos --include-narinfo --runtime-only nixpkgs#hello

For a full set of options, see:

genealogos --help

genealogos-api

Genealogos can also run as an API server using the genealogos-api binary. genealogos-api provides two categories of endpoints. A blocking endpoint and one based on jobs.

Blocking

Currently, there is only a single blocking endpoint: /api/analyze?installable=<installable>. By default, genealogos-api binds itself on localhost:8000.

For example, using curl, the api can be invoked like this:

curl "http://localhost:8000/api/analyze?installable=nixpkgs%23hello"

Note that the # in nixpkgs#hello is URL Encoded.

Additionally an optional bom_format query parameter can be provided to specify the bom format to use. Example:

curl "http://localhost:8000/api/analyze?installable=nixpkgs%23hello&cyclonedx_version=v1_4"

Currently supported are [cyclonedx_1.3_json, cyclonedx_1.3_xml, cyclonedx_1.4_json, cyclonedx_1.4_xml], with cyclonedx_1.4_json being the default.

Jobs

The jobs based API consists of three endpoints: /api/jobs/create, /api/jobs/status, and /api/jobs/result.

Creating a job is done in a similar fashion to the blocking api:

curl "http://localhost:8000/api/jobs/create?installable=nixpkgs%23hello"

This endpoint also supports the bom_format query parameter. The response of this API call is a job_id, which needs to be passed to further calls to indentify the desired job.

Getting the status of a job is done as such:

curl "http://localhost:8000/api/jobs/status/0"

where 0 was the job_id provided in the previous call. This API can return one of stopped, running and done.

Finally, getting the result is done with the result endpoint:

curl "http://localhost:8000/api/jobs/result/0"

genealogos-frontend

Genealogos ships with a pure html/javascript web frontend. By default, this frontend uses 127.0.0.1 to connect to the genealogos-api. Changing this default can be done using the settings button in the top of the webpage.

The Web UI currently only supports analyzing from a flake ref and attribute path, analyzing from a trace file is not yet supported.

The frontend can be opened by opening the index.htmlfile in your favourite web browser. Alternatively, if the genealogos-api was built with the frontend feature-flag, the frontend can be accessed at the root of wherever the api is hosted (e.g. http://localhost:8000/).

NixOS Module

The flake in this project provides a NixOS Module to host Genealogos. Once the module has been added to your NixOS configuration, Genealogos can be enabled with:

services.genealogos.enable = true;

For further options see ./nix/genealogos-module.nix.

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue.

Testing

Genealogos is tested against fixtures in genealogos/tests/fixtures/nixtract/success/. With each .in file containing nixtract output and each .out file containing the corresponding expected genealogos output. Testing against these fixtures is done automatically by nix build, but can also manually be performed using cargo test. Typically, genealogos output is non-deterministic (the UUID is random, and the order of elements in lists is random), which makes testing a little more annoying. To overcome this hurdle, when running cargo test, or when setting the GENEALOGOS_DETERMINISTIC environment variable, the output of genealogos is made deterministc. This is done by setting the UUID to all zeroes, and sorting the dependsOn lists.

In order to make working with these fixtures a little nicer, the nix develop .#scripts devShell provides two scripts. verify-fixture-files, which verifies the .out files with the cyclonedx-cli tool to ensure genealogos produces valid CycloneDX. And update-fixture-files, which should be ran when an update to genealogos changes its output. Note that this second script requires that genealogos-cli is buildable.

License

Distributed under the MIT License. See LICENSE for more information.

Contact

GitHub: https://github.com/tweag/genealogos

genealogos's Issues

Add HTTP endpoints for long-running jobs (create, status, get result)

We will keep the existing /api/analyze endpoint, and extend it with three additional ones:

• /api/unblocking/create
• /api/unblocking/status
• /api/unblocking/result

XML support

Is your feature request related to a problem? Please describe.
The CycloneDX specification allows both a JSON and XML format. We currently only output JSON, but XML should also be supported.

Describe the solution you'd like
Add a flag to genealogos --output-format [json,xml] defaulting to json.

Additional context
https://cyclonedx.org/docs/1.5/xml/

Add an option to run nixtract from genealogos-cli

genealogos: get a CycloneDX SBOM from Nix via a browser

While the API is the first step to making SBOM from Nix available to everyone, we are convinced it should target an audience as broad as possible, beyond those who can use an HTTP API directly.

To that end, the web service should provide a UI that facilitates the process for everyone.

Deliverables:

• Release of a new version of genealogos that includes a browsable web service to generate a CycloneDX SBOM from a reference to a Nix flake
• Guide on how to run the web server
• Documentation for the web interface

Tasks

Beta Give feedback

No tasks being tracked yet.

cargo-dist is broken

Describe the bug
cargo-dist uses a cargo build command under the hood. However, given an issue in cargo, cargo build is broken in our repository.

Instead, we use cargo hack build to perform a build of all packages.

Ideally, cargo-dist would be able to use cargo-hack directly.

To Reproduce
Run: cargo dist build.

Implement `dependencies` field of the CycloneDX spec

This is one of the more important fields of the spec, which denotes the dependencies between all the components in the components field.

Create a web server with a simple HTTP endpoint for a single call

genealogos: write a NixOS module to self-host Genealogos

It should be easy to run your own Genealogos instance. Because we love NixOS, we want to provide a NixOS module so that deploying Genealogos should be as easy as add services.genealogos.enable = true; to a NixOS configuration.

Deliverables:

• Nix code of a NixOS module in the source repository to run a Genealogos instance
• Documentation explaining how to use and configure the module
• Link to a GitHub Pull Request that submits the NixOS module to nixpkgs

genealogos: add support for multiple CycloneDX versions in the SBOM generator

The CycloneDX format is evolving rapidly and often, we would like to allow Genealogos users to choose the version of CycloneDX for the SBOM to generate.

Deliverables:

• Release of a new version of genealogos that includes the feature to choose the version of CycloneDX to use for the output
• Updated documentations to explain the feature

Include nixtract options

Is your feature request related to a problem? Please describe.
nixtract has two notable options that aren't exposed by genealogos.

• --runtime-only/-r which ensures nixtract only produces runtime dependencies in the graph
• --include-nar-info/-n which informs nixtract to query substituters for narinfo.

Describe the solution you'd like
These should be ported to genealogos

provide a CLI to generate a CycloneDX SBOM from a Nix URL-like Flake reference

Tasks

Beta Give feedback

1. Create a good CLI (--help etc)
2. Call nixtract automatically

Consider switching from serde_cyclonedx to cyclonedx-sbom

Since we started this project, the official rust crate cyclonex-sbom for CycloneDX has gotten into a state where it is suitable for our purpose.

But there are some things to consider:
Advantages:

• Support for XML and JSON, see also #39
• Code de-duplication, we currently need a lot of duplicate code to generate both 1.4 and 1.5, using this crate would resolve that issue.
• We can drop our dependency on some other crates as well

Disadvantages:

• The official crate has (as of yet) only support for 1.3 and 1.4, with the current newest standard being 1.5, however support for 1.5 is on the horizon with CycloneDX/cyclonedx-rust-cargo#646.

Include nixtract's narinfo ouput in our components

Blocked by tweag/nixtract#29

genealogos: incorporate metadata from built derivations

nixtract outputs metadata from the definitions in the Nix code, but some other information like closure size or runtime dependencies require building Nix derivations. In order to enrich the generated SBOM, genealogos needs to be able to incorporate such information, usually available in a binary cache server, called substituter in the Nix ecosystem.

Deliverables:

• Release of a new version of genealogos that includes which incorporates metadata from a Nix substituter to the output SBOM when available
• Updated documentations to explain the feature

Tasks

Beta Give feedback

1. Include NarInfo in the output nixtract#29
   type:feature +1
   
2. Include nixtract's narinfo ouput in our components #35
   +0
   
3. Update documentation where applicable.

Output PURLs for components

CycloneDX analysis programs like snyk use the purls of the components to lookup CVE's to report, there exists no nix PURL (yet). But as an alternative we can use the generic PURL Type, which we can extract from the src attribute of derivations. This might result in false positives (because Nixpkgs often includes its own patches into the source), experimentation is needed.

Subtask of #2

genealogos: get a CycloneDX SBOM from Nix via HTTP

A long term goal would be for anyone to be able to host a service that makes the generation of SBOM from Nix accessible to anyone.

To that end, genealogos should provide an HTTP API to generate SBOM, so everyone can easily get an SBOM from a HTTP request.

Deliverables:

• Release of a new version of genealogos that includes an HTTP API server to generate a CycloneDX SBOM from a reference to a Nix flake
• Guide on how to run the web server
• Documentation for the API endpoints

Tasks

Beta Give feedback

1. Add an option to run nixtract from genealogos-cli #25
   +0
   
2. Create a web server with a simple HTTP endpoint for a single call #28
   +0
3. Add HTTP endpoints for long-running jobs (create, status, get result) #30
   +0
   

genealogos: output a CycloneDX SBOM from Nix via a terminal

Existing tools to generate SBOM from Nix (sbomnix, nixbom, bombon), are limited to derivations that have been realized in the Nix store, meaning packages that have been built and are available locally. nixtract, an open-source project developed at Tweag, can extract metadata about derivations without building them. Combining this data with Nix builtin capability to extract a dependency graph, we can build a SBOM generator that can be used by anyone, in the form of a command line tool.

Our first goal is to release an initial version of genealogos with minimal functionality.

Tasks

Beta Give feedback

1. provide a CLI to generate a CycloneDX SBOM from a Nix URL-like Flake reference #8
   1 of 2
   type:feature-request +1
   
2. Implement dependencies field of the CycloneDX spec #3
   type:feature-request +1
3. Complete component information #2
   type:feature-request +1
   
4. Output PURLs for components #4
   type:feature-request +1
   

Complete component information

The components array of the cycloneDX v1.5 spec contains the information of all components in the sbom.

The information below relates both to nixtract

• type (required) - Since nixpkgs does not expose this, all nixpkgs components are currently applications.
• mime_type - impossible to determine without accessing the Nix store
• bom_ref - the derivation's output path
• supplier - data not present in nixpkgs
• author - data not present in nixpkgs
• publisher - data not present in nixpkgs
• group - see publisher above
• name (required) - the package name
• version - the package version from the metadata
• description - the package description from the metadata
• scope - cannot be derived using nix eval, can be left empty for the default of "required"
• hashes - might be doable for CA derivations, otherwise impossible without access to the nix store
• licenses
• copyright - data not present in nixpkgs
• cpe - data not present in nixpkgs, but might be extractable from the official CPE dictionary
• purl - unachievable, the nix purl isn't standardized (yet) (https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#other-candidate-types-to-define)
• swid - TBD, not present in nixpkgs, for exceptional packages maybe in the store
• modified - deprecated, should not be present
• pedigree - maybe for later
• external_references
• components - not extractable from nixpkgs, we cannot derive this information
• evidence - TBD, maybe when/if we perform more in-depth analysis
• releaseNotes - based on the source, might be extractable from nixpkgs, low priority
• modelCard - only applicable to components of type "machine-learning-model", which we don't produce yet
• data - only applicable to components of type "data", which we don't produce yet
• properties - TBD: https://github.com/CycloneDX/cyclonedx-property-taxonomy
• signature - if desired, official tools can be used to add a signature to Genealogos' output. Genealogos will not include signing capabilities.