philips-software/bom-base

Caching repository for bill-of-materials metadata

BOM-Base

Description: BOM-Base is an experimental caching repository for bill-of-materials metadata

Status: Experimental research prototype

Powered by Philips SWAT Eindhoven

While producing a bill-of-materials, a private metadata cache is desirable to augment the publicly available metadata with local curations and store metadata for packages that are not publicly released.

The service consists of a metadata store with "harvesters" to collect metadata. The unavailability and modification of metadata automatically triggers harvesters to fill the cache.

The harvesting mechanism starts from a client requesting metadata for a specific package. If the package is unknown, one or more harvesters start collecting metadata from external sources. The harvester for the relevant package management repository obtains the basic package metadata, and another harvester might additionally pull various curated fields for the same package from ClearlyDefined. A scoring mechanism ensures the most reliable metadata is kept. The availability of a source code location (and no scanned license) could trigger the license scanning harvester to download and scan the source code for licenses and other copyright information. If the scanned license does not match the license declared in the originating repository, it can be contested by a harvester that checks consistency between the "declared" and "detected" license fields. When a client later requests the same package, it receives the latest updated metadata.

The user interface allows human inspection and curation of the metadata. A manual change of such metadata can in turn trigger other processes to complete additional fields.

(See the architecture document for a detailed technical description.)

Dependencies

The service requires at least Java 11.

Installation

Build the executable JAR

The Flutter web user interface should be first built through the install_ui script in the /ui directory. (This script checks and builds the web application and installs it into the /src/main/resources/static directory of the backend.)

Next, the backend can be built through the Maven mvn clean install command, and yields a "fat" executable jar containing all dependencies.

The backend server starts as a standard Java executable:

java -jar BOM-base-<version>.jar

Some useful command line parameters are:

• --server.port=9090 changes the http port (from default 8080) to 9090.
• --bom-base.scan-licenses=false disables the source code license scanner, reducing the machine load during development and testing.
• --bom-base.harvest-clearly-defined=false disables the clearly-defined lookup

Install ScanCode Toolkit license scanner

Scanning licenses from source files is delegated to ScanCode Toolkit.

Follow any of these installation instructions to install the command line application.

Then make sure the scancode and extractcode commands are accessible from any directory by updating the path or creating symbolic links in an appropriate location.

Docker

After building the project, you can build and run the application using Docker.

Build docker image:

docker build -f docker/Dockerfile -t bom-base .

Run docker container:

docker run -p 8080:8080 bom-base

Image from docker hub

The latest released version is also available from Docker Hub:

docker run -p 8080:8080 philipssoftware/bom-base:latest

Configuration

(Empty)

Usage

The service exposes a REST API and a user interface on port 8080.

Proper operation can be checked by e.g.:

curl http://localhost:8080/packages/pkg%253Anpm%252Fmarked%25400.7.0 | jq

Harvesters will then start collecting the metadata for the pkg:npm/[email protected] package if its metadata was not yet available. Else it returns the existing metadata for the package.

How to test the software

Unit tests for this Maven are run by the mvn clean test command.

Note that ScanCode Toolkit must be installed for all tests to pass. (See installation instructions)

Known issues

(BOM-Base is still under development.)

Disclaimer

BOM-Base is an experimental tool, and not suited for production.

Contact / Getting help

Submit an issue in the issue tracker of this project.

License

See LICENSE.md.

Credits and references

• BOM-Base relies for scanning of license information from source code on ScanCode Toolkit.
• Many thanks go out to the nice people at OSS Review Toolkit for their work and being an inspiration to try a different approach for managing bill-of-materials metadata.
• If you are looking for tools to build a bill-of-materials, you might want to have a look at the SPDX-Builder project that can (among various other solutions) use BOM-Base metadata to build rich bill-of-materials documents in the SPDX format.

bom-base's Issues

Show statistics on dashboard about workload

Would be nice to see how many tasks are remaining in the queue, and how many packages are currently stored.

Idea would be to provide this information on a status API, and have the main client screen periodically poll that information.

Add harvester for NPM

Autoconvert "LicenseRef-scancode-public-domain" to "Public Domain" license

The ScanCode Toolkit scanner could automatically convert this internal identifier into the more common phrasing of a component being generic "Public Domain".

Move known issues from readme to issues.

Current situation

There's a list of known issues in the readme.

• Retry failed harvesting attempts (e.g. when source offline)
• #40
• #41
• #42
• #43
• #44
• #45
• #46
• #48
• #47
• #49

Proposed Situation

Move the known issues from the readme to issues to make them more contributor friendly to resolve

Extra

It more easy to see who's working on what.

Harvesters don't provide version info in (GIT) source location

Many package managers provide a generic repository URL as source location. When using this URL for download, this will yield the latest version of the default branch in the GIT repository. This is most probably not the source code for the actual package version.

The downloader understands a syntax of appending @ and the version number to a GIT URL, but this should be provided by the harvester.

Gotcha: Sometimes a Git URL is specified by a package manager with a "user" prefix, like: [email protected]/...

[NpmHarvester] Support UNMET dependencies.

Problem

Some npm trees have dependencies which are unmet. UNMET PEER DEPENDENCY

This will end-up with an 405 error in the harvester.

Example tree

  ├─┬ @svgr/[email protected]
  │ ├── @babel/[email protected] deduped
  │ ├─┬ @babel/[email protected]
  │ │ └── @babel/[email protected]
  │ ├─┬ @babel/[email protected]
  │ │ ├── @babel/[email protected]
  │ │ ├── UNMET PEER DEPENDENCY @babel/core@^7.13.0
  │ │ ├─┬ @babel/[email protected]

Log

2021-07-08 15:07:38.305 ERROR 13310 --- [         pool-2] .a.i.SimpleAsyncUncaughtExceptionHandler : Unexpected exception occurred invoking async method: public void com.philips.research.bombase.core.meta.registry.QueuedTaskRunner.execute(com.github.packageurl.PackageURL,java.util.function.Consumer,java.util.function.Consumer)

com.philips.research.bombase.core.npm.NpmException: Failed to harvest pkg:npm/unmet%20peer%20dependency%20%40babel%2Fcore@%5E7.13.0
	at com.philips.research.bombase.core.npm.domain.NpmHarvester.harvest(NpmHarvester.java:61) ~[classes!/:0.1.1-SNAPSHOT]
	at com.philips.research.bombase.core.npm.domain.NpmHarvester.lambda$onUpdated$0(NpmHarvester.java:45) ~[classes!/:0.1.1-SNAPSHOT]
	at com.philips.research.bombase.core.meta.registry.QueuedTaskRunner.lambda$execute$0(QueuedTaskRunner.java:38) ~[classes!/:0.1.1-SNAPSHOT]
	at java.base/java.util.Optional.ifPresent(Optional.java:176) ~[na:na]
	at com.philips.research.bombase.core.meta.registry.QueuedTaskRunner.execute(QueuedTaskRunner.java:36) ~[classes!/:0.1.1-SNAPSHOT]
	at com.philips.research.bombase.core.meta.registry.QueuedTaskRunner$$FastClassBySpringCGLIB$$b8050159.invoke(<generated>) ~[classes!/:0.1.1-SNAPSHOT]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.7.jar!/:5.3.7]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) ~[spring-aop-5.3.7.jar!/:5.3.7]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.7.jar!/:5.3.7]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.7.jar!/:5.3.7]
	at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) ~[spring-aop-5.3.7.jar!/:5.3.7]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]
Caused by: com.philips.research.bombase.core.npm.NpmException: NPM server responded with status 405
	at com.philips.research.bombase.core.npm.domain.NpmClient.query(NpmClient.java:56) ~[classes!/:0.1.1-SNAPSHOT]
	at com.philips.research.bombase.core.npm.domain.NpmClient.getPackage(NpmClient.java:45) ~[classes!/:0.1.1-SNAPSHOT]
	at com.philips.research.bombase.core.npm.domain.NpmHarvester.harvest(NpmHarvester.java:50) ~[classes!/:0.1.1-SNAPSHOT]
	... 14 common frames omitted

Add auto-converter for obvious (but formally wrong) SPDX license names

Add harvester for NuGet

Add section in readme about analyse scripts.

In #74 we've added analyse scripts.
Please add some documentation on how to use this and why you want to use this.

Background

The scripts are part of a blog we're writing on the differences between Black Duck license information and other sources. We can also add a link to that blog in this README.

Harvesting metadata from inner source repositories

Add harvester for APK

Persist metadata in database

Retry failed harvesting attempts (e.g. when source offline)

Automatically resolve license URL

When a license is found to be a (valid) URL, a harvester (or just the base repository harvester) could pass the URL to the license scanner to read the referenced license and use the result of the scan instead.

Build UI in CI/CD

Make sure the UI is also build in the CI

Related

Similar to philips-software/bom-bar#26

Add harvester for Cargo

Add harvester for Debian

Handle false information (e.g. non-existing source location)

Add harvester for Maven

Verified releases do not work.

In #76 we tried to fix creating verified releases.

This does not seem to work see ( https://github.com/philips-software/bom-base/releases/tag/v0.2.1 )

Add scancode in dockerfile

Add scancode in dockerfile