OSS Review Toolkit

Linux (OpenJDK 8)	Windows (Oracle JDK 9)

Introduction

The goal of the OSS Review Toolkit (ORT) is to verify Free and Open Source Software licence compliance by checking project source code and dependencies.

At a high level, it works by analyzing the source code for dependencies, downloading the source code of the dependencies, scanning all source code for license information, and summarizing the results.

The different tools that make up ORT are designed as libraries (for programmatic use) with minimal command line interfaces (for scripted use, doing one thing and doing it well).

The toolkit is envisioned to consist of the following libraries:

Analyzer - determines dependencies of a software project even when multiple package managers are used. No changes to software project required.
Downloader - fetches the source code based on output generated by the Analyzer.
Scanner - wrapper around existing copyright / license scanners which takes as input the output of the Downloader and produces standardized output such as SPDX.
Evaluator * - Evaluates the scan results from Scanner as OK or NOT OK based on user specified approval / rejection ruleset.
Advisor * - Retrieves security advisories based on Analyzer results.
Reporter - Presents the output from Analyzer and Scanner in various formats, making it easy to identify the dependencies, licenses, copyrights and issues.
Documenter * - Generates the outcome of the review, e.g. Open Source notices and annotated SPDX files that can be included into your distribution.

* Libraries to be completed by Q3 2018.

Installation

Follow these steps to get started with the OSS Review Toolkit:

Ensure OpenJDK 8 or Oracle JDK 8u161 or later (not the JRE as you need the javac compiler) is installed and the JAVA_HOME environment variable set.
Clone this repository recursively, i.e. with submodules (git clone --recurse-submodules).
Change into the repo directory on your machine and run ./gradlew installDist to setup the build environment (e.g. get Gradle etc.) and build / install the start scripts for ORT. The individual start scripts can then be run directly from their respective locations as follows:
- ./analyzer/build/install/analyzer/bin/analyzer --help
- ./downloader/build/install/downloader/bin/downloader --help
- ./scanner/build/install/scanner/bin/scanner --help
- ./reporter/build/install/scanner/bin/reporter --help
Alternatively, as of Gradle 4.9 the different tools can be run directly:
- ./gradlew analyzer:run --args="--help"
- ./gradlew downloader:run --args="--help"
- ./gradlew scanner:run --args="--help"
- ./gradlew reporter:run --args="--help"
Make sure that the locale of your system is set to en_US.UTF-8, using other locales might lead to issues with parsing the output of external tools.

Supported package managers

Currently, the following package managers / build systems can be detected and queried for their managed dependencies:

Bundler (Ruby)
dep (Go)
Glide (Go)
Godep (Go)
Gradle (Java)
Maven (Java)
NPM (Node.js)
Composer (PHP)
PIP (Python)
SBT (Scala)
Stack (Haskell)
Yarn (Node.js)

Supported license scanners

ORT comes with some example implementations for wrappers around license / copyright scanners:

Supported remote caches

For reusing already known scan results, ORT can currently use one of the following backends as a remote cache:

Artifactory

Usage

analyzer

The Analyzer determines the dependencies of software projects inside the specified input directory (-i). It does so by querying whatever supported package manager is found. No modifications to your existing project source code, or especially to the build system, are necessary for that to work. The tree of transitive dependencies per project is written out as ABCD-style YAML (or JSON, see -f) files to the specified output directory (-o) whose inner structure mirrors the one from the input directory. The output files exactly document the status quo of all package-related meta-data. They can and probably need to be further processed or manually edited before passing them to one of the other tools.

downloader

Taking a single ABCD-syle dependencies file as the input (-d), the Downloader retrieves the source code of all contained packages to the specified output directory (-o). The Downloader takes care of things like normalizing URLs and using the appropriate VCS tool to checkout source code from version control.

scanner

This tool wraps underlying license / copyright scanners with a common API. This way all supported scanners can be used in the same way to easily run them and compare their results. If passed a dependencies analysis file (-d), the Scanner will automatically download the sources of the dependencies via the Downloader and scan them afterwards. In order to not download or scan any previously scanned sources, the Scanner can be configured (-c) to use a remote cache, hosted e.g. on Artifactory or S3 (not yet implemented). Using the example of configuring an Artifactory cache, the YAML-based configuration file would look like:

scanner:
  cache:
    type: Artifactory
    url: "https://artifactory.domain.com/artifactory/generic-repository-name"
    apiToken: $ARTIFACTORY_API_KEY

reporter

The reporter generates human-readable reports from the scan-record file generated by the scanner (-s). It is designed to support multiple output formats. Currently a static HTML report (-f STATIC_HTML) and an Excel report (-f EXCEL) are supported.

Getting Started

Please see GettingStarted.md for an introduction to the tools.

Development

The toolkit is written in Kotlin and uses Gradle as the build system. We recommend the IntelliJ IDEA Community Edition as the IDE which can directly import the Gradle build files.

The most important root project Gradle tasks are listed in the table below.

Task	Purpose
assemble	Build the JAR artifacts for all projects
detektCheck	Run static code analysis on all projects
test	Run unit tests for all projects
funTest	Run functional tests for all projects
installDist	Build all projects and install the start scripts for distribution

License

See the LICENSE file in the root of this project for license details.

romaniakovlev / oss-review-toolkit Goto Github PK