trustification/scale-testing

Utilities for testing trustification at scale

scale-testing

Utility for testing trustification at scale.

This tool is to help replicating existing SBOMs (SPDX or CycloneDX) file in order to augment an existing data set by multiplying the number SBOMs files.

For instance let's say we have a total of 1000 SBMS (500 SPDX and 500) and we'd like to obtain a total of 10K SBOMs files for our scale test, so we can run the tool using a replication size of 10.

The tool replicates existing SBOMs, by copying each file content and change its file name and its key records.

Usage

After installing trustification/scale-testing repo,

We can run the tool, by providing the size of the replication, the source directory and the destination directory :

We need to provide a source of SBOMs, here located in /SBOMs which contains SBOM files in json format.

Also the target directory must not exists, this is to ensure we're not erasing an existing test set.

$ rm -rf /data-set
$ cargo run -- 10 /SBOMs/ /data-set/

The latter will replicate 10 times each SBOM file available in /SBOMs/.

Each replicated SBOM file will be created under its corresponding batch directory under `/data-set`.

> Example

```sh
$ cargo run -- 2 ./SBOMs ./data-set
   Compiling scale-testing v0.1.0 (/home/gildub/github.com/gildub/scale-testing)
    Finished dev [unoptimized + debuginfo] target(s) in 0.24s
     Running `target/debug/scale-testing 2 ./SBOMs ./data-set`
Replication multiplier 2
Source directory ./SBOMs
Destination directory ./data-set
successfully wrote to metadata file
successfully wrote to metadata file
Amending version:     "version": 1,
successfully wrote to ./data-set/batch1/A7ED160707AB4BC.replicate1.cdx.json
Amending version:     "version": 1,
successfully wrote to ./data-set/batch2/A7ED160707AB4BC.replicate2.cdx.json
Amending name:   "name": "quarkus-2.13",
Amending documentNameSpacekey:   "documentNamespace": "https://access.redhat.com/security/data/sbom/beta/spdx/quarkus-2.13-1a6ac4c55918a44fb3bada1b7e7d12f887d67be4",
successfully wrote to ./data-set/batch1/quarkus-2.replicate1.13.json
Amending name:   "name": "quarkus-2.13",
Amending documentNameSpacekey:   "documentNamespace": "https://access.redhat.com/security/data/sbom/beta/spdx/quarkus-2.13-1a6ac4c55918a44fb3bada1b7e7d12f887d67be4",
successfully wrote to ./data-set/batch2/quarkus-2.replicate2.13.json

$ tree data-set/
data-set/
├── batch1
│  ├── metadata
│  │  └── metadata.json
│  ├── A7ED160707AB4BC.replicate1.cdx.json
│  └── quarkus-2.replicate1.13.json
└── batch2
    ├── metadata
    │  └── metadata.json
    ├── A7ED160707AB4BC.replicate2.cdx.json
    └── quarkus-2.replicate2.13.json

5 directories, 6 files

Using bombastic_walker

Prepare initial SBOMs data set

Provide initial set of SBOMs files to be replicated, i.e /SBOMs/. Use only SPDX files, for now, because the CycloneDX files tested were rejected with following error: JSON: Unsupported CycloneDX version: 1.4

Replicate SBOMs

Run the replication tool to multiply your existing SBOM files set

cargo run -- 10 /SBOMs /data-set

Use the replicated SBOMs

The bombastic_walker could exploit each replicated SBOMs batch, for example in devmode :

RUST_LOG=info cargo run -p trust bombastic walker --sink http://localhost:8082 --source /data-set/batch1/ --devmode -3

scale-testing's Issues

hyperfoil change the test to hit the same paths as goose loadtest

I'm still working on this
(temporary file...) https://github.com/helio-frota/dotfiles/commits/main/hyperfoil.yml

Test pm-mode again after embedded OIDC server merge

trustification/trustify#332

How to use/create the files and directories `/SBOMs/ /data-set/` ?

rust-toolchain, clippy same as trustify

for us to avoid creating multiple vscode profiles, also to help us understanding trustify clippy issues mainly

Use the benchmark with the current trustification - stable branch

It is doable using plain username and password and running locally.
After merging this #1

We can go with trustification repository at least to have something.

Would be good to find a good way to join the work

The hulk-mode [lots of data] from Gilles + Hyperfoil hitting the search and some other ui features
( pm-mode first ofc )

Request for basic/normal permissions

Hi folks I can't select a reviewer:

thanks

Find a way to make it meaningful per release

Similar to cargo llvm-cov test that shows coverage, it would be good to have a result for each release.

not exactly this but something like:

v0.1.0-alpha.11 - slower
v0.1.0-alpha.10 - faster
v0.1.0-alpha.9   - same as previous 

or

v0.1.0-alpha.11 - 9k requests
v0.1.0-alpha.10 - 12k requests
v0.1.0-alpha.9   - 10k requests