nordix/bomres

Software Bill-of-Material Resolver

bomres's Issues

Two types of resolved SBOM

There is two kinds of package dependencies.

• Direct ( what you specify in the product configuration file )
• Transitive ( packages being added by dependencies of the direct )

Transitive could also introduce new packages.

The current implementation of the resolved.json indicates with a key if the all dependencies being included. The download command should be blocked if transitive packages are missing.

Support for podman

Ubuntu supports a CLI emulation of docker
apt install podman-docker

There is some minor issues with missing directories that must be fixed

Build-directory
product/build

The alpine repository is cloned, and cached. The default dir is /tmp/alpine. The following dirs must be created.

mkdir -p /tmp/alpine/src
mkdir -p /tmp/alpine/checkout
mkdir -p /tmp/alpine/cache

Parser for APKBUILD in separate module

The file parse_apkbuild.py is the most complex module of the resolver, since it match metadata from Alpines package manager with parsed data from aports APKBUILD manifest.

The link between the package manager and the package could be one of three cases.

• The commit hash for a specific packages is used to match the source code
• Commit hash for a repository combined with tag is used to match
• Miss, no resolve is possible.
  The output of this module is the cache file, used in the final resolve phase.

cache_index_file = "%s/APKINDEX-%s.json" % (args.cache, apkindex['hash'])

The parse_apkbuild_manifest function should be migrated to separate file, enabling unit testing.

def scan_aports(checkout_dir, apkindex):

parse_apkbuild_manifest(name, repository, filename, repo_hash_dict, apkindex, "package")
parse_apkbuild_manifest(name, repository, filename_commit, repo_hash_dict, apkindex, "repo")
parse_apkbuild_manifest(name, repository, filename_commit, repo_hash_dict, apkindex, "broken-link-between-aports-and-apkbuild")

APKBUILD parser

The preflight check reports two error, it is probably related to the parsing of APKBUILD files.

$ make check

verify.log
failed main ncurses
failed main busybox

Alpine 3.17 fails

When building the first payload image from Internet, the mkimage-alpine.bash complain about missing etc directory
tar: etc: not found in archive

gunicorn

I noticed that Nordix have gunicorn , therefore I did not integrate uwsgi.
Lets discuss which server to use , until then add both servers.

Top makefile missing

To better understand the structure of the project a top Makefile should be added.

$ make build
( Builds the tools , and then the three layers for the resolver )
$ make deploy
( tag and push two containers )
$ make clean
( Remove temporary files )

depends_dev

Some aports packages divides build dependencies into
depends_dev
makedepends

depends_dev="
cyrus-sasl-dev
libevent-dev
libsodium-dev
openssl1.1-compat-dev
util-linux-dev
"
makedepends="
$depends_dev

openldap: Building main/openldap 2.6.0-r0 (using abuild 3.9.0-r0) started Mon, 04 Apr 2022 16:48:49 +0000
openldap: Checking sanity of /var/local/aports/main/openldap/APKBUILD...
openldap: Analyzing dependencies...
ERROR: openldap: Missing dependencies (use -r to autoinstall them): cyrus-sasl-dev libevent-dev libsodium-dev
ERROR: openldap: builddeps failed

Restructure repo structure to comply with python best practices

This is a proposal to restructure the repository such that the Python code resides in a more common place for Python applications (and make it pypi compatible).

Parsing problem of APKBUILD when empty line in list of sources

ans@hans-VirtualBox:/tmp/apa$ make abuild
...

musl: Checking sha512sums...
musl-v1.2.3.tar.gz: OK
handle-aux-at_base.patch: OK
relr-1.patch: OK
relr-2.patch: OK
relr-3.patch: OK
relr-4.patch: FAILED
sha512sum: can't open 'relr-4.patch': No such file or directory

source="musl-$_commit.tar.gz::https://git.musl-libc.org/cgit/musl/snapshot/$_commit.tar.gz
handle-aux-at_base.patch

    relr-1.patch
    relr-2.patch
    relr-3.patch
    relr-4.patch

    ldconfig
    __stack_chk_fail_local.c
    getconf.c
    getent.c
    iconv.c
    "

relr-4.patch is missing in parsed data

            {
                "remote": "relr-3.patch",
                "local": "relr-3.patch"
            },
            {
                "remote": "ldconfig",
                "local": "ldconfig"
            },

There is some unit tests in the test directory

bomres/services/sbom_resolver/service/test
python3 ../build/bomres/bomres/lib/create_apkcache.py

Reduce footprint of Alpine build container

The base_os_alpine is currently 723 Mb since it includes packages required for service mode.

Error when creating aggregate.json

Two images are required to maintain a product

• The product itself
• The tools container required to build the product from source

Exception generated during tool creation

File "/opt/base_os/scripts/aggregate_bom.py", line 493, in format_dep
if 'apkindex' in metadata and s_patch[0] == 'r':
IndexError: string index out of range

Integration test

We need integration tests, Lets start with the token endpoint

Download options

The goal of this project is to rebuild in a isolated environment. To prove true isolation the rebuild should start in a empty directory. The content is then populated with source code being analyzed.

To better understand the functionality different options should be added in the template Makefile

• Internal patches ( provided by Alpine )
• Additional files ( provided by Alpine )
• Build control file ( APKBUILD , provided by Alpine )
• External files ( typically compressed tarballs download from github, curl.se etc )
• External patches ( There is some rare cases where non Alpine patches being downloaded and applied )

$ make download: internal_patch internal_files internal_build external_patch external_files
( Separate rules for granular control below )

Product and tool

For maintenance of a product, tools are required. Many components have common dependencies such as compiler, but there is also requirement of addition components. Openssl for instance, requires perl during build. The purpose of this issue is to autogenerate a desired Bill-of-material based on the the complete SBOM from the product.

• Build a small product container capable of serving static content ( apk packages )
• Generate the complete SBOM for the product above
  * Download ALL source code and APKBUILD required for rebuild
  * Generate a tailored build tool for the above product
• Create ( or use existing PEM formatted ) keys for signing of packages and APKINDEX.
• Preflight check prior to rebuild
• Rebuild all packages in product, based on downloaded source code using the newly created tool
• Sign all packages
• Rebuild the original product from packages created by the rebuild process.

Convert TODO to issues

Distfiles

The build process in Alpine consists of several small tasks. The fetch task retrieve the external tarball and save it in /var/cache/distfile, all packages share the same directory. The complete SBOM ( resolved.json ) is organized around packages.
In order for the complete SBOM to be used for analyze of open source as well as support the Alpine build, postprocessing is required.

Investigate mapping of current bomres data format to standard SBoM formats

bomres currently uses a custom json-based data format for its output. Going forward, we should align with a standard format, such as SPDX or CycloneDX. To this end, we need to investigate the mapping of the current format and it's specifics (e.g., patches) to SPDX (or CycloneDX).

Truncated versions in external paths

Some external sites provides tarballs with full semantic versioning ( 2.37.4 ) as part of the filename.
The parent directory does not include the minor version ( 2.37 ).

APKBUILD generates truncated versioning by the _v variable. This must be added to the APKBUILD parser.

------------------ APKBUILD ---------------------------------
pkgname=util-linux
pkgver=2.37.4

case $pkgver in
..) _v=${pkgver%.};;
.) _v=$pkgver;;
esac

source="https://www.kernel.org/pub/linux/utils/util-linux/v$_v/util-linux-$pkgver.tar.xz

-------------------- External source --------------------------

https://mirrors.edge.kernel.org/pub/linux/utils/util-linux/v2.37/util-linux-2.37.4.tar.xz

-------------------- Complete SBOM -------------------------------
"code": [
{
"remote": {
"type": "generic",
"url": "https://www.kernel.org/pub/linux/utils/util-linux/v$_v/util-linux-2.37.4.tar.xz"
},
"local": {
"type": "file",
"path": "main/util-linux/util-linux-2.37.4.tar.xz"
}
}

---------------------- abuild ---------------------------------
abuild verify fails since no file exists.

sha512sum: WARNING: 1 of 1 computed checksums did NOT match

Multiple files on the same line for the source key in APKBUILD.

This Bug is listed in the TODO file

The package busybox-initscripts source key spans many lines, where each line may contain many fields.
The parse_apkbuild.py module must handle the case below.

source="acpid.initd crond.initd dnsd.initd httpd.initd inetd.initd klogd.initd
mdev.initd ntpd.initd rdate.initd syslog.initd udhcpd.initd loadkmap.initd
watchdog.initd crond.confd klogd.confd ntpd.confd rdate.confd syslog.confd
watchdog.confd loadkmap.confd mdev.conf dvbdev usbdev persistent-storage

Prevent download during rebuild

All external source code is current written to the aports directory in the same directory as APKBUILD resides, however the cache must also be populated.

tool/builder/build.sh

Generated, do not edit

mkdir -p /var/local/aports/$3/$1
cp -r /aports/$3/$1/* /var/local/aports/$3/$1

This file must be added

cp /aports/$3/$1/* /var/cache/distfiles/

The build file is generated by Makefile.bootstrap
BUILDER_SCRIPT = tool/builder/build.sh
.PHONY: $(BUILDER_SCRIPT)
$(BUILDER_SCRIPT) :

While fixing the caching, also adjusted the indexing and handling of DESCRIPTION

rm -f /var/local/packages/$3/$2/APKINDEX.tar.gz
cd /var/local/aports/$3/$1 && abuild -F -P /var/local/packages

rm -f /var/local/packages/$3/$2/APKINDEX.tar.gz
cd /var/local/aports/$3/$1 && abuild -F -P /var/local/packages -D 3.18.2-575-g02de16b1332 index

Sign packages

In the current implementation there is some code that allows import of keys. It is not completed.

Command that generates and install keys

abuild-keygen -a -n -q -i

Shell script that wraps openssl

/usr/bin/abuild-keygen

Rule that generates keys externally, and then integrates with Alpine.

keygen:
mkdir -p $(CWD)/tool/pki
openssl genrsa -out $(CWD)/tool/pki/iafw.rsa 1024
openssl rsa -in $(CWD)/tool/pki/iafw.rsa -pubout > $(CWD)/tool/pki/iafw.rsa.pub
chmod 755 -R $(CWD)/tool/pki

Add endpoint(s) for rediness/liveness probes

This would improve the reliability of the service when running in a Kubernetes or Openshift environment.
Ideally the endpoints should not use authentication. A simple endpoint returning status 200 is fine to begin with I think.

install and trigger files defined in APKBUILD

In order to rebuild packages in a isolated environment, all source code and APKBUILD could be copied to a new directory outside the aports repository.

Some simple packages such as grep have been verified, but some packages specify local additions outside the source= key.

busybox lists several files with the keys below
install=
triggers=

The values is a list of local files that spans several lines.

Fix OpenAPI linting errors

There are errors related to reply codes reported by our linter service

If _pkgname exists in APKBUILD, add alias name to complete SBOM

Some external software used in other distributions get different names assign by the aggregator.

pkgname=Name of the Alpine package
_pkgname=External name of the project / package

Rationale: Different aggregators find the same external component by the alias.

Add APKBUILD to complete SBOM

If the content of a package directory in aports could be populated by processing the complete SBOM the isolation will be improved.
Instead of building in the cloned aports directory, the build could start from a empty directory.

Create SBOM for build container

Currently tools are not included, but the resolver is inspired from the "Canadian Cross" model.

In APKBUILD for curl, the following packages are listed as dependencies.

depends="ca-certificates"
depends_dev="openssl-dev nghttp2-dev zlib-dev brotli-dev"
checkdepends="nghttp2 python3"
makedepends_host="$depends_dev"
makedepends_build="autoconf automake groff libtool perl"

The tools/base_os_alpine/test/Containerfile file defines the current builder.

FROM alpine:3.14
RUN apk --update add alpine-sdk build-base pcre-dev autoconf automake

The abuild utility allows installation of packages required for building a specific package ( abuild deps ).

Extract additional metadata from APKINDEX

The reference suite now included lighttpd for serving APK-packages. This package depends on libldap which is a submodule of openldap.

The same information could be extracted from aports, but the syntax is more strict in APKINDEX.

Missing dependency of subpackage

Problem: discover during signing of APKINDEX.tar.gz during hardening of lighttpd

APKBUILD
build() { abuild-meson \ -Db_lto=true \ -Dwith_pcre2=true \ -Dwith_webdav_locks=enabled \ -Dwith_webdav_props=enabled \ . output meson compile -C output

WARNING: No provider for the dependencies:
so:libxml2.so.2
WARNING: Total of 1 unsatisfiable package names. Your repository may be broken.

P:lighttpd-mod_webdav o:lighttpd D:so:libc.musl-x86_64.so.1 so:libsqlite3.so.0 so:libxml2.so.2

P:lighttpd o:lighttpd D:/bin/sh so:libc.musl-x86_64.so.1 so:libpcre2-8.so.0 so:libz.so.1

Workaround

By removing webdav support, the subpackage is no longer included in APKINDEX
build() { abuild-meson \ -Db_lto=true \ -Dwith_pcre2=true \ . output meson compile -C output }

make and checkdepends in APKBUILD spans many lines

Some components ( nginx ) requires many packages during build. The current parses does not support multiple lines

makedepends="
brotli-dev
gd-dev
geoip-dev
libmaxminddb-dev
libxml2-dev
libxslt-dev
linux-headers
luajit-dev
openssl-dev
pcre-dev
perl-dev
pkgconf
zeromq-dev
zlib-dev
"

Multithreaded backend

The current backend is single threaded and some requests such as /index may prevent /readyness to response causing kubelet to restart pod.