iris-gmbh/meta-osselot

bitbake layer repository for intergrating osselot into the build process

meta-osselot's Issues

Add option to override used package name and version

Add support for OSSELOT_NAME and OSSELOT_VERSION variables, for overriding package name and version for osselot identification within recipe.

Optimize IO heavy checksum tasks

Checksum tasks currently do a lot of read and write operations on disk, due to the way checksum information is stored (each file checksum is individually written and read from disk. Use a per-package JSON files for data exchange instead.

Evaluate options in regards to dealing with bitbake patch files

Currently, meta-osselot does not have a mechanism in place for dealing with patch files in bitbake layers

Fix bitbake QA issue

Investigate and fix:

WARNING: /home/jasper/playground/test-osselot/demo/kas/build/../poky/meta/recipes-core/meta/cve-update-db-native.bb: QA Issue: cve-update-db-native: native/nativesdk class is not inherited last, this can result in unexpected behaviour. Cl
asses inherited after native/nativesdk: cve-check.bbclass [native-last]

Make OSSELOT_HASH_EQUIVALENCE work with all supported hash algorithms, regardless of OSSELOT_HASH_ALGORITHM setting

Currently, OSSELOT_HASH_EQUIVALENCE will only work if the configured equivalences are defined in the hash algorithm defined in OSSELOT_HASH_ALGORITHM. This especially posses an issue for predefined hash equivalences, when a user decides to change the OSSELOT_HASH_ALGORITHM to anything other than md5.

Exception: AttributeError: module 'hashlib' has no attribute 'file_digest'

BB_VERSION = "2.7.2"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "universal"
TARGET_SYS = "arm-resy-linux-gnueabi"
MACHINE = "multi-v7-ml"
DISTRO = "resy"
DISTRO_VERSION = "4.3.66"
TUNE_FEATURES = "arm armv7a vfp thumb neon callconvention-hard"
TARGET_FPU = "hard"
meta
meta-poky
meta-yocto-bsp = "master:3aff9b01e559529bf1ce2aa9b318a0157092add9"
meta-multi-v7-ml-bsp-master = "master:e1831dcf9a0d9ce998e3a7f0d2f42e3caa3a26a4"
meta-u-boot-wic-bsp-master = "master:850440d2a48150520d7d9b989f93e11d8d4fc562"
meta-resy-master = "master:ceacfc5dd8cd85679def4d942f40e370cb532a02"
meta-oe
meta-networking
meta-filesystems
meta-python = "master:dea8afa45ef5a3226c88b062c13284d68f380e18"
meta-my-yocto-layer
meta-my-yocto-images-layer
workspace = ":"
meta-osselot = "master:8608fffeeb1a2a41dc5283eea0601098a76e00a4"

Checking sstate mirror object availability: 100% |#############################################################################################################################| Time: 0:00:02
Sstate summary: Wanted 222 Local 0 Mirrors 0 Missed 222 Current 176 (0% match, 44% complete)
Removing 3 stale sstate objects for arch x86_64: 100% |########################################################################################################################| Time: 0:00:00
NOTE: Executing Tasks
ERROR: binutils-cross-arm-2.41-r0 do_osselot_create_s_checksums: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function:
0001:
*** 0002:do_osselot_create_s_checksums(d)
0003:
File: '/workdir/sources/meta-osselot/classes/osselot.bbclass', lineno: 152, function: do_osselot_create_s_checksums
0148: except:
0149: bb.warn(f"Could not open file {item.path}")
0150: else:
0151: with fb:
*** 0152: digest = hashlib.file_digest(fb, osselot_hash_algorithm).hexdigest()
0153: checksums[os.path.relpath(item.path, s)] = {}
0154: checksums[os.path.relpath(item.path, s)][osselot_hash_algorithm] = digest
0155: write_json(osselot_s_checksums_file, checksums)
0156:}
Exception: AttributeError: module 'hashlib' has no attribute 'file_digest'

ERROR: Logfile of failure stored in: /workdir/build/multi-v7-ml-debug-training-master/tmp/work/x86_64-linux/binutils-cross-arm/2.41/temp/log.do_osselot_create_s_checksums.3833
ERROR: Task (/workdir/sources/poky-training-master/meta/recipes-devtools/binutils/binutils-cross_2.41.bb:do_osselot_create_s_checksums) failed with exit code '1'
NOTE: Tasks Summary: Attempted 237 tasks of which 229 didn't need to be rerun and 1 failed.
NOTE: The errors for this build are stored in /workdir/build/multi-v7-ml-debug-training-master/tmp/log/error-report/error_report_20240208151920.txt
You can send the errors to a reports server by running:
send-error-report /workdir/build/multi-v7-ml-debug-training-master/tmp/log/error-report/error_report_20240208151920.txt [-s server]
NOTE: The contents of these logs will be posted in public if you use the above command with the default server. Please ensure you remove any identifying or proprietary information when prompted before sending.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 2 seconds

Summary: 1 task failed:
/workdir/sources/poky-training-master/meta/recipes-devtools/binutils/binutils-cross_2.41.bb:do_osselot_create_s_checksums
Summary: There was 1 ERROR message, returning a non-zero exit code.

Exclude build-time packages (as far as possible)

Exclude build-time packages (e.g. -native), as these are not relevant for license compliance.

Add workflow for package version mismatches

Currently, meta-osselot will only produce output for exact package name and version matches. However, it would make sense to also provide data on version mismatch for the Fossology reuse feature, as described in the Osselot wiki

Wrong source list & race condition in do_osselot_create_s_checksums when S == WORKDIR

Some layer "internal" recipes (e.g. https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/initscripts/initscripts_1.0.bb?h=kirkstone) only add source code from within the layer repository. In these cases they set S = ${WORKDIR}. This causes do_osselot_create_s_checksums to create checksums for all of the WORKDIR, including the checksums output itself. This can also cause race conditions, when folders are deleted during the do_osselot_create_s_checksums process.

To fix this, do_osselot_create_s_checksums should only consider top level files as source files if S is set to ${WORKDIR}

Add support for OSSELOT_IGNORE within recipes

Add support for recipe variable OSSELOT_IGNORE for skipping querying internal or image recipes.

Allow defining multiple output formats

Currently, it is only possible to define a single output format. Add support for defining multiple formats using a space separated value.

Add checksum equivalence functionality

Add functionality to configure source file checksum equivalence. Use-case: file content changed but without changes to licenses or copyright

meta-osselot should work with Python 3.8

Currently we use functions which are only available in Python versions >= 3.8.

Since bitbake python tasks use the python binary provided by the operating system rather than python-native built by bitbake, we should make meta-osselot work with whatever the Yocto project considers minimum python requirements. See: #33 (comment)

meta-osselot only works with single-project recipes

Currently, meta-osselot tries to map a recipe to a single curated project. This means, that recipes that rely on more than one project will inevitably end up with un-curated data, even if both projects by themselves are available within the Osselot database.

I am not sure how much of a need there is for this use-case.

If this is something we want to support, we would probably have to add the ability to specify more than one OSSELOT_NAME and OSSELOT_VERSION for each recipe.

make skipped package suffixes configurable

Currently, skipping non-target packages is hardcoded in source. Make it configurable.

fix Python bbappends

Currently, Python bbappends will falsely apply to any Python module, not just the python main package

Evaluate options to identify build-relevant files

Currently, meta-osselot treats all source files as license compliance relevant by default. However, in reality only a subset of files will end up in the target binary. Automatically identifying these would drastically reduce the amount of work when curating a version diff.