Giter Site home page Giter Site logo

cukinia's Introduction

cukinia logo License

Cukinia - a Linux firmware validation framework

Cukinia is designed to help Linux-based embedded firmware developers run simple system-level validation tests on their firmware.

Cukinia integrates well with embedded firmware generation frameworks such as Buildroot and Yocto, and can be run manually or by your favourite continuous integration framework.

Project objectives

Cukinia works if it offers the following value:

  • It is very simple to use
  • It requires no dependencies other than busybox
  • It integrates easily with CI/CD pipelines
  • It helps developers creating better software

Usage

cukinia [options] [config file]

Useful options:

  • -f junitxml: format results as JUnit XML (useful for Jenkins & others)
  • -f csv: format results as CSV text
    • --no-header: omit CSV header line
  • -o file: output results to file instead of stdout

Screenshot

Screenshot

Basic config

To run Cukinia, create a configuration describing your tests, and invoke it. By default, cukinia reads /etc/cukinia/cukinia.conf. Alternatively, a config file can be passed to cukinia as its argument.

A cukinia config file supports the following statements:

Test statements

  • cukinia_user <username>: Validates that user exists
  • cukinia_group <groupname>: Validates that group exists
  • cukinia_user_memberof <username> <group...>: Validate that user is member of groups
  • cukinia_kmod <kernel module>: Validates that kernel module is loaded
  • cukinia_kconf <kernel config symbol> <y|m|n>: Validates that kernel config symbol is set to given tristate value
  • cukinia_kversion <version>: Validate kernel version (only check maj.min, e.g 5.14)
  • cukinia_process <pname> [user]: Validates that process runs (optional user)
  • cukinia_kthread <pname>: Validates that kernel thread runs
  • cukinia_python_pkg <pkg>: Validates that Python package is installed
  • cukinia_test <expr>: Validates that test(1) expression is true
  • cukinia_http_request <url>: Validates that url returns a 200 code
  • cukinia_cmd <command>: Validates that arbitrary command returns true
  • cukinia_cmdline <param[=val_regex]>: Validates kernel cmdline contains param (optional value)
  • cukinia_listen4 <proto> <port>: Validates that tcp/udp port is open locally
  • cukinia_mount <source> <mount point> [fstype] [options]: Validate the presence of a mount on the system
  • cukinia_symlink <link> <target>: Validate the target of a symlink
  • cukinia_systemd_failed: Raise a failure if a systemd unit is in failed state
  • cukinia_systemd_unit <unit>: Validate systemd unit is active
  • cukinia_i2c <bus_number> [device_address] [driver_name]: This checks i2c bus or (optional) device, and (optionally) verifies it uses the indicated driver
  • cukinia_gpio_libgpiod -i [input_pins] -l [output_low_pins] -h [output_high_pins] -g [gpiochip](default:gpiochip0): Validate the gpio configuration via libgpiod (ex: cukinia_gpio_libgpiod -i "0 3 4" -l "10" -h "2 50" -g gpiochip1)
  • cukinia_gpio_sysfs -i [input_pins] -l [output_low_pins] -h [output_high_pins] -g [gpiochip](default:gpiochip0): Validate the gpio configuration via sysfs (ex: cukinia_gpio_sysfs -i "20 34" -h "3 99 55")
  • cukinia_knoerror <priority>: Validate kernel has booted without important errors (the priority argument is the log level number to check)
  • cukinia_sysctl <parameter> <value>: Validate kernel sysctl parameter is set to value
  • cukinia_netif_has_ip <interface> [-4|-6] [flags]: Validate that interface has ip config parameters
    • example: cukinia_netif_has_ip eth0 -4 dynamic
    • example: cukinia_netif_has_ip eth0 -6 "scope global"
  • cukinia_netif_is_up <interface>: Validate network interface state is up
  • cukinia_dns_resolve <hostname>: Validate that hostname can be resolved
  • not: Can prefix any test to invert the issue it will produce (a [!] is appended to the default test description)
  • verbose: Can prefix any test to preserve stdout/stderr
  • as <string>: Can prefix any test to change its textual description
  • id <string>: Can prefix any test to add a test id in the different outputs

Condition statements

  • when <condition>: Can prefix any test to <condition> it
  • unless <condition>: Just like when, but the opposite

If the condition is not met, the test status will be reported as SKIP.

A few examples using when and unless:

on_eval_board() { grep -q EVK /sys/firmware/devicetree/base/model; }
arch_is_arm64() { test "$(uname -m)" = "aarch64"; }

unless "on_eval_board" \
  as "Custom LED controller was detected" \
    cukinia_test -d /sys/class/leds/superled

when "arch_is_arm64" \
  unless "on_eval_board" \
    cukinia_kmod some_driver 

Utility statements

  • cukinia_conf_include <files>: Includes files as additional config files
  • cukinia_run_dir <directory>: Runs all executables in directory as individual tests
  • cukinia_log <message>: Logs message to stdout
  • _ver2int <version>: Convert numeric version string to int, for use with e.g. cukinia_test $(_ver2int ${kernel_version}) -gt $(_ver2int 4.19.7)

Logging customization

  • logging prefix "string": prefix logs with "string"
  • logging class "string": change the junitxml class name to "string" for the next tests
  • logging suite "string": change the junitxml test suite to "string" for the next tests

Useful variables

  • $cukinia_tests: number of tests attempted
  • $cukinia_failures: number of tests that failed

Environment variables

  • $CUKINIA_ALWAYS_PASS: if set, every test will succeed

Example cukinia.conf

# Ensure our basic users are present
cukinia_user appuser1
cukinia_user appuser2

# This should always be the case
cukinia_test -f /etc/passwd

# If this user exists, then something went wrong
not cukinia_user baduser

# Those config snippets are deployed by our packages
cukinia_conf_include /etc/cukinia/conf.d/*.conf

# Is our embedded webservice up?
as "Checking webapp" cukinia_http_request http://localhost:8080/sanitycheck

# Run executable tests for myapp1
cukinia_run_dir /etc/cukinia/myapp1.d/

# Check for misc. mount points
cukinia_mount sysfs /sys
cukinia_mount /dev/sda1 /boot ext4 rw sync

# Check for ssh and dns servers
cukinia_listen4 tcp 22
cukinia_listen4 udp 53

# Check the link interfaces point to /tmp/interfaces
cukinia_symlink /etc/network/interfaces /tmp/interfaces

# Add a id linked to the test
id "SWR_001" as "Checking systemd units" cukinia_systemd_failed

# End
cukinia_log "ran $cukinia_tests tests, $cukinia_failures failures"

More advanced config

A config file is actually a POSIX shell script that is sourced by cukinia, so any logic can be used in a test file scenario. This is useful for example to make certain groups of tests depend on preliminary checks:

if cukinia_test -x /usr/bin/myapp; then
	cukinia_user myuser
	cukinia_process myapp myuser
	cukinia_http_request http://localhost:8080/testme
else
	cukinia_log "$(_colorize red "myapp not found :(")"
fi

License

Copyright (C) 2017-2024 Savoir-faire Linux, Inc.

Cukinia is released under the Apache 2 license.

cukinia's People

Contributors

abhishekojhasfl avatar bachp avatar deribaucourt avatar ebail avatar eroussy avatar gportay avatar joufellasfl avatar kevlhop avatar lucbeaufils avatar macpijan avatar markfeathers avatar p-d-g avatar rubusch avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cukinia's Issues

Suite name get rid of spaces

When logging a new suite which name contains spaces, the spaces are removed by cukinia.

cukinia.conf

logging suite "suite with spaces"
cukinia_user www-data

When running this command :
$ cukinia -f junitxml -o res.xml cukinia.conf
The output is
res.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="suitewithspaces" package="suitewithspaces" errors="0" tests="1" failures="0" timestamp="2023-04-07T13:56:05" time="0">
    <testcase classname="cukinia" name="Checking user &quot;www-data&quot; exists" time="0">
    </testcase>
  </testsuite>
</testsuites>

And the spaces are gone ("suitewithspaces")

It should not be a problem to keep them because they are enclosed with quotes.

`cukinia_netif_is_up` fails when it shouldn't

From the cukinia code:

_cukinia_netif_is_up() {
	local interface="$1"

	_cukinia_prepare "Checking if interface \"$interface\" is ${__not:+NOT }up"
	ip -o link show "$interface" up | grep -q "state UP" 2>&1
}

On my device, there is UP but no state UP:

root@my-yocto-beagleboneblack:~# ip -o link show eth0 up
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000\    link/ether 78:a5:04:c1:16:d0 brd ff:ff:ff:ff:ff:ff

Regression on cukinia_process

Since 84061bc I do have a regression on my test based on cukinia_process:

  • I used to have python3 based application.
    For instance: python3 /usr/bin/my-app

  • I had my cukinia test working:
    cukinia_process "python3 /usr/bin/my-app"

But cukinia_process is now based on pidof which can only check that python3 is running (not python3 /usr/bin/my-app)

I also have this issue with node based application.

logs incomplete

For instance,
cukinia_test -f /etc/passwd
is printed out:
[PASS] Running "test -f

weird behaviour for piped commands

Intuitively, I wanted to check if cukinia supports piped commands.

This sequence

as grep cukinia_cmd echo toto | grep toto
as cat  cukinia_cmd echo toto | cat 
as awk  cukinia_cmd echo toto | awk '/toto/'
as wc   cukinia_cmd echo toto | wc
as sed  cukinia_cmd echo toto | sed 's/o/i/g'
cukinia_log "$cukinia_tests"

gives:

[PASS] cat
        1         2        21
[PASS] sed
0

It looks like something, probablyecho toto, is executed because of PASS but the result is ignored (cukinia_test==0). So, the output is confusing because cat and sed look fine. And it's not clear why some commands ouput something while other don't.

Quoting the expression ("cmd1|cmd2") helps cukinia_test incrementing but always returns FAIL regardless of cmd1 and cmd2.

The use of "|" in the syntax should be either forbidden by documentation either fixed somehow.

test suite assumes too much about host system

Hi,

The test suite assumes many details from the host system it runs on, which makes it fragile. For example, running it in the Guix build container, it fails like:

starting phase `check'
----> cukinia_cmd <----
[PASS] Running "true" is successful
[PASS] Running "false" is NOT successful [!]
----> cukinia_user <----
[FAIL] Checking user "root" exists
[PASS] Checking user "nonexistent" NOT exists [!]
----> cukinia_group <----
[FAIL] Checking group "games" exists
[PASS] Checking group "nonexistent" NOT exists [!]
----> cukinia_process <----
[FAIL] Checking process "systemd" running as any user
[FAIL] Checking process "systemd" running as root
[PASS] Checking process "systemd" NOT running as nonexistent [!]
[PASS] Checking process "nosuchprocess" NOT running as any user [!]
----> cukinia_http_request <----
[FAIL] HTTP 200 on localhost:631 (cups)
[PASS] Checking http url "http://localhost:633/" does NOT return 200 [!]
----> cukinia_test <----
[PASS] Checking file "/etc/passwd" exists
[PASS] Checking "/etc/passwd" is NOT a symlink [!]
----> cukinia_python_pkg <----
[FAIL] cukinia_python_pkg: interpreter not found
[PASS] cukinia_python_pkg: interpreter not found [!]
----> cukinia_mount <----
[FAIL] Checking proc on /proc type any (any)  mounted
[FAIL] Checking proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)  mounted
[PASS] Checking /dev/nonexistent on /mnt type any (any) NOT mounted [!]
----> cukinia_symlink <----
[FAIL] Checking link "/tmp/cukinia.27.link" does point to "/tmp/guix-build-cukinia-0.6.0.drv-0/tmp.3KCYj4Oohg"
[PASS] Checking link "/tmp/guix-build-cukinia-0.6.0.drv-0/tmp.3KCYj4Oohg" does NOT point to "/tmp/cukinia.27.link" [!]
[PASS] Checking link "/tmp/cukinia.27.link" does NOT point to "/dev/null" [!]
----> cukinia_kmod <----
[FAIL] Checking kernel module "vfat" is loaded
[PASS] Checking kernel module "nonexistent" is NOT loaded [!]
----> cukinia_kconf <----
[PASS] Checking kernel config "CONFIG_NONEXISTENT" NOT set to "y" [!]
----> cukinia_kversion <----
[PASS] Checking if kernel version is 6.0
----> cukinia_listen4 <----
[FAIL] Checking tcp v4 port "631" is open
[PASS] Checking tcp v4 port "632" is NOT open [!]
[FAIL] Checking udp v4 port "5353" is open
[FAIL] Checking tcp|udp v4 port "67" is open
[PASS] Checking tcp|udp v4 port "63530" is NOT open [!]
[PASS] Checking tcp v4 port "63530" is NOT open [!]
----> cukinia_systemd_* <----
[FAIL] Checking if systemd unit "atd.service" is active
[PASS] Checking for no failed systemd units
----> cukinia_i2c <----
[PASS] Checking if i2c-2-0xXX exists [!]
----> cukinia_gpio_* <----
[FAIL] Checking if gpio pins are well configured via libgpiod
[FAIL] Checking if gpio pins are well configured via sysfs
----> cukinia_knoerror <----
[PASS] Checking that kernel has booted with no error (using grep)
----> _ver2int <----
[PASS] _ver2int test 44.10.5 > 44.6.5
[PASS] _ver2int test 43.6.5 < 43.6.6
[PASS] _ver2int test 44.3.0 == 44.3
[PASS] _ver2int test 0.99.1 < 1.0.0
[PASS] _ver2int test 1.99 > 1.98.999.999
----> cukinia_conf_include <----
cukinia: warning: can't include "conf.d/*.conf"
----> cukinia_run_dir <----
----> color tests <----
* _colorize red
* _colorize green
* _colorize blue
* _colorize cyan
* _colorize yellow
* _colorize gray
* _colorize purple
----> failure detection (must all FAIL) <----
[FAIL] Running "false" is successful
[FAIL] Running "true" is NOT successful [!]
[FAIL] Running "test 0 -eq 1" returns success
[FAIL] Checking process "nosuchprocess" running as any user
[FAIL] cukinia_python_pkg: interpreter not found
[FAIL] Checking link "/dev/zero" does point to "/dev/null"
[FAIL] Checking if systemd unit "nosuchunit.service" is active
----> tests/failures counters <----
ran 50 tests, 23 failed

The host doesn't use systemd, there's no root user, etc. It'd be nicer to have a setup script that creates props to be validated, so that everything is controlled and deterministic, on any system.

Allow checking kernel processes

cukinia_process allows checking that a user process is running, however it explicitly doesn't allow testing kernel processes:

		# skip kernel threads
		[ -x $procdir/exe ] || continue

It can be legitimate to verify that some key kernel threads are running, for instance [watchdogd].

I suggest adding a cukinia_kprocess function dedicated to kernel processes.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.