theslowgrowth/daisyhat

Automated hardware testing for Electro-Smith Daisy and other embedded platforms

daisyHat

A framework for automated testing on Electro Smith Daisy (or similar) hardware
Explore the docs »

View Example Code · Report Bugs · Request Features

About The Project

daisyHat aims to provide a framework for writing and executing automated tests for embedded hardware, with premium support for the Electro Smith Daisy platform. Tests can be run locally or in github actions and cover a wide range of test scenarios from simple one-device tests to complex hardware setups including custom measurement equipment and test fixtures.

daisyHat provides ...

• a device-side C++ library that comes with assertion macros, flow control and other tooling to write tests on the hardware,
• a host-side python library to upload device firmware images, collect test results and orchestrate tests that involve other hardware,
• CMake functions and utilities to create a CMake project that contains multiple daisyHat tests which can be built and run with one command respectively,
• a Docker image that can be used to deploy ephemeral github actions runners that are safe to use on public repositories,
• setup scripts to prepare and install this Docker image on a Raspberry Pi 4B and turn it into the heart of a automated hardware testbed for github repositories.

WOKR IN PROGRESS This project is in very early stages and not production ready. Don't expect everything to be plug-and-play yet. Please help out where you can!

Example test

The simplest test consists of a single firmware image that performs the entire test on a Daisy Seed. View more complex example code here.

firmware.cpp

#include <daisy_seed.h>
#include <daisyHat.h>

daisy::DaisySeed seed;

int main()
{
    seed.Configure();
    seed.Init();

    daisyhat::Init(seed, "test1");
    int a = 1;
    int b = 1;
    EXPECT_EQ(a, b);
    daisyhat::FinishTest();
}

CMakeLists.txt

# register the test with CMake.
# This will upload the test to a Daisy Seed with name "Alice" and collect the results over USB-Serial
daisyhat_add_test(
    NAME test1
    SEED Alice
    SOURCES 
        firmware.cpp
)

Getting started

Core concepts:

• Tests live in a git repository where daisyHat and libDaisy are available (e.g. as submodules)
• A project-level daisyHat.config.json file describes the hardware setup
• Depending on the test scenario, each test consists of one or multiple firmware images and additional host-side test execution scripts, if needed
• All tests are accumulated into a CMake project so that building and running is entirely handled by CMake
• GitHub actions integration is realised with an ephemeral test runner based on a docker image that can easily be deployed to a Raspberry Pi and is safe to use for public repositories

Setting up a test project

1. Create a new repository for your tests
2. Add libDaisy as a submodule in the lib/libDaisy folder
3. Add the daisyHat repo as a submodule in the lib/daisyHat folder
4. Add a CMakeLists.txt in the repository root that includes lib/libDaisy and lib/daisyHat as sub directories (with add_subdirectory()) (you can copy and edit this file)
5. For each test, create a new directory tests/<testName> and add to it
   1. The C++ source code for your test firmware (take a look here)
   2. A CMakeLists.txt file that registers a test firmware with the daisyhat_add_test() function (you can copy this file)
6. Include each of the test directories in the root CMakeLists.txt
7. Future addition: Describe the hardware configuration of your test setup in the daisyHat.config.json at the project root

Running the tests locally

1. Setup your toolchain with CMake, make, gcc-arm-none-eabi, openocd and Python3.9
2. Connect to each Daisy Seed board via USB
3. Connect to each Daisy Seed board via an STLinkv3 JTAG programmer
4. Configure the project: cmake -D TOOLCHAIN_PREFIX=<path-to-gcc-arm-none-eabi> -D CMAKE_TOOLCHAIN_FILE="lib/libDaisy/cmake/toolchains/stm32h750xx.cmake" -S . -B build -G "Unix Makefiles"
5. Build libDaisy, daisyHat and the tests: cmake --build build
6. Run the tests: ctest --output-on-failure

Running tests automatically via github ations

1. Prepare the target github repository by generating a personal access token to be able to register new github actions runners (see here)
2. Install Ubuntu Server 20.04 on a Raspberry Pi 4B
3. Download and execute the setup script (see here)
4. Connect the Pi to each Daisy Seed board via USB
5. Connect the Pi to each Daisy Seed board via an STLinkv3 JTAG programmer

Detailed instructions can be found here. Please note that the setup is currently not fully automated and still work in progress.

Project Structure

• cmake/ contains cmake functions
• docs/ contains documentation and guides
• docker/ contains files to build the github action runner docker image
• examples/ contains a usage example with two tests (one succeeds, one fails)
• python/ contains the python library for the host computer / test runner
• scripts/ contains scripts to setup and run a daisyHat test runner
• src/ contains C++ library for the device firmware

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

daisyhat's Issues

Add CI tests

• Build against the most recent version of libDaisy, maybe older versions as well?
• How to test? daisyHat_Examples?

Add automatic container startup on boot

Auto login the ubuntu user after booting the pi and execute ~/daisyHat/startDockerContainer.sh

Tooling: Add code for device-host synchronisation

During a test, it will often be necessary to synchronize the host computer and firmware(s) on the seed(s) - making them wait for each other to ensure the correct flow of the test. This is particularily important for tests that involve custom hardware fixtures that need to be reconfigured during the test.

This involves two parts:

1. device side C++ code:
   • awaitHost()
   • signalHost()
2. host side python code:
   • awaitDevice(deviceName)
   • signalDevice(deviceName)

Should probably be done after #5

Introduce more test types

Right now, a test uploads to the daisy seed and executes there. This is not everything.
We should add support for more test types and clarify in the docs what kind of tests are available:

1. single daisy seed only: A single firmware image that's uploaded to the seed and executed there. The host computer only collects the test results via USB serial
   • this is the most simple test case that's simple to setup but limited in the range of tests that can be done
2. multi seed: multiple firmware images are uploaded to an arbitrary number of seeds, results are collected from a subset (or all) of them
   • For more complex tests that do inter-seed communication vis SPI, I2C etc.
   • Still simple to setup: just specify multiple firmware images and the devices they should be uploaded to respectively.
3. hybrid: One or more firmware images that are uploaded to one or more seeds and executed there. The host computer runs a custom script that does the testing, e.g. by configuring other hardware, collecting results from the seed as well as doing its own comparisons
   • For test cases that involve testing against/with custom hardware fixtures
   • This requires one or more firmware images as well as a custom test runner code on the host side => harder to setup but very flexible
4. custom: A completely custom test that's basically just a script that does everything.
   • For anything else. Really, this could do anything but it needs a lot of setup code.
5. ... ?

Cleanup setup instructions

Add semantic versioning and automatic releases

Provide json config for the hardware setup

Introduce a configuration file format to describe the hardware setup for the tests.
The format should be able to provide separate configurations for each of the seeds in the hardware setup, where each configuration includes:

• a human-readable name for the device ("Alice", "Bob", ...)
• daisy seed USB serial device to collect results from
• programming device / command used to flash firmwars to this device (do we limit this to stlinkv3 for the time being? At least we need to tell apart multiple programmers for multi-seed testing)

This configuration file should be included at the project level. Then each test could specify which firmware image to upload to which seed just by using the "name". This would simplify the individual tests a lot.

We will need

• the format specification and documentation for it
• python code to read and represent the device configuration.
• A CMake function that specifies the path for the hardware configuration file in the project root CMakeLists.txt and stores it in a variable to be used the the other CMake functions. Or do we expect a fixed file name for simplicity? Something like daisyhat.config.json?

Docker container security hardening

Improve the container security by limited outgoing/incoming network access to only the domains that are required for the github actions API

Improve host/device synchronization

Right now, the daisy seed waits for a couple of seconds before it starts the test execution, so that the host computer has some time to attach to the USB serial. It would be benefitial to have the host trigger the start of the test by writing a magic string to the daisy seed. This would improve both the test execution times as well as the robustness.