westerndigitalcorporation/stat

Standalone unit-testing framework for software written in C

STAT (Unit-test Framework)

StandAlone unit-Testing framework for software written in C

STAT is designed to promote an instant assimilation of Test-Driven Development in conditions of large-scale codebases that are constrained with heavy load of legacy code. The framework was conceived with special regard to limitations of embedded environments and to implementation-concepts widely accepted in embedded programming. It is based on a really great open-source project named Unity (which can be found also on GitHub).

Table of Contents

• 1. Background
• 2. Installation
• 3. Usage
  • 3.1. User-Guides
  • 3.2. Additional Helpful Documentation
• 4. Integration
  • 4.1. OS
  • 4.2. Build Tools
  • 4.3. IDEs
• 5. Maintainer
• 6. Contribution
  • 6.1. Contributors
  • 6.2. Imported Components
  • 6.3. Unity Harness
  • 6.4. Contributing
• 9. License
• 8. Definitions

1. Background

We tried several unit-test frameworks, and there are quite a few really great products. So, we stated the following principles as our requirements for the framework we were looking for:

• Simplicity – test-setup shall be simple, fast and intuitive
• Speed – execution shall be fast and focused to support TDD short cycles
• Lightweight – portability to embedded platforms shall be simple (planned for the future)
• Comprehensive feedback – better logging ⇒ lesser step-by-step debugging ⇒ better efficiency
• Reproducibility– tests shall be reproducible and deterministic
• Test-code sharing – reduce the inevitable code-duplication (e.g. test-doubles)
• Automation – test automation shall be very simple to achieve
• CUT-Isolation – decouple from noise-impact of other FW-components, HW or OS

The last one we found most critical for our needs due to constraints described in Conceptual Model. The CUT-Isolation principle was the one that we couldn't achieve with existing alternative frameworks.

In addition, we had certain limitations that hardened integration of other solutions in our environment. Our lab machines were limitted to run only Windows OS, Python 2.7, MS Visual Studio and our target build tool-chain. No lua, Ruby, CMake or any other things of that kind.

Eventually, it was decided to build our own framework that will meet all these requirements. But we didn't do everythign from scratch. We derived our solution from Unity Harness.

2. Installation

Please refer the Getting Started user-guide. It describes how to install and to setup the framework, and how to compose unit-tests over it.

3. Usage

3.1. User-Guides

• Command-line - full description of all possible command-line arguments to control behavior of STAT
• Unity Assertions Reference - the original 'Unity' user-guide with usage-description of assertion macros
• STAT-Mock Library - user-guide describing how to use built-in Mock library of STAT framework

3.2. Additional Helpful Documentation

• Conceptual Model - conceptual model underlying the design of the framework, a valuable knowledge to become Pro with STAT
• Unity Assertions Cheat Sheet - a cook-book with macros worth printing and framing
• Unity Configuration Guide - the default configuration of Unity defined within STAT can be overwritten

4. Integration

4.1. OS

STAT-framework is built to run both on Windows and Linux.

4.2. Build Tools

STAT-framework has an integrated support for build-tools, which are determined based on OS, localy instlled tools and user-configuration:

• On Windows: Microsoft Visual Studio including all versions from 2005 up to 2019 (any edition, i.e. Community, Professional and Enterprise).

• On Linux: GNU Compiler, i.e. GCC

STAT is designed to minimize its requirements from the system. Therefore, it uses a precompiled GNU Make Tool (included in the repo).

This decision was dictated by restrictions of our lab, which didn't allow installing any additional software.

MSVS Versioning

If not explicitly specified in file ".statconfig", the latest MSVS version on the machine gets identified and gets set as active build-tools. Otherwise, the specified version is selected instead. The latter is useful to fixate on common single version for all developer and lab machines.

4.3. IDEs

Framework provides services to generate project files for certain IDEs, by the means of command-line. The following IDEs are currently supported:

• Microsoft Visual Studio of version corresponding to the selected build-tools
  • The generated project file for this IDE supports step-by-step debugging
• Source Insight 4.0 - it is a unique IDE, which also is a code-analyzer, actually one of a kind
• Visual-Studio Code - the generated workspace for this framework supports step-by-step debugging on Linux

5. Maintainer

   Arseniy Aharonov - the maintainer and the main contributor
   

6. Contribution

6.1. Contributors

An open source project thrives on contribution of different people, and it may come in many forms. Some develope the very code, others provide constructive feedback, and some share wisdom through guidance and consultation. And all of these are very valuable contributions made by very valuable contributors:

• Eitan Talianker
  
• Dr. Eitan Farchi
  
• Udi Arie
  
• Oran DeBotton
  
• Luna Benarroch
  
• Leah Adler
  

6.2. Imported Components

STAT wasn't build from scratch. Several valuable components underlay the framework:

• Unity harness source-files - a key component for the framework
  • License: MIT
  • Source: on GitHub
• Visual Studio Locator (a.k.a. vswhere) - a utility to locate Visual Studio in newer versions of MSVS.
  • License: MIT
  • Source: on GitHub
• GNU Make tool - controls generation of executables from sources based on makefiles
  • License: GPLv3+ license
  • Source: on GNU.org

6.3. Unity Harness

Though we couldn't find something that will answer all our requirements, we found the one that was very close.
Unity harness is that great "almost"-match with the following clear advantages:

• Minimalistic in size and dependencies
• Can be compiled almost on any platform
• Provides rich and strong assertion mechanism
• Prints very comprehensive log and results
• Tolerant to test-failures
  • Failing test (if properly built) doesn't crash the subsequent tests

An additional advantage worthy of a separate mentioning is the very fact that Unity is written in C, which is the same language we use for our production code.
Writing in one language gives the same sense of developing, no matter whether it is the production code or unit-tests, without constant switching between different language paradigms.

6.4. Contributing

Feel free to impact! Open an issue.

9. License

The base-code of the STAT framework is licensed under MIT license. In addition, this repo contains a binary of GNU-Make tool, which is distributed under GPLv3+ license.

8. Definitions

These definitions are provided to clarify terms used above.

• STAT – stands for standalone testing framework
• CUT – means Code Under Test; usually it is a module or a sub-module that is tested
• DOC – means Depended-on Component; in the literature this term is used to describe external dependencies that our CUT depends on
• CUT-Isolation – is a method of decoupling the CUT from the other parts of FW-Code for unit-testing purposes, meaning how we deal with DOCs in our unit-tests
• FW - Firmware
• Test-Double – is a substitution of an operational version of DOC with a test-version that emulates a real interface, but serves testing goals
• Test-package – in terms of STAT-framework, it represents a group of unit-tests that run as a single executable; it usually covers a specific CUT
• TDD - Test-Driven Development, see also TDD for more details

stat's Issues

Support for directory view when using VS 2019 is missing / not working

makestat -vs creates a vs 2019 solution, when vs2019 is installed on the computer.
Currently, this solution is not opened with correct directory view - to show the files arranged by their actual folder. instead files are presented flat on the solution explorer.

Add support for GCC on Linux

To enable the framework on Linux it must be able to compile on Linux. GNU Compiler was chosen as a default one for Linux platform.

BUG: Get-Spy-Data API returns NULL instead of failing if no call was really performed

If no call to a certain derived test-double is performed, every API that returns on-call statistics (e.g. spy-data) should fail on read attempt instead of returning any data.

Makefile parsing inconsitency with comments

If a comment is interleaved with the values of a macro defintion, makefile parser doesn't ignore its contents.
This behavior conflicts with IDE-writers, since it attempts to treat the comment as a valid entry and fails on it.

For instance:

SOURCES = \
  file_x.c\
  # file_y.c\   <--this comment will be treated by parser as a file, and it shouldn't
  file_z.c

STAT: Add additional user-case examples to improve documentation and to test corner cases

Additional examples of user-case setup along with some corner case scenarios might provide a good improvement for documentation and also as additional tests for handling of different corner cases.

Might consider to split this task to two separate efforts/tickets.

Incremental build fails in certain scenarios

Testing the latest version (1.2.0) on a real product revealed the following issues:

• Incremental build failed due to overkill checking-option (/WX) that was added for linker
  • It caused the makefiles with duplication of source declarations to fail due to duplication of object files
• There is a bug in the logic responsible for extraction of dependecies that are used by incremental build
  • The compiler command-line doesn't include definitions specified for the test-packages, causing it to fail
• A workaround is needed for certain cases of bad-syntax encountered in user-makefiles

Ocasional failure in incremental build

NMAKE shows an inconsistent behavior in dealing with assignment of values to environment variables. This caused to invariant behavior of the incremental build, which is affected by the amount of times certain environment variable is set (not as part of a batch file). Therefore, sometimes incremental build works correctly, but sometimes it fails.

Pass framework unit-tests on Linux

Adjust the framework to make it run on Linux

STAT: Support for incremental build

The current version of NMAKE script is very basic that rebuilds each test-package in full.
It would be nice to add the framework ability to provide several options of building:

• incremental build
• rebuild
• clean and rebuild (currently the only target that issued by default)

BUG:FWK: Framework doesn't fail test-package on complication fails if running incremental build

If the previous framework run for a test-package succeeded to compile and to pass, then the subsequent incremental run won't fail in case the compilation fails, but will run previously compile test-package executable.

Temporary workaround: add a '-cc' command-line argument to each framework re-run.

Add CI workflow for regression on Linux platform

MOCK: Modify NUMERIC-type macros to operate natively with negative/float values too

We used STAT_ADD_NUMERIC_MOCK to mock a variable that is from float type.
For some reason, we didn't get the correct value.
after a debug session, we saw that the converting from float to unsign was incorrect, and required the same method like in STAT_ADD_MOCK when the value is copied as void.

STAT: Add support for standalone MS Build Tools

Today, on Windows STAT supports full MSVS variations (i.e. build tools with IDE). To support better DevOps and lower expenses it has been requested by some users to add support standalone MS Build Tools (i.e. w/o IDE).
This is especially relevant now, when a support for VSCode has been added to the framework.

Make: Performance degradation on Windows platform

After the framework has migrated to GMake tool, a new design of the core makefiles causes a significant performance degradation on Windows platform.

Migrate from NMake to GMake

GMake (a.k.a. GNU Make tool) is a cross-platform Make tool. Migrating to it, will simplify the procedure of entending the framework to platforms other than Windows.

MSVS: Compilation errors are not clickable in Visual-Studio GUI

In the output window of Visual-Studio when one clicks on a link with the filename and the line of a complication error, the IDE fails to open the location of the file.
Instead, it fires an error that the file cannot be found:

Produce symbolic link to source

Add a support to STAT that will produce symbolic link to source to some dedicated folder (like I do with the headers) and compile them from there, this isolating source files from header files.

Make: Critical issue with dummy interfaces

A new quite critical bug has spotted within makefiles that disturbs it work:

• If there is a dummy header file within the dummies folder any test-package that needs a production version of this file will end up using the dummy version and crash

Add some IDE support for debugging on Linux platform (e.g. VS Code)