nemequ/munit

µnit is a small testing framework for C

munit's Issues

Support forking on Windows

Windows doesn't support fork(), but there might be something similar enough for us to use…

Don't abort() in assertions if we can avoid it

There are some disadvantages to calling abort() in the assertion macros when not forking… perhaps the most problematic is that, when we're not in a forked child, the contents of stderr will be swallowed since the test runner exits before it can splice it to stderr. Also, the test runner exits and no more tests are run, which is obviously not what we want.

I think the easiest way to address this would be to setjmp with some thread-local storage right before we exec the test case, then longjmp back in the assertion.

Skipping tests

It should be possible to call munit_test_skip (or maybe have tests return a MUNIT_SUCCESS/FAILURE/SKIPPED enum value) from inside of a test to skip it instead of requiring success/failure.

Check exit status from forked child even if test was successful

Currently, if a test function returns MUNIT_OK we count it as successful, even if the process exits with a non-zero value (like if AddressSanitizer detected a leak…). We should probably count such events as errors.

First time compilation error

I downloaded munit and included the .c file in my sources and the header in my includes but when I went to compile, I got an error message saying "error: variably modified 'bytes' at file scope" and a few warnings. Here is a paste of the error message. I am on Intel MacOS 11.4 using C11 with GCC.

Colorize result

OK -> green, SKIP -> yellow, FAIL/ERROR -> red

Not difficult, just need to detect terminal capabilities before doing it.

Rewrite PRNG

The underlying algorithm is fine (AFAIK), but

• munit_rand_uint32 is not composable but I use it like it is. Functions like rand_double, rand_memory, etc., should get a local copy of the state, generate as many random numbers as they need, then CAS the new state back to the global variable.
• Expose munit_rand_uint32 instead of munit_rand_int. The latter could vary in size across different platforms, yielding different results. rand_int_range should be safe as long as min and max are not huge (outside of [INT32_MIN, INT32_MAX])
• Sticking to 32 bits means we could use something like https://stackoverflow.com/questions/2509679/how-to-generate-a-random-number-from-within-a-range#6852396 for uniform distribution.

unnecessary (or faulty?) conditional

This is a fun one! Check this one out: https://github.com/nemequ/munit/blob/master/munit.c#L755

r - (start - stop)... is mathematically equal to r + (stop - start). The conditional is unnecessary. :)

Found it using gcc -Wduplicated-branches.

Use higher-resolution clock on Windows

GetTickCount / GetTickCount64 has millisecond resolution.

https://msdn.microsoft.com/en-us/library/windows/desktop/dn553408%28v=vs.85%29.aspx explains a better way.

Error when disabling timing

I defined the MUNIT_DISABLE_TIMING macro and i get compile errors.

munit.c:904:10: error: variable ‘wc’ has initializer but incomplete type
   struct PsnipClockTimespec wc = { 0, };
          ^~~~~~~~~~~~~~~~~~
munit.c:903:29: error: storage size of ‘wc’ isn’t known
   struct PsnipClockTimespec wc = { 0, };

munit.c:906:24: error: ‘PSNIP_CLOCK_TYPE_WALL’ undeclared (first use in this function)
   psnip_clock_get_time(PSNIP_CLOCK_TYPE_WALL, &wc);

PGCC Travis job fails: "make: pgcc: Command not found"

See https://travis-ci.org/github/nemequ/munit/builds/764414729

Divide by zero in munit_rand_state_at_most()

.. when calling munit_rand_int_range(any_negative_int, 0);

Address sanitizer:

munit/munit.c:676:42: runtime error: division by zero
SUMMARY: AddressSanitizer: undefined-behavior munit/munit.c:676:42 in 
Error: child killed by signal 8

Valgrind:

==60== Process terminating with default action of signal 8 (SIGFPE)
==60==  Integer divide by zero at address 0x802DA55A0
==60==    at 0x406CDD: munit_rand_state_at_most (munit.c:676)
==60==    by 0x406D74: munit_rand_at_most (munit.c:698)
==60==    by 0x406E21: munit_rand_int_range (munit.c:714)

The % max when max=0 is the issue on line 676:

const munit_uint32_t min = (~max + 1U) % max;

Fails to compile with the latest MSVC 2019 x64 and C11 (C99 is OK)

Hi! Thanks for the great library!

With the latest MSVC (Community) update munit fails to compile when the C standard version is set to C11. With C99 it works fine. C11 used to work fine before the update.

CMakeLists.txt:

cmake_minimum_required(VERSION 3.10)

project(MUNIT-TEST VERSION 1.0.0
             LANGUAGES C)

set(CMAKE_C_STANDARD 11) # HERE
set(CMAKE_C_EXTENSIONS OFF)

add_executable(munit-test munit.c main.c)

main.c:

#include "munit.h"

int main(int argc, char *argv[])
{
    return 0;
}

cmake -A x64 ..:

D:\projects\munit-test\build>cmake -A x64 ..
-- Building for: Visual Studio 16 2019
-- Selecting Windows SDK version 10.0.19041.0 to target Windows 6.1.7601.
-- The C compiler identification is MSVC 19.28.29334.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: D:/projects/munit-test/build

cmake --build . --config Release:

Microsoft (R) Build Engine версии 16.8.2+25e4d540b для .NET Framework
(C) Корпорация Майкрософт (Microsoft Corporation). Все права защищены.

  munit.c
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\um\winbase.h(9461,5): warning C5105: macro expansion producing 'defined' has undefined behavior [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(403,81): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(403,81): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(465,118): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(465,118): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(474,139): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(480,83): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(480,83): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(152,38): error C2054: expected '(' to follow '_Thread_local' [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(152,64): error C2085: 'munit_error_jmp_buf_valid': not in formal parameter list [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(152,64): error C2143: syntax error: missing ';' before '=' [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(153,35): error C2054: expected '(' to follow '_Thread_local' [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(153,54): error C2085: 'munit_error_jmp_buf': not in formal parameter list [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(169,111): error C2085: 'munit_logf_exv': not in formal parameter list [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(169,111): error C2143: syntax error: missing ';' before '{' [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(204,17): warning C4013: 'munit_logf_exv' undefined; assuming extern returning int [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(223,34): error C2065: 'munit_error_jmp_buf_valid': undeclared identifier [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(224,34): error C2065: 'munit_error_jmp_buf': undeclared identifier [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(224,34): warning C4047: 'function': '_JBTYPE *' differs in levels of indirection from 'int' [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(224,15): warning C4024: 'longjmp': different types for formal and actual parameter 1 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(239,32): error C2065: 'munit_error_jmp_buf_valid': undeclared identifier [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(240,32): error C2065: 'munit_error_jmp_buf': undeclared identifier [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(240,32): warning C4047: 'function': '_JBTYPE *' differs in levels of indirection from 'int' [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(240,13): warning C4024: 'longjmp': different types for formal and actual parameter 1 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(939,103): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(939,103): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(956,74): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(956,74): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1077,97): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1077,97): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1430,9): error C2065: 'munit_error_jmp_buf': undeclared identifier [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1430,9): warning C4047: 'function': '_JBTYPE *' differs in levels of indirection from 'int' [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1430,9): warning C4024: '_setjmp': different types for formal and actual parameter 1 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1434,33): error C2065: 'munit_error_jmp_buf_valid': undeclared identifier [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1700,72): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1700,72): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1836,79): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(1836,79): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(2053,72): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.c(2053,72): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
  main.c
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\um\winbase.h(9461,5): warning C5105: macro expansion producing 'defined' has undefined behavior [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(403,81): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(403,81): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(465,118): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(465,118): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(474,139): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(480,83): error C2057: expected constant expression [D:\projects\munit-test\build\munit-test.vcxproj]
D:\projects\munit-test\munit.h(480,83): error C2466: cannot allocate an array of constant size 0 [D:\projects\munit-test\build\munit-test.vcxproj]
  Generating Code...

Child suites do not iterate as expected

I have test suites like this:

static MunitSuite other_suites[] = {
    {
        "/morton-tests-randomized",
        tests_randomized,
        NULL,
        1024,
        MUNIT_SUITE_OPTION_NONE
    },
    { NULL, NULL, NULL, 0, MUNIT_SUITE_OPTION_NONE }
};


static const MunitSuite suite = {
    "/morton-tests",
    tests_static,
    other_suites,
    1,
    MUNIT_SUITE_OPTION_NONE,
};

I have two changes which solve this problem. The first one is very simple:

diff --git a/munit.c b/munit.c
index 8acf15c..a436462 100644
--- a/munit.c
+++ b/munit.c
@@ -1290,6 +1290,8 @@ munit_test_runner_run_suite(MunitTestRunner* runner,
   size_t pre_l;
   char* pre = munit_maybe_concat(&pre_l, (char*) prefix, (char*) suite->prefix);

+  runner->suite = suite;
+
   /* Run the tests. */
   for (const MunitTest* test = suite->tests ; test != NULL && test->test != NULL ; test++) {
     if (runner->tests != NULL) { /* Specific tests were requested on the CLI */

I am VERY suspicious about the fact that this worked at all. While it did exactly what I wanted, it looks like one of those really suspicious one-line changes that while it fixes one thing it also fscks everything else. I checked everywhere for bits of code similar to runner->suite and the only ones I found are the one passed in through munit_test_runner_run and the bit of munit_test_runner_exec which checks the number of iterations to run a test for. So I think it's safe. Probably?

It seems mildly ironic that munit itself doesn't have any unit tests I can run to try and catch this.

To clarify, when the ROOT test suite (the one simply named suite in the above sample) has its iterations field set to a number greater than one, all of the tests in all suites are run multiple times as expected. A consequence of this change, I guess, kinda unwanted? Would be that if the root suite had its iterations set to a number n and the child suite had m iterations, we wouldn't get n * m iterations of the child suite, only m.

A solution to this second problem is the second solution overall that I found, which is a little bit more complex:

diff --git a/munit.c b/munit.c
index 8acf15c..f2f7fad 100644
--- a/munit.c
+++ b/munit.c
@@ -1290,6 +1290,10 @@ munit_test_runner_run_suite(MunitTestRunner* runner,
   size_t pre_l;
   char* pre = munit_maybe_concat(&pre_l, (char*) prefix, (char*) suite->prefix);

+  size_t save_iters = runner->iterations;
+
+  runner->iterations *= suite->iterations;
+
   /* Run the tests. */
   for (const MunitTest* test = suite->tests ; test != NULL && test->test != NULL ; test++) {
     if (runner->tests != NULL) { /* Specific tests were requested on the CLI */
@@ -1316,6 +1320,7 @@ munit_test_runner_run_suite(MunitTestRunner* runner,

  cleanup:

+  runner->iterations = save_iters;
   munit_maybe_free_concat(pre, prefix, suite->prefix);
 }

@@ -1457,7 +1462,7 @@ munit_suite_main_custom(const MunitSuite* suite, void* user_data,
     .suite = NULL,
     .tests = NULL,
     .seed = 0,
-    .iterations = 0,
+    .iterations = 1,
     .parameters = NULL,
     .single_parameter_mode = false,
     .user_data = NULL,

This one leaves behind a little bit of suddenly extraneous code (which, if this solution is acceptable to y'all, I can take care of.) However, it works by setting the initial iterations value to 1 instead of 0, and then every time we recurse over a test suite and run the child suites, we save the old iterations value on the callstack, and multiply the current iterations value to get that n * m effect I wanted. A consequence of this is that calling a test executable built with munit with --iterations 0 will probably cause the iteration behavior to revert to that which was present before the patch (since 0 * m == 0.) I haven't tested this, though. It also feels really silly to be saving the iterations field on the callstack.

Error when compiling on Windows with TDM-GCC-32

gcc -std=c99 -o example munit.c example.c
munit.c: In function 'munit_replace_stderr':
munit.c:1252:17: warning: implicit declaration of function 'fileno' [-Wimplicit-
function-declaration]
     int errfd = fileno(stderr_buf);
                 ^

gcc (tdm-1) 5.1.0
tdm-gcc-5.1.0-3 (32-bit)

Suites in suites

It should be possible to include suites in other suites; that would make organizing code in which tests were split across multiple files much easier.

Color on Windows

On Windows, we should probably try to detect if the command window supports ANSI escapes, and default to not using color if we are not sure. The situation is a bit tricky, as different console emulators set different environment variables.

• Some programs (like ConEmu) set the environment variable ANSICON (related to https://github.com/adoxa/ansicon, which also appears to set CLICOLOR)
• Some set TERM (mostly bash, but for instance MSYS2 bash returns 0 for isatty(), explained here)

Most importantly though, the default command prompt does not support ANSI colors and returns true for isatty().

Asserted Macros are Expanded Before Stringification

When most assertion macros are called, they call another macro, and in the process expand the arguments originally passed to the assertion. Since stringification happens in munit_assert_type_full, after the user's macro has been expanded, there's no way to stringify the original expression.

For example:

uint8_t SOME_LONG_UGLY_EXPANSION = 0;
#define FOO SOME_LONG_UGLY_EXPANSION
munit_assert_uint8(FOO, ==, 7);

Will result in the message:

assertion failed: SOME_LONG_UGLY_EXPRESSION == 7 ...

This isn't the behavior I expected. Furthermore if the expansion of the macro contains format-string characters lots of warnings are produced.

I've fixed this in my local copy by adding two additional terms to many internal-use macros through which I pass #a and #b, sometimes adding an additional level of indirection as necessary to preserve the public API.

Rewrite test runner

Current version is really just a prototype, now that I'm a bit more familiar with all the features it needs to provide I can come up with something much cleaner.

Valgrind report 'still reachable' warning on test_name

I have strange behavior with valgrind. Basically, I defined my test suite as follow:

  /* Set the test cases */
  MunitTest test_cases[] =
    {
     { "trace/instr", test_instr, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
     { "trace/hash", test_hash, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
     { "trace/hashtable", test_hashtable, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
     { NULL, NULL, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
    };

  /* Set the test suite */
  const MunitSuite test_suite[] =
    {
     { "libafl/", test_cases, NULL, 1, MUNIT_SUITE_OPTION_NONE },
     { NULL, NULL, NULL, 0, MUNIT_SUITE_OPTION_NONE }
    };

But, the problem is that valgrind detect a "still reachable" memory area in each process. For example the last test gives:

[ OK    ] [ 0.00903018 / 0.00862850 CPU ]
libafl/trace/hashtable               ==8031== 
==8031== HEAP SUMMARY:
==8031==     in use at exit: 23 bytes in 1 blocks
==8031==   total heap usage: 21 allocs, 20 frees, 5,050 bytes allocated
==8031== 
==8031== 23 bytes in 1 blocks are still reachable in loss record 1 of 1
==8031==    at 0x483577F: malloc (vg_replace_malloc.c:299)
==8031==    by 0x4848C03: munit_maybe_concat (munit.c:1110)
==8031==    by 0x484A7D8: munit_test_runner_run_test (munit.c:1556)
==8031==    by 0x484B028: munit_test_runner_run_suite (munit.c:1677)
==8031==    by 0x484B1A0: munit_test_runner_run (munit.c:1696)
==8031==    by 0x484CAA5: munit_suite_main_custom (munit.c:2026)
==8031==    by 0x484CCCA: munit_suite_main (munit.c:2054)
==8031==    by 0x10BD45: main (test_trace.c:189)
==8031== 
==8031== LEAK SUMMARY:
==8031==    definitely lost: 0 bytes in 0 blocks
==8031==    indirectly lost: 0 bytes in 0 blocks
==8031==      possibly lost: 0 bytes in 0 blocks
==8031==    still reachable: 23 bytes in 1 blocks
==8031==         suppressed: 0 bytes in 0 blocks
==8031== 
==8031== For counts of detected and suppressed errors, rerun with: -v
==8031== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[ OK    ] [ 0.01238986 / 0.01177837 CPU ]
3 of 3 (100%) tests successful, 0 (0%) test skipped.
==8028== 
==8028== HEAP SUMMARY:
==8028==     in use at exit: 0 bytes in 0 blocks
==8028==   total heap usage: 12 allocs, 12 frees, 4,612 bytes allocated
==8028== 
==8028== All heap blocks were freed -- no leaks are possible
==8028== 
==8028== For counts of detected and suppressed errors, rerun with: -v
==8028== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Note that the size of the "still reachable" memory area is exactly the size of the string enclosing the name of the test (plus the \0 character at the end).

One way to make the error vanish seems to simply declare the test suite as follow:

 /* Set the test cases */
  MunitTest test_cases[] =
    {
     { "libafl/trace/instr", test_instr, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
     { "libafl/trace/hash", test_hash, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
     { "libafl/trace/hashtable", test_hashtable, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
     { NULL, NULL, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
    };

  /* Set the test suite */
  const MunitSuite test_suite[] =
    {
     { "", test_cases, NULL, 1, MUNIT_SUITE_OPTION_NONE },
     { NULL, NULL, NULL, 0, MUNIT_SUITE_OPTION_NONE }
    };

The root name of the test suite is empty and this time valgrind gives a clean report for each test:

All heap blocks were freed -- no leaks are possible

I tried to see what was the problem but couldn't find a good way to solve it (everything seems to be linked to the munit_maybe_free_concat(() function).

Website does not display properly with Safari on macOS

When you click any of the "v" buttons or any of the menu buttons on the bottom the site scrolls up and then the content is immediately obscured by the giant purple cover sheet.

Safari 12.1.1 on macOS High Sierra 10.13.6

Update MWE in documentation

Hi,

the MWE doesn't compile. It seams the main function has been renamed to munit_suite_main.

Cheers

 The return value will be "EXIT_FAILURE" if any tests fail, or "EXIT_SUCCESS" if all tests succeed. This makes the value suitable for returning directly from your main() function.

In the simplest case you'll end up with something like this:

int main (int argc, const char* argv[]) {
  return munit_main(&suite, NULL, argc, argv);
}

User Data and Fixtures

munit_assert_double_equal doesn't care about scale of the values

Example:

munit_assert_double_equal(1e100, 1.0000000001e100, 1); // fails

For the same reasons that the next test passes the test above should pass too.

munit_assert_double_equal(1, 1.0000000001, 1);

The better ways of comparing floating point numbers are described here: https://stackoverflow.com/a/253874/4626533

can't compile on Ma OS X with --Werror flag

got errors:

munit.c:652:47: error: missing field 'denom'
      initializer [-Werror,-Wmissing-field-initializers]
  static mach_timebase_info_data_t tbi = { 0, };
                                           


munit.c:676:47: error: missing field 'denom'
      initializer [-Werror,-Wmissing-field-initializers]
  static mach_timebase_info_data_t tbi = { 0, };
                                              


munit.c:907:39: error: missing field
      'nanoseconds' initializer [-Werror,-Wmissing-field-initializers]
  struct PsnipClockTimespec wc = { 0, };
                                      


munit.c:1175:53: error: missing field
      'nanoseconds' initializer [-Werror,-Wmissing-field-initializers]
  struct PsnipClockTimespec wall_clock_begin = { 0, }, wall_clock_end = { 0, };
                                                    


munit.c:1175:78: error: missing field
      'nanoseconds' initializer [-Werror,-Wmissing-field-initializers]
  struct PsnipClockTimespec wall_clock_begin = { 0, }, wall_clock_end = { 0, };
                                                                             


munit.c:1176:52: error: missing field
      'nanoseconds' initializer [-Werror,-Wmissing-field-initializers]
  struct PsnipClockTimespec cpu_clock_begin = { 0, }, cpu_clock_end = { 0, };
                                                   


munit.c:1176:76: error: missing field
      'nanoseconds' initializer [-Werror,-Wmissing-field-initializers]
  struct PsnipClockTimespec cpu_clock_begin = { 0, }, cpu_clock_end = { 0, };
                                                                           

clang -v:
Apple LLVM version 9.0.0 (clang-900.0.39.2) Target: x86_64-apple-darwin17.4.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Add a --no-buffer command line option

In case MUNIT_NO_BUFFER is not defined, it'd be nice to still achieve the same effect at run-time by passing a "--no-buffer" option.

My use case is that I'd like to build munit with -DMUNIT_NOFORK, but occasionally pass --no-buffer when running tests to get the stderr of a test failing because of the underlying code calling a failing assert().

Add CMake module for automatically discovering test cases from a CMake target

Description

I have written a CMake module along with a helper cmake file with function munit_discover_tests that could be used to automatically discovering test case from any CMake test target that uses Munit for unit testing, just like gtest_dicover_tests in GoogleTest CMake module in this gist.

It uses a post-build command to run <Munit test executable> --list to get list of all tests in the executable and then writes add_test command on each of them in an include file that is included by CTest whenever CTest runs.

munit_discover_tests has CMake parameters for all command-line flags in Munit except --help & --color auto|always|never, as well as an EXTRA_ARGS parameter for accomodating future changes in Munit CLI.

License

CC0

Add option to exit on failure

If a test fails, sometimes people might want to just abort the whole suite instead of trying them all.

`Unresolved symbol clock_gettime` with MinGW 8.1 x64 (win32 threads)

MinGW 8.1 x64 from https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/8.1.0/threads-win32/seh/

Command-line interface for test runner

A command-line interface for selecting tests and setting options would be nice.

Redirect printf to other function (like uart) to take log

I find a C test framework for embedded, I find this but have a problem with the printf, puts function like a other function in stdio.h. So how can I redirect the printf.

munit on an embedded system without files or filesystem

Hi.

I've been looking for a unit test framework that I could use on my embedded ARM Cortex-M3 board, which runs FreeRTOS without a filesystem. munit looked like a good choice so I went ahead and wrote a bunch of tests. Unfortunately I wasn't able to run anything, because it wants to create a temporary file to redirect stderr with, but my platform is unable to work with files at all.

This is mainly my mistake, perhaps I should've evaluated munit better before choosing it for my testing needs, but before I rewrite my tests to use another test framework, is there any straightfoward way I can get munit to work on an embedded target that does not run a memory-protected OS like Linux and does not have a filesystem?

Add Git tags to generate munit releases in GitHub

Hello,

Could you add Git tags to generate munit releases in GitHub?

Thanks

munit_assert_string_equal on two null pointers provoque SEGFAULT

Hi, I just cloned master from this repo and used you code on my debian machine.

Hi, take a look at that gdb session that illustrate the problem:

11			munit_assert_string_equal(argc[i], expected[i]);
2: argc[i] = 0x0
3: expected[i] = 0x0
(gdb) n

Program received signal SIGSEGV, Segmentation fault.
__strcmp_sse2_unaligned ()
    at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:31
31	../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S: Aucun fichier ou dossier de ce type.

I understand the bug but I feel that it maybe a good idea to add a check that checks for the NULL == NULL(actually checking that two pointers are equals should be stronger and include more use cases) case on your asserts since a lot of function return NULL when they should return whatever the pointer to indicate a fail. And for automation testing sometimes by using your asserts on my expected result list I ran into the problem twice already. I don't see drawbacks of checking the case but you are more suited to think about what could go wrong. It would save some testcase re-arranging. Thanks for your work tho.

Verbosity level

A method for setting a verbosity level that controls the amount of information printed.

TAP support

TAP support could be interesting.

The tear down function is not executed if a munit assertion fails in the test function

If a tear down function is provided, and the associated test function executes a munit assertion like munit_assert_init(1, ==, 2), then the tear down function is not executed.

This might be a problem if some resources were allocated (e.g. temporary directories) which always need to be released.

Also, in case the test program is compiled with MUNIT_NO_FORK, then even non-persistent resources (like memory allocated by a setup function) could be leaked.

Add version information

Perhaps better left off until munit is stable, but it would be nice if there were a macro in the header that reflects the version number. This way, people who copy the files into their own project have a way of knowing which version they are using.

gcc -Wshadow produces a warning

I get a warning when compiling munit.c with gcc -Wshadow (GCC version 7.4.0, but happens consistently with higher versions too).

$ gcc -c -Wshadow -fpic munit.c
munit.c: In function ‘munit_test_runner_run_test_with_params’:
munit.c:1426:24: warning: declaration of ‘orig_stderr’ shadows a previous local [-Wshadow]
     const volatile int orig_stderr = munit_replace_stderr(stderr_buf);
                        ^~~~~~~~~~~
munit.c:1299:7: note: shadowed declaration is here
   int orig_stderr;
       ^~~~~~~~~~~

Provide runtime options for positively or negatively selecting tests or suites

I recently found myself in a position where I wanted to run only a subset of my unit tests (while I was actively iterating on them) because certain tests ran very slowly, and slowed me down. To do this, I ended up changing the suites passed to munit_suite_main(), but it sure would be nice to be able to pass a command-line flag to do this instead.

I'm willing to implement this feature myself, so this issue is primarily to gauge interest level (and get suggestions regarding an API).

I'm thinking something like this:

Support inclusion/exclusion by a psuedo-glob on the fully-qualified test name. Something like --include /suite1/* would include all tests or sub-suites of /suite1. --exclude /suite2/long_test would exclude a single test. --include * would be the equivalent of the current default behavior.

Include and exclude can be chained, with the most recent flag taking precedence. --exclude /suite1/* --include /suite1/important_test would only run /important_test from /suite1.

Any thoughts? Does this seem like it could be a worthwhile feature?

Support for non-string parameterized test parameters

It was a bit surprising this wasn't supported. I don't see an easy way (or safe way) to handle pameters that aren't strings. For instance, you can't do this:

static int bar_params[] = {
  0, 1, 2, 3, 4, 5, MUNIT_PARAMETERS_END
};

static MunitParameterEnum test_params[] = {
  { "bar", bar_params },
  { NULL, NULL },
};

Note the use of MUNIT_PARAMETERS_END since NULL would obviously not work (especially given some compilers might resolve that to 0 which is really confusing). Honestly, that wouldn't work so well either (at least, it's not ideal). I would really suggest array lengths be passed instead.

Add an option for a test to expect to fail

TBH I've never used this functionality in other frameworks, but some people seem to like it…

iOS/Mac OS support

Hi

Is it possible to use munit for iOS test?

Can i compile it for i386 with clang?

Parameterized tests

I've been thinking about how to integrate µnit in Squash. Squash is an interesting case because it has many plugins, each of which contain 1 or more codec, each of which operates at 1 or more different compression level. We currently only test each codec at the default level, but it would be nice to test them all.

Parameterized tests would let us support use cases like this in a fairly clean way. Allowing callbacks to generate a list of parameter values would make things a lot cleaner for some users (including Squash). The API is going to require some thought, but I think it would be worth the effort.

gcc-11 and "declared with mismatched bound" warnings

munit hits some new warnings in gcc-11:

  CC       tests/unit-tests/munit/munit.o
../tests/unit-tests/munit/munit.c:1836:47: warning: argument 4 of type ‘char * const[argc + 1]’ declared with mismatched bound ‘argc + 1’ [-Wvla-parameter]
 1836 |                         int argc, char* const argv[MUNIT_ARRAY_PARAM(argc + 1)],
      |                                   ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../tests/unit-tests/munit/munit.c:118:
../tests/unit-tests/munit/munit.h:478:51: note: previously declared as ‘char * const[argc + 1]’ with bound ‘argc + 1’
  478 |                             int argc, char* const argv[MUNIT_ARRAY_PARAM(argc + 1)],
      |                                       ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../tests/unit-tests/munit/munit.c:2053:40: warning: argument 4 of type ‘char * const[argc + 1]’ declared with mismatched bound ‘argc + 1’ [-Wvla-parameter]
 2053 |                  int argc, char* const argv[MUNIT_ARRAY_PARAM(argc + 1)]) {
      |                            ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../tests/unit-tests/munit/munit.c:118:
../tests/unit-tests/munit/munit.h:463:86: note: previously declared as ‘char * const[argc + 1]’ with bound ‘argc + 1’
  463 | int munit_suite_main(const MunitSuite* suite, void* user_data, int argc, char* const argv[MUNIT_ARRAY_PARAM(argc + 1)]);
      |                                                                          ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I do not yet understand why argc + 1 would be different from argc +1 . Still happens even if I remove MUNIT_ARRAY_PARAM -- which of course it does because on my platform that macro does nothing.

orig_stderr shadows previous definition

This declaration: https://github.com/nemequ/munit/blob/master/munit.c#L1418

shadows this one: https://github.com/nemequ/munit/blob/master/munit.c#L1291

this makes gcc -Wshadow complain.

MunitTest.name and MunitSuite_.prefix should be const char*

I believe it's because we have the pedantic flag with gcc, but compiling unit tests against munit I get warnings about discarded qualifiers. This is because I initialize a struct with a const string similar to what's in the example. It doesn't show up in default gcc, so I believe it's one of my stricter compilation flags (likely pedantic).

It's not exactly as simple as changing the type qualifier in the struct to be const char* instead of char* since when I took a brief scan over some of the functions, it looks like there could actually be some modifications to the strings.

Fix should be simply to allocate new strings on the heap and copy/concatenate things over. I'm willing to help out when I can.

TODO tests

Useful for TDD. Expected to fail, but (unlike #23) not counted against the totals. Success should result in a warning that you probably forgot to remove the todo flag.

Dump memory slice when munit_assert_memory_equal fails

Something like

Expected:                                                               
0000   00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f  ................
0010 X 10 11 12 13 14 15 16 58<18 19 1a 1b 1c 1d 1e 1f  .......X........
0020 X 20 21 58<23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f   !X#$%&'()*+,-./
0030   30 31 32 33 34 35 36 37                          01234567        

     Got:                                                               
0000   00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f  ................
0010 X 10 11 12 13 14 15 16 17<18 19 1a 1b 1c 1d 1e 1f  ................
0020 X 20 21 22<23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f   !"#$%&'()*+,-./
0030   30 31 32 33 34 35 36 37                          01234567     

I think colorized output would be very helpful here, but we would have to have some sort of marker character (like the '<' above) when colors aren't available.

Support for Apple M1 chipset

When compiling and linking test executables I get the following error:

Undefined symbols for architecture arm64:
"_munit_errorf_ex", referenced from:
_test_compare in test.c.o
"_munit_suite_main", referenced from:
_main in test.c.o
ld: symbol(s) not found for architecture arm64

I guess the support for Apple's M1 is not there yet, is it?

Type of argv parameter

The rule that allows you to pass a pointer to a function that expects a const pointer is only applied at top level, so if you try to pass an argv declared in the standard way (char *argv[]) to munit_suite_main, GCC will warn about it.