How to find the input that generates an assert failure in the test tests/file/large_int.bas

[freebasic]

How to find the input that generates an assert failure in the test tests/file/large_int.bas

[rversteegen]

Sorry for the slow reply; I've been really busy.

Oh, that's a cool project! Did fbc end up in that dataset, with just 8 other programs, just because it meets all the criteria? It seems strange though, because fbc (the compiler) is written in FreeBasic, not C, so its inclusion appears invalid. Only the runtime libraries are written in C. FB is exceedingly closely related to C though; it is basically a subset of C++98 with BASIC syntax (optionally QuickBASIC-compatible) and standard library. The bug you are investigating may be in either FB or C code, I don't know.
Also, it's not accurate to describe FB as a "legacy language compiler" since it's its own, new language rather than a QuickBASIC revival project :)

You're going to have to edit file/large_int.bas, by adding this line before the failing CU_ASSERT_DOUBLE_EQUAL:
if (abs(cdbl(check(i))-cdbl(n)) <= abs(cdbl(check(i) \ 10000000))) then print "failure with " & check(i) & " " & n
This just copies the test that CU_ASSERT_DOUBLE_EQUAL expands to.

[jayrm]

@CharlesDDNoble, yes, interesting project. We consider unit testing an important part of our development, so very cool to see fbc used for input and study in your research.

The test-suite's CU_ASSERT_* macros generate a generic message that is suitable for most cases where we want to know the expression that failed at a specific file/line number. But as you have discovered, since the context of the assertion is dependent on an input file, the full context is not reported.

The older versions of fbc use CUnit for the testing framework. Around December 2017, we re-implemented it as fbcunit written in freebasic instead of C, and using a simplified API, However, most of the same macro names are still used across both.

CUnit (fbc 1.05.0 and earlier)

fbc's CU_ASSERT_DOUBLE_EQUAL is defined in inc/CUnit/CUnit.bi

#define CU_ASSERT_DOUBLE_EQUAL(actual, expected, granularity) CU_assertImplementation(-(fabs(cdbl(actual) - (expected)) <= fabs(cdbl(granularity))), __LINE__, "CU_ASSERT_DOUBLE_EQUAL(" #actual "," #expected "," #granularity ")", __FILE__, "", CU_FALSE)

I think the most direct way to get more specific results but still in standard format is to define a new assertion macro, for example, CU_ASSERT_DOUBLE_EQUAL_MSG to include an extra message to report the specific values:

#define CU_ASSERT_DOUBLE_EQUAL_MSG(actual, expected, granularity, msg) CU_assertImplementation(-(fabs(cdbl(actual) - (expected)) <= fabs(cdbl(granularity))), __LINE__, "CU_ASSERT_DOUBLE_EQUAL(" #actual "," #expected "," #granularity ") " & msg, __FILE__, "", CU_FALSE)

Then use the new macro, for more specific results:

CU_ASSERT_DOUBLE_EQUAL_MSG( check(i), n, check(i) \ 10000000, "check(" & i & ")=" & check(i) & ", n=" & n )

fbcunit (fbc 1.06.0 and later)

This might make a nice addition, or some variation of it, to current fbcunit test-suite API. This framework is located in https://github.com/freebasic/fbc/tree/master/tests/fbcunit

Solution would be similar:

CU_ASSERT_DOUBLE_EQUAL is one of many #defines that wraps the test-suite's fbcu.CU_ASSERT_ generic assertion function found in tests/fbcunit/inc/fbcunit.bi:

#define CU_ASSERT_DOUBLE_EQUAL( a, e, g ) fbcu.CU_ASSERT_( (abs(cdbl(a)-cdbl(e)) <= abs(cdbl(g))), __FILE__, __LINE__, __FUNCTION__, "CU_ASSERT_DOUBLE_EQUAL(" #a "," #e "," #g ")" )

Where fbcu.CU_ASSERT_ is declared as:

namespace fbcu
	declare sub CU_ASSERT_ _
		( _
			byval value as boolean, _
			byval fil as zstring ptr, _
			byval lin as long, _
			byval fun as zstring ptr, _
			byval msg as zstring ptr _
		)
end namespace

value := expression that evaluates to true (pass) or false (fail)
fil   := filename (of the test source file)
lin   := line number (of the test source file)
fun   := function name (of the test source file)
msg   := custom message

[CharlesDDNoble]

@rversteegen @jayrm Thanks a lot! I really appreciate the quick feedback!

@rversteegen I am not actually a member of the team that published the paper and created the repository, but I believe FBC was included because it has suffient C source code and some defects in the version history include only changes to C source code (Python and PHP are also projects included for this reason). Many Auto Program Repair (APR) systems are made specifically for C and the purpose of the ManyBugs repository is to provide a system to test APR's.

@jayrm Thanks for the info on the the FBC testing framework, this will come in handy when I am looking into the newer versions.

I ultimately used the method suggested by @rversteegen of just printing the information before the actual assert. I added the code mentioned before the assert (changing the '<=' to '>' to report failures). This is what I used:

if (abs(cdbl(check(i))-cdbl(n)) > abs(cdbl(check(i) \ 10000000))) then print "failure with " & check(i) & " (" & i & ") " & n

Thanks again!

[rversteegen]

I'd forgotten about the fbcunit rewrite, so that was helpful for me too!