"Test output" does not show the name of the func generating the output

[throwtheswitch]

"Test output" does not show the name of the func generating the output

[mvandervoord]

Can you provide a bit more detail. I'm not sure what circumstances you're looking at here.

[hmijail]

What kind of detail? This is the output I see:

...

TEST OUTPUT

[test_TBuffer.c]

• "0/F/TBuffer.c:24/Failed to malloc TBuf: No Memory"
• "0/F/TBuffer.c:24/Failed to malloc TBuf: No Memory"
• ""

---

FAILED TEST SUMMARY

[test_TBuffer.c]
Test: test_TBuf_Create_OK
At line (83): "Expected 0x00000004 Was 0x00000040. Function tmalloc Argument size. Function called with unexpected argument value."

---

OVERALL TEST SUMMARY

TESTED: 3
PASSED: 2
FAILED: 1
IGNORED: 0

---

BUILD FAILURE SUMMARY

Unit test failures.

Note that the filename test_TBuffer.c is given under TEST OUTPUT, but not the test function names.
Meanwhile, for the FAILED TEST SUMMARY, both test file and test function names are given, which would be the correct behavior I guess.

On 29 Apr 2016, at 15:19, Mark VanderVoord [email protected] wrote:

Can you provide a bit more detail. I'm not sure what circumstances you're looking at here.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

[hmijail]

Sorry, looks like the answer I sent through mail got broken? I'll repeat here.

What kind of detail? This is the output I see:

...
-----------
TEST OUTPUT
-----------
[test_TBuffer.c]
  - "0/F/TBuffer.c:24/Failed to malloc TBuf: No Memory"
  - "0/F/TBuffer.c:24/Failed to malloc TBuf: No Memory"
  - ""

-------------------
FAILED TEST SUMMARY
-------------------
[test_TBuffer.c]
  Test: test_TBuf_Create_OK
  At line (83): "Expected 0x00000004 Was 0x00000040. Function tmalloc Argument size. Function called with unexpected argument value."

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  3
PASSED:  2
FAILED:  1
IGNORED: 0

---------------------
BUILD FAILURE SUMMARY
---------------------
Unit test failures.

Note that the filename test_TBuffer.c is given under TEST OUTPUT, but not the test function names.
Meanwhile, for the FAILED TEST SUMMARY, both test file and test function names are given, which would be the correct behavior I guess.

[mvandervoord]

Ah. Ceedling doesn't make any attempt to correlate data that you are manually dumping to stdout with the data that it is looking for. It simply captures it and displays it so that you can use it for whatever you like. This is intentional. In order to correlate it to a test, Ceedling would have to keep that data around, then apply it to the test when it is completed. In addition to this being a lot of overhead for a feature that is seldom used, it could easily be wrong if the output was dumped from setUp, tearDown, the RUN_TEST macro, etc. In any of those cases, you would see the extra output incorrectly assigned to the next completed test.

The output summary is really just there for the convenience of the test writers. The majority of the time, it is completely empty.

[hmijail]

OK, I think I understand.

My assumption was that, just as each stderr output is connected to a test function, the stdout would also be connected, and that the problem of not seeing which test function printed something was just a problem in the test result printer.

From what you say, I understand now that in fact the stdout is NOT captured specifically for each test function, and that doing so would be too much overhead. Right? That's a pity, I assumed it would be the same procedure already being done for stderr.

Just to be clear, I can kinda work-around this myself in my tests by adding the printf("%s",__func__); thing as the first line of each test, just to know that the coming text was output by such-and-such test. It's just strange boilerplate. Though I understand that usually (ideally) unit tests don't even use stdout.

So, as a way to reduce the boilerplate in a generic way: would it be possible to have the name of the test be passed to the setUp() function? So I could do something like

void setUp(char * test_func){
    printf("%s",test_func);
}

[hmijail]

(... or maybe this isn't even about stderr and stdout and then I'm totally off??)

[mvandervoord]

Actually all the output from stdout is being captured. The part that makes it more challenging is that the name of the test is announced with the results, so it happens at the END of the test. (This is necessary for quite a few reasons, all of which are odd little details. Long ago, we used to print the test name at the beginning and then the pass/fail result at the end of the test... everything else would get inserted in between. this didn't really work very well because people's results parsers would get all messed up by anything they output in the middle).

ANYWAY, it happens at the end now... which means that anything you output would have to get stored away separately, then associated with that test once it was announced. This would be correct most of the time... but if the user chose to write something from setup or teardown or anything like that, it would still get associated with the nearest test, which may not be what they were looking for.

Maybe we add a TEST_INFO_MESSAGE macro to output an info line with whatever you want?

[hmijail]

How would that TEST_INFO_MESSAGE macro be used?

[mvandervoord]

It would output the text you pass it, but prefix it with the usual info... maybe something like filename:line:function:INFO:message? I'm just thinking aloud here.

(The advantage being that you could force this sort of output to appear in the appropriate place in Ceedling's Pretty Print)

[hmijail]

If it is something that could be used in setUp(), then OK. But things like filename, line, function, can already be done with standard compiler-provided macros, so... I don't know.

In general, IMO anything that maximizes info in case of debugging and minimizes boilerplate is good :).