Include Code Coverage Infrastructure

[alejmrm]

Include Code Coverage Infrastructure

[cvoltz]

I use Bullseye Code Coverage http://www.bullseye.com/ with Unity and
it works fine. My script to run the unit tests does the following:

1. set the COVFILE so I know where the coverage data is:
   export COVFILE=/tmp/test.cov
2. set the COVDIR directory so I know where my reports will go
   export COVDIR=coverage
3. turn on the code coverage compiler
   cov01 --on
4. select which files I want (and don't want) in the code coverage report
   covselect --no-banner -a !test/
   ...
5. build and run the unit tests
   rake
6. turn off the code coverage compiler
   cov01 --off
7. generate various coverage reports (by directory, by function, by
   file, index of all of the above, and an XSL file for easy data
   crunching)
   covhtml --no-banner $COVDIR
   covdir -v --no-banner --html -o $COVDIR/directory_coverage.html
   covfn -v --no-banner --html -s -o $COVDIR/function_coverage.html
   covsrc -v --no-banner --html -o $COVDIR/file_coverage.html
   covxml --no-banner --xsl -o $COVDIR/coverage.xml

This doesn't require the code to be instrumented manually. Bullseye
replaces the calls to the compiler with their tool which automatically
instruments the code and then calls the original compiler. When the
program is run, the instrumentation dumps the results into the file
given by COVFILE. Later when the reports are generated, they read the
same file to get the data. This is similar to how Coverity works.
Bullseye supports all the mainstream compilers and many embedded ones
and it runs on Linux, UNIX (and derivatives), and Mac OS X. The product
works great. The only downside is the cost and the fact that you have to
purchase a license for each target OS you need to run it on. Their tech
support is good too.

If Bullseye is too expensive for you or you refuse to use something
which isn't open source, you can look into gcov
http://gcc.gnu.org/onlinedocs/gcc/Gcov-Intro.html#Gcov-Intro. Of
course, it only works with gcc. It also isn't as full featured.

Note that ceedling already has plugins for /gcov/ and for /bullseye/.
Look in the /plugins/ directory. Unfortunately, they aren't documented
very well so you pretty much need to understand ceedling and read
through the source of the plugins to figure out how to use them. Look
at issue #3 ThrowTheSwitch/Ceedling#3 for
some info on how to setup the Bullseye plugin.

Christopher Voltz mailto:[email protected], Master Engineer
HP http://www.hp.com, ISS Storage, Linux and VMware Driver Development
voice: 281-514-8369

On 06/04/2013 01:44 PM, Alex Rod wrote:

After working with unity for a while I am seeing that some of my test
are not covering all the cases that I should. So this request is to
include some sort of code coverage infrastructure in Untiy.

I don't have any specifics about how to implement this feature, but
what I have seem in the web --wikipedia mainly, by searching for code
coverage-- is some sort of file transformation on the fly to get an
instrumented version of the file that you want to analyze, then run
through your unit test harness, or some sort of this variation.

—
Reply to this email directly or view it on GitHub
#40.

[mjago]

Spot on

On 4 Jun 2013, at 20:23, Christopher Voltz wrote:

I use Bullseye Code Coverage http://www.bullseye.com/ with Unity and
it works fine. My script to run the unit tests does the following:

1. set the COVFILE so I know where the coverage data is:
   export COVFILE=/tmp/test.cov
2. set the COVDIR directory so I know where my reports will go
   export COVDIR=coverage
3. turn on the code coverage compiler
   cov01 --on
4. select which files I want (and don't want) in the code coverage report
   covselect --no-banner -a !test/
   ...
5. build and run the unit tests
   rake
6. turn off the code coverage compiler
   cov01 --off
7. generate various coverage reports (by directory, by function, by
   file, index of all of the above, and an XSL file for easy data
   crunching)
   covhtml --no-banner $COVDIR
   covdir -v --no-banner --html -o $COVDIR/directory_coverage.html
   covfn -v --no-banner --html -s -o $COVDIR/function_coverage.html
   covsrc -v --no-banner --html -o $COVDIR/file_coverage.html
   covxml --no-banner --xsl -o $COVDIR/coverage.xml

This doesn't require the code to be instrumented manually. Bullseye
replaces the calls to the compiler with their tool which automatically
instruments the code and then calls the original compiler. When the
program is run, the instrumentation dumps the results into the file
given by COVFILE. Later when the reports are generated, they read the
same file to get the data. This is similar to how Coverity works.
Bullseye supports all the mainstream compilers and many embedded ones
and it runs on Linux, UNIX (and derivatives), and Mac OS X. The product
works great. The only downside is the cost and the fact that you have to
purchase a license for each target OS you need to run it on. Their tech
support is good too.

If Bullseye is too expensive for you or you refuse to use something
which isn't open source, you can look into gcov
http://gcc.gnu.org/onlinedocs/gcc/Gcov-Intro.html#Gcov-Intro. Of
course, it only works with gcc. It also isn't as full featured.

Note that ceedling already has plugins for /gcov/ and for /bullseye/.
Look in the /plugins/ directory. Unfortunately, they aren't documented
very well so you pretty much need to understand ceedling and read
through the source of the plugins to figure out how to use them. Look
at issue #3 ThrowTheSwitch/Ceedling#3 for
some info on how to setup the Bullseye plugin.

Christopher Voltz mailto:[email protected], Master Engineer
HP http://www.hp.com, ISS Storage, Linux and VMware Driver Development
voice: 281-514-8369

On 06/04/2013 01:44 PM, Alex Rod wrote:

After working with unity for a while I am seeing that some of my test
are not covering all the cases that I should. So this request is to
include some sort of code coverage infrastructure in Untiy.

I don't have any specifics about how to implement this feature, but
what I have seem in the web --wikipedia mainly, by searching for code
coverage-- is some sort of file transformation on the fly to get an
instrumented version of the file that you want to analyze, then run
through your unit test harness, or some sort of this variation.

—
Reply to this email directly or view it on GitHub
#40.

—
Reply to this email directly or view it on GitHub.

[alejmrm]

Yeap, I have looked at several tools, including the one mentioned above ( bullseye ) and cvtool. Pretty much the description provided by Chris is what I have understood about them and the workflow is similar to what I was thinking as a good implementation for it. Also a good reading is the "C unit testing on a shoestring" whitepaper from Ark Khasin.

The intention of this Open item is to highlight the need of such tool embedded in the Unity framework if possible. But also based in Chris comments I will look deeper into bullseye and evaluate it.

[hemaggopal]

Existing code invokes cov01 commands through ant. Now I have to exclude few files. covselect is throwing the following error. Could someone provide pointers on this?

error: unknown option "--add \!**/filename.extn1.*/"

//Trying to exclude files with name "filename.extn1.*" in any folder
<exec failonerror="false" failifexecutionfails="false" executable="covselect">
    <arg value="--add  \!**/filename.extn1.*/"/>
</exec>

//Enabling the coverage
<exec failonerror="false" failifexecutionfails="false" executable="cov01">
    <arg value="-1" />
</exec>

//Building the files
....

//Disabling the coverage
<exec failonerror="false" failifexecutionfails="false" executable="cov01">
    <arg value="-0" />
</exec>

[cvoltz]

Since the original issue was closed 3 years ago, it would probably be best if your new request was created as a new issue.