atroxaper/raku-racoco

Raku Code Coverage tool.

raku-racoco's Issues

Error after successfully run tests

Running all tests with RAKULIB=/home/wolf/repos/module2rpm racoco
showing an error:

All tests successful.
Files=10, Tests=82,  10 wallclock secs
Result: PASS
===SORRY!===
Library path ｢/home/wolf/repos/module2rpm/lib/.precomp｣ has ambiguous .precomp folder with more than one CompUnit Repository

Removing the ./lib/.precomp folder helped.

Including ./lib does not work

Hi,
I want to use racoco to get a test coverage of my module but unfortunately, it fails to run my tests because the local lib folder is not in the search path:

racoco --lib=./lib

===SORRY!=== Error while compiling /home/wolf/repos/module2rpm/t/00-tar.t
Could not find Module2Rpm::Archive::Tar in:
    inst#/home/wolf/.raku
    inst#/home/wolf/.rakubrew/versions/moar-2020.12/install/share/perl6/site
    inst#/home/wolf/.rakubrew/versions/moar-2020.12/install/share/perl6/vendor
    inst#/home/wolf/.rakubrew/versions/moar-2020.12/install/share/perl6/core
    ap#
    nqp#
    perl5#

It does not work without --lib and not with it. Only using the RAKULIB environment variable works.

RaCoCo fails when tests passed

Hi!
I am not sure if this a RaCoCo issue, however - http://sparrowhub.io:2222/report/37

HTH

Aleksei

`racoco` fails silently

It simply bails out without any message. I've tried several combinations:

racoco -I
racodo -l
racoco --exec 'zef test .'

It simply does not do a thing. This is

Welcome to Rakudo™ v2022.07.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2022.07.

Cannot install because 09 test fails

Hi,
Unfortunately, I am unable to install this module with tests. I always get the error:

zef install App::RaCoCo
===> Searching for: App::RaCoCo
===> Testing: App::RaCoCo:ver<1.3.1>:auth<github:atroxaper>
[App::RaCoCo] # Failed test 'without index file'
[App::RaCoCo] # at t/09-coverable-index.t line 58
[App::RaCoCo] # Failed to create directory '/.racoco' with mode '0o777': Failed to mkdir: Permission denied
[App::RaCoCo] # You failed 1 test of 6
===> Testing [FAIL]: App::RaCoCo:ver<1.3.1>:auth<github:atroxaper>
Aborting due to test failure: App::RaCoCo:ver<1.3.1>:auth<github:atroxaper> (use --force-test to override)

raku version:

raku -v
Welcome to Rakudo(tm) v2020.12.
Implementing the Raku(tm) programming language v6.d.
Built on MoarVM version 2020.12.

OS:

Linux linuxboxhome 5.11.6-arch1-1 #1 SMP PREEMPT Thu, 11 Mar 2021 13:48:23 +0000 x86_64 GNU/Linux

Make output file variable

The directory where we're testing need not be writable; that's usually the case in JJ/alpine-raku#28
It would help if an environment variable would say where we're storing coverage.log and/or the directory where .racoco is being created.

Warn messages

I've gotten some warn messages while trying to generate the code coverage of Red using --exec='zef test -v .'. It worked and output Coverage: 57%

RaCoCo + App::mi6 bug?

Hi @atroxaper ! I found this while playing with SparkyCI:

• racoco -l html fails for App::Mi6 - http://sparrowhub.io:2222/report/336
• The same build without rakcoco succeeds zef test .. - http://sparrowhub.io:2222/report/335

HTH

Aleksei

Use TAP::Harness directly

Currently this is shelling out to prove6, but that's really not necessary. It can just use TAP::Harness directly to run the tests instead. This would also mean you can much more easily inject code into those tests, which may make the implementation of the coverage analysis easier.

Make coverage level testing a bit easier

Here's what I'm looking for

1. Initialize the configuration with a certain level, say 80
2. Make keeping that level easier with a --save-level option of something like that that would save to the configuration file the level we're interested in

That way it's easier to maintain the level, and create tests that fail if the coverage level drops. This could be the default test or a custom level command in the configuration.

Running more than once changes reported coverage

Running racoco --html seems to change the calculated coverage percentage for future runs of either output type:

Terminal-LineEditor $ RAKULIB=. racoco
t/00-use.rakutest .......................... ok 
t/01-single-line-text-buffer.rakutest ...... ok    
t/02-sltb-with-cursors.rakutest ............ ok   
t/03-single-line-text-input.rakutest ....... ok   
t/10-duospace-layout.rakutest .............. ok  
t/11-scrolling-single-line-input.rakutest .. ok   
All tests successful.
Files=6, Tests=656,  8 wallclock secs
Result: PASS
Coverage: 66.9%

Terminal-LineEditor $ RAKULIB=. racoco --html
t/00-use.rakutest .......................... ok 
t/01-single-line-text-buffer.rakutest ...... ok     
t/02-sltb-with-cursors.rakutest ............ ok   
t/03-single-line-text-input.rakutest ....... ok   
t/10-duospace-layout.rakutest .............. ok  
t/11-scrolling-single-line-input.rakutest .. ok  
All tests successful.
Files=6, Tests=656,  4 wallclock secs
Result: PASS
Visualisation: .../Terminal-LineEditor/.racoco/report.html
Coverage: 65.6%

Terminal-LineEditor $ RAKULIB=. racoco
t/00-use.rakutest .......................... ok 
t/01-single-line-text-buffer.rakutest ...... ok   
t/02-sltb-with-cursors.rakutest ............ ok   
t/03-single-line-text-input.rakutest ....... ok   
t/10-duospace-layout.rakutest .............. ok  
t/11-scrolling-single-line-input.rakutest .. ok  
All tests successful.
Files=6, Tests=656,  4 wallclock secs
Result: PASS
Coverage: 65.6%

Terminal-LineEditor $ RAKULIB=. racoco --html
t/00-use.rakutest .......................... ok 
t/01-single-line-text-buffer.rakutest ...... ok   
t/02-sltb-with-cursors.rakutest ............ ok   
t/03-single-line-text-input.rakutest ....... ok   
t/10-duospace-layout.rakutest .............. ok  
t/11-scrolling-single-line-input.rakutest .. ok  
All tests successful.
Files=6, Tests=656,  4 wallclock secs
Result: PASS
Visualisation: file:///home/geoff/foss/git/Terminal-LineEditor/.racoco/report.html
Coverage: 65.6%

Note how the reported coverage changes from 66.9% to 65.6% upon first using --html and then stays like that on future runs.

Allow for pluggable reporters

In the Perl world, the cover command provided by App::Cover can be called like cover -report Foo which will use Devel::Cover::Report::Foo to handle the report. This allows other people to develop their own reporters, which can be used without changing the core tool.

racoco has App::RaCoCo::Report::ReporterHTML to handle the HTML report, but maybe we could have a similar mechanic here? That way, we could write eg. an App::RaCoCo::Report::Coveralls that does not need to be shipped with racoco, but that could be used with something like racoco --report=Coveralls to send data to that service.

What do you think?

Test fail on OpenSuse Linux

Hi,
Sorry, installing App::RaCoCo fails on my OpenSuse Linuxbox:
Kernel: 5.13.5-1
OS: NAME="openSUSE Leap"
VERSION="15.2"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.2"
PRETTY_NAME="openSUSE Leap 15.2"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.2"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"

zef install App::RaCoCo --force-install
===> Searching for: App::RaCoCo
===> Testing: App::RaCoCo:ver<1.3.1>:auth<github:atroxaper>
===SORRY!=== Error while compiling /home/wb/.zef/store/App-RaCoCo-1.3.1.tar.gz/App-RaCoCo-1.3.1/t/09-coverable-index.t
Unable to parse expression in argument list; couldn't find final ')' (corresponding starter was at line 58)
at /home/wb/.zef/store/App-RaCoCo-1.3.1.tar.gz/App-RaCoCo-1.3.1/t/09-coverable-index.t:58
------> ndexFile.new(:lib($*TMPDIR.add('test')) ⏏}, 'without index file'
===> Testing [FAIL]: App::RaCoCo:ver<1.3.1>:auth<github:atroxaper>
Aborting due to test failure: App::RaCoCo:ver<1.3.1>:auth<github:atroxaper> (use --force-test to override)

Raku version:

raku -v
Welcome to 𝐑𝐚𝐤𝐮𝐝𝐨™ v2021.08.
Implementing the 𝐑𝐚𝐤𝐮™ programming language v6.d.
Built on MoarVM version 2021.08.

Error while reading precomp directory contents

Thank you for your work on racoco! I've been waiting for a tool like this one for a while now. 🗡️

I tried recently to run it as part of my CI setup, but I ran into a strange error that I cannot actually reproduce locally. You can see the error in question on the logs for that job here: https://gitlab.com/jjatria/raku-game-entities/-/jobs/1735233677, but the relevant part is

$ racoco --exec='prove6 -Ilib' --html --fix-compunit
Failed to get the directory contents of '/builds/jjatria/raku-game-entities/lib/.precomp': Failed to open dir: No such file or directory
  in sub MAIN at /usr/share/perl6/site/sources/44A4D1B35D8BDF9C08695E7730B9BD23D1D0C3D6 (App::Racoco::Cli) line 122
  in block <unit> at /usr/share/perl6/site/resources/1737D3959927C9BF0C4E4A1C2EA9DB935C657379 line 2
  in sub MAIN at /usr/share/perl6/site/bin/racoco line 3
  in block <unit> at /usr/share/perl6/site/bin/racoco line 1

Any idea what might be the reason?