sagotch/ocveralls

**Deprecated** bisect_ppx now have built-in coveralls.io support: https://github.com/aantron/bisect_ppx/pull/176

ocveralls's Issues

Add -o option

If -o file.json present, write json to file.json instead of stdout.

Also allow to use - to explicitly print ou stdout.

Use BISECT_FILE env variable instead of given explicit .out files.

Ocveralls should not be built with bisect :)

I think that ocveralls is shipping linked with bisect itself. This is really annoying because it means that executing ocveralls itself is producing bisect files. Worse, these cannot be used because we don't have the cmp files.

It means we can't run it more than once.

$ ocveralls --repo_token dummy --prefix _build bisect*.out --output dump.json                                                              
$ echo $?
0
$ ocveralls --repo_token dummy --prefix _build bisect*.out --output dump.json
Fatal error: exception Sys_error("_build/src/ocveralls.cmp: No such file or directory")
$ ls bisect000*
bisect0001.out  bisect0002.out
$ cat bisect0002.out                                                                                                                   
BISECT-RTD?????A@????L                    ?0src/ocveralls.m@@@@@@@@@@@@@@@@@@@@AA@@@@@@@@AAA@AAA@AAAAAAAAAAAAAAA@@AAAAAAAAAAAAAAAAAA@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@?4src/ocverallsJSON.ml??2src/ocverallsCI.ml?A?6src/ocverallsBisect.m?b?         ??                                              JJ      JJJ%JJJJJJAA@@@@@@@@@A@@@@@@@@@@@@@@@@@@@@@@

Add --send option

--send option would automatically send output to coveralls.io.

Ocveralls and Bisect disagree on coverage metrics.

My scripts generate a single bisect file:

$ ls bisect*
bisect0001.out

If I run bisect-report I get these values for the coverage:

$ bisect-report -I _build bisect0001.out -text report
$ egrep "Summary|File|total" report
Summary:
- total: 353/374 (94.39%)
File 'bindings/ffi_bindings.ml':
- total: 161/161 (100.00%)
File 'lib/ffi_generated.ml':
- total: 35/37 (94.59%)
File 'lib/ffi_generated_types.ml':
- total: 55/62 (88.71%)
File 'lib/pci.ml':
- total: 57/67 (85.07%)
File 'lib_test/test_pci.ml':
- total: 45/47 (95.74%)

But when I use ocveralls it generates very different metrics on this build.
Here is the commit SHA for the above:

$ git rev-parse HEAD
e58ecc98ca565a963ec92718e9945b4b84459640

And here is the corresponding build on coveralls.io (you can see it has the
same git SHA):

https://coveralls.io/builds/2651693

Here it thinks, for example, that the coverage of lib/pci.ml is just 20% but the bisect-report says it is 85%. Also just by visual inspection it is clear that the lines are being marked wrong. For example, the lines marked in lib_test/test_pci.ml are clearly wrong. These lines are clearly executed by the OUnit framework.

This was generated using:

$ ocveralls --prefix _build bisect* --send

I've used --repo_token /dev/null --output ocveralls_dump to collect the
output and uploaded that, the bisect report and the bisect file to gist:

https://gist.github.com/simonjbeaumont/85fca139e173b7573763

Any ideas?

Latest release is not 3.3.4 :)

The tag on 0.3.4 is 3.3.4. It might be too late to change that, but perhaps it's worthwhile to add an 0.3.4 tag at the same commit? Also, see the releases page.

Running outside of CI gives unhelpful error message

I just tried running ocveralls locally to make sure it was working before pushing to github, and I got a Not_found error with no helpful description. Luckily, the source is rather clear, so I tracked it down to this call to List.find. It would be helpful if the tool could instead print a helpful error which says something about running only in CI with the relevant env vars defined.

Is the reported coverage fair?

Bisect_ppx has the notion of code points for which it collects runtime data. Hitting some of these code points in OCaml has quite a different meaning as when hitting a statement in a procedural language. For example, hitting a sequence point implies that a list or sequence of statements will be evaluated that can easily span several lines. I therefore believe that a fair measure for coverage is what fraction of all code points are executed - independent of line numbers.

I understand that cveralls wants line numbers but in general there will be more line numbers than code points. This means, it is almost impossible to achieve high coverage with data reported by ocveralls. In addition, ocveralls reports a line with several code points on only as hit when all its points have been hit - exaggerating the problem.

More detailed error messages

A recent Travis build of ctypes failed with an ocveralls assertion error:

Fatal error: exception File ["src/ocveralls.ml", line 135, characters 14-20](https://github.com/sagotch/ocveralls/blob/c80388b8d3f699a2e95a532bce98439966c441f3/src/ocveralls.ml#L135): Assertion failed

It'd be useful for the error message to include more information about what went wrong.

Array index out-of-bounds when combining *.out files from multiple runs

I sometimes find that bisect runs of the same program will include count vectors of different lengths for the same file, triggering an array index exception in ocveralls.

Recompiling with -g gives me

$ OCAMLRUNPARAM=b ~/sagotch/ocveralls/ocveralls.native --prefix _build _build/xenvm.coverage0001.out _build/xenvm.coverage0002.out
Fatal error: exception Invalid_argument("index out of bounds")
Raised by primitive operation at file "src/ocverallsBisect.ml", line 73, characters 49-55
Called from file "array.ml", line 93, characters 21-43
Called from file "list.ml", line 84, characters 24-34
Called from file "src/ocverallsBisect.ml", line 83, characters 8-60
Called from file "list.ml", line 55, characters 20-23
Called from file "list.ml", line 55, characters 32-39
Called from file "list.ml", line 55, characters 32-39
Called from file "list.ml", line 55, characters 32-39
Called from file "list.ml", line 55, characters 32-39
Called from file "src/ocveralls.ml", line 60, characters 4-29

Is zero-padding short vectors the right thing to do? I'll submit a pull request for your review.