Differences between `coverage lcov` and `py2lcov` for consideration

[zackw]

Differences between `coverage lcov` and `py2lcov` for consideration

[henry2cox]

One thing you might want to do, is to simply generate a differential report between the two. (There are obviously no source differences - so any differences in numbers are down to the tools.) You probably want to turn off function coverage as that will somewhat pollute the data given that py2lcov will report it but coverage.py wont (so will look like purely additions or deletions, depending on the direction of compare .... entirely predictable, and completely unhelpful).

The differential report will call out all the inconsistencies.

[zackw]

I'm very confused that you keep saying py2lcov will report function coverage data when for me it definitely does not and I don't see how it possibly can given that coverage xml doesn't either.

Anyway, here's the result of genhtml --baseline-file B.lcov --diff-file /dev/null --ignore-errors empty -o ABdiff --flat --no-function-coverage A.lcov. There's definitely something weird going on here; just on a quick glance, several of the GIC and GUC lines are the beginnings of docstrings, which should be treated as lines that generated no code.

ABdiff.tar.gz

Adding --branch-coverage to the invocation gives me

Reading tracefile A.lcov.
genhtml: ERROR: (corrupt) unable to read trace file 'A.lcov':
genhtml: ERROR: (inconsistent) "A.lcov":67: unexpected line number '0' in .info file record 'BRDA:0,0,0,1'

which is the aforementioned nedbat/coveragepy#1846 issue.

[henry2cox]

For the first question: if the tool can find the source file, then it parses the file to find function decls (...making some assumptions about indentation rules). It then generates a function coverage record, claiming that the function is hit as many times as the first line in the function was.
If your version of py2lcov does not contain that support - which version are you using, and does it look like this one?
Note that the tool assumes that the file it found is the one referred in the input data. Generally true - but maybe not, if there is some delay between test execution and data extraction, and if people are not so careful with revision management.
In a larger environment: this is also why it is a good idea to use the --version-script feature to tag the coverage data such that subsequent users can do some validity checking. (Long back, we had some very strange validation results that took some time to figure out. Never again.)

For the second: if we have GIC and UIC lines, then 'A.lcov' is claiming that that there are line coverpoints in those locations. From your description: that seems not correct. Not sure why that happens (would need to look at testcase).

For the third: you can tell the tool to --ignore inconsistent - and then it will skip that record. (There may be subsequent issues that also need to be ignored, and it is possible that you then hit a fatal error that cannot be ignored.