[Question] Up-to-date overview: How to use ReportGenerator with Azure DevOps?

[georg-jung]

[Question] Up-to-date overview: How to use ReportGenerator with Azure DevOps?

[danielpalme]

I just released version 5.3.0 of ReportGenerator.

The Azure DevOps task now has a new setting publishCodeCoverageResults.
This allows to publish the report in 'Code Coverage' tab directly.
If set to true, it makes the 'PublishCodeCoverageResults' task(s) obsolete. They can be removed from your pipeline.

Example:

- task: reportgenerator@5
  displayName: ReportGenerator
  inputs:
    reports: 'coverage.xml'
    targetdir: 'coveragereport'
    publishCodeCoverageResults: true

[jchenathub24]

• As soon as the PublishCodeCoverageResults@2 task finished (but not the whole job/pipeline), the Code Coverage tab contains a ReportGenerator report. When specifying a license key using .netconfig it even contains a pro version report.
• At the same time, a "Code Coverage Report_" artifact is created.
• As soon as the pipeline(/job?) finishes, the Code Coverage tab's content is replaced with the report format that can be seen in your screenshot.

@georg-jung I come cross the same behaviour.

When I test steps like:

1. dotnet test: generate coverage file
2. using PublishCodeCoverageResults@2

I can see that PublishCodeCoverageResults@2 internally use ReportGenerator, it shows a footer in the report saying that. But later, the whole report is override when pipeline finished. Feel like a bug for that task.

Also, I try the disable.coverage.autogenerate: 'true', it does not change any outcome from PublishCodeCoverageResults@2.

So the options I have tested and working for me:
Option1:

1. dotnet test: generate coverage file.
2. reportgenerator: generate report file from coverage file in step 1. (ReportGenerator 5.2.1.0 by the time of this reply)
3. disable re-generate report: by setting the env variable.
4. PublishCodeCoverageResults@1: publish the report generated in step 2.

Option2:

1. dotnet test: generate coverage file.
2. PublishCodeCoverageResults@1: this task will generate report base on coverage xml file from step 1 and publish to ADO. But the report is generated by ReportGenerator 5.1.14.0

Option3:

1. dotnet test: generate coverage file.
2. PublishCodeCoverageResults@2: this can generate report and publish some report to code coverage tab, but using ADO layout.

Opeion 2 and 1 have no much different. But Option1 give your more customization option in ReportGenerator task. To me, Option 2 is easier to setup then option 1 and do the job, if you don't need too much customization on ReportGenerator task and dont mind to use ReportGenerator 5.1.14.0.

One thing is missing and I believe is useful is that the 'collapose all' and 'expand all' link. If I generate the report locally on my laptop, I can use them to quickly collapose the list. But they are not in ADO's coverage tab, meaning that I have to use scrol bar.

[danielpalme]

Thanks for your detailed issue and the research behind it.

You are right, there are several options.
My experience is, that PublishCodeCoverageResults@1 or PublishCodeCoverageResults@2 are required to get the "Code Coverage" tab

PublishCodeCoverageResults@1 uses an (outdated) version of ReportGenerator internally.
If you don't specify disable.coverage.autogenerate: 'true' it will (re-)generate the coverage report based on the Coberatura file.
If you want to use the latest version of ReportGenerator you should always set disable.coverage.autogenerate: 'true'.

PublishCodeCoverageResults@2 does not use ReportGenerator internally. If you have different information, I would be interested.
Instead it generates it's own report which looks like this:

As far as I know it's not possible to publish a full report with PublishCodeCoverageResults@2, since there is no Report Directory parameter.

Regarding your questions:

• Whats the minimal, optimal but full example to use ReportGenerator with Azure DevOps today? I've seen https://reportgenerator.io/usage but in my testing using only the reportgenerator task didn't produce e.g. the Code Coverage tab.

Correct. You'll have to combine it with PublishCodeCoverageResults@1.
See also: https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension

• Report type HtmlSummary sounds like it could help with microsoft/azure-pipelines-tasks#4945, if it's acceptable to e.g. have just the summary in the "Code Coverage" tab but the full report in the zip artifact. Is there an easy way to set this up?

You could generate two reports in separate directories. Then you would probably have to rename the summary.html file to index.html before publishing the it with PublishCodeCoverageResults@1

- task: reportgenerator@5
  displayName: ReportGenerator
  inputs:
    reports: 'src\target\reports\coverage\coverage.xml'
    targetdir: '$(Build.SourcesDirectory)/coveragereport'
    reporttypes: 'HtmlInline_AzurePipelines;HtmlSummary;Cobertura'
    customSettings: 'settings:createSubdirectoryForAllReportTypes=true'

# Now rename file `$(Build.SourcesDirectory)/coveragereport/HtmlSummary/summary.html` to `$(Build.SourcesDirectory)/coveragereport/HtmlSummary/index.html`

# Create artifact of directory $(Build.SourcesDirectory)/coveragereport/HtmlInline_AzurePipelines

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage results'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(Build.SourcesDirectory)/coveragereport/Cobertura/Cobertura.xml'
    reportDirectory: '$(Build.SourcesDirectory)/coveragereport/HtmlSummary'
  env:
    DISABLE_COVERAGE_AUTOGENERATE: 'true'

• disable.coverage.autogenerate: 'true' doesn't seem to work for me with PublishCodeCoverageResults@2. Is there a workaround for V2 too? Is it better to use V1 with reportgenerator@5?

As mentioned above, I'm pretty sure that you have to use PublishCodeCoverageResults@1.
disable.coverage.autogenerate: 'true' does not work here. And there is no option to specify a report directory any more. Am I missing something?!

• I have a DotNetCoreCLI@2 test task like the following. The PublishCodeCoverageResults@2 seems to first publish ReportGenerator's report, which is later overwritten. How to fix this while still publishing the trx test results?

I guess that's not possible. See also your next question.

• My current production pipeline config is similar to the following. It does fulfil most of my requirements except that the contents of "Code Coverage" are replaced by an autogenerated report when the pipleine finishes. How to get that right?

This is also my experience. See my answers above. I always use PublishCodeCoverageResults@1 instead of PublishCodeCoverageResults@2.

[rikrak]

FWIW I would have benefitted from such a guide. It took me some time to understand that the PublishCodeCoverageResults@2 task was flawed, and the older version is preferred in my scenario (which was similar to the one described above). The main difficulty I faced in researching the production of a Devops Pipeline code coverage report, was that I didn't realise that a lot of the examples out in the wild (using the V2 task) are quite simplistic and don't cover what I'd consider a "real-world" scenario. In addition I thought any examples using the older V1 task were out-of-date :-/

[danielpalme]

@cremor
I updated the documentation here:
https://github.com/danielpalme/ReportGenerator/wiki/Integration#attention

I think if you don't care how the HTML report looks and if you don't need to set any additional ReportGenerator settings, then you can simply use PublishCodeCoverageResults@2 now, without using reportgenerator@5 at all, because ReportGenerator will still be used to read/convert and merge the coverage reports.

The new version has several disadvantages regarding the report in the Code Coverage tab within Azure DevOps

• No branch and method coverage
• No details page for each class

[danielpalme]

Also wish there was collapse all & filter options in the report.

That's because Azure DevOps blocks all JavaScript.
If you download the full report artifact, those features will become available.

[georg-jung]

Thanks for your detailed response!

PublishCodeCoverageResults@2 does not use ReportGenerator internally. If you have different information, I would be interested.

My example pipeline above with

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'release'
        publishTestResults: true
      displayName: dotnet test Release

    - task: PublishCodeCoverageResults@2
      displayName: 'Publish code coverage report'
      inputs:
        summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
        failIfCoverageEmpty: true

- and notably without any reportgenerator@5 task - leads to the following behaviour:

• As soon as the PublishCodeCoverageResults@2 task finished (but not the whole job/pipeline), the Code Coverage tab contains a ReportGenerator report. When specifying a license key using .netconfig it even contains a pro version report.
• At the same time, a "Code Coverage Report_<id>" artifact is created.
• As soon as the pipeline(/job?) finishes, the Code Coverage tab's content is replaced with the report format that can be seen in your screenshot.
• The tasks log also contain lines like "Initializing report builders for report types: HtmlInline_AzurePipelines" that sound quite ReportGenerator related.
• In the end, I can view the PublishCodeCoverageResults@2-style report in the Code Coverage tab or download the ReportGenerator report from artifacts.

=> Thus, a V2 disable.coverage.autogenerate: 'true' might be quite interesting.

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/dotnet-core-cli-v2?view=azure-pipelines reads

#publishTestResults: true # boolean. Optional. Use when command = test. Publish test results and code coverage. Default: true.

Not sure how this might affect the behaviour.

I changed my pipeline now like this and it works as exected. The Code Coverage tab does contain the ReportGenerator report even after the end of the job (and also still accounts for .netconfig).

    - task: DotNetCoreCLI@2
      inputs:
        command: test
        projects: '$(solution)'
        arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
        testRunTitle: 'release'
        publishTestResults: true
      displayName: dotnet test Release

    # https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension
    - task: reportgenerator@5
      displayName: '[Coverage] ReportGenerator'
      inputs:
        reports: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
        targetdir: '$(Agent.TempDirectory)/coveragereport'
        reporttypes: 'HtmlInline_AzurePipelines;Cobertura;Badges'
        assemblyfilters: '-xunit*'

    - task: PublishCodeCoverageResults@1
      displayName: '[Coverage] Publish results'
      inputs:
        codeCoverageTool: Cobertura
        summaryFileLocation: '$(Agent.TempDirectory)/coveragereport/Cobertura.xml'
        reportDirectory: '$(Agent.TempDirectory)/coveragereport'
      env:
        DISABLE_COVERAGE_AUTOGENERATE: 'true'

Thanks for pointing me to https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension. I actually overlooked this first (even though it's linked in the README 🙄 - I guess there are just many places providing pieces of information, some of which are more up to date while others aren't).

[danielpalme]

@MTomBosch
It's the richer report (Sample report)

[A-Rai-col]

I can now answer this. Sadly PR line coverage does not show up with reportgenerator@5 while it does work with PublishCodeCoverageResults@2.

Still, cheers for sharing that something like code coverage for pull requests was a thing. That would be a very useful feature indeed.
Hopefully, Microsoft eventually lets us publish ReportGenerator reports using PublishCodeCoverageResults@2 task which might have that feature built in.

Until then, thanks to @danielpalme we can now get rid of the deprecated warning that came with the V1 task. 🙌

[danielpalme]

=> Thus, a V2 disable.coverage.autogenerate: 'true' might be quite interesting.

There is no such option. I looked at the source code: https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/PublishCodeCoverageResultsV2

Internally PublishCodeCoverageResults@1 uses https://github.com/microsoft/azure-pipelines-coveragepublisher to create the coverage reports.

[danielpalme]

@rikrak
Yes there are several options.

But the overall process is documented here:
https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-vsts

The FAQs also answer the most common question:
https://github.com/danielpalme/ReportGenerator/wiki/FAQ#azure-devops-extension-seems-to-ignore-my-settings

[A-Rai-col]

Thanks @jchenathub24 this clears my doubts about PublishCodeCoverageResults@2 ,I faced the same issue when using V2 task as it seems to use reportgenerator internally; First it generates the HtmlInline_AzurePipelines report and publishes that to the Code Coverage tab then after a couple of seconds it repopulates the tab with the default ADO layout.

I'm having to resort to using PublishCodeCoverageResults@1 as mentioned in your Option1 cause the ReportGenerator report is just so much better. Also wish there was collapse all & filter options in the report.

[cremor]

@danielpalme I think this issue should be reopened because the ReportGenerator documentation regarding Azure DevOps tasks should be updated. A few weeks ago Azure DevOps pipeline builds that use V1 of the PublishCodeCoverageResults tasks started showing deprecation warnings. See also the announcement here: https://devblogs.microsoft.com/devops/new-pccr-task/

Important part in that announcement:

Q: I’m using Azure DevOps Server version 2022. What will happen to me?
A: In a future version of the Azure DevOps Server, we will also remove the V1 version of the publish code coverage results task. There will be a transitioning period where the V1 version will still run. During the transition period you’ll need to move from the V1 version to the V2 version, as when the transition period ends the V1 version will be removed and any pipeline containing that task will fail.

So there should be a documentation that shows how to correctly use ReportGenerator with PublishCodeCoverageResults V2.

I think if you don't care how the HTML report looks and if you don't need to set any additional ReportGenerator settings, then you can simply use PublishCodeCoverageResults@2 now, without using reportgenerator@5 at all, because ReportGenerator will still be used to read/convert and merge the coverage reports.

[MTomBosch]

When publishing the report "your way" is then still the html file regenerated from azure devops (which does not contain the branch and method coverage and details per class) or is your richer report uploaded?

[cremor]

The Azure DevOps task now has a new setting publishCodeCoverageResults.
This allows to publish the report in 'Code Coverage' tab directly.
If set to true, it makes the 'PublishCodeCoverageResults' task(s) obsolete. They can be removed from your pipeline.

Thanks!
Does this also work with code coverage line indicators in pull requests?
I'd test it myself but TBH the feature seems to be flaky even with the official tasks 😅

Documentation
Example when it works:

[danielpalme]

@cremor

Does this also work with code coverage line indicators in pull requests?

I don't know. I have not tested this. Would like to hear if this also appears in PRs.

[A-Rai-col]

The Azure DevOps task now has a new setting publishCodeCoverageResults.
This allows to publish the report in 'Code Coverage' tab directly.
If set to true, it makes the 'PublishCodeCoverageResults' task(s) obsolete. They can be removed from your pipeline.

Thanks! Does this also work with code coverage line indicators in pull requests? I'd test it myself but TBH the feature seems to be flaky even with the official tasks 😅

Example when it works:

Interesting.
In the ms docs

Which coverage tools and result formats can be used for validating code coverage in pull requests?
Code coverage for pull requests capability is currently only available for Visual Studio code coverage (.coverage) formats. This can be used if you publish code coverage using the Visual Studio Test task, the test verb of dotnet core task and the TRX option of the publish test results task. Support for other coverage tools and result formats will be added in future milestones.

They say it only supports .coverage formats though, so this will probably only work if you use the VSTest task.

[cremor]

They say it only supports .coverage formats though

That's wrong or outdated. The screenshot I posted above is from a project that doesn't do anything with a .coverage file. At least not explicitly.
Here is the pipeline the project currently uses:

  - task: DotNetCoreCLI@2
    displayName: 'Run tests'
    inputs:
      command: 'test'
      # Collect code coverage in OpenCover format because this is what SonarCloud supports.
      arguments: '--no-restore --configuration $(buildConfiguration) --collect:"XPlat Code Coverage;Format=opencover"'
      publishTestResults: true

  - task: PublishCodeCoverageResults@2
    displayName: 'Publish code coverage report'
    inputs:
      summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.opencover.xml'

I'm planning to replace PublishCodeCoverageResults@2 with reportgenerator@5.

[A-Rai-col]

  - task: PublishCodeCoverageResults@2
    displayName: 'Publish code coverage report'
    inputs:
      summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.opencover.xml'

I see, this might be an undisclosed feature they're adding the PublishCodeCoverageResults@2 task. 💭 As neither the V2 task docs nor the release notes for it, has any mention of code coverage for pull requests.

[cremor]

PublishCodeCoverageResults@2 uses https://github.com/microsoft/azure-pipelines-coveragepublisher (which uses ReportGenerator.Core).
reportgenerator@5 uses the same publishing method as PublishCodeCoverageResults@1 used. I couldn't figure out where the source code of that is.

[cremor]

I don't know. I have not tested this. Would like to hear if this also appears in PRs.

I can now answer this. Sadly PR line coverage does not show up with reportgenerator@5 while it does work with PublishCodeCoverageResults@2. I don't know how PublishCodeCoverageResults@2 does it but I assume it is something in https://github.com/microsoft/azure-pipelines-coveragepublisher

At least I know that it is not related to the files which are published as artifacts. Because currently PublishCodeCoverageResults@2 doesn't publish any artifacts at all. Although that might be a bug, see microsoft/azure-pipelines-tasks#19547. Along with many other active issues for PublishCodeCoverageResults@2. It seems to be quite bad 😁