Number of Test for TypeScript is not showing in Sonar

[pablissimo]

Number of Test for TypeScript is not showing in Sonar

[iazim]

Hi,
Any update on the enhancement for the unit test numbers?
Thanks

[pyrocks]

Hi,

This seems related to my problem in a way - if not I can open a new issue.
My project is written in TypeScript and uses karma which runs mocha and Istanbul + junit reporters to generate the coverage reports (coverage.json, lcov.info, HTML report) and a junit-style test report (test-results.xml).

Your plugin works great with the code coverage (lcov.info) but no matter what I do I can't seem to have the test-results.xml be recognized by sonarqube. In the scan log I don't see it even try to read the xml.

I tried to use the following properties - but with no luck:
sonar.junit.reportsPath
sonar.surefire.reportsPath
sonar.javascript.karmajstestdriver.reportsPath
sonar.genericcoverage.reportPaths

Is it because it's not supported by your plugin, or I'm doing something wrong?
Thanks in advance,
Mor

[Pablissimo]

Happy to use this issue to track it, unsure when I'll get to it (though pull requests obviously welcomed!).

[marklagendijk]

Issue I created before I found this one (#82):

Java projects in Sonar have the following metrics, which this plugin does not yet provide:

(Number of) Unit Tests
Unit Test Errors
Unit Test Failures
Skipped Unit Tests
Unit Test Success (%)
Unit Test Duration

It would be nice if this plugin would also support these.
My guess is that for Java projects it uses JUnit test execution reports to get these metrics. We could add this functionality to this plugin in the same way as has been done for LCOV.
Users could generate JUnit test reports. There is a Karma reporter for this.

So I agree that JUnit format would be the way to go. I think it is more widely supported than the Sonar specific format.

[Pablissimo]

It's not something that's currently imported via the plugin, and getting it in there may be quite specific to the runner being used to execute the tests (and what reporting it supports). Happy to take suggestions on formats to accept here.

We are currently using Junit reporting. Using sonar.javascript.karmajstestdriver.reportsPath for importing the same for JavaScript. But same property does not work for TypeScript.

[Pablissimo]

Will probably focus on that one first, as Chutzpah will also output JUnit-formatted test summaries and it's a really easy format to handle.

Thanks and it will be good to have the report imported to sonarqube. Also do you recommend any ts rules other than the ones come with the plugin?

[Pablissimo]

There is a way I think to do it at the minute with different plugins - they've removed JUnit support from the normal JavaScript plugin, preferring instead to use the 'Generic Coverage Plugin' and requiring that your build process transform the JUnit XML into its standardised format.

If you're using Grunt there's a note at the bottom of this page on an NPM package to do the conversion, though you could probably do it with an XSLT as well. Still less pleasant than the plugin doing it for you but may get you out of a hole.

[Pablissimo]

Sorry - also, no there aren't any extra rules I tend to use although the Microsoft ones are interesting:

https://www.npmjs.com/package/tslint-microsoft-contrib

We are generating the Junit XML file and using the sonar.javascript.jstestdriver.reportsPath to import the report. The report is not showing up because the sonar.language=ts ? We are generating the JUnit report for TypeScript language.

[Pablissimo]

The plugin doesn't currently support pulling in test results, only coverage - I need to extend it to pull in details of test runs. I'm not sure whether you can use another plugin to pull that in against files managed by SonarTsPlugin, never tried but doesn't sound good from your testing!

[pyrocks]

Hi,

It would be nice if you could extend the plugin to support it...

With that being said - after a week of experimenting I finally made some breakthrough - the thing I missed was to actually install the "sonar-karma-test-report-plugin" :|
So now with sonar.javascript.karmajstestdriver.reportsPath the scan finally shows some parsing information but I still get some errors - but I guess it's for the other plug-in's git :)

Cheers,
Mor

This is really great information. sonar.javascript.karmajstestdriver.reportsPath works fine for Java script and all other languages. When used against the type script it is not importing the XML. The XML file do exist in the test_results folder but still unable to import it. Is it because of the source mappings in the XML file?

On Nov 22, 2016, at 5:16 AM, pyrocks [email protected] wrote:

sonar.javascript.karmajstestdriver.reportsPath

[pyrocks]

Hi,

I am not sure exactly regarding the reason - the scan log is not meaningful to me at this stage.
Here are some snippets - maybe you'll make more sense of it.

Snippet from test-result.xml:

<?xml version="1.0"?>
<testsuite name="PhantomJS 2.1.1 (Linux 0.0.0)" package="" timestamp="2016-11-22T13:13:30" id="0" hostname="hostname" tests="75" errors="0" failures="0" time="0.664">
  <properties>
    <property name="browser.fullName" value="Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1"/>
  </properties>
  <testcase name="bills components BillItem Should have columns for date, orderId, price &amp; view link" time="0.02" classname="PhantomJS_2_1_1_(Linux_0_0_0).bills"/>
  <testcase name="bills components BillList Should have table headers for date, order, price" time="0.009" classname="PhantomJS_2_1_1_(Linux_0_0_0).bills"/>
  <testcase name="bills components BillList Should have header table body with BillItems" time="0.002" classname="PhantomJS_2_1_1_(Linux_0_0_0).bills"/>
...
<testcase name="services components Header By default 3 buttons exist" time="0.001" classname="PhantomJS_2_1_1_(Linux_0_0_0).services"/>
  <system-out>
    <![CDATA[
]]>
  </system-out>
  <system-err/>
</testsuite>

A snippet from the scan log:

...
16:01:12.594 INFO: Sensor KarmaJunitReporterJsTestDriverSensor
16:01:12.594 INFO: Parsing Unit Test run results in Surefire format from folder /build/build_reports/test-results
16:01:12.594 DEBUG: test directory prefix is tests/
16:01:12.595 DEBUG: Attempting to determine suite name from "tests/index.js"
16:01:12.595 DEBUG: Removed "tests/" test directory prefix from relativePathKey
16:01:12.595 DEBUG: Found test file: "tests/index.js" with relativePathKey "index.js"
16:01:12.596 DEBUG: Trying to get suite name from "//"
16:01:12.596 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.596 DEBUG: Trying to get suite name from "//  require.context is a special webpack function"
16:01:12.596 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.596 DEBUG: Trying to get suite name from "//  it allows us to dynamically decide which files are considered dependency of this file and therefore will"
16:01:12.596 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.596 DEBUG: Trying to get suite name from "//  be processed by webpack"
16:01:12.597 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.597 DEBUG: Trying to get suite name from "//  In our case, we should search for all app/**/*.spec.ts files"
16:01:12.597 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.597 DEBUG: Trying to get suite name from "//  However since we want to generate a code coverag report we includes all application TS[x] files"
16:01:12.597 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.597 DEBUG: Trying to get suite name from "//  that's way a TS file that was not covered by our tests is still visible inside coverage report"
16:01:12.597 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.597 DEBUG: Trying to get suite name from "//"
16:01:12.597 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.597 DEBUG: Trying to get suite name from "const testsContext = require.context('../app', true, /\.ts[x]?$/)"
16:01:12.598 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.598 DEBUG: Trying to get suite name from "testsContext.keys().forEach(testsContext);"
16:01:12.598 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.598 WARN: Suite name could not be determined for "tests/index.js"
16:01:12.598 DEBUG: Attempting to determine suite name from "tests/int-polyfil.js"
16:01:12.598 DEBUG: Removed "tests/" test directory prefix from relativePathKey
16:01:12.598 DEBUG: Found test file: "tests/int-polyfil.js" with relativePathKey "int-polyfil.js"
16:01:12.599 DEBUG: Trying to get suite name from "!function(e){function n(r){if(t[r])return t[r].exports;var a=t[r]={exports:{},id:r,loaded:!1};return... -->a very long line which I truncated
...
16:01:12.601 DEBUG: Regex could not find suite name: ^(describe\((\'|\")(.*)(\'|\").*$)
16:01:12.601 WARN: Suite name could not be determined for "tests/int-polyfil.js"
16:01:12.712 WARN: Test result will not be saved for test class "PhantomJS_2_1_1_(Linux_0_0_0).bills", because SonarQube associated resource has not been found in suite-to-file map using key "bills".
16:01:12.712 WARN: Test result will not be saved for test class "PhantomJS_2_1_1_(Linux_0_0_0).services", because SonarQube associated resource has not been found in suite-to-file map using key "services".
16:01:12.712 WARN: Test result will not be saved for test class "PhantomJS_2_1_1_(Linux_0_0_0).(Component) AppButton", because SonarQube associated resource has not been found in suite-to-file map using key "(Component) AppButton".
16:01:12.712 WARN: Test result will not be saved for test class "PhantomJS_2_1_1_(Linux_0_0_0).login", because SonarQube associated resource has not been found in suite-to-file map using key "login".
16:01:12.712 WARN: Test result will not be saved for test class "PhantomJS_2_1_1_(Linux_0_0_0).(Component) InputGroup", because SonarQube associated resource has not been found in suite-to-file map using key "(Component) InputGroup".
16:01:12.712 WARN: Test result will not be saved for test class "PhantomJS_2_1_1_(Linux_0_0_0).(Component) Header", because SonarQube associated resource has not been found in suite-to-file map using key "(Component) Header".
16:01:12.712 INFO: Sensor KarmaJunitReporterJsTestDriverSensor (done) | time=118ms

[pyrocks]

Hi @Pablissimo,

Can you mark this as an enhancement request (and perhaps change the title) to enable loading a junit style report? I think the most common report format is the Surefire XML Format - so similar to Java's sonar.junit.reportsPath we would have sonar.ts.reportsPath ?

Since the code is TypeScript - if there is no Java/JavaScript file in the project - none of the other plugins are relevant (if there's no Java file in the project, sonar.junit.reportsPath is ignored for instance).

This will help me a lot.
If needed - I can open a new Issue for it...

Thanks,
Mor

[Pablissimo]

Cool - another vote is good. I'm currently doing a bit of refactoring to get things in a better shape for this sort of work, so once that's done will start looking at this.

[pyrocks]

I suggest you change the title of this Issue to reflect the enhancement request... (and avoiding future duplicate ones).

[nixel2007]

Notice:
Sonar also takes very simple generic test results
http://docs.sonarqube.org/plugins/servlet/mobile#content/view/1441839
If you gonna to implement something on plugin side - this can be the easiest way

[alexrun]

Is there any work done on this feature? @Pablissimo

[Pablissimo]

None yet, nope

[PascalGugenberger]

Any news here? We'd also love to have unit test reports (# of tests passed, skipped, duration) in Sonar.

[Pablissimo]

No sadly not had time to address any of this yet (or indeed a couple of outstanding defects on the plugin).

[krystan]

This thread just saved me from continuing trying to make this work, I take it as the issue is still open, this still an issue

[squelix]

I had the same problem and now i'm using karma-sonarqube-unit-reporter, which generate a Generic test file for Sonar.
You can now pass the file through sonar.testExecutionReportPaths and you can now see all the tests in Sonar.

[shrutigv]

@vilchik-elena
@stas Vilchik stas-vilchik
@squelix

/karma.config file/
sonarQubeUnitReporter: {
sonarQubeVersion: 'LATEST',
outputFile: './.tmp/sonarQube/coverage/test-report.xml',
useBrowserName: false
},
reporters: ['progress', 'coverage', 'sonarqubeUnit']

/sonar.project.properties file/

sonar.projectKey=Philips:UC_PAL_IS
sonar.projectName=Philips_rtc_sdk
sonar.projectVersion=1.8.0
sonar.ts.tslintconfigpath=tslint.json
sonar.sources=src
sonar.tests=src
sonar.sourceEncoding=UTF-8
sonar.exclusions=**/node_modules/**, **/dependencies/**/*.js,**/release/*,**/release/**/*.js
sonar.test.inclusions=src/**/**/*.spec.ts,src/**/**/**/*.spec.ts,src/core/classes/baseapi.spec.ts,src/core/classes/conference.spec.ts, src/core/classes/conferencemanager.spec.ts ,src/core/classes/event.spec.ts ,src/core/classes/participant.spec.ts ,src/core/classes/philipsrtcapi.spec.ts ,src/core/common/orchestration-service.spec.ts ,src/core/common/consolelogprovider.spec.ts ,src/core/constants/index.spec.ts,src/providers/fm/classes/fmavapi.spec.ts,src/providers/fm/core/fmbaseapi.spec.ts,src/providers/twilio/twilio.spec.ts,src/providers/fm/features/fmpresenceapi.spec.ts
sonar.typescript.lcov.reportPaths=.tmp/sonarQube/coverage/lcov.info
sonar.testExecutionReportPaths=.tmp/sonarQube/coverage/test-report.xml

//Project structure

project source dir
| - .tmp
| karma.config
| src
| sonar.project.properties

Sonar execution runs successfully and generates report but i dont see the unit test coverage report. Please help.

xml

<testExecutions version="1">
  <file path="src/core/classes/baseapi.spec.ts">
    <testCase name="Async testing for baseapi Token - Async test for token to be defined" duration="3741"/>
  </file>
  <file path="src/core/classes/conference.spec.ts">
   <testCase name="Async testing for conference Token - Async test for token to be defined" duration="2406"/>
  </file>
</testExecutions>

Not sure if am missing something. PLease help

[dmberton]

I am having the exact same issue as @shrutigv with a similar configuration.
Running the scanner locally, the output is as follow:

INFO: Sensor Generic Test Executions Report
WARN: Property 'sonar.genericcoverage.unitTestReportPaths' is deprecated. Please use 'sonar.testExecutionReportPaths' instead.
INFO: Parsing ########################################\coverage\unit-test-report.xml
INFO: Imported test execution data for 0 files
INFO: Test execution data ignored for 43 unknown files, including:
AboutOfficeCardComponent
AboutComponent
AppComponent
ArchiveComponent
ArticleComponent
INFO: Sensor Generic Test Executions Report (done) | time=93ms

(Characters ### are just obfuscation of my local path.)

My unit-test-report.xml generated by karma-sonarqube-unit-reporter looks like this:

<testExecutions version="1"> <file path="AboutOfficeCardComponent"> <testCase name="AboutOfficeCardComponent should create" duration="226"/> <testCase name="AboutOfficeCardComponent should not equate the injected service instance with the service provider" duration="105"/> <testCase name="AboutOfficeCardComponent should equate the injected service instance with the TestBed instance" duration="93"/> <testCase name="AboutOfficeCardComponent should equate the injected service instanc with the instance from fixture.debugElement.injector.get()" duration="92"/> <testCase name="AboutOfficeCardComponent should have expected imageUrl" duration="82"/> </file> <file path="AboutComponent"> <testCase name="AboutComponent should create" duration="99"/> </file> .... </testExecutions>

[shrutigv]

@dmberton xml path seems to be wrong.. Point it to right file path.