webdriverio-boneyard/wdio-sumologic-reporter

A WebdriverIO v4 plugin. Reporter that uploads results to Sumo Logic for statistical analyses

WDIO Sumologic reporter

A WebdriverIO plugin to report test results to Sumologic for data analyses. It helps you to find flaky tests over time using Sumologics data analyses features.

Installation

The easiest way is to keep wdio-sumologic-reporter as a devDependency in your package.json.

{
  "devDependencies": {
    "wdio-sumologic-reporter": "~0.0.1"
  }
}

You can simple do it by:

$ npm install wdio-sumologic-reporter --save-dev

Instructions on how to install WebdriverIO can be found here.

Configuration

First we have to create a new collector that collects all logs of your tests. To do that click on Manage in the navigation bar and go to Collection. There you need to add a new "Hosted Collector". Apply a suited name, e.g. "test integration logs", description and a category, e.g. "wdio". Click on Save to create the collector.

Next step is to add a source. It makes sense to have an own source for each of your environment (e.g. branch build, integration). Click on the "Add Source" link next to your collector and add an HTTP Source. Apply again a suiteable name and description and set a "Source Category" that reflects the environment. Leave the other options in default state and click on save.

A modal pops up with the source endpoint. Copy that url and paste it into your wdio.conf.js so the reporter know where to send the data.

Following code shows the default wdio test runner configuration. Just add 'sumologic' as reporter to the array and add your source endpoint:

// wdio.conf.js
module.exports = {
  // ...
  reporters: ['spec', 'sumologic'],
  reporterOptions: {
    sumologic: {    
      // define sync interval how often logs get pushed to Sumologic
      syncInterval: 100,
      // endpoint of collector source
      sourceAddress: process.env.SUMO_SOURCE_ADDRESS
    }
  },
  // ...
};

After running the first tests with the reporter you should be able to check out the tests logs with the following query:

_source=wdio
| parse "\"type\":\"*:*\"" as type,status
| json auto

I will provide some useful dashboard templates for Sumologic soon.

---

For more information on WebdriverIO see the homepage.

wdio-sumologic-reporter's Issues

Sumologic reporter doesn't consistently transactionize all tests

The problem

Sumologic reporter is not consistently transactionizing all tests - some blocks within a given spec will get all three events, where others do not.

Environment

• WebdriverIO version: 4.2.8
• Node.js version: 6.3.0
• Standalone mode or wdio testrunner: testrunner
• if wdio testrunner, running synchronous or asynchronous tests: sync
• Additional wdio packages used (if applicable): mocha, spec/dot, sumologic

Details

each block within a spec is determined to have a test:start, test:(result), and test:end - not all blocks within a spec are consistently receiving this event tag. after running a spec several times, i see that some blocks are getting only 1 or 2 of the expected 3 test state events.

Link to Selenium/WebdriverIO logs

spec reporter output (consistently 5 tests reported)

[Chrome #0a] Session ID: 8d49a74edc2e4cbb0da6a8f88f862ad894282cfd
[Chrome #0a] Spec: /Users/jterry/huge.amc-webdriverio/test/specs/homeHeroCarouselLarge_Test.js
[Chrome #0a] Running: Chrome
[Chrome #0a]
[Chrome #0a] AMC Homepage Hero Carousel - Full
[Chrome #0a] ✓ Carousel Next is visible at >1260
[Chrome #0a] ✓ Carousel Previous is visible at >1260
[Chrome #0a] ✓ Carousel Next works
[Chrome #0a] ✓ Carousel Previous works
[Chrome #0a] ✓ Carousel Buttons work
[Chrome #0a]
[Chrome #0a]
[Chrome #0a] 5 passing (33s)
[Chrome #0a]

here is a gist that should hopefully just point out something silly i shouldn't do:
https://gist.github.com/jonyet/da3b9859a26de94ba48dd33ac5e93afd

gist with the csv output from sumo:
https://gist.github.com/jonyet/089eda1a0e2e00e69cc0a34b38d07b9f

you can see in the csv that groups 3, 4, and 5 are the examples of the issue.

relevant block from wdio.conf.js:

  framework: 'mocha',
  reporters: ['spec', 'sumologic'],
    reporterOptions: {
        allure: {
            outputDir: './test/reports/allure-results/'
        },
        sumologic: {
          syncInterval: 100,
          sourceAddress: process.env.SUMO_SOURCE_ADDRESS
        }
    },

Code To Reproduce Issue [ Good To Have ]

will work on reproducing this with an example you can use, yourself

Please remember that, with sample code; it's easier to reproduce bug and much faster to fix it.