genetec/azure-devops-extension-publishtestresultscreenshot

Azure DevOps extension used in CI pipeline to upload screenshot in test results report.

PublishTestResultScreenshot for Azure DevOps

DEPRECATED: This Azure DevOps extension is no longer maintained. An alternative solution is to use the Publish Test Result task which now supports file attachments.

Get more insight on your UI test failure with screenshots.

Installation

You can get the task from the Visual Studio MarketPlace.

How to use

Requirements

• Capture your screenshot on test failure using the following path automatedTestStorage/automatedTestName (see json response example)
  • Currently only support .png images
• The task must be placed after the PublishTestResult task in your pipeline
• If you enabled Fail build on test failure on PublishTestResult task preceding PublishTestResultScreenshot, you will need to also enable the Continue on error to make sure PublishTestResultScreenshot is not skipped.

Compatibility

Version 0.1.x is compatible with:

• Azure DevOps Service Cloud
• Test type: JUnit
• Microsoft-hosted agents: macOS 10.14 & 10.13, ubuntu16.04, win2019

NOTE that emulators are currently only available on macOS agents, so for Android UI Testing you will be limited to 10.14 or 10.13, but the task is not dependent on macOS agents.

Configure

Available inputs:

Required

• organization: specify your organizationName (can be found from devops instance url http://dev.azure.com/{yourOrg})

Optional

• screenshotFolder: you can change where the task looks for screenshot, defaults to "./app/build/reports/androidTests/connected/screenshots/failures/"

Using classic editor

Using YAML

- task: Genetec.publish-test-result-screenshot.pipeline-extension.PublishTestResultScreenshot@0
  displayName: 'Publish test result screenshot'
  inputs:
    organization: {yourOrganizationName}
    screenshotFolder: {yourCustomPath}

Specifications

This task uses azure-devops-node-api which, under the hood, uses the Azure DevOps REST Api.

Task Results

• Succeeded - All the screenshots were successfully uploaded
• SucceededWithIssues (2 self explained possibles messages)
  • Some/All screenshots upload failed AND/OR
  • Some/All screenshots files were missing
• Skipped - No test failures were found
• Error - Usually related to authentification, connection, networkError etc.

How to build

Prerequisites

You will need to install nvm, and npm with node to run this project.

• Install nvm
• Install node (npm is bundled)
  • nvm install node
  • nvm use node
• Install Typescript
  • npm install typescript -g

Verify versions

I used the latest version but just make sure that all commands return a version number.

• nvm --version
• npm --version
• node --version

Getting started

Clone the repo

git clone [email protected]:Genetec/azure-devops-extension-publishtestresultscreenshot.git

Setup for local use

1. Uncomment the header of index.ts and set input values

// **********************************************************************************
// ************************* FOR LOCAL USE ******************************************
// process.env.ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN = ""
// process.env.SYSTEM_TEAMPROJECT = ""
// process.env.INPUT_ORGANIZATION = ""
// process.env.INPUT_SCREENSHOTFOLDER = ""
// process.env.BUILD_BUILDID = ""
// **********************************************************************************

2. Assign your access token to ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN

Compile the typescript into javascript

• cd PublishTestResultScreenshotV0
• npm install
• tsc

Execute the task locally

• node index.js

Deploy and test on DevOps

Before submitting your PR, make sure to test your changes on your DevOps instance using the following command:

tfx build tasks upload --overwrite --task-path "myPathToTheTask" --service-url "https://myOrg.visualstudio.com/DefaultCollection" --auth-type pat -t "xxx"

The PAT (personal access token) requires only the Manage Agent Pool permission.

How to contribute

Check Contribution Guide

License

Published under Apache 2.0 License

azure-devops-extension-publishtestresultscreenshot's Issues

Migration to Node 10

This task uses Node 6 execution handler, which will be removed March 31st 2022

Please migrate this addon to Node 10.

On-prem support

I noticed your blurb that 0.1 only works in the cloud.

Is there any plan to make it compatible with on site installs? We are on Azure DevOps Server 2019.

Issues block mapping and emulator image

Good evening,
I would like to thank you in advance and I would position this as a starter question. I refer to the medium blog entry and I tried to solve it already by myself with your documentation.

I have integrated a UnitTest into my current project. Now I would be very happy if I could use your program in Azure. You can find the YAML file in the appendix. First I had the following problem when I didn't use your Python script but your "Screenshot Task". Encountered error(s) while parsing pipeline YAML:
/azure-pipelinesUITest.yml: (Line: 47, Col: 1, Idx: 2029) - (Line: 47, Col: 2, Idx: 2030): While parsing a block mapping, did not find expected key.

Line 47 is the beggining of the task.

continueOnError: true

If I exclude this task there is another problem with downloading and installing the emulator image.

Thank you so much!

Add support for Release pipelines

Agent pool: Hosted Azure Pipeline, release pipelines
Agent specification: ubuntu
Context: Pyhon Selene tests
Test result format: jUnit, generated by unittest-xml-reporting

Describe the bug
The task fails with a single error message "Cannot read property 'length' of null"

To Reproduce
Steps to reproduce the behavior:

1. Add the task right after "Publish Test Results" task
2. Set the name of the organisation, one word of eight characters.
3. Set the path to "$(System.DefaultWorkingDirectory)/test_results".
4. Run the release pipeline

Expected behavior
The png files in "$(System.DefaultWorkingDirectory)/test_results" are present in the test run. As advertised.

Unfortunately I couldn't find anything more detailed than the task log:

2020-09-04T16:03:16.6261975Z ##[section]Starting: Publish screenshot
2020-09-04T16:03:16.6269610Z ==============================================================================
2020-09-04T16:03:16.6270109Z Task         : Publish screenshots for test failure
2020-09-04T16:03:16.6270626Z Description  : Get more insight on your test failures with contextual screenshots.
2020-09-04T16:03:16.6271037Z Version      : 0.1.0
2020-09-04T16:03:16.6271341Z Author       : Genetec
2020-09-04T16:03:16.6272436Z Help         : https://github.com/Genetec/azure-devops-extension-publishtestresultscreenshot/issues
2020-09-04T16:03:16.6273040Z ==============================================================================
2020-09-04T16:03:17.4087552Z ##[error]Cannot read property 'length' of null
2020-09-04T16:03:17.4127763Z ##[section]Finishing: Publish screenshot

Incorrect import FS

There is a compile error regarding an unsupported import syntax with ES6

index.ts:13:8 - error TS1192: Module '"fs"' has no default export.
13 import fs from "fs";
          ~~
Found 1 error.

Allow multiple test outcomes to have screenshots attached

Is your feature request related to a problem? Please describe.
Allowing users to specify what outcomes they would like screenshots attached for would be terrific. For example, if users are generating screenshots on success, or even ad hoc during test execution, it would be great to be able to add those as attachments.

Describe the solution you'd like
Add an input for outcome type

Describe alternatives you've considered
Grabbing all test outcomes and iterating through them until there is a match between the screenshot and the test

Additional context
I implemented something similar using a modified version of the python script in the original article. I'll add that below, with additional comments. I'd be more than happy to work on implementing this solution, but I'd need some time to make sure I wasn't breaking anything, as I haven't worked on creating custom devops extensions before.

# Get runIds for current build 
# (we execute our tests against multiple flavors, so multiple runs in a single build)
api = f'{baseUrl}/test/runs?buildUri={buildUri}&api-version=5.1'
response = requests.get(api, headers=header)
print("GET", api, "-->", response.status_code)
json_response = json.loads(response.text)['value']
runId = []
for test_result in json_response:
    if "UI Tests" in test_result['name']:
        runId.append(test_result['id'])
# Get results for current build, as an array of [runId, urlWithoutOutcome]
getResultsApi = []
for run in runId:
    getResultsApi.append([run, f'{baseUrl}/test/runs/{run}/results?api-version=5.1&outcomes='])

# Re-usable function to upload image attachment
def uploadAttachment(imgPath, resultId, testName, runId):
    # Only execute if we can find the image
    if os.path.isfile(imgPath):
        print("Screenshot exists, will upload: ", imgPath)
        with open(imgPath, "rb") as image_file:
            encoded_string = base64.b64encode(image_file.read())
            api = f'{baseUrl}/test/runs/{runId}/results/{resultId}/attachments?api-version=5.1-preview.1'
            data = {'stream': encoded_string.decode('utf-8'), 'fileName': f'{testName}.png',
                    'comment': 'Uploaded by REST from pipeline', 'attachmentType': 'GeneralAttachment'}
            response = requests.post(api, headers=header, json=data)
            print("POST", api, "-->", response.status_code)
    else:
        print("Screenshot does not exist, will not upload")
        print("Image Path is: ", imgPath)

for result in getResultsApi:
    # Get failed tests and upload screenshot
    apiFailures = f'{result[1]}3'
    responseFailures = requests.get(apiFailures, headers=header)
    print("GET", apiFailures, "-->", responseFailures.status_code)
    failures = json.loads(responseFailures.text)['value']
    print("failures is:\n", str(failures))
    for failedTest in failures:
        resultId = failedTest['id']
        className = failedTest['automatedTestStorage']
        testName = failedTest['testCaseTitle']
        # Custom image path
        imgPath = f"{screenshotsDir}/failures/{className}/{testName}.png"
        print('imgpath = ' + imgPath)
        uploadAttachment(imgPath, resultId, testName, result[0])

    # Get successful tests and upload screenshots
    apiSuccesses = f'{result[1]}2'
    responseSuccesses = requests.get(apiSuccesses, headers=header)
    print("GET", apiSuccesses, "-->", responseSuccesses.status_code)
    successes = json.loads(responseSuccesses.text)['value']
    print("successes is:\n", str(successes))
    for successfulTest in successes:
        resultId = successfulTest['id']
        className = successfulTest['automatedTestStorage']
        testName = successfulTest['testCaseTitle']
        # Custom image path
        imgPath = f"{screenshotsDir}/successes/{className}/{testName}.png"
        uploadAttachment(imgPath, resultId, testName, result[0])

(as an aside, i'm looking for a solution to add my ad hoc screenshots to the script)

Add support for parallel test stages

Agent pool: Private Pool
Agent specification: macOS
Context: Android
Test result format: jUnit

Describe the bug
Parallel test stages with publish test results screenshot task on each stage attaches the screenshot multiple times.
For example, the current # of stages that run in parallel is 3 and all of them have publish test screenshot task. Each also has publish test result task.
Stage to finish first (let's call this test_stage0) will try to attach screenshot to its failed tests. Let's say 4 tests failed on this one. So it will try to attach 4 screenshots. This stage will succeed.
Then, the stage to finish second (test_stage1) has the same failed tests as test_stage_0. The publish test result screenshot will detect 8 failed tests here including failed tests of test_stage0 since widget gets results by BuildId. Now it will attach the screenshots to the failed tests of both test_stage0 and test_stage1 since they have the same filename for the failed tests. test_stage0 now has 2 screenshots attach for each failed test.
Same case will happen if stage to finish last (test_stage2) has the same failed tests as test_stage0 and test_stage1. Screenshot for test_stage2 will also be added to test_stage0 and test_stage1.

Total # of screenshots attached to each test failure on each test_stage:
test_stage0 = 3
test_stage1 = 2
test_stage2 = 1

To Reproduce
Steps to reproduce the behavior:

1. Setup a pipeline with 3 parallel stages. All should have publish test result and publish test screenshot and all should run the same test scripts but on different devices.
2. Run the pipeline.

Expected behavior
Screenshot should only attach once to each failed step for every stage.

Noticed that test results are being fetched by BuildId. Perhaps we can also consider the test runId from PublishTestResults task since this is also part of the stage.

Add support for configurable screenshot filename

Agent pool: Private pool
Agent specification: macOS
Context: Android - codeceptjs-appium
Test result format: jUnit (generated by using mocha-junit-reporter, mocha-multi)

Describe the bug
Publish test result screenshot task completes but with a warning that says

1 tests failed. Will proceed with screenshot upload.
##[warning]All screenshots were missing. 
Task completed. Published 0/1 screenshots

To Reproduce
Steps to reproduce the behavior:

1. Add task right after 'Publish Test Results' task
2. Provide organization
3. Provide screenshot folder
4. Run build pipeline
   **Part of yaml being used by the pipeline related to publishing of test results

    ...
    - task: PublishTestResults@2
      inputs:
        testResultsFormat: 'JUnit'
        testResultsFiles: '**/TEST-*.xml'
        searchFolder: '$(System.DefaultWorkingDirectory)/output'
        publishRunAttachments: true
        # failTaskOnFailedTests: true
      condition: always()
    - task: Genetec.publish-test-result-screenshot.pipeline-extension.PublishTestResultScreenshot@0
      displayName: 'Publish test result screenshot'
      inputs:
        organization: retailshelf
        screenshotFolder: $(System.DefaultWorkingDirectory)/output/

Sample artifact for test results:

Expected behavior
It should be able to locate and publish appropriate screenshot for each failed test.

Plugin Not working

I configured my plugin to run on vsts
I've set the organization name and an existing folder containing png files
Result: ##[error]Cannot read property 'value' of null

[error] read ECONNRESET occurs (rarely reproduced)

Agent pool: [Private pool]
Agent specification: [ubuntu]
Context: [Web]
Test result format: [jUnit]

Describe the bug
Sometimes task fails with the error (rarely reproduced):
##[error]read ECONNRESET

To Reproduce
Steps to reproduce the behavior:

1. Launch tests.
2. Wait till execution is finished.
3. Observe test results.

• stage: Build
  jobs:
  • job: TestRun
    timeoutInMinutes: ${{variables.timeout}}
    pool: BUILD-LINUX
    steps:
    • task: DockerCmd@7
      displayName: 'Test execution' #Custom task
      continueOnError: true
      inputs:
      dockerfile: 'Dockerfile'
      dockerImage: 'myImage'
      imageSource: 'Dockerfile'
      dockerRunArgs: --ipc=host --memory=6g --shm-size=1g
      dockerCmd: |
      xvfb-run --auto-servernum npm run test ${{parameters.suite}}
      /bin/bash -c "if [[ -d "/e2e/results/screenshotsGlobalSetup" ]]; then aws s3 cp /e2e/results/screenshotsGlobalSetup s3://myBucket --recursive; fi"
      dockerCmdExecuteAll: true
      mountCustomVolume: true
      externalMountLocation: '$(Build.ArtifactStagingDirectory)/results/'
      internalMountLocation: '/e2e/results'
    • task: Bash@3
      displayName: 'Fail build if no test run'
      inputs:
      targetType: 'inline'
      script: if [[ ! -d "$(Build.ArtifactStagingDirectory)/results/jest-junit-report" ]]; then exit 1; fi
    • task: PublishTestResults@2
      displayName: 'Publish test results'
      inputs:
      searchFolder: '$(Build.ArtifactStagingDirectory)/results/jest-junit-report'
    • task: Genetec.publish-test-result-screenshot.pipeline-extension.PublishTestResultScreenshot@0
      displayName: 'Publish test result screenshots'
      inputs:
      organization: myorg
      screenshotFolder: '$(Build.ArtifactStagingDirectory)/results/screenshots'

Expected behavior
No error occurs.

Adding Support for Azure Devops On Premises

Dear Developer Team,
would it be possible to release this extension for on premise maschines? Or is there a way to achieve this?
I would love to use the functionality of your extension for our enviroment to optimize our testing!

Best regards!

Custom domain not supported

Agent pool: [Private pool]
Agent specification: [windows]
Context: [ Web]
Test result format: [jUnit]

Describe the bug
It's not possible to use the task screenshots when using custom domain e.g.
https://custom.visualstudio.com instead of https://dev.azure.com

To Reproduce
Steps to reproduce the behavior:

1. Add the task Publish screenshots for test failure to the pipeline
2. Add custom domain in the Organization field e.g. https://custom.visualstudio.com/{organization}
3. Run the pipeline
   

Publish screenshot with different name from testname

Hi!
I would like to upload .png files with random name, but task search only files with same name.