datadog/synthetics-ci-github-action

Use Browser and API tests in your CI/CD with Datadog Continuous Testing

Overview

Trigger Synthetic tests from your GitHub workflows with the Datadog CI Synthetics command.

Setup

To get started:

1. Add your Datadog API and Application Keys as secrets to your GitHub repository. For more information, see API and Application Keys.
2. In your GitHub workflow, use DataDog/synthetics-ci-github-action.

Your workflow can be simple or complex.

Simple workflows

Example workflow using public IDs

name: Run Synthetic tests using the test public IDs
jobs:
  e2e_testing:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Datadog Synthetic tests
        uses: DataDog/[email protected]
        with:
          api_key: ${{secrets.DD_API_KEY}}
          app_key: ${{secrets.DD_APP_KEY}}
          public_ids: 'abc-d3f-ghi, jkl-mn0-pqr'

Example workflow using an existing synthetics.json file

name: Run Synthetic tests using an existing synthetics.json file
jobs:
  e2e_testing:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Datadog Synthetic tests
        uses: DataDog/[email protected]
        with:
          api_key: ${{secrets.DD_API_KEY}}
          app_key: ${{secrets.DD_APP_KEY}}

For an example test file, see this test.synthetics.json file.

Note: By default, this workflow runs all the tests listed in {,!(node_modules)/**/}*.synthetics.json files (every file ending with .synthetics.json except for those in the node_modules folder). You can also trigger a list of Synthetic tests by specifying a public_id or using a search query.

Complex workflows

Example workflow using the test_search_query

name: Run Synthetic tests by test tag
jobs:
  e2e_testing:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Datadog Synthetic tests
        uses: DataDog/[email protected]
        with:
          api_key: ${{secrets.DD_API_KEY}}
          app_key: ${{secrets.DD_APP_KEY}}
          test_search_query: 'tag:e2e-tests'

Example workflow using a test search query and variable overrides

name: Run Synthetic tests using search query
jobs:
  e2e_testing:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Datadog Synthetic tests
        uses: DataDog/[email protected]
        with:
          api_key: ${{secrets.DD_API_KEY}}
          app_key: ${{secrets.DD_APP_KEY}}
          test_search_query: 'tag:staging'
          variables: 'START_URL=https://staging.website.com,PASSWORD=stagingpassword'

Example workflow using a global configuration override with config_path

This GitHub Action overrides the path to the global datadog-ci.config.json file.

name: Run Synthetic tests with custom config
jobs:
  e2e_testing:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Datadog Synthetic tests
        uses: DataDog/[email protected]
        with:
          api_key: ${{secrets.DD_API_KEY}}
          app_key: ${{secrets.DD_APP_KEY}}
          config_path: './synthetics-config.json'

For an example test file, see this global.config.json file.

Inputs

Name	Type	Requirement	Description
api_key	string	required	Your Datadog API key. This key is created by your Datadog organization and should be stored as a secret. Default: none.
app_key	string	required	Your Datadog Application key. This key is created by your Datadog organization and should be stored as a secret. Default: none.
public_ids	string	optional	Comma-separated list of public IDs for Synthetic tests you want to trigger. If no value is provided, the action looks for files named with synthetics.json. Default: none.
test_search_query	string	optional	Trigger tests corresponding to a search query. Default: none.
subdomain	string	optional	The name of the custom subdomain set to access your Datadog application. If the URL used to access Datadog is myorg.datadoghq.com, the subdomain value needs to be set to myorg. Default: app.
files	string	optional	Glob pattern to detect Synthetic test configuration files. Default: {,!(node_modules)/**/}*.synthetics.json.
datadog_site	string	optional	The Datadog site to send data to. Default: datadoghq.com.
config_path	string	optional	The global JSON configuration to be used when launching tests. See the example configuration file for more details. Default: datadog-ci.json.
variables	string	optional	Comma-separated list of global variables to use for Synthetic tests. For example: START_URL=https://example.org,MY_VARIABLE=My title. Default: [].
junit_report	string	optional	The filename for a JUnit report if you want to generate one. Default: none.
tunnel	boolean	optional	Use the Continuous Testing Tunnel to execute your test batch. Default: false.
polling_timeout	number	optional	The duration (in milliseconds) after which the action stops polling for test results. At the CI level, test results completed after this duration are considered failed. Default: 30 minutes.
fail_on_critical_errors	boolean	optional	Fail the CI job if no tests were triggered, or results could not be fetched from Datadog. Default: false.
fail_on_missing_tests	boolean	optional	Fail the CI job if at least one specified test with a public ID (using public_ids or listed in a test file) is missing in a run (for example, if it has been deleted programmatically or on the Datadog site). Default: false.
fail_on_timeout	boolean	optional	Fail the CI job if at least one test exceeds the default test timeout. Default: true.

Contributing

See CONTRIBUTING.md

Further reading

Additional helpful documentation, links, and articles:

• Continuous Testing and CI/CD Configuration
• Best practices for continuous testing with Datadog

synthetics-ci-github-action's Issues

Test results aren't fetched

Hi,
I am using DataDog/[email protected] for triggering the datadog test. The workflow fails as the test has failed because of timeout and upon clicking Results_URL the below error message appears on the Datadog app page

"The CI result you are trying to access cannot be found.
It might not have been received yet. Please retry in a few moments."

The tests are continuously failing with time out issue. Can you please help with this?

Some issue after latest release?

I'm not sure if this in on my end yet, but our pipelines started with this error this morning.

 Unable to resolve actions. Repository not found: datadog/synthetics-ci-github-action.

I notice you have a new release, maybe the releases got messed up somehow? We were on 0.3.0, but switched to 0.4.0 to try and fix but didn't work.

      - name: Run Datadog Synthetics tests (Conformance test UK and US pre-prod)
        uses: DataDog/[email protected]
        with:
          api_key: ${{ secrets.DD_API_KEY }}
          app_key: ${{ secrets.DD_APP_KEY }}
          public_ids: 'omitted'

Action Deprecation Warning

Every time my test runs in GHA I get this output:

(node:2203) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)

Don't know how serious a deprecation / security issue it is, but it would be nice not to see it.

unable to use test_search_query with multiple tags

The example given only provide 1 tag, tried to follow the syntax of 'key1:value1,key2:value2,....' for multiple tags which seems does not work.

Question: Synthetic tests in Datadog pass but CI report from the action fails due to a timeout

Hi there!
I'm currently using DataDog/[email protected] to run synthetics tests in a Github CI/CD workflow.
Some of the tests report a failure due to a timeout - what's strange is that the tests actual show as "Passed" in the Datadog dashboard.
wondering if there is a way to configure the result reporting timeout or if there is a limit we have to respect. Thanks in advance!

Failed run in CI due to result timeout

The same run in the Datadog dashboard

Synthetic tests in the datadog passes but the workflow using the datadog github action fails

Hi ,

I have been using DataDog/[email protected] for triggering the datadog tests from github action workflow. The datadog test passes but the workflow fails.
I saw there has been a similar issue reported already - #103.
The workflow fails with "failed: 1" and not because of timeout. However in the datadog it shows status as timed out. Is there any fix for this ?

polling_timeout does not seem to get picked up by action

Hi,

I am using Datadog synthetics action version 0.12.0 and added polling_timeout to my action to increase the timeout from 30 min to 60 min before action fail, in a script like so:

name: Run Datadog Synthetics tests
        uses: DataDog/[email protected]
        with:
          api_key: ${{secrets.DD_API_KEY}}
          app_key: ${{secrets.DD_APP_KEY}}
          public_ids: 'foo, bar'
          polling_timeout: 3600000

However the action still fails after 30 minutes (see example run below)

did I do something wrong or can it be a bug?

Failed Synthetic tests doesn't fail the Github action step

Config:

      - name: Run Datadog Synthetic tests
        uses: DataDog/[email protected]
        with:
           api_key: ${{secrets.SYNTHETIC_DD_API_KEY}}
           app_key: ${{secrets.SYNTHETIC_DD_APP_KEY}}
           public_ids: 'xxx-xxx-xxx'

Output:

View pending summary in Datadog: https://app.datadoghq.com/synthetics/explorer/ci?batchResultId=907abcf1-9843-4002-8001-1b4385ce01ac

- Waiting for 1 test result (jqh-uv5-fqx)…

  Waiting for 1 test result (jqh-uv5-fqx)…


=== REPORT ===
Took 290941ms

✖ [non-blocking] [jqh-uv5-fqx] [Browser] US XYZ staging UI Preview - location: N. California (AWS) - device: chrome.laptop_large
  ⎋ Total duration: 70705 ms - View test run details: https://app.datadoghq.com/synthetics/details/jqh-uv5-fqx/result/1032458531758739575?from_ci=true 
    ✓ | 3422ms - Navigate to start URL
    https://xya.com/xxx
    ✓ | 1309ms - Type text on input #user_password
    {{ USER_PASSWORD }}
    ✓ | 1339ms - Click on input "commit"
    ✓ | 4635ms - Navigate to link
    https://xyz.com/xxxxx
    ✖ | 60000ms - Click on span #ember14555555
    Error: Cannot locate element.

View full summary in Datadog: https://app.datadoghq.com/synthetics/explorer/ci?batchResultId=907abcf1-9843-4002-8001-1b4[38](https://github.com/xyz/xyz-app/actions/runs/6052540194/job/16426220784?pr=53512#step:3:39)5ce01ac

Continuous Testing Summary:
Test Results: 0 passed, 0 failed, 1 failed (non-blocking)
Max parallelization configured: 1 test running at the same time
Total Duration: 4m 51s

Increase your parallelization to reduce your total duration: https://app.datadoghq.com/synthetics/settings/continuous-testing


Datadog Synthetics tests succeeded: criticalErrors: 0, passed: 0, failedNonBlocking: 1, failed: 0, skipped: 0, notFound: 0, timedOut: 0
Results URL: https://app.datadoghq.com/synthetics/explorer/ci?batchResultId=907abcf1-9843-[40](https://github.com/xyz/xyz-app/actions/runs/6052540194/job/16426220784?pr=53512#step:3:41)02-8001-1b[43](https://github.com/xyz/xyz-app/actions/runs/6052540194/job/16426220784?pr=53512#step:3:44)85ce01ac
##[debug]Node Action run completed with exit code 0
##[debug]Finishing: Run Datadog Synthetic tests

Expected:
Since synthetic tests failed, we would expect it to exit with code 1 so it can fail the GHA.

##[debug]Node Action run completed with exit code 1

Question: How to specify multiple test search query values

Hello,
I am trying to pass multiple search query values in the action but datadog seems to be not able use it properly to find the test.

jobs:
  test:
    timeout-minutes: 60
    runs-on: xxxx
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Datadog Synthetic tests
        uses: DataDog/[email protected]
        with:
          api_key: ${{secrets.DATADOG_API_KEY}}
          app_key: ${{secrets.DATADOG_SERVICE_ACCT_APP_KEY}}
          test_search_query: 'tag:app:cdq'  # <= this works !
          #test_search_query: 'type:browser env:devl team:my-team tag:app:cdq'  <= this doesn't work
          #test_search_query: 'type:browser, env:devl, team:my-team, tag:app:cdq'  <= this doesn't work

Node.js 16 actions are deprecated: datadog/synthetics-ci-github-action

Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: datadog/[email protected]. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.

Log error even if it's not explicitly a ci error

Running the action now, on failure that is not a CiError, it will not report anything other than
Error: Running Datadog Synthetics tests failed.
This is due to this block in main.ts only reporting on synthetics.CiError.

  } catch (error) {
    if (error instanceof synthetics.CiError) {
      reportCiError(error, reporter)
    }
    core.setFailed('Running Datadog Synthetics tests failed.')
  }

It would be better if the actual error was also logged, i.e using core.info(error) or something

How to use START_URL in workflow?

Have been struggling to get a synthetic triggered with a different start url as described in the docs.

Adding this to my synthetics.yml file triggers the correct test but it only runs on the start url defined in the Synthetic UI, not the one provided in variables:

name: Run Synthetic tests using search query
jobs:
  e2e_testing:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Datadog Synthetic tests
        uses: DataDog/[email protected]
        with:
          api_key: ${{secrets.DD_API_KEY}}
          app_key: ${{secrets.DD_APP_KEY}}
          test_search_query: 'tag:staging'
          variables: 'START_URL=https://mywebsite.com'  // this has no effect

I can trigger it via postman using the Datadog api but I'd appreciate a nudge in the right direction with this one 🙏

Thank you!

unable to add synthetics-config.json file

I'm attempting to add a synthetics-config.json file to overwrite "failOnTimeout": false. I followed the example provided in the Datadog Synthetics CI action.

Here is the code snippet from the deploy.yml file which lives in ./.github/workflows/deploy.yml

e2e_testing:
    name: Datadog Synthetics Tests
    runs-on: ubuntu-latest
    needs: [setup, integration, unit-test, buildNext, buildDesign, buildLegacy]
    steps:
      - name: Run Datadog Synthetics tests
        uses: DataDog/[email protected]
        with:
          api_key: ${{secrets.DATADOG_API_KEY}}
          app_key: ${{secrets.DATADOG_APP_KEY}}
          config_path: ./.github/workflows/synthetics-config.json
          public_ids: 'et8-nd3-dmm, yxg-99h-tnv, 4b3-y2u-jk9, reh-i3i-pw5'

When the action runs, I keep running into the following error: Error: Unable to parse config file! Please verify config path : ./.github/datadog-config/synthetics-config.json

Where am I supposed to place the synthetics-config.json file in my repository and how should I reference it from the config_path key in the action description? Thank you!

Feature Request: Better support for retries

I want to use this action in my deployment pipeline to prevent unsafe pushes, the idea being we will run all synthetics with the tag blocks_prod_deploy before a code push and verify they pass before proceeding. However, I'm running into issues from flaky synthetics. Sometimes this action will fail due to a synthetic "timing out", and the linked test run will show that the first run failed but a second configured fast retry succeeded. Example:

Could we add some better handling of retries? Some ideas:

• The action respects the retry configuration of the synthetic and retries as needed before reporting failure.
• The action takes in an input letting the user choose how long to wait for each synthetic.
• The action reports the IDs of the failed synthetics as an output so the user can retry them as needed.

Right now, with none of these options implemented, I have to retry all synthetics, which is time-consuming.

Expose results via outputs

It would be extremely helpful if the data shown in the action's logs could also be set as outputs for accessibility in other parts of a workflow. For example, adding the report as a comment on a PR, extracting links and appending to a PR, external notification of pass/fail status.

I do not know the entire structure of synthetic tests, but based on this run as an example, I would see the following outputs being useful:

• result: succeeded|failed
• resultsUrl: https://app.datadoghq.com/synthetics/explorer/ci?batchResultId=1eba11d9-e0f8-43c0-ba2c-dfce4f4c0dab
• criticalErrors: int
• passed: int
• failedNonBlocking: int
• failed: int
• skipped: int
• notFound: int
• timedOut: int
• report: string (the entire string from === REPORT == and below)
• testRuns: array (output as a json string)

[
   {
      "passed":2,
      "failed":0,
      "batchUrl":"https://app.datadoghq.com/synthetics/explorer/ci?batchResultId=1eba11d9-e0f8-43c0-ba2c-dfce4f4c0dab",
      "runs":[
         {
            "id":"2r9-q7u-4nn",
            "detailsUrl":"https://app.datadoghq.com/synthetics/details/2r9-q7u-4nn/result/3868113174340701979?from_ci=true",
            "device":"chrome.laptop_large",
            "duration":1204,
            "location":"Paris (AWS)",
            "name":"[DO_NOT_DELETE] Used in `datadog-ci` repo for CI testing",
            "result":"passed|failed"
         }
      ]
   }
]

Pass variables to Synthetics from workflow

Would it be possible to pass a list of variables to Synthetics from the workflow without having to define a config file?
Thx

Question: Can I run with the `--tunnel` flag?

Can I run with the --tunnel flag? If yes, how?
Thank you!

Buffer is deprecated error on version 0.9.2

Run DataDog/[email protected]
(node:1714) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

Getting this error when running the tests without any extra variables:

      - name: Run Datadog Synthetics tests
        uses: DataDog/[email protected]
        with:
          api_key: $DATADOG_API_KEY
          app_key: $DATADOG_APP_KEY
          test_search_query: 'tag:ci-test'

Add failOnCriticalErrors and failOnMissingTests as inputs

Hi! Right now if I follow your first example, the job passes even if authorization fails due to failOnCriticalErrors defaulting to false. Could you add an input to this action to allow me to set failOnCriticalErrors to true? Adding failOnMissingTests would also be beneficial, since that also is defaulted to false and I would very much like the job to fail if someone accidentally deleted the synthetic!

Thanks!

Add documentation link for the format of a *.synthetics.json file

I could not find the specification anywhere. It would be really helpful to link out or spec out the format that the file should be in.

I would love to be able to record it with the datadog interface or chrome extension and dump out to a synthetics.json file.

Thanks!

Global variable ids don't exist

Hi there! We're having some issues that we can't quite wrap our heads around. Test in all repos fail with this error (Global variable ids don't exist --

Run DataDog/[email protected]
(node:3592) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
[2ut-s7t-fs8] Found test "[Int] [omitted]" (1 config override)


- Waiting for 1 test result (2ut-s7t-fs8)…

  Waiting for 1 test result (2ut-s7t-fs8)…


 ERROR: unable to trigger tests
[2ut-s7t-fs8] Failed to trigger tests: query on https://intake.synthetics.datadoghq.com/api/v1/synthetics/tests/trigger/ci returned: "Global variable ids don't exist: "

None of the tests was changes recently, so we can't figure out what's wrong. Is this a DD server-side issue?