microsoft/webvalidate

Web Validate (WebV) is a web request validation tool that can be used to run end-to-end tests and long-running smoke tests

Web Validate - A web request validation tool

Web Validate (WebV) is a web request validation tool that we use to run end-to-end tests and long-running performance and availability tests.

There are many web test tools available. The two main differences with WebV are:

• Integrates into a single pane of glass

  • WebV publishes json logs to stdout and stderr
  • WebV publishes the /metrics endpoint for Prometheus scraping
    • This allows you to build a single pane of glass that compares server errors with client errors
    • This also allows you to monitor applications on the edge with centralized logging which reduces edge network traffic significantly

• Deep validation of arbitrary result graphs

  • WebV is primarily designed for json API testing and can perform deep validation on arbitrary json graphs

• WebV is published as a nuget package and can be installed as a dotnet global tool

• WebV can also be run as a docker container

• If you have dotnet core sdk installed, running as a dotnet global tool is the simplest and fastest way to run WebV

WebV Quickstart

The easiest way to try WebV is to fork this repo and Open in Codespaces

WebV is already installed in this GitHub Codespace

Install WebV as a dotnet global tool

# this allows you to execute WebV from the shell
dotnet tool install -g webvalidate

Run a sample validation test against microsoft.com

# change to a directory with WebV test files in it
pushd src/app

# run a test
webv --server https://www.microsoft.com --files msft.json --verbose

Run more complex tests against the GitHub API by using:

# github tests
webv --server https://api.github.com --files github.json --verbose

Run a test that fails validation and causes a non-zero exit code

webv --server https://www.microsoft.com --files failOnValidationError.json --verbose-errors

Experiment with WebV

# get help
webv --help

# change back to the root of the repo
popd

WebV Quickstart (docker)

Run a sample validation test against microsoft.com

# run the tests from Docker
docker run -it --rm ghcr.io/cse-labs/webvalidate --server https://www.microsoft.com --files msft.json --verbose

Run more complex tests against the GitHub API by using:

# github tests
docker run -it --rm ghcr.io/cse-labs/webvalidate --server https://api.github.com --files github.json --verbose

Run a test that fails validation and causes a non-zero exit code

docker run -it --rm ghcr.io/cse-labs/webvalidate --server https://www.microsoft.com --files failOnValidationError.json --verbose-errors

Experiment with WebV

# get help
docker run -it --rm ghcr.io/cse-labs/webvalidate --help

In the above examples, the json files are included in the docker image

Use your own test files

# assuming you want to mount the current directory to the container's /app/TestFiles
# this will start bash so you can verify the mount worked correctly
docker run -it --rm -v $(pwd):/app/TestFiles --entrypoint bash ghcr.io/cse-labs/webvalidate

# run a test against a local web server running on port 8080 using ./myTest.json
docker run -it --rm -v $(pwd):/app/TestFiles --net=host  ghcr.io/cse-labs/webvalidate --server localhost:8080 --files myTest.json

Configuration

See Command Line Parameters for more details

• Web Validate uses both environment variables and command line options for configuration

  • Command flags take precedence over environment variables

• Web Validate works in two distinct modes

  • The default mode processes the input file(s) in sequential order one time and exits
  • The --run-loop mode runs in a continuous loop until stopped or for the specified duration

• Some environment variables and command flags are only valid if --run-loop is specified and WebV will exit and display usage information

• Some parameters have different default values depending on the mode of execution

Validation Files

See Validation Test Files for more details.

• WebV uses validation files to define what requests should be run as a part of the testing
  • Each line in the file details the request and the expected results to be validated by WebV
  • This can be as simple as validating the returned status code is 200 or as complex as checking each returned value in a nested json object or array

Integration with Application Monitoring

See Application Monitoring for more details.

• We use WebV to run geo-distributed tests against our Web APIs
  • These tests run 24 x 7 from multiple regions and provide insight into network latency / health as well as service status
  • The results integrate with our single pane of glass via log forwarding (Fluent Bit) and metrics (Prometheus)

By doing this, not only can we ensure against a large cloud bill, but we can track how usage and performance change over time, ensuring application functionality and performance through testing in production.

Running as part of an CI-CD pipeline

WebV will return a non-zero exit code (fail) under the following conditions

• Error parsing the test file(s)
• If an unhandled exception is thrown during a test
  • Please use GitHub Issues to report as a bug
• --max-errors is exceeded
  • To cause the test to fail on any validation error, set --max-errors 1 (default is 10)
• Any validation error on a test that has FailOnValidationError set to true

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit Microsoft Contributor License Agreement.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services.

Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines.

Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.

Any use of third-party trademarks or logos are subject to those third-party's policies.

webvalidate's Issues

Accept non-stringified request and response in webv test cases

Description

• A clear and concise description of what the bug is
  Currently, people need to convert request and response to a stringified format by escaping double quotes etc to make them work in webv

Expected Behavior

json request response should be accepted as it is in webv test cases without the need for them to be stringified.

Reproduce

Enter request body and response without stringifying them. webv throws error.

Add message to log when webvalidate stops execution due to default configuration of config.MaxErrors (10) being reached

Problem Statement

• When 100s of tests are executed due to the default value of config.MaxErrors (which seems to be 10) the tests stop execution after 10 failures. At this point there is no message in the log that webvalidate stopped execution due to the error threshold being breached

Proposed Solution

• Add a message to the log indicating that value of MaxErrors has been reached and hence further tests are not being executed

Additional Context

• Without this one cannot understand looking at pipeline logs what went wrong

Next Steps

• Team consensus to proceed
• Complete Design Review Template (if applicable)
• Schedule Design Session

Publish webvalidate container image to official microsoft registry mcr.microsoft.com

Problem Statement

• The current documentation at https://github.com/microsoft/webvalidate, to execute WebV from a docker container sources the images from docker hub retaildevcrew/webvalidate
  docker run -it --rm retaildevcrew/webvalidate --server https://www.microsoft.com --files msft.json

Proposed Solution

-It will be good if the WebV docker image is available from mcr.microsoft.com , which as per https://github.com/microsoft/containerregistry is " the primary Registry for all Microsoft Published docker images that offers a reliable and trustworthy delivery of container images "

Alternative Proposals

-The images can be pushed and made available from the Microsoft account on docker hub

Additional Context

Next Steps

• Team consensus to proceed
• Complete Design Review Template (if applicable)
• Schedule Design Session

Convert to dotnet 6

• Convert to dotnet 6
• Update packages
• Update docs as needed

Enable CodeQL action

Description

• CodeQL doesn't currently support dotnet 6
• Re-enable once dotnet 6 is supported

Add TestCaseName in Test definition and in test results

Problem Statement

• At present if one is to run multiple tests, the only way to easily relate on which tests ran successfully and which failed is to relate using the sequence and the urls/methods for tests. Which makes it a little less intuitive.

Proposed Solution

• Adding testcasename in the test definition and return the same along with the test results.

Next Steps

• Team consensus to proceed
• Complete Design Review Template (if applicable)
• Schedule Design Session

Provide option to publish webv report in Junit/xunit/nunit format which can be consumed in tools like Azure Pipelines

Problem Statement

• Provide option to publish webv report in Junit/xunit/nunit format which can be consumed in tools like Azure Pipelines

Proposed Solution

• Provide option to publish webv reports in one of the standard formats (Junit etc). This will allow the reports to be easily consumed in tools like Azure Pipelines
• Currently we use a custom script to publish webv report in Junit format, which appears as follows in the Azure Pipelines Dashboard
  
  

Next Steps

• Team consensus to proceed
• Complete Design Review Template (if applicable)
• Schedule Design Session

maxMilliSeconds is not working

Description

setting maxMilliSeconds is validation.
validation still True when duration > maxMilliSeconds

Expected Behavior

validation True when duration < maxMilliSeconds
validation False when duration > maxMilliSeconds

The json log output doesn't render the "Method/Verb"

Problem Statement

• At present if you run the default webvalidate tests - the tabbed output contains the HTTP Verb in it as shown below

2021-04-20T13:48:18.0703350Z https://www.microsoft.com 302 0 541 0gBxQOdCViU6KdxGwb+l/TQ.0 - 4 Static GET /

• While for the same tests with json log the output doesn't contain the HTTP Verb in it

{"date":"2021-04-20T13:42:59.9092105Z","server":"https://www.microsoft.com","statusCode":302,"failed":false,"validated":true,"correlationVector":"PLRsY/m1FEaJJ2ANKL6ehQ.0","errorCount":0,"duration":579,"contentLength":0,"category":"Static","quartile":4,"path":"/","errors":[]}

Proposed Solution

• It'll be helpful to bring the information at par in both kind of outputs and include the HTTP Verb in the JSON log as well.
  so it looks like this
  {"date":"2021-04-20T13:42:59.9092105Z","server":"https://www.microsoft.com","Method":"GET","statusCode":302,"failed":false,"validated":true,"correlationVector":"PLRsY/m1FEaJJ2ANKL6ehQ.0","errorCount":0,"duration":579,"contentLength":0,"category":"Static","quartile":4,"path":"/","errors":[]}

Readme.md Commandline Parameters --log-json is incorrect

Description

• In Readme.md file - the commandline Parameters section has this option --log-json which is incorrect

Expected Behavior

• The option doesn't work - the right option is --json-log

Reproduce

Run the command
dotnet run -- --server https://www.microsoft.com -f msft.json --log-json
File Not Found: --log-json
Invalid json in file: --log-json
2021-04-20T13:32:49.6672052Z https://www.microsoft.com 302 0 1097 0N0y/HbV350+fuzWle2w2FA.0 - 4 Static GET /
2021-04-20T13:32:50.0490469Z https://www.microsoft.com 200 0 357 207566 HiZCk7X5nkmfamoPDTjhmg.0 - 4 Static GET /en-us
2021-04-20T13:32:50.0752890Z https://www.microsoft.com 200 0 25 4793 H1mQVeAkVE+lSamMDqT/SA.0 - 1 Static GET /robots.txt
2021-04-20T13:32:50.0965871Z https://www.microsoft.com 200 0 21 17174 xmWxXL2hJEuFJlOmLUXIlg.0 - 1 Static GET /favicon.ico

Run it with right option
dotnet run -- --server https://www.microsoft.com -f msft.json --json-log
{"date":"2021-04-20T13:42:59.9092105Z","server":"https://www.microsoft.com","statusCode":302,"failed":false,"validated":true,"correlationVector":"PLRsY/m1FEaJJ2ANKL6ehQ.0","errorCount":0,"duration":579,"contentLength":0,"category":"Static","quartile":4,"path":"/","errors":[]}
{"date":"2021-04-20T13:43:00.3553432Z","server":"https://www.microsoft.com","statusCode":200,"failed":false,"validated":true,"correlationVector":"r2qzc2RF2UOsN3YXtkSI7A.0","errorCount":0,"duration":371,"contentLength":207553,"category":"Static","quartile":4,"path":"/en-us","errors":[]}
{"date":"2021-04-20T13:43:00.4223001Z","server":"https://www.microsoft.com","statusCode":200,"failed":false,"validated":true,"correlationVector":"5T236xLYPkmXQE8wqSisTA.0","errorCount":0,"duration":34,"contentLength":4793,"category":"Static","quartile":1,"path":"/robots.txt","errors":[]}
{"date":"2021-04-20T13:43:00.4848782Z","server":"https://www.microsoft.com","statusCode":200,"failed":false,"validated":true,"correlationVector":"Hgg\u002BY5lsB0WlNuJT/y2\u002BvA.0","errorCount":0,"duration":43,"contentLength":17174,"category":"Static","quartile":2,"path":"/favicon.ico","errors":[]}

Additional Context

• doc changes