linuxsuren/api-testing

YAML based API testing tool. 开源接口调试、测试工具。Mock Server

English | 中文

This is a API testing tool. 🚀

Features

• Supported protocols: HTTP, gRPC, tRPC
• Multiple test report formats: Markdown, HTML, PDF, Stdout
• Mock Server in simple configuration, and Open API support
• Support converting to JMeter files
• Response Body fields equation check or eval
• Validate the response body with JSON schema
• Pre and post handle with the API request
• Run in server mode, and provide the gRPC and HTTP endpoint
• VS Code extension support
• Multiple storage backends supported(Local, ORM Database, S3, Git, Etcd, etc.)
• HTTP API record
• Install in multiple use cases(CLI, Container, Native-Service, Operator, Helm, etc.)
• Monitoring integration with Prometheus, SkyWalking

Get started

Install it via hd or download from releases:

hd install atest

or, you can install it in Kubernetes. See also the manifests.

see the following usage:

API testing tool

Usage:
  atest [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  func        Print all the supported functions
  help        Help about any command
  json        Print the JSON schema of the test suites struct
  run         Run the test suite
  sample      Generate a sample test case YAML file
  server      Run as a server mode
  service     Install atest as a Linux service

Flags:
  -h, --help      help for atest
  -v, --version   version for atest

Use "atest [command] --help" for more information about a command.

below is an example of the usage, and you could see the report as well:

atest run -p sample/testsuite-gitlab.yaml --duration 1m --thread 3 --report md

API	Average	Max	Min	Count	Error
GET https://gitlab.com/api/v4/projects	1.152777167s	2.108680194s	814.928496ms	99	0
GET https://gitlab.com/api/v4/projects/45088772	840.761064ms	1.487285371s	492.583066ms	10	0
consume: 1m2.153686448s

Use in Docker

Use atest as server mode in Docker, then you could visit the UI from 8080:

docker run --pull always -p 8080:8080 ghcr.io/linuxsuren/api-testing:master

Use atest-collector in Docker:

docker run -p 1234:8080 -v /var/tmp:/var/tmp \
  ghcr.io/linuxsuren/api-testing atest-collector \
  --filter-path /api \
  -o /var/tmp/sample.yaml
# you could find the test cases file from /var/tmp/sample
# cat /var/tmp/sample

Template

The following fields are templated with sprig:

• API
• Request Body
• Request Header

Functions

You could use all the common functions which comes from sprig. Besides some specific functions are available:

Name	Usage
randomKubernetesName	{{randomKubernetesName}} to generate Kubernetes resource name randomly, the name will have 8 chars
sleep	{{sleep(1)}} in the pre and post request handle

Verify against Kubernetes

It could verify any kinds of Kubernetes resources. Please set the environment variables before using it:

• KUBERNETES_SERVER
• KUBERNETES_TOKEN

See also the example.

TODO

• Reduce the size of context.
• Support customized context.

Limit

• Only support to parse the response body when it's a map or array.

Community Exchange

Feel free to talk to us about any questions you may have about API Testing in the following ways.

Mailing List

[email protected], Feel free to discuss everything related to API Testing via this mailing list.

GitHub discussion

GitHub Discussion

api-testing's Issues

Support to print all supported expr functions

Similar to the Go template functions, it would be great if we could print all the supported functions which come from antonmedv/expr. See also:

• https://github.com/LinuxSuRen/api-testing/blob/master/cmd/function.go

• api-testing/pkg/server/remote_server.go

  Line 611 in 78f517c

  func (s *server) FunctionsQuery(ctx context.Context, in *SimpleQuery) (reply *Pairs, err error) {

• expr-lang/expr#413

Add step-by-step introductions for the UI

The step-by-step introductions are very friendly for the newcomers. We could provide the introductions for the basic functions.

References

• https://introjs.com/docs/

Feat: Add github action to detect broken connections

In order to avoid connection failure in the project, you need to add github action to detect.

Support loading test cases from IO stream

The test case files are might stored in a HTTP server. It would be good to support loading it from an URL in this case. See also the following example usage:

atest run -p http://foo/test.yaml

See also

api-testing/pkg/testing/loader_file.go

Line 29 in d564f69

data, err = os.ReadFile(l.paths[l.index])

Support converting test cases between atest and postman

We could add a new sub-command. Such as: atest convert --from postman --input example.json --to atest --output example.yaml.

In the future, we could consider supporting converting between any kind of data. And the atest format will play the middle role. It means A could to be converted to atest, then convert atest to your desired format.

References

• Postman examples
• JSON schema

Support sending test report to a HTTP server

Sometimes, we might want to send the test report to an HTTP server.

How to

It's easy to do this. Please implement the following interface first:

api-testing/pkg/runner/simple.go

Line 161 in 2520e80

type TestReporter interface {

then, add the corresponding CLI flag.

api-testing/cmd/run.go

Line 74 in 2520e80

flags.StringVarP(&opt.pattern, "pattern", "p", "test-suite-*.yaml",

Use cases

• Send test report to LinuxSuRen/gogit#35. So users could see the report from a Gitlab merge request.

Support a Windows MSI format of the package

See also https://github.com/LinuxSuRen/http-downloader/blob/c2c33156de7282ca439945ca05dc52782431e90f/.github/workflows/release.yaml#L33

Got error when running the sample grpc testsuite file

I got an error when trying the following command:

go run . run -p sample/grpc-sample.yaml

See the output:

sample/grpc-sample.yaml pattern 1
found suites: 1
routing end with 13.817028ms
API Average Max Min QPS Count Error
gRPC Runner.GetVersion 6.065096ms 6.065096ms 6.065096ms 0 1 0
gRPC Runner.FunctionsQuery 3.48314ms 3.48314ms 3.48314ms 0 1 0
gRPC Runner.FunctionsQueryStream 2.891849ms 2.891849ms 2.891849ms 0 1 1
consume: 14.054419ms
Error: failed to run 'FunctionsQueryStream', proto: syntax error (line 1:1): unexpected token [

By the way, I recommend adding the corresponding testing into the GitHub workflows. For instance, below might be a good place to add:

api-testing/.github/workflows/build.yaml

Line 39 in 3f2877e

- name: API Test

Support to generator code or commands base on the testcase

Sometimes, we might need to use the HTTP request in different cases. For instance, take the HTTP request as javascript code lines running in the browser. Or, use the curl command to run in Linux.

All the languages could send an HTTP request. But we could support some of the most popular ones:

• curl command
• JavaScript
• Golang
• Python
• Java

A very important thing that we need to consider, is how to make sure the generated code lines work well.

Add authentication support in the gRPC server

Currently, all clients could connect to the gRPC server with any auth. It's not safe for the production environment. We could simply support username/password or other auth method.

See also:

• https://github.com/LinuxSuRen/api-testing/blob/master/cmd/server.go
• https://grpc.io/docs/guides/auth/#extending-grpc-to-support-other-authentication-mechanisms
• https://github.com/grpc/grpc-go/blob/master/Documentation/grpc-auth-support.md

Support checking the license header

We could use the following project to do it.

https://github.com/apache/skywalking-eyes?tab=readme-ov-file#check-license-headers

By the way, I prefer to use the latest fixed version (for instance 0.4.0) instead of a branch.

Support download the extension files from OCI registry

Consider we could push any arbitrary files into an image registry (such as Docker hub). So, we could push the extension files into it, and then download it once it is required.

There are some benefits if we do this:

• The GitHub release is not anymore the only place for saving extension files.
• There are many image registries we could use. For instance, dockerhub, aliyun registry, quay.io

Below is an example of how to pull files from an OCI registry.

https://pkg.go.dev/oras.land/oras-go/v2#example-package-PullFilesFromRemoteRepository

Add websocket support

See also https://github.com/LinuxSuRen/gows

Create an operator of API Testing

Operator is a very popular way to install applications. Please feel free to help us to create an operator if you want to have a try.

References

• https://operatorhub.io/

Support to install atest in Docker and Kubernetes

atest could install as a Linux or macOS service according to the OS until v0.0.12. See also the following output. See the code lines of this sub-command

# atest service
Error: not support action: ''
Usage:
  atest service [flags]

Aliases:
  service, s

Flags:
  -a, --action string        The action of service, support actions: install, start, stop, restart, status
  -h, --help                 help for service
      --script-path string   The service script file path

atest could run in Docker or Kubernetes as well. So, it's good to support installing in Docker and Kubernetes. You could find the Kubernetes manifest files.

In order to specify the target environment, we could add a new flag --target. For example:

atest service install --target (os | docker | kubernetes)

Considering the compatibility, we could make the default value of --target to be os.

Support sending HTTP request through unix socket

Some services provide the HTTP through unix socket. For instance:

• Podman

Below is a sample code:

package main

import (
	"bufio"
	"fmt"
	"io"
	"net"
)

func main() {
	conn, err := net.Dial("unix", "/run/podman/podman.sock")
	if err != nil {
		panic(err)
	}

	fmt.Fprintf(conn, "GET /v4.0.0/libpod/secrets/name/json?showsecret=true HTTP/1.0\r\n\r\n")
	reader := bufio.NewReader(conn)

	var status string
	if status, err = reader.ReadString('\n'); err == nil {
		fmt.Println(status)
	} else {
		fmt.Println(err)
	}

	var header string
	if header, err = reader.ReadString('\n'); err == nil {
		fmt.Println(header)
	} else {
		fmt.Println(err)
	}

	var body []byte
	if body, err = io.ReadAll(reader); err == nil {
		fmt.Println(string(body))
	} else {
		fmt.Println(err)
	}
}

Support to generate a PDF report

I found some open-source projects below. Please see them if you're interested in it.

• https://github.com/jung-kurt/gofpdf

We could put the feature into a separate CLI which is only rendering a JSON to a PDF or HTML. Such as: atest run --report json | atest-pdf-report -

Not support request method "patch"

Add more samples

add more samples, like how to write post, put and delete test

Create a plugin of DevStream for installing API Testing

DevStream is a CNCF project to help users to install tools. Please freel free to create a plugin of it to install API Testing.

References

• https://docs.devstream.io/en/release-0.11/plugins/plugins-list/

Command "atest service action" is not support in CentOS

OS: CentOS 7.9
Version: 0.0.11
Install Mode: Install it via hd

I try the command atest service --action start|stop|restart|status, and print the error information. Like this:

Then I see the code in the programs is systemctl start atest, I tried to run it, and the result is :

Support gRPC test cases

gRPC is a very popular protocol. It would be great if we could have this feature. Please feel free to see also the following ideas if you're interested in it:

• Support all common feature of gRPC
• CLI and server side supports both needed
• VSCode extension support will be optinal
• Have a good compitable with the current YAML specification
• Docuemntation output

Online Meetings

• GLCC API Testing ~ gRPC 课题讨论会-1
• GLCC 活动中 API-Testing 项目的第二次沟通

Add secret managment plugin

See also
https://github.com/hashicorp/vault-examples/blob/main/examples/_quick-start/go/example.go

Create a JMeter plugin to import from API Testing

Reference

• https://jmeter.apache.org/usermanual/jmeter_tutorial.html
• https://mp.weixin.qq.com/s/aW-p4HxbqsyAcipZjygA6Q
• https://github.com/Bugazelle/hello-jmeter-plugin

Making the UI as an extension point as well

Now, we have serval extension points:

• storage
• secret

Actually, providing UI extension is also very important. Different protocals have different elements, it's better to load the corresponding page instead of putting all things together. For example, gRPC, HTTP, and webSocket might be very different with each other.

Suggested extension points might be:

• [UI] Store editing and creating page
• [UI] TestCase page

There are some potential solutions:

• SSR aka Server-Side Rendering

Support loading test cases from git repository

Expected usage: atest run -p https://github.com/LinuxSuRen/api-testing/tree/master/sample --auth personal-token

We need to consider the git branch and auth. Usually, the branch name could found from the URL. We could manipulate the git repository via jenkins-x/go-scm.

UI

In many cases, people will deploy a private Gitlab, and provide it to the team. Having an oauth integration might be a good idea for the users. They could login with their own account. So, we don't need to consider the user management.

See also the similar issue #94

Add a flag to support save report content to file

An example usage:

atest run -p sample.yaml --report-file report.md

This feature is very useful when users want to use it in a CI/CD system.

Replace the go model from custom to the official

It's not necessary to define the swagger related sturcts by ourself. We don't need to consider the details. Please see also the following links if you're interested in it.

• https://github.com/go-openapi
• https://github.com/LinuxSuRen/api-testing/blob/01da53f39e651a1178ee1d41033cf74990914c13/pkg/apispec/swagger.go

Add a mapping between Go and JMeter template functions

#170 could convert the API Testing file into a JMeter file. But there're some issues that we need to fix. For instance, they have different template functions. As I mentioned in the title, we could have a mapping. See below:

atest	JMeter
{{randAlpha 6}}	${__RandomString(6,abcdefghijksf,)}

Support run command set item name

I have a test yaml have many items, like create, list, update and delete. I want to only run create by item name.

command like

atest run -p test-suite-demo.yaml   --report md --name "create demo"

Support to install atest as a macOS service

We already support it on the Linux platform. See also here.

Please feel free to read the following references below if you're interested in this issue:

• About Daemons and Services

Support before/after at the test suite level

It could be similar with #69. See the following example:

name: Gitlab
before:
  items:
    - "waitPod('default-ns', 'app=gitlab', '2m')"
items:
- name: projects
  request:
    api: https://gitlab.com/api/v4/projects
  before:
    items:
      - "sleep(1)"

There are some desired functions needed:

• waitPod: wait until the pod is ready
• createK3d
• delK3d: delete a k3d cluster

References

• https://github.com/k3d-io/k3d

Support sending test report to a gRPC server

Sometimes, we might want to send the test report to a gRPC server.

How to

It's easy to do this. Please implement the following interface first:

api-testing/pkg/runner/simple.go

Line 161 in 2520e80

type TestReporter interface {

then, add the corresponding CLI flag.

api-testing/cmd/run.go

Line 74 in 2520e80

flags.StringVarP(&opt.pattern, "pattern", "p", "test-suite-*.yaml",

Similar issues

• #91

How to test

We could try to send the test report to etcd.

Docs: Readme file link is 404

https://github.com/LinuxSuRen/api-testing/blob/master/sample/kubernetes/manifest.yaml

Support set different image registry server

This project publishes images to the following container registries:

• docker.io
• ghcr.io

And, the following is kind of the proxy image address. So linuxsuren.docker.scarf.sh is an available container registry as well.

linuxsuren.docker.scarf.sh/linuxsuren/api-testing

You could add it to the following section

api-testing/helm/api-testing/values.yaml

Line 7 in dd7cd46

image:

Expect

Users could use the following flags to choose different container registry:

helm install atest oci://docker.io/linuxsuren/api-testing \
    --version image.registry=ghcr.io

Support setting the proxy for the test suite

Sometimes an API can only be accessed through a proxy. It would be good to read the proxy from env or a user-defined value.

Support to export the test records into the Prometheus

We could add an extension for putting test records into Prometheus.

Support start the proxy server on UI

Postman has a wonderful feature that could start an HTTP proxy server. This project already has the backend function. We just need to expose the service on the UI.

See also

Support to take the user-defined test report template

Currently, all the test reports are embed in the code lines. But it's quite normal that uesrs want a new test report style. A simple solution is that allowing pass a user-defined template.

A suggestion is adding a flag to the following command:

atest run --report-template (my-template.md | my-template.html | https://foo/my-template.html)

References

• api-testing/pkg/runner/writer_markdown.go

  Line 33 in 02d7e23

  var markdownReport string

• api-testing/pkg/runner/writer_html.go

  Line 33 in 02d7e23

  var htmlReport string

Feature: Request for Executing an API Request after Another API Request and Supporting Delayed Execution

大佬您好：
我希望请求在特定的API端点上增加延迟功能，某个api 在另外一个api之后进行，并运行延迟一段时间（比如某个 api先create,然后多一段时间执行另外一个api请求）。这个功能将极大地提升处理API调用的灵活性和控制性。

目前，在向API端点发送请求后需要引入延迟，以模拟特定情景或确保与外部系统的同步。然而，现有的实现缺乏将这种延迟纳入考虑的能力。

提议的解决方案是扩展API测试框架，以在每个API端点上包含可配置的延迟选项。此外，支持后续执行操作将非常有益，允许用户定义在请求发送后立即执行的特定任务。

整合这些功能将使用户能够创建更全面的测试场景，包括涉及延迟和后续操作的情景，最终提高框架的整体测试能力。

感谢您考虑这个请求。比如 增加after:xxApi; delay;1minute; 表示该api 在xxApi 1分钟后执行

Support to install atest as a Windows service

We already support it on the Linux platform. See also here.

Please feel free to read the following references below if you're interested in this issue:

• How to Run Any Program as a Background Service in Windows

Add OCI-based helm support

This is great. And we don't need a web server for the Helm repository. Please feel free to help on this if you're interested in this.

References

• https://helm.sh/blog/storing-charts-in-oci/

Provide the UI page of gRPC testing

Thanks to @Ink-33, this project has the gRPC feature. But the web UI is still very important. Please see also the following ideas if anyone is interested in it:

• Having the similar UI style of the HTTP feature

We will have a release after this issue.

References

• #81

Support to generate JSON format report

Expect usage: atest run --report json

api-testing/pkg/runner/writer_std.go

Line 45 in 7680987

func (w *markdownResultWriter) Output(result []ReportResult) (err error) {

Provide a gRPC endpoint to getting the version

Currently, we only have one endpoint for running the test cases. The VS Code extension could communicate with it. It would be great if the VS Code side could get the version and prompt users if they upgrade atest according to the version comparison.

Please feel free to read the following references before trying to fix this issue:

• https://github.com/LinuxSuRen/api-testing/blob/master/pkg/server/server.proto
• https://grpc.io/docs/languages/go/quickstart/

Support to generate a report for the APIs test results

For example, we can generate a report looks like this:

| API | Test result|
|---|---|
| `/v1alpha1/users` | `pass` |

Improve the response body validation

See the following

https://github.com/LinuxSuRen/api-testing/blob/581b6527ae1243e543608ca3ad98f4022ceff166/pkg/runner/http.go#L345C46-L345C53

We could use gjson to get JSON value.

Support GraphQL testCase

GraphQL is also a popular API spec. It would be good to support it.

Adoption

See the GraphQL users:

• https://docs.github.com/en/graphql
• https://docs.gitlab.com/ee/api/graphql/index.html
• https://graphql.org/users/

References

• https://graphql.org/learn/

Support converting test cases between atest and curl command

See also the similar issue #101

The dark theme is hard to view

This project is using element-plus.