freenet-public/got-swag

:monkey_face: A tool to test Swagger-powered APIs automatically through monkey testing and custom test suites

Got Swag?

A tool to test Swagger-powered APIs automatically through monkey testing. Also allows for custom tests written directly in Swagger files or in separate test suites. Includes command-line and programmatic interfaces. Install via npm install got-swag -g.

Usage

got-swag <url> ... [-m] [-t <ms>] [-T] [-v] [-w]
  Test a Swagger URL or file (YAML). Additional files are merged.

Options:
  -m, --monkey        Run monkey tests on GET endpoints
  -l, --monkey-limit  Maximum number of parameter combinations for each
                      monkey GET, default is 50
  -t, --timeout <ms>  Set a timeout (in milliseconds) for test step execution,
                      default is 2000 ms
  -T, --trace         Trace: Log requests and responses
  -V, --version       Show version
  -w, --watch         Watch the Swagger files and rerun tests on changes

Most Mocha options are valid. See https://mochajs.org/#usage for details.

Monkey Testing

The most basic usage of got-swag is monkey testing: Each GET endpoint of a service is validated using minimal variable input, if any, and the definitions from the services' Swagger file. The endpoints are requested with random authentication/variable combinations until one combination leads to a response status code less than 400.

Just invoke got-swag on a URL with the -m switch:

got-swag http://petstore.swagger.io/v2/swagger.json -m

See monkeystore.yaml for an example of input variables.

Custom Tests

Additionally, got-swag allows to embed custom tests in Swagger files or separate test suites. The test steps are written in JS using a small domain-specific language. Every step is evaluated, even if a previous step failed.

For example, see petstore.yaml (embedded) and yoda.yaml (separate).

Test Syntax Reference

Assertions

• ok( actual )
• equal( actual, expected )
• notEqual( actual, expected )
• deepEqual( actual, expected )
• notDeepEqual( actual, expected )
• strictEqual( actual, expected )
• notStrictEqual( actual, expected )
• deepStrictEqual( actual, expected )
• notDeepStrictEqual( actual, expected )
• match( actualString, expectedPattern )

Validation

• validate( data, schema )
  • Validate JSON data against a JSON schema
  • If data or schema are omitted (strictly equal to undefined), the last response is validated against the current operation's response schema

Requests

• request( options )
  • Requests the current endpoint
  • options is optional, see http
  • options.data sets the request body
• Shortcuts:
  • get( url, headers )
  • post( url, data, headers )
  • put( url, data, headers )
  • delete( url, headers )
  • options( url, headers )
  • head( url, headers )
  • Use null for url to request the current endpoint
  • headers are optional

Authentication

• auth( securityDefinitionId, credentials, scopes )
  • Authenticates against a security definition
  • scopes are optional and inferred from the API if possible

Utility

• encodeURIComponent( s ) encodes a string for URI transmission
• log( value ) logs a value
• stringify( value ) alias of JSON.stringify
• parse( string ) alias of JSON.parse
• byteLength( string ) alias of Buffer.byteLength for computing 'Content-Length' header manually
• monkeyAuth() tries to authenticate using known method/credentials
• monkeyGet() tries to GET using known parameters

Variables

• vars: Variables reusable for all tests
  • You can write to vars in test steps, see example
• req: Last request data
• res: Last response data
  • res.statusCode: Integer response status code
  • res.headers: Response headers
  • res.body: String response body
  • res.json: Parsed JSON response, if any
• api: Complete Swagger API

Extension Syntax

You can define extension Swagger files on top of existing Swagger files using the '#/path': value syntax. For reference, see extended-petstore.yaml.

Programmatic Usage

var gotSwag = require( 'got-swag' );

// test api and report as JSON
gotSwag.test( 'swag.yaml', 'vars.yaml' ).then( function ( report ) {
  console.log( report );
} );

// describe mocha tests in current suite
describe( 'My test suite', function () {
  gotSwag.describe( 'swag.yaml', 'vars.yaml', { parent: this } );
} );

Notes

• Currently, got-swag only supports JSON
• The DSL is sandboxed using vm
• If you see something like .../node_modules/got-swag/lib/validate.js:24 throw new Error( result.errors ); in your console, it's a Node.js Bug

got-swag's Issues

Add support for OAuth 2.0 implicit/authorization code flow

Support would be nice. However I'm unsure if this is generally possible without human interaction, as login/consent is often application-specific.

make logging output configurable

The Standard Output of a (successful) monkeytest does not neccessarily need the full list of all requests tested. Sommetimes you only need a simple OK (no http4** or http5** Status found).

On other hands in case of on error there might be the need of an explicit log level.
So it would be nice to have configurable log levels.

validate() does not work without request context

Make it work stand-alone plz.

Forward argv to mocha

... in order to enable other report formats etc.

Feature request: expose byteLength in sandbox

First of all, this library rocks!!!

I'm having a use case on which I need to calculate the Content-Length manually.

Buffer.byteLenth would be the solution :)

I played with the source code, exposed it as parse or stringify methods and it works like a charm.
If I make a pull request would you include it?

Thanks in advance

Not able to install got-swag

Following is the error log. Please help

---

udo npm install got-swag -g .
npm ERR! addLocal Could not install .
npm ERR! Error: ENOENT, open 'package.json'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues

npm ERR! System Linux 3.19.0-59-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "got-swag" "-g" "."
npm ERR! cwd /home/pradeep/Documents/swagger-tester
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.4.21
npm ERR! path package.json
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /home/pradeep/Documents/swagger-tester/npm-debug.log
npm ERR! not ok code 0

Exit code 2

... makes no sense.

Monkey tests should validate all responses

Otherwise, non-conforming error responses slip through.

Add POST monkey testing

Notes:

• POSTs are possibly destructive
• A whitelist of hosts may be beneficial to prevent running POST monkey tests against a production environment

date-time format is not validated correctly

got-swag should validate against ISO8601 but fails with some uncommon notations like +0000 (instead of +00:00), which is valid(?) ISO8601.

Test case generation

Sometimes it would be beneficial to generate test cases dynamically. Consider for example zip code ranges or numbers 1 to 100. Need to r&d an API to simplify that.

Support http PUT

Do you plan to support PUT in the future? It would be really helpful for our monkey tests 👍

Greetings, Felix

Example data in monkeyGet

Extend monkeyGet to use example data from given parameter and schema definitions.

host parameter in request() does not work

request( { host: ... } ) does not work correctly. Workaround is to use hostname instead.

Error running programmatic usage

I've been testing and I can't find anything on the error I get, both using mocha and pure JS.

 
  1) My test suite "before all" hook:
     TypeError: Promise.all is not a function
      at Function.all (native)
      at bundleApis (c:\Users\gollivier\Desktop\loopHPC2\bikube\MESO_Bikube_Tools_verif\node_modules\got-swag\lib\bundle
Apis.js:13:18)
      at mocha (c:\Users\gollivier\Desktop\loopHPC2\bikube\MESO_Bikube_Tools_verif\node_modules\got-swag\lib\mocha.js:13
:10)
      at Context.<anonymous> (c:\Users\gollivier\Desktop\loopHPC2\bikube\MESO_Bikube_Tools_verif\node_modules\got-swag\l
ib\describe.js:8:5)
      at callFnAsync (c:\Users\gollivier\AppData\Roaming\npm\node_modules\mocha\lib\runnable.js:349:8)
      at Hook.Runnable.run (c:\Users\gollivier\AppData\Roaming\npm\node_modules\mocha\lib\runnable.js:301:7)
      at next (c:\Users\gollivier\AppData\Roaming\npm\node_modules\mocha\lib\runner.js:298:10)
      at Immediate._onImmediate (c:\Users\gollivier\AppData\Roaming\npm\node_modules\mocha\lib\runner.js:320:5)

And

return Promise.all( apis.map( function ( api ) {
                          ^

TypeError: apis.map is not a function
   at bundleApis (c:\Users\gollivier\Desktop\loopHPC2\bikube\MESO_Bikube_Tools_verif\node_modules\got-swag\lib\bundleAp
is.js:13:28)
   at Object.test (c:\Users\gollivier\Desktop\loopHPC2\bikube\MESO_Bikube_Tools_verif\node_modules\got-swag\lib\test.js
:12:10)
   at Object.<anonymous> (c:\Users\gollivier\Desktop\loopHPC2\bikube\MESO_Bikube_Tools_verif\test\test.js:5:9)
   at Module._compile (module.js:409:26)
   at Object.Module._extensions..js (module.js:416:10)
   at Module.load (module.js:343:32)
   at Function.Module._load (module.js:300:12)
   at Function.Module.runMain (module.js:441:10)
   at startup (node.js:139:18)
   at node.js:968:3

Any lights ?

Improve validation errors

With validate(), e.g. "element is not an instance of object" is not very helpful. Need to expose more details here.

On Windows, running got-swag with working directory C:\ fails

... with Unable to resolve $ref pointer "c://path/to/swagger.yaml"

Improve README

• Motivation
• Use cases (e.g. contract-first, TDD, shareable tests, verifiable by other parties, ...)
• Benefits over regular test suites

Path-level parameters are ignored

Parameters on path-level (and possibly other levels, need to check this) are ignored, e.g.

paths:
  /test
    parameters:
      ... # ignored
    get:
      parameters:
        ... # work

This seems to be valid Swagger, so got-swag should respect that, e.g. in monkey tests.

Host as parameter

Hello,

it will be very nice, if the got-swag tool could take the host param from the command-line arg like -h google.de instead of the yaml

Greetz
Fival

Monkeystore example doesn't parse string parameters

When running the monkeystore example on the petstore:
got-swag http://petstore.swagger.io/v2/swagger.json examples/monkeystore.yaml -m
the status does not get filled in
{
"MONKEY_LAST": {
"parameters": {
"status": null
},
"statusCode": 200
}
}