yelp/swagger-spec-compatibility

Python library to check Swagger Spec backward compatibility

swagger-spec-compatibility

About

swagger-spec-compatibility is a Yelp maintained library that provides tools to automatically detect the safety of Swagger/OpenAPI 2.0 specification changes with respect to backwards compatibility.

swagger-spec-compatibility aims to give developers confidence that their spec changes are safe and that clients built with previous versions of the Swagger spec can continue to communicate correctly.

Disclaimer

The library is not supposed to cover all the possible cases of backward incompatibility.
This is because OpenAPI 2.0 specifications are very expressive and flexible that leads to many cases of backward incompatibility.

The detection rules currently supported are built due to the need to cover common breaking changes (that we've experienced internally at Yelp) or support received from contributors.

If you're experiencing breaking changes and you would have the tool help you figure it out before being late, feel free to open issues on the project. You can also open pull requests implementing the rules, we're always open to contributors.

Documentation

More documentation is available at swagger-spec-compatibility.readthedocs.org.

How to Install

# to install the latest released version
$ pip install swagger-spec-compatibility

# to install the latest master [from Github]
$ pip install git+https://github.com/Yelp/swagger-spec-compatibility

Example Usage

The commands below assume that the library is already installed

$ swagger_spec_compatibility -h
usage: swagger_spec_compatibility [-h] {explain,info,run} ...

Tool for the identification of backward incompatible changes between two swagger specs.

The tool provides the following level of results:
- WARNING: the Swagger specs are technically compatible but the are likely to break known Swagger implementations
- ERROR: new Swagger spec does introduce a breaking change respect the old implementation

positional arguments:
  {explain,info,run}  help for sub-command
    explain           explain selected rules
    info              Reports tool's information
    run               run backward compatibility detection

optional arguments:
  -h, --help          show this help message and exit

$ swagger_spec_compatibility explain -r REQ-E001 REQ-E002
Rules explanation
[REQ-E001] Added Required Property in Request contract:
    Adding a required property to an object used in requests leads client request to fail if the property is not present.
[REQ-E002] Removed Enum value from Request contract:
    Removing an enum value from a request parameter is backward incompatible as a previously valid request will not be
    valid. This happens because a request containing the removed enum value, valid according to the "old" Swagger spec, is
    not valid according to the new specs.

$ old_spec_path=docs/source/rules/examples/REQ-E001/old.yaml
$ new_spec_path=docs/source/rules/examples/REQ-E001/new.yaml

# Run with all the registered rules
$ swagger_spec_compatibility run ${old_spec_path} ${new_spec_path}
ERROR rules:
    [REQ-E001] Added Required Property in Request contract : #/paths//endpoint/post/parameters/0/schema
$ echo $?
1

# Run with a subset of registered rules
$ swagger_spec_compatibility -r=MIS-E001 -r=MIS-E002 run ${old_spec_path} ${new_spec_path}
$ echo $?
0

$ swagger_spec_compatibility info
swagger-spec-compatibility: 1.3.0
Python version: CPython - 3.6.9
Python compiler: GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)
Discovered rules:
    MIS-E001: swagger_spec_compatibility.rules.deleted_endpoint.DeletedEndpoint
    MIS-E002: swagger_spec_compatibility.rules.changed_type.ChangedType
    REQ-E001: swagger_spec_compatibility.rules.added_required_property_in_request.AddedRequiredPropertyInRequest
    REQ-E002: swagger_spec_compatibility.rules.removed_enum_value_from_request.RemovedEnumValueFromRequest
    REQ-E003: swagger_spec_compatibility.rules.removed_properties_from_request_objects_with_additional_properties_set_to_false.RemovedPropertiesFromRequestObjectsWithAdditionalPropertiesSetToFalse
    RES-E001: swagger_spec_compatibility.rules.added_properties_in_response_objects_with_additional_properties_set_to_false.AddedPropertiesInResponseObjectsWithAdditionalPropertiesSetToFalse
    RES-E002: swagger_spec_compatibility.rules.removed_required_property_from_response.RemovedRequiredPropertyFromResponse
    RES-E003: swagger_spec_compatibility.rules.added_enum_value_in_response.AddedEnumValueInRequest

Development

Code is documented using Sphinx.

virtualenv is recommended to keep dependencies and libraries isolated.

Setup

# Initialize your dev environment
$ make minimal

# Ensure that you have activated the virtualenvironment
$ source ./venv/bin/activate

Tip: If you have aactivator installed the virtual environment will be automatically activated

How to define a new compatibility rule

Use the following steps to define a new rule:

1. Generate the rules skeletons by running python -m create_new_rule.

   The tool will be creating the detection rule class, tests, etc. Check the tool output for the exact list of created files.

2. Implement the rule logic (swagger_spec_compatibility/rules/{filename}.py) and ensure testing coverage (tests/rules/{filename}_test.py).

3. Update docs/source/rules/examples/{rule_code}/(new|old).yaml example Swagger spec change and update docs/source/rules/examples/{rule_code}/tester.py tester file.

   The objective of those files is to provide a simple spec change that triggers the backward incompatible detection rule through the usage of a bravado client (check the other testers for examples).

   NOTE: The testers are executed by automated tests, so tester.py should complete without errors and that the spec changes are triggering the newly created rule.

4. Add documentation for the defined rule in swagger_spec_compatibility/rules/{filename}.py and docs/source/rules/{error_code}.rst.

   Try to be consistent with the style of the others documentation pages.

5. [Optional] Add integration tests to ensure that no regressions will be introduced and/or to validate edge cases of the new rule.

   Integration tests are defined as follow: case-<incremental number>-<number of expected reports>-reports-<short description> directory with two files: old.yaml and new.yaml. The two files represent two versions of the swagger specs that need to be checked for backward compatibility.

Contributing

1. Fork it (http://github.com/Yelp/swagger-spec-compatibility/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Add your modifications
4. Commit your changes (git commit -m "Add some feature")
5. Push to the branch (git push origin my-new-feature)
6. Create new Pull Request

License

Copyright 2019 Yelp, Inc.

Swagger Spec Compatibility is licensed with a Apache License 2.0.

swagger-spec-compatibility's Issues

Adding values to enum triggers MIS-E002

Describe the bug
We added values to an enum in a response. We disabled rule RES-E003 to allow that, however it still triggers MIS-E002.

To Reproduce
Please provide information about:

1. Initial Swagger Spec (old version)

/my_endpoint:
   get:
      ...
      responses:
         '400':
              description: Bad Request
              schema:
                  $ref: '#/definitions/Error'

definitions:
  Error:
    type: object
    required:
      - type
    properties:
      type:
        type: string
        enum:
          - error1
          - error2

2. Modified Swagger Spec (new version)

/my_endpoint:
   get:
      ...
      responses:
         '400':
              description: Bad Request
              schema:
                  $ref: '#/definitions/Error'

definitions:
  Error:
    type: object
    required:
      - type
    properties:
      type:
        type: string
        enum:
          - error1
          - error2
          - error3

3. Executed command
   swagger_spec_compatibility run old.yaml new.yaml -b RES-E003

the output:

[MIS-E002] Changed type: #/paths//my_endpoint/get/responses/400/schema/properties (documentation: https://swagger-spec-compatibility.readthedocs.io/en/latest/rules/MIS-E002.html)

Note: Please provide a minimal case of spec as it helps more in identifying the issue and fixing it.

Expected behavior
I expected the compability check to pass as rule RES-E003 is disabled.

Platform:

• OS: Linux
• Python interpreter: Python 3.8.7 (default, Jan 12 2021, 17:16:32) [GCC 8.3.0]
• Library Version:

Name: swagger-spec-compatibility
Version: 1.2.3
Summary: Python library to check Swagger Spec backward compatibility
Home-page: https://github.com/Yelp/swagger-spec-compatibility
Author: Yelp, Inc.
Author-email: [email protected]
License: Copyright Yelp, Inc. 2018
Location: /usr/local/lib/python3.8/site-packages
Requires: termcolor, six, bravado-core, swagger-spec-validator, typing-extensions, bravado, venusian
Required-by:

Implement detection rule for changing type of additional properties

Change value of additionalProperties

Rationale

If the object is defined with additionalProperties set to False then the object will not allow presence of properties not defined on the properties section of the object definition.
Changing the value of additionalProperties could lead to incompatible changes when the new specs define a less stringent requirement.

Mitigation

TODO

The main goal is to identify changes like

• {...} -> {"additionalProperties": False, ...}
• {...} -> {"additionalProperties": {xxx}, ...}
• {"additionalProperties": {...}, ...} -> {"additionalProperties": {something different}, ...}
• etc

Add exclude rules CLI argument

Currently the library's CLI interface does allow to selectively select which rules to validate with a whitelist approach.
This means that we cannot selectively disable the execution of a single rule in a easy way (we could enable all the rules except the one that we want to disable, but it would not scale with versions update as new rules will be ignored as well).

The idea of this issue is to add a new CLI flag (ie. --exclude-rules) which blacklists certain rules from being executed

Implement rules for removing a consume/produce format that was previously supported

If the service no longer support a mime-type/data format, e.g. "application/json" or xml, it would be a breaking change for the users. There should be rules detecting these changes.

Implement detection rules for adding / removing x-nullable attribute

Changing the nullability of an object or property might lead to backward incompatible specs.

For easier understanding we can use the following example

definitions:
  model:
    type: object
    x-model: Model
    properties:
      prop:
        type: string

with the above example we should consider {"prop": "a"} or {} as valid Model and {"prop": null} as invalid.
But adding x-model: true into prop property it makes {"prop": null} a valid Model.

This change is:

• backward compatible if the object is used in requests parameters, as old clients would continue to send Model instances
• backward incompatible if the object is used in responses, as old clients might start receiving null values which were not supposed to receive

The objective of this issue is to track the implementation of two complementary rules

• add x-nullable into response models
• removing x-nullable from request models

Internal reference: CORESERV-9265

Improve error message format

The error message currently printed out doesn't directly give user the information they need. For example, if a user changes the type of a schema, the user would see:

[MIS-E002] Changed type: #/paths//pet/{id}/put/parameters/1/schema/properties/age (documentation: https://swagger-spec-compatibility.readthedocs.io/en/latest/rules/MIS-E002.html)

The "#/path//..." is hard to follow and not providing much useful information. Users just need to know 1) the property that has changed, 2) the endpoint & the method that are being affected by the change, 3) the error rule and 4) the link to the documentation explaining the rule. Similar case for other error rules.
We could simplify and clarify the error message so that it's more user-friendly, similar to another widely-adopted library swagger-diff has done it.

Possibly missing check for nested required field.

Changing a schema such that the object defined inside an (optional) list now requires a certain attribute did not trigger a compatibility warning. Although ok in this specific case this is unexpected behavior in general.

Implement detection rules for adding/removing discriminated types

Add a new discriminated type (model) used in responses

Rationale

Polymorphism, done using the discriminator keyword, allows returning different objects that are united by the fact that a string property is required. The content of the property represent the specific type of the object.
Adding a new discriminated type, that will be used in responses, is backward incompatible as a new object type could be returned to clients that are operating with "old" Swagger spec and they would not be able to properly handle the object.

Mitigation

This condition is very similar to the case of "Adding enum value used in responses" but instead of strings, it’s about objects.
A possible mitigation consists of ensuring that the clients communicate to the backend the discriminated types that are handled by the implementation and offloading the filtering on the business logic (we cannot guarantee that on Swagger spec level).
Another option would be to ensure that clients properly handle all the types of discriminated objects, as they have a property in common, but this could easily masquerade errors in the client implementation which could be hard to fix (especially for mobile apps).

---

Removing a discriminated type (model) used as request parameter

Rationale

Polymorphism, done using the discriminator keyword, allows returning different objects that are united by the fact that a string property is required. The content of the property represent the specific type of the object.
Removing a discriminated type, that will be used in requests, is backward incompatible as previously valid objects could be sent by clients that are operating with "old" Swagger spec and the backend service will fail to validate the request.

Mitigation

This condition is very similar to the case of "Removing enum value used as request parameter" but instead of string it has objects.
Like for the enum case, the general recommended approach is: don't do it and add business logic to ignore such objects.

unable to run this tool on any files

Forgive me, I'm not a python programmer so this probably due to my misunderstanding:

I ran pip install git+https://github.com/Yelp/swagger-spec-compatibilityf (first installed the released version and got the same result; the line numbers match the source better in this version), and running it on two very simple yaml files produces:

ansible|ghoren-ltm:Downloads ghoren$ swagger_spec_compatibility run old.yaml new.yaml
Traceback (most recent call last):
  File "/Users/ghoren/.ansible/py3/bin/swagger_spec_compatibility", line 11, in <module>
    load_entry_point('swagger-spec-compatibility==1.2.0', 'console_scripts', 'swagger_spec_compatibility')()
  File "/Users/ghoren/.ansible/py3/lib/python3.6/site-packages/swagger_spec_compatibility/__main__.py", line 17, in main
    exit_code = args.func(args)  # type: int
  File "/Users/ghoren/.ansible/py3/lib/python3.6/site-packages/swagger_spec_compatibility/cli/run.py", line 80, in execute
    rules=rules(cli_args),
  File "/Users/ghoren/.ansible/py3/lib/python3.6/site-packages/swagger_spec_compatibility/rules/__init__.py", line 33, in compatibility_status
    for rule in rules
  File "/Users/ghoren/.ansible/py3/lib/python3.6/site-packages/swagger_spec_compatibility/rules/__init__.py", line 33, in <dictcomp>
    for rule in rules
  File "/Users/ghoren/.ansible/py3/lib/python3.6/site-packages/swagger_spec_compatibility/rules/deleted_endpoint.py", line 30, in validate
    endpoints_left_spec = get_endpoints(left_spec)
TypeError: unhashable type: 'Spec'
ansible|ghoren-ltm:Downloads ghoren$ 

I see that Spec is imported from bravado core, so I uninstalled swagger_spec_compatibility, installed bravado_core, and reinstalled swagger_spec_compatibility. I get the same result.

I'm using python from Ansible version 3.6.5, on Mac Mojave:

ansible|ghoren-ltm:Downloads ghoren$ python --version
Python 3.6.5
ansible|ghoren-ltm:Downloads ghoren$ which python
/Users/ghoren/.ansible/py3/bin/python

What am I not understanding about how to install this?

Implement detection rules for adding new required header parameters in request

The current rules only account for newly added required parameters in the objects sent in a request, but no the request header. For example, when the api has a newly required header parameter, it could be a backward incompatible change for users sending requests that were accepted by the old api.