ioggstream/openapi-resolver

A python 3 module for resolving $refs in OpenAPI specs and create bundles.

OpenAPI Resolver

This module recursively parses openapi specs resolving references.

Test

Tests run locally via

    tox

Or via circleci-local

    circleci build 

Usage

The module has an embedded script that can be run via

    $ python -m openapi_resolver --help

    usage: __main__.py [-h] src_file [dst_file]

    Recursively resolves and bundles OpenAPI v3 files.

    positional arguments:
      src_file    An OpenAPI v3 yaml file.
      dst_file    Destination file, default is stdout.

    optional arguments:
      -h, --help  show this help message and exit

To create an openapi bundle from a spec file just run

    $ python -m openapi_resolver sample.yaml

You can use this module to normalize two specs before diffing, eg:

    $ python -m openapi_resolver one.yaml normal-one.yaml
    $ python -m openapi_resolver two.yaml normal-two.yaml
    $ diff normal-one.yaml normal-two.yaml

Use with docker

Build the image with:

$ docker build --tag openapi-resolver . 

then run docker mapping the openapi.yaml directory to the /code volume.

docker run -it --rm -v $(dirname path-to-openapi.yaml):/code \
    openapi-resolver /code/openapi.yaml /code/bundle.yaml

openapi-resolver's Issues

Use yaml.safe_load

I expect

yaml.safe_load everywhere

Collect referenced items under components

I expect

openapi resover to collect all referenced items under components

Multi-level ref expansion doesn't handle path correctly

When my top-level OpenAPI file has a ref:

$ref: user/User.yaml

and that file also has a ref:

$ref: CreateUser.yaml

openapi-resolver looks for CreateUser.yaml in the top-level directory instead of the user subdirectory.

$ python -m openapi_resolver api.yaml
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__main__.py", line 32, in <module>
    main(args.src_file, args.dst_file)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__main__.py", line 17, in main
    resolver.resolve()
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 98, in resolve
    self.traverse(self.openapi, cb=self.resolve_node)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 194, in traverse
    self.traverse(i, k, parents, cb, context)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 194, in traverse
    self.traverse(i, k, parents, cb, context)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 194, in traverse
    self.traverse(i, k, parents, cb, context)
  [Previous line repeated 6 more times]
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 227, in traverse
    self.traverse(ancestor[needle], key, parents, cb, context)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 194, in traverse
    self.traverse(i, k, parents, cb, context)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 194, in traverse
    self.traverse(i, k, parents, cb, context)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 194, in traverse
    self.traverse(i, k, parents, cb, context)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 210, in traverse
    ancestor[needle] = cb(key, node, context)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 263, in resolve_node
    _yaml = self.get_yaml_reference(n)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 243, in get_yaml_reference
    self.yaml_cache[host] = open_file_or_url(host)
  File "/home/boris/iron_ox/server/.venv/lib/python3.6/site-packages/openapi_resolver/__init__.py", line 63, in open_file_or_url
    with open(host) as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'CreateUser.yaml'

Error resolving definition

Hi @ioggstream

I'm using your package with our openapi.yaml spec (attached):

docker run -it --rm -v $(pwd)/0.0.3:/code openapi-resolver /code/openapi.yaml /code/openapi-res.yaml

and I noticed that It generates a wrong structure.

The schema:

  parameters:
    . . .
    provenance__parameters:
      default: null
      description: Name/URI/DOI of the parameters | varchar(255)
      example: ToDo
      type: string

is pushed into parameters: object but should be in schemas:.

In addition, in the object ObjectProvenance:

    ObjectProvenance:
      properties:
        description:
          $ref: '#/components/schemas/provenance__description'
        hostname:
          $ref: '#/components/schemas/provenance__hostname'
        method:
          $ref: '#/components/schemas/provenance__method'
        model:
          $ref: '#/components/schemas/provenance__model'
        name:
          $ref: '#/components/schemas/provenance__name'
        parameters:
          $ref: '#/components/parameters/provenance__parameters'

the property called parameters contains a wrong ref.; It should be:

• $ref: '#/components/schemas/provenance__parameters'

instead of:

• $ref: '#/components/parameters/provenance__parameters'

Thank you,
Valentino

openapi.yaml.zip

Add tests for __main__.py

I expect

• to add some tests in https://github.com/ioggstream/openapi-resolver/blob/master/tests/test_cases.py
  for testing https://github.com/ioggstream/openapi-resolver/blob/master/openapi_resolver/__main__.py

Tests could just verify that main() works as expected.

Remove redundant path from entries not in components

I expect

References like:

components:
  headers:
    X-RateLimit-Limit:
      $ref: 'https://raw.githubusercontent.com/teamdigitale/openapi/0.0.3/docs/definitions.yaml#/headers/X-RateLimit-Limit'

to be resolved correctly

Instead

They are resolved like

components:
  headers:
    headers/X-RateLimit-Limit:
       ...

Fix context management

I expect

The context (namely the URL associated to the $ref to process) to be fully managed in the stack.

Instead

There's an instance variable too.

Riferimento esterno ('$ref')

Buongiorno Roberto

Una richiesta di comportamento su openapi-resolver.

Caso 1

Ho un file denominato openapi.yaml che contiene:

    schemas:
        myLongitude:
            $ref: 'https://teamdigitale.github.io/openapi/master/definitions.yaml#/schemas/Longitude'

eseguo openapi-resolver e ottengo:

    schemas:
      Longitude:
        description: |-
          Longitude of a point expressed in:

          * the ETRS89 system for Italian and European territories
          * and in WGS84 for the others.
        example: 9.19289
        externalDocs:
          url: http://dati.gov.it/onto/clvapit#long
        format: double
        maximum: 180
        minimum: -180
        type: number
      myLongitude:
            $ref: '#/components/schemas/Longitude'

Tutto ok.

Caso 2

Il file openapi.yaml contiene un riferimento ad un mio file definitions.yaml:

    schemas:
        myLongitude:
            $ref: 'https://ingv.github.io/openapi/definitions.yaml#/schemas/longitude'

il file definitions.yaml contiene un riferimento esterno:

    schemas:
        longitude:
            $ref: 'https://teamdigitale.github.io/openapi/master/definitions.yaml#/schemas/Longitude'

eseguo openapi-resolver e ottengo:

    schemas:
      Longitude:
        description: |-
          Longitude of a point expressed in:

          * the ETRS89 system for Italian and European territories
          * and in WGS84 for the others.
        example: 9.19289
        externalDocs:
          url: http://dati.gov.it/onto/clvapit#long
        format: double
        maximum: 180
        minimum: -180
        type: number
      myLongitude:
            $ref: '#/components/schemas/longitude'
      longitude:
            $ref: 'https://teamdigitale.github.io/openapi/master/definitions.yaml#/schemas/Longitude'

il problema e' che ho un Warning da Swagger su Longitude in quanto nel codice non viene usata. Per il resto, tutto funziona perfettamente; Swagger espande il riferimento esterno per longitudine e tutto torna.

Resolve fragment-only relative references in external sources.

I expect

relative references in external sources are correctly resolved.

Instead

they are not

Reindent with black & isort

I expect

everything to be reformatted with black and isort automagically.

Support remote files (http uri)

I wish

To support remote files

Example

    python -mopenapi_resolver https://foo.com/openapi.yaml

More descriptive errors when local components are missing

I expect

a more descriptive error when local components are missing

Note

a simple solution could be to validate yaml before parsing.
This should be done:

• adding openapi-spec-validator in test-requirements.txt
• providing further infos only if openapi-spec-validator is installet

Important:

• openapi-spec-validator should be an OPTIONAL dependency

Proxied references are not bundled.

I expect

Indirect references to be resolved, eg:

openapi.yaml#/components/schemas/Pointer2
-> indirect.yaml#/schemas/Pointer1
-> store.yaml#/schemas/Pointer

Eg:

--- openapi.yaml
components:
  schemas:
    Pointer2:
       $ref: indirect.yaml#/schemas/Pointer1
--- pointer.yaml
schemas:
  Pointer1:
    $ref: original.yaml#/schemas/Pointer
--- original.yaml
schemas:
  Pointer:
    type: string
    format: pointer

should contain

--- resolved.yaml
components:
  schemas:
    Pointer:
      type: string
      format: pointer