kong/go-apiops

Kong's Go based APIOps library

go-apiops

Home of Kong's Go based APIOps library.

What is APIOps

API Lifecycle Automation, or APIOps, is the process of applying API best practices via automation frameworks. This library contains functions to aid the development of tools to apply APIOps to Kong Gateway deployments.

See the Kong Blog for more information on APIOps concepts.

What is this library?

The go-apiops library provides a set of tools (generation and transformation) for working with API specifications and Kong Gateway declarative configurations. Conceptually, these tools are intended to be organized into a pipeline of individual steps configured for a particular users needs. The overall purpose of the library is to enable users to build a CI/CD workflow which deliver APIs from specification to deployment. This pipeline design allows users to customize the delivery of APIs based on their specific needs.

What is the current status of this library?

The library is an Apache 2.0 license. The library functionality will be be made available through the deck cli tool.

Installation & Usage

Currently, the functionality is released as a library and as part of the decK CLI tool. The repository also contains a CLI named go-apiops for local testing. For general CLI usage, the decK tool should be used.

Local Build

• make sure Go-lang tools are installed
• Checkout the Git repository (switch to a specific tag to select a version if required)
• use the makefile to build the project via make build

Reporting issues

Issues using the go-apiops CLI or the library can be reported in the Github repo.

Releasing new versions

The releases are automated. To create a new release:

• tag at the desired place to release

  git tag vX.Y.Z

• push the tag and CI will create a new release

  git push vX.Y.Z

• verify the release on the releases page, possibly edit the release-notes (which will be generated from the commit history)

go-apiops's Issues

Unable to create request-validator plugin when using +json media type

According to the RFC6839

[RFC4627] defines the "application/json" media type. The suffix
"+json" MAY be used with any media type whose representation follows
that established for "application/json".

So media type merge-patch+json should be considered as application/json

Test spec

openapi: 3.0.2
info:
  title: Test Spec
  version: 0.0.1

x-kong-plugin-request-validator:
  config:
    verbose_response: true

paths:
  /anything:
    post:
      requestBody:
        required: true
        content:
          application/merge-patch+json:
            schema:
              type: object
              properties:
                testdata:    
                  type: integer
        
      responses:
        '200':
          description: ok
          content:
            application/json:
              schema:
                type: object
                properties:
                  name:
                    type: string
                  value:
                    type: integer

_ignore with command history not compatible with deck v1.19.1

When running our generated files with the command history feature, deck errors that the _ignore property is not allowed. For example:

deck diff -s platform/kong-combined.yaml --konnect-runtime-group-name default --konnect-token $KONNECT_PAT
Error: reading file: validating file content: 1 errors occurred:
        (root): Additional property _ignore is not allowed

If I remove the entire _ignore key, then the file is processed as expected.

openapi2kong Key-auth plugin not generated based on security/securitySchemes in OAS

• I have searched the existing issues

go-apiops version: v0.1.20
Running OS: windows 10 build 10.0.19045

Current behavior

key-auth plugin not added.

Expected Behavior

key-auth plugin should be added.

Steps To Reproduce

Source OAS file:

{
    "openapi": "3.0.2",
    "info": {
        "title": "Test security",
        "version": "1",
        "description": "Test security plugin generation based on security/securitySchemes"
    },
    "paths": {
        "/v1/secuire-path": {
            "get": {
                "security": [
                    {
                        "ApiKeyAuth": []
                    }
                ],
                "summary": "Secuire path",
                "responses": {
                    "204": {
                        "description": "Success case",
                        "content": {}
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {},
        "securitySchemes": {
            "ApiKeyAuth": {
                "type": "apiKey",
                "name": "apikey",
                "in": "header",
                "x-kong-security-key-auth": {
                    "name": "key-auth",
                    "enabled": true,
                    "protocols": [
                        "http",
                        "https"
                    ],
                    "config": {
                        "anonymous": null,
                        "hide_credentials": true,
                        "key_in_body": false,
                        "key_in_header": true,
                        "key_in_query": false,
                        "key_names": [
                            "apikey"
                        ],
                        "run_on_preflight": true
                    }
                }
            }
        }
    },
    "servers": [
        {
            "url": "http://localhost"
        }
    ],
    "security": [
        {
            "ApiKeyAuth": []
        }
    ],
    "x-kong-name": "security-test-service"
}

go-apiops command: go-apiops.exe openapi2kong --format json -s ./source.oas3.json -o ./kong.json --verbose 2
command logs:

2023/08/29 16:34:12 "level"=2 "msg"="received OpenAPI2Kong options" "options"={"Tags":[],"DocName":"","UUIDNamespace":"6ba7b810-9dad-11d1-80b4-00c04fd430c8"}
2023/08/29 16:34:12 "level"=1 "msg"="tags after parsing x-kong-tags" "tags"=[]
2023/08/29 16:34:12 "level"=2 "msg"="no document name specified, trying x-kong-name"
2023/08/29 16:34:12 "level"=1 "msg"="document name (namespace for UUID generation)" "name"="security-test-service"
2023/08/29 16:34:12 "level"=1 "msg"="processing path" "path"="/v1/secuire-path"
2023/08/29 16:34:12 "level"=2 "msg"="path name (namespace for UUID generation)" "name"="security-test-service_v1-secuire-path"
2023/08/29 16:34:12 "level"=1 "msg"="processing operation" "method"="GET" "path"="/v1/secuire-path" "id"=""
2023/08/29 16:34:12 "level"=2 "msg"="operation base name (namespace for UUID generation)" "name"="security-test-service_v1-secuire-path_get"
2023/08/29 16:34:12 "level"=2 "msg"="finished processing document"

generated file:

{
  "_format_version": "3.0",
  "services": [
    {
      "host": "localhost",
      "id": "5419f3a2-2022-5fde-9fda-2ec6382459cc",
      "name": "security-test-service",
      "path": "/",
      "plugins": [],
      "port": 80,
      "protocol": "http",
      "routes": [
        {
          "id": "2e6d5170-acec-5d35-99da-e5f53ddb4aa0",
          "methods": [
            "GET"
          ],
          "name": "security-test-service_v1-secuire-path_get",
          "paths": [
            "~/v1/secuire-path$"
          ],
          "plugins": [],
          "regex_priority": 200,
          "strip_path": false,
          "tags": []
        }
      ],
      "tags": []
    }
  ],
  "upstreams": []
}

Additional information

Using inso 3.16.0 on same file : inso generate config --format json -k 3 .\source.oas3.json -o .\kong.json generates following content:

{
    "_format_version": "3.0",
    "services": [
        {
            "name": "security-test-service",
            "protocol": "http",
            "host": "localhost",
            "port": 80,
            "path": "/",
            "plugins": [
                {
                    "name": "key-auth",
                    "config": {
                        "key_names": [
                            "apikey"
                        ],
                        "anonymous": null,
                        "hide_credentials": true,
                        "key_in_body": false,
                        "key_in_header": true,
                        "key_in_query": false,
                        "run_on_preflight": true
                    },
                    "tags": [
                        "OAS3_import",
                        "OAS3file_temp.oas3.json"
                    ]
                }
            ],
            "routes": [
                {
                    "tags": [
                        "OAS3_import",
                        "OAS3file_temp.oas3.json"
                    ],
                    "name": "security-test-service-v1-secuire-path-get",
                    "methods": [
                        "GET"
                    ],
                    "paths": [
                        "~/v1/secuire-path$"
                    ],
                    "strip_path": false,
                    "plugins": [
                        {
                            "name": "key-auth",
                            "config": {
                                "key_names": [
                                    "apikey"
                                ],
                                "anonymous": null,
                                "hide_credentials": true,
                                "key_in_body": false,
                                "key_in_header": true,
                                "key_in_query": false,
                                "run_on_preflight": true
                            },
                            "tags": [
                                "OAS3_import",
                                "OAS3file_source.oas3.json"
                            ]
                        }
                    ]
                }
            ],
            "tags": [
                "OAS3_import",
                "OAS3file_source.oas3.json"
            ]
        }
    ]
}

Minor documentation issue in learn service OAS yaml example

https://github.com/Kong/go-apiops/blob/cd3a7f217d4a711c4f1b3ffa5a817e738268d2fa/docs/learnservice_oas.yaml#L128C46-L128C66

documents the eventual name as "awesome-learnservice-upserttracks" but the name in x-kong-name at the top is awesome_learnservice

Add alias oas2kong for command openapi2kong

The Kong inso tool uses the shorter version of the command. To help support a migration path, add an alias for the oas2kong version of the same command.

openapi2kong should allow multiple "foreign key" plugins

In Konnect some of our "gateway services" use many instances of the rate-limiting-advanced plugin and have multiple "foreign keyed" resources. This is a helpful architecture for doing things like rate limiting tiers for a single service.

Example would be an RLA that is "scoped" to: consumerA + serviceA. Another instance of an RLA is "scoped" to: consumerB + serviceA. There does not seem to be a way to do this using openapi2kong. It seems to try to "promote" both and the second one fails. This issue seems to happen even with 2 different plugins that have consumers.

To replicate add two RLA plugins that are configured with "consumers" to two paths in an openApi file.

Simply uncommenting the consumer on 1 of the 2 allows the command to succeed. I'm using:

deck 1.38 on a Mac. The command issued is pretty standard:
deck file openapi2kong -s <openApi-file.yaml> -o <output-file.yaml> --inso-compatible

FWIW - it would be great if "root" level plugins could be put into the openApi file using some "x-kong-plugins:" or similar. The final structure in deck for these types of configurations look like:

...
plugins: <-- at the root of the deck file

• config:
  limit:
  • 5
    ...
    consumer: consumerA
    service: serviceA
    name: rate-limiting-advanced
• config:
  limit:
  • 10
    ...
    consumer: consumerB
    service: serviceA
    name: rate-limiting-advanced

Error produced:
panic: interface conversion: interface {} is nil, not string

goroutine 1 [running]:
github.com/kong/go-apiops/openapi2kong.Convert.func1(0x1000100?, 0x14bf62588?)
github.com/kong/[email protected]/openapi2kong/openapi2kong.go:1126 +0x1d0
sort.insertionSort_func({0x14000123a90?, 0x140003ae080?}, 0x0, 0x3)
sort/zsortfunc.go:12 +0xc0
sort.pdqsort_func({0x14000123a90?, 0x140003ae080?}, 0x140001975a8?, 0x0?, 0x0?)
sort/zsortfunc.go:73 +0x260
sort.Slice({0x103d9db20?, 0x140001975a8?}, 0x14000123a90)
sort/slice.go:29 +0xbc
github.com/kong/go-apiops/openapi2kong.Convert({0x14000174000, 0x769e, 0x769f}, {{0x0, 0x0, 0x0}, {0x0, 0x0}, {0x6b, 0xa7, ...}, ...})
github.com/kong/[email protected]/openapi2kong/openapi2kong.go:1122 +0x1a18
github.com/kong/deck/cmd.executeOpenapi2Kong(0x140001bcb00?, {0x1400010bef0?, 0x4?, 0x103a49a2a?})
github.com/kong/deck/cmd/file_openapi2kong.go:61 +0x3f4
github.com/spf13/cobra.(*Command).execute(0x14000133808, {0x1400010bea0, 0x5, 0x5})
github.com/spf13/[email protected]/command.go:983 +0x840
github.com/spf13/cobra.(*Command).ExecuteC(0x140001ba008)
github.com/spf13/[email protected]/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/[email protected]/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
github.com/spf13/[email protected]/command.go:1032
github.com/kong/deck/cmd.Execute({0x1040d8b10, 0x140003eeaf0})
github.com/kong/deck/cmd/root.go:265 +0x64
main.main()
github.com/kong/deck/main.go:31 +0x20

Kong 2.x compatibility with openapi2kong

Hi!

There is an incompatibility with Kong 2.x for the generated routes. When running deck validate --online, there's this error

validate entity 'routes (<redacted>)': HTTP status 400 (message: "schema violation (paths.1: should start with: /)")

This is due to a breaking change in Kong 3.x

Kong will no longer use an heuristic to guess whether a route.path is a regex pattern. From now 3.0 onwards, all regex paths must start with the "~" prefix, and all paths that don't start with "~" will be considered plain text.

For now we're still using 2.8.x but would like to adopt decK specifically with the openapi2kong functionality. Would a flag to disable the autoprefixing of ~ here be something you'd consider? Maybe generalized to something like --kong-version in case more changes are needed.

If you're not planning to support Kong < 3, might want to indicate Kong version compatibility in docs since this is in the decK docs.

decK is compatible with Kong Gateway (OSS) >= 1.x and Kong Enterprise >= 0.35..

Add switch to disable command history

Command history populates a file with a pipeline of commands ran. For example:

- command: merge
  files:
  - filename: .github/artifacts/kong/flight-data-flights-kong.yaml
    info:
    - command: openapi2kong
      input: flight-data/flights/openapi.yaml
      output: .github/artifacts/kong/flight-data-flights-kong.yaml
      uuid-base: ""
      version: kceD v0.1.11 (347b296)

One issue here is that multiple runs with different versions of the tool will result in different results. Thus, subsequent runs with no changes (other then the version of the tool ran) will result in non-idempotent results. This could trigger downstream processes unnecessarily (including deployment to Kong).

[openapi2kong] Server Port only supports up to 32767

Hi,

If we use a value larger than 32767 as the port in the url under servers, the service port gets set to 32767. We should be able to specify ports up to 65535.

openapi

servers:
  - url: 'http://127.0.0.1:50000'

generated decK state

services:
  - port: 32767
...

It seems that it's because of this line in code:

go-apiops/openapi2kong/service.go

Line 233 in dab2c18

service["port"], _ = strconv.ParseInt(targets[0].Port(), 10, 16)

Setting bit size to 16 doesn't support any higher port numbers. There's actually an error that is returned but that is ignored.

Deck file merge - Entity already exists

Hi Guys,

I am setting up a gitlab ci pipeline that will allow publishing apis in kong. The error message is Error: building state: inserting route Swagger_Petstore-listPets: entity already exists.

In my pipeline, I have to convert an openapi (oas) specification to a kong specification (this is done with the command deck file openapi2kong -s oas.yaml -o kong-spec.yaml).
Then I make a backup of the current Kong configuration (deck dump -o current-spec.yaml) and I merge current-spec.yaml and kong-spec.yaml files to get the final file to synchronize on kong (deck file merge current-spec.yaml kong-spec.yaml -o merge-spec.yaml).

At the first occurrence of publication, it goes without a hitch. (deck diff -s merge-spec.yaml; deck sync -s merge-spec.yaml). On the other hand for all other occurrences of file merge, it does not work since the elements defining the api some already exist in kong.

So with the deck tool is it possible to do a file merge by solving the problem of duplication and prioritizing the specification to keep according to its position in the list of files provided as a parameter to deck.

Deck version : v1.25.0
Kong version : v3.2

Thanks in advance

routes default for strip_path to be taken from x-kong-route-defaults

Issue: strip_path for routes not taking defaults from x-kong-route-defaults given in OAS

deck version: 1.25.0

OAS spec:

openapi: 3.0.0
info:
  description: "Predict the nationality of a name. Nationalize.io predicts the nationality of a person given their name. Use the API for analytics, ad segmenting, demographic statistics etc."
  version: 1.0.0
  title: Nationalize
  termsOfService: http://swagger.io/terms/
  contact:
    email: [email protected]
  license:
    name: Apache 2.0
    url: http://www.apache.org/licenses/LICENSE-2.0.html
servers: [{url: "https://api.nationalize.io"}]
x-kong-tags: [ nationality ]
x-kong-route-defaults:
  strip_path: true
  protocols: [https]
paths:
  /nationalize:
    get:
      tags:
        - nationality
      summary: Get nationality of user
      description: "Get nationality of user"
      operationId: getNationality
      parameters: 
        - in: query
          name: "name"
          schema: 
            type: string
          required: true
      responses:
        "200":
          content: 
            application/json: 
              schema: 
                $ref: '#/components/schemas/Response'
          description: Valid Response
        "401":
          content: 
            application/json: 
              schema: 
                $ref: '#/components/schemas/Error'
          description: Invalid API Key
        "422":
          content: 
            application/json: 
              schema: 
                $ref: '#/components/schemas/Error'
          description: Invalid input
components:
  schemas:
    Response:
      type: object
      properties: 
        count: 
          type: number
        name:
          type: string
        country:
          type: array
          items: 
            type: object
            properties:
              country_id:
                type: string
              probability:
                type: number
    Error: 
      type: object
      properties: 
        error: 
          type: string

cli cmd: deck file openapi2kong --spec nationalize.yml --output-file nationalize-config.yml

It shows TODO here - Maybe this can be supported sooner, this would help in ApiOps

https://github.com/Kong/go-apiops/blob/b95b5904b54834999e9dc60db4caded6a7b826b5/openapi2kong/openapi2kong.go#L844C11-L844C21

list-tags is missing some tags

$ ./deck file openapi2kong -s ../go-apiops/docs/learnservice_oas.yaml --select-tag=thijs,Tieske | ./deck file list-tags
Tieske
$

Expected output would be;

thijs
Tieske

Deck openapi2kong regex_priority forcing unsigned int

When using deck v1.35 and looking to convert an openapi file with the command deck openapi2kong file-name.yaml

The regex_priority when set to a negative number, will become a very large integer on a path given that that path has regex within it. For example if I have a path of /test/{tester} and it has a regex_priority of -10, the converted file output will look like the following: regex_priority: 18446744073709551605

This seems to be the max size of unsigned int minus 10.

The code snippet I believe is this here:

go-apiops/openapi2kong/openapi2kong.go

Lines 1104 to 1106 in 05ef8ef

	if regexPriority == regexPriorityWithPathParams {
	// this is a path with parameters, so we need to lower the priority
	route["regex_priority"] = currentRegexPrio - 1

This can be replicated with the file attached and using the command deck file openapi2kong -s openapi.yml

openapi.txt

Docker and local build result in different binary behaviors

For example:

❯ docker run kong/kced:v0.1.11 version
kceD kong/kced:v0.1.11 (347b2965c3713219aff3844306878ca492e782a2)

and

❯ kced version | cat -v
kceD v0.1.11 (347b296)

I believe that the current docker build process results in a new build with different build variables, instead of packaging up an already built version into the docker image.

We should package a pre-built version as part of the docker build instead of rebuilding.

Schema Regex patterns are not translated to regex paths

When using this library to convert an OAS spec to Kong declarative yaml, if a regex schema pattern is set in the OAS parameters for a path like so:

paths:
  /document-services/v1/generate/{template_id}:  
    parameters:
        - in: path
          name: template_id
          schema:
            type: string
            pattern: '[0-9a-zA-Z_-]+'
          required: true
          description: ID of the template to generate document from given content

The resulting conversion to Kong declarative yaml does not take into account this pattern when generating the path in route yaml:

    paths:
    - ~/document-services/v1/generate/(?<template_id>[^#?/]+)$

Unsure if this is a bug, a feature, not the intention of this library, or would have some other consequences that I am not aware of.

Please advise if this is something that would be considered a good addition to this library.

CVE-2021-3538 on package github.com/satori/go.uuid

Hello,

Go module github.com/satori/go.uuid version 1.2.0 is unmaintained and has a weak number generator, thus should be replaced.

CVE details: https://nvd.nist.gov/vuln/detail/CVE-2021-3538
Reported issue: satori/go.uuid#120 (which also links to satori/go.uuid#73 for more context.

As others have advised, this could potentially be replaced with https://github.com/gofrs/uuid or https://github.com/google/uuid.

Hopefully this provides enough context.

With regards,
Benjamin

patch command will not add values to null key

Take this input config (input.yaml)

# This will be used as the base file for
# all default _plugin_configs.  The actual
# defaults will be patched in incrementally
_plugin_configs:

And this patch file (patch.yaml):

_format_version: "1.0"
patches:
  - selectors:
    - $._plugin_configs
    values:
      default-jwt:
        name: jwt
        config:
          enabled: true

Ran as:

deck file patch -s input.yaml patch.yaml

Results in:

_plugin_configs: null

If I change input.yaml to:

# This will be used as the base file for
# all default _plugin_configs.  The actual
# defaults will be patched in incrementally
_plugin_configs:
  dummy:

The result will be:

_plugin_configs:
  default-jwt:
    config:
      enabled: true
    name: jwt
  dummy: null

The patch command should be able to create a new key in an existing null valued key.

CLI(openapi2kong) hangs if argument specified with nothing on STDIN

As a user, i've made this mistake:

./kced openapi2kong ./docs/mock-a-rena-oas.yml

Where I thought I was specifying a spec file, but the command hangs waiting for input on STDIN which will never come.

Instead, I wanted this:

./kced openapi2kong -s ./docs/mock-a-rena-oas.yml

Can we detect this situation and help the user with an error?

The patch command should always fail when a given patch is not applied

Examples

Assume in.yaml:

_plugin_configs: {}
data:
  hello: "world"
  stuffs: ["abc", "123"]
  foo: null
  bar:
  obj:
    dog: "barks"

A value that is null, an array, or a scalar cannot have an object patched. This should result in an error and a non-zero exit code. Currently the tool returns a 0 exit code and the unaffected input.

cat in.yaml | deck file patch --selector "$.data.foo" --value 'xyz:"abc"'
_plugin_configs: {}
data:
  bar: null
  foo: null
  hello: world
  obj:
    dog: barks

This ticket should also capture any additional ways that the patch command fails silently that we are aware of.

These are bugs and not breaking changes. Any existing users that "rely" on this broken behavior have the opportunity to resolve the incorrect usage of the tool because of the new raised failure.

Inconsistent results with recursive descent with filter expression

In the repo we have a sample deck file: docs/summertime-kong.yml that has one service and one route and the route is defined under the service.

Applying a patch with a selector that uses recursive descent and a filter expression results in (I believe) inconsistent results:

The first example's selector is explicit in the path to the routes:

> kced patch -s docs/summertime-kong.yml --selector '$.services[*].routes[?(@.name=="summer-time_get")]' --value 'field:"foo"'
_format_version: "3.0"
services:
- host: mockbin.org
  id: b1525aee-d304-11ed-afa1-0242ac120002
  name: summer-time
  path: /requests
  plugins: []
  port: 443
  protocol: https
  routes:
  - field: foo
    id: b7d87736-d304-11ed-afa1-0242ac120002
    methods:
    - GET
    name: summer-time_get
    paths:
    - ~/summer-time$
    plugins: []
    regex_priority: 200
    strip_path: false
    tags: []

In the second example we use recursive descent which should result in the same behavior, but do not get the expected results:

❯ kced patch -s docs/summertime-kong.yml --selector '$..routes[?(@.name=="summer-time_get")]' --value 'field:"foo"'
_format_version: "3.0"
services:
- host: mockbin.org
  id: b1525aee-d304-11ed-afa1-0242ac120002
  name: summer-time
  path: /requests
  plugins: []
  port: 443
  protocol: https
  routes:
  - id: b7d87736-d304-11ed-afa1-0242ac120002
    methods:
    - GET
    name: summer-time_get
    paths:
    - ~/summer-time$
    plugins: []
    regex_priority: 200
    strip_path: false
    tags: []

Using https://jsonpath.com/ to validate, I see the following: