kubeshop/kusk-gateway

Kusk-gateway is an OpenAPI-driven API Gateway for Kubernetes

Dear users and contributors of Kusk and Kusk Gateway!

We continue to believe in the idea of OpenAPI driven Gateway. However, we have decided to pause ongoing active development and support for the time being.

We want to express our sincere gratitude to everyone who has contributed to Kusk, whether by submitting code, reporting issues, or simply using it. We are proud of the work we accomplished together, and hope that our efforts have had a positive impact on your work and your community.

If you're willing to improve the project, please feel free to contribute. We will do our best to review and approve incoming PRs as our time and resources allow.

Thank you for your understanding and support!

The Kusk Team at Kubeshop

Welcome to Kusk Gateway - a self-service API gateway powered by OpenAPI and Envoy

Website |  Documentation |  Twitter |  Discord |  Blog

Welcome to Kusk Gateway

Kusk Gateway enables you to design and configure your APIs from single OpenAPI manifest and you can:

• ⚡ Rapidly prototype your APIs by mocking your API responses, allowing your teams to instantly start building on top of your APIs without your services being implemented
• 🔒 Protect your endpoints with automatic request and response validations
• ⚠️ Configure critical policies like request timeouts and CORS with no coding required
• 🧐 Centrally control your APIs from an Open Source dashboard

Get Started

Check out our Getting Started section to install Kusk Gateway, deploy an example API and connect a service to the API.

Contributing

Go to contribution document to read more how can you help us 🔥

Feedback

Whether it helps you or not - we'd LOVE to hear from you. Please let us know what you think and of course, how we can make it better. Please join our growing community on Discord.

Development

See SKAFFOLD.md.

kusk-gateway's Issues

CORS support

• For each endpoint with CORS we should have method OPTIONS (if not already specified in OpenAPI) that allows to get CORS headers.
• Envoy configuration makes CORS policy applicable per vhost and per route as well, we should make clear distinction when to use what - right now we configure everything per route.

API: build envoy config from API CRDs

API CRD: validation webhook

CI: setup actions to build and test the code

Traffic management: add rewrites

Add rewrites of path to backed. Note that trip_prefix option is does this currently too.

Fix parser.go panics

I.e. if service: disabled is not present the parser it would panic.
There are a few other places it might get broken too.

Websockets: basic support - specify them somehow in the OpenAPI spec, route traffic to them, metrics

Metrics: basic API usage / performance metrics

Traffic management: Circuit breaker

static routing support

OpenAPI is not feature complete and using its definition it won't be possible to host the site with API AND front.
I.e. static route to index.html should exist, e.g.:

path:
  '/': 
   backend: front.namespace:80

Define how should we supply it - either with OpenAPI CRD or separately.

API CRD: handle deletes

Local development Makefile target

We can run the operator locally with make run, but we need to disable websockets, because CRDs are installed with them and API server can't reach the local development server.

Or make them work via routing the traffic from the API server to local running operator somehow (probably more complicated).

Traffic management: Retries

Protect API gateway and underlying services from malicious / malformed JSON

It could make sense to incorporate some level of protection into kusk-gateway, i.e. something like this.

Other security features could be: SYN flood protection, max payload size, countermeasures for slowloris attack, etc.
This would make kusk-gateway more appealing for hosting production workloads.

Observability: Dashboard - metrics

docs: x-kusk extension

Add admission control (webhook) support for StaticRoutes CRD

Introduce internal Route parameter to envoy/config/config.go#AddRoute

docs: README.md

Deletion of last API CRD causes Envoy to receive malformed configuration

STR:

1. Have a fresh cluster
2. make run
3. kubectl apply -f examples/httpbin && kubectl rollout status -w deployment/httpbin
4. Check that Envoy has received the updated configuration
5. kubectl delete -f examples/httpbin/
6. Check that Envoy is now receiving malformed configuration

Request payload validation according to OpenAPI spec

Create a prototype where request is validated against the spec.
See https://github.com/olensmar/openapi-sidecar as a reference.

Change license to MIT in kubebuilder boilerplate and already generated files

We have a MIT license, but boilerplate.txt contains Apache license by default. Also we've already pushed some files with that boilerplate and they need to be changed manually.

Add RateLimit support

Add support for local RateLimiting (without e.g. Redis)

Static Envoy Fleet Deployment

For an internal MVP, we've agreed to have a static Envoy Fleet deployment fetching settings from xDS API server.
This task's scope includes:

• Envoy Fleet deployment
• static configuration for it to fetch settings from xDS API server
• Makefile adjustments to easily deploy and iterate over it

Possibility to use requests validation based on OpenAPI schema

CI: Add GH action for releasing

We need automation for releasing new versions of kusk-gateway's docker image

The basic version should be building and pushing a tagged docker image to a public docker repo

CLI parameters framework improvements

It is better to switch to cobra CLI framework in order to run "local" mode separately instead of relying on "--in" key in parameters.

TLS: with static certificates

Introduce a new CRD for configuring TLS.

Based on the context and TLS workflows, people will be creating the certificates secrets manually and at the same time LetsEncrypt (CertManager) certificates will be updated dynamically, so we need to send to Envoy information on new certs when this happens.

Since TLS is mapped to the Listener config and EnvoyFleet doesn't manage listener config, but EnvoyConfigManager does when Envoy is ready, we need to send this as part of Envoy application configuration - not as part of Fleet configuration.

Fields for CRD

• Secret ref containing certs
• Cipher suites order string (users will likely exclude or include ciphes depending on their usecases.)

https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/common.proto has some of those,

I think for the first iteration we need only the secret ref and ciphers order (which again has a sane default - we can make it optional for the user, but allow to configure).

Then manager has to pull this on startup/creation and update and send this to Envoy with listener configuration.

The Envoy config manager sends the whole configuration in one go (we don't do delta configuration currently), it must be a part of main configuration routine in https://github.com/kubeshop/kusk-gateway/blob/main/controllers/config_manager.go#L56

Document happy path for testing and first iteration of product

Add CORS support

Add x-kusk CORS configuration

EnvoyFleet CRD programmatic deployment

Once #45 is done we need to think how do we actually perform an EnvoyFleet deployment in cluster.
I've found a rather simple example of that in redis-operator.

Observability: Dashboard - services status (mirror Kubernetess service status)

Validation for suboptions isn't called

While top level x-kusk validation is called, it is skipped for suboptions creating the situation when incorrect values can be submitted.

Handle path regexps for routes

/pet/{petID} must be handled during route modification, right now only strict /pet/ is done.

options: restructure it for a better UX

example api crd should look like this:

apiVersion: gateway.kusk.io/v1
kind: API
metadata:
  name: httpbin-sample
spec:
  # service name and port should be specified inside x-kusk annotation
  fleet: default
  
  spec: |
    swagger: '2.0'
    info:
      title: httpbin.org
      description: API Management facade for a very handy and free online HTTP tool.
      version: '1.0'
    host: httpbin.org
    schemes:
    - http
    - https
    x-kusk:
      hosts:
        - example.com // vhost is created dynamically, but you can override TLS and other settings in EnvoyFleet per-host if you want to
      
      disabled:
      
      mock:
      
      validation:
      
      redirect: // redirect and upstream nodes are mutually exclusive
      
      upstream: // service upstream is k8s-native, host is a generic one, they are mutually exclusive
        service:
          namespace: default
          name: petstore
          port: 8080
        host:
          hostname: example.com
          port: 8080
          
      qos:
        retries:
        timeouts:
        rate_limits:
        
      cors:
      
      path:
        rewrite:
        // remove trim_prefix, superseded by rewrite
        prefix: /env3/api // renamed from base

Static in-cluster Envoy deployment

x-kusk: allow user to specify upstream (backend) in a user-friendly way

We should allow user to specify upstream (backend?) in a few ways:

1. Native Kubernetes Service Name, Namespace and port, i.e.:

backend:
  service:
    name: petstore
    namespace: default
    port: http

2. Fully customizable host, something like:

backend:
  host: somehost.com:80

The first one would be so much easier and more user-friendly for those that route their traffic to the services inside their clusters (most of the cases) and the latter would allow for custom use cases like a different cluster, non-k8s workloads and etc.

Ensure correct regex for routes with parameters

When setting routes we use naive implementation of regex as [A-z0-9]+ for route with parameters (/pet/{id}). This creates conflicts for existing routes (e.g. /pet/findByStatus) since is occasionally caught by /pet/{id} route when it has CORS but /pet/findByStatus doesn't. We need to make sure that regex covers the type that is set in the parameters in OpenAPI.

Helm installer packaging

Traffic management: add redirects (redirect to actual endpoint)

Add trim path configuration

Add x-kusk trim path configuration

Make envoy deployment accessible without port-forward

Currently it's done via port-forwarding. Perhaps we can use k3d's node port forwarding: https://k3d.io/usage/guides/exposing_services/

This is intended for local development / demo purposes only.

Routes order must be more specific

As per https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/route_matching

When Envoy matches a route, it uses the following procedure:
The HTTP request’s host or :authority header is matched to a virtual host.

Each route entry in the virtual host is checked, in order. If there is a match, the route is used and no further route checks are made.

Independently, each virtual cluster in the virtual host is checked, in order. If there is a match, the virtual cluster is used and no further virtual cluster checks are made.

That means if route with path prefix matching was early in the list of routes for vhost, it will win all routes, even if they're more specific (exact path or regex). We need to ensure that routes in vhost are sorted by:

1. Exact path first
2. Longest regex path first
3. Longest prefix path first
   This must ensure that prefix path / will not match any other routes.

Mocking prototype

Implement the way to mock paths with predefined responses as POC.
E.g. using x-kusk construct:

x-kusk:
  mock:
    response: "{ blablabla: true }"

Add Timeouts configuration

Add x-kusk Timeouts configuration

OpenAPI spec to Envoy configuration converter

Have a package that would using an envoy configuration manager create an envoy configuration from OpenAPI spec (without any extensions for now).

Mocking Service based on API spec, enabled explicitly in API CRD, mock responsess provided from OpenAPI examples

EnvoyFleet CRD - basic setup and schema

Provide basic CRD schema and validation with kubebuilder.

Change APIs to v1alpha1

We should change our APIs to v1alpha1 before the general release so that we'd be able to update them later without breaking a compatibility promise (which is implicit if we settle on v1 API).

Have a schema with OpenAPI w/ x-kusk extension

We need to have the definitions of x-kusk extension objects suitable for kusk-gateway to improve user experience. The outcome of this task has to be schema file.