emissary-ingress/emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy

emissary's Issues

Ambassador needs to support Datadog statsd sink

There will be others needed, too, but Datadog is first.

Users need a way to recover from configuration errors

We had a fun situation here where a tester made a bad change to Ambassador's config (specifically, they fed Ambassador an extauth URL that was malformed) and its Envoy crashed. In this case, you can't recover without some serious magic.

Thus:

• move Ambassador's admin port out from behind Envoy
• let Ambassador keep running even if Envoy crashes
• have a way to force Ambassador to kill & restart the Envoy, and/or to force Ambassador to exit so that Kubernetes restarts the pod

I find myself really wanting to understand the formal rules for how the url mapping works (e.g. do I need to start with a slash or end with a slash, etc)

Ambassador's endpoints should change to /mapping/

The docs talk about mappings, but the endpoints say services. That's silly.

Prototype: Ambassador Authentication Filter

First stage of GH-9

• Prototype code to validate the external authentication mechanism and learn how to implement properly as well as learn Envoy toolchain.
• Prototype Documentation
• Plan for moving from Prototype to GA.

internal service for the admin api

External Authentication Service Documentation

We have a prototype authentication service, but we have no documentation for how to use it.

DoD:

• Document how to configure Ambassador to use the auth service
• Document how to use the reference auth service
• Document the REST API between the auth filter and the auth service, so that custom implementations can be written if desired.

Users want to have extauth talk to its auth server with gRPC

Having REST and gRPC would be make life easier for some users than simply REST.

• define and publish the protobuf for such a gRPC
• make extauth able to use it
• make Ambassador able to describe which to use
• interop testing

mysteriously wedged sds

My ambassador got into a mysteriously wedged state. It was responding just fine to /ambassador/* queries, but it refused to route to my upstream "hello" cluster.

After some head scratching I managed to track this down to the sds service being wedged. I was able to diagnose this because a) the sds service was hanging when I queried it, and b) I got the appended stack trace from the logs.

This raises the obvious question of why it got wedged in this way, but perhaps more importantly this points to the need for some kind of health checking on the various envoy "plugin" services.

[rhs@venture hello]$ kubectl logs ambassador-sds-3085809130-cp47x  
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 699, in inner
    srv.serve_forever()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 536, in serve_forever
    HTTPServer.serve_forever(self)
  File "/usr/lib/python2.7/SocketServer.py", line 238, in serve_forever
    self._handle_request_noblock()
  File "/usr/lib/python2.7/SocketServer.py", line 297, in _handle_request_noblock
    self.handle_error(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 232, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 267, in handle_one_request
    return self.run_wsgi()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 209, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 200, in execute
    write(data)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 168, in write
    self.send_header(key, value)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 401, in send_header
    self.wfile.write("%s: %s\r\n" % (keyword, value))
IOError: [Errno 32] Broken pipe

Users need correct handling of service ports

Creating a service in Ambassador shouldn't need a port, because the K8s service record has the port. That requires Ambassador-SDS to honor that port number though (currently, it doesn't).

Make Ambassador installable with Helm

In order to to make Ambassador easier to distribute, install and reuse packaging at a Helm Chart is important.

Output of this Issue should be:

• Helm chart to install Ambassador.
• Documentation on how to use the Helm chart for installation and upgrades.

Users need build documentation

It's not hard to build and run a modified Ambassador... but since it's not documented, it might as well be.

Users need URL rewriting

Envoy supports the prefix_rewrite directive to morph one URL prefix into another. We need to support that.

(We need to support many other directives, too, but this is probably first.)

Python API to operate Ambassador programatically

Desired features

• Get Ambassador URL
• Add certificate for Ambassador to k8s secrets
• Add CA certificate for client auth to k8s secrets
• Map and unmap resources
• Retrieve existing mappings
• Retrieve Ambassador internal stats

This will be useful for setting up release tests and soak tests, as well as providing the foundation for the actl command.

Users need docs for how to get statsd data to their own sink

Ambassador just needs a statsd-sink service to be defined, but we should document that.

Datawire should be running a continuous soak of an Ambassador

Ambassador is meant to have a role that is operationally critical, so we should have one running all the time actually handling significant amounts of traffic while both its services and itself are coming, going, being upgraded, etc.

This would also be a good place to test various types of failures.

Users need to be able to configure statsd

...notably including:

• should stats be sent anywhere?
• which statsd sink should be used?
• where should the aggregated stats be sent?

Users need statsd support

Ambassador needs to be able to send stats off to statsd, whatever statsd the user wants to use.

first time image down load takes a long time, the README instructions should accommodate this

Ambassador sends a POST request to the External Auth, but need a GET request to my external auth.

Can this be configurable?

Operational Documentation: Installation and Upgrades

Ambassador needs documentation around Installation and Upgrade that is written from an Operational / Administrative perspective for production usage.

• How to install Ambassador
• How to update Ambassador (Ambassador container, Envoy, PostgreSQL)

traceback from clean install on an empty minikube cluster

I've included the logs below. The deployment goes into the CrashLoopBackoff and never seems to come alive... I see this in the logs...

[rhs@venture ambassador]$ kubectl logs ambassador-381468975-trd2g ambassador 
/application
[2017-05-04 01:20:23.017][9][warning][main] initializing epoch 0 (hot restart version=7.2490552)
[2017-05-04 01:20:23.030][9][warning][main] all clusters initialized. initializing init manager
[2017-05-04 01:20:23.034][9][warning][main] all dependencies initialized. starting workers
[2017-05-04 01:20:23.034][9][warning][main] starting main dispatch loop
2017-05-04 01:20:23 ambassador 0.8.0 INFO: initializing on ambassador-381468975-trd2g (resolved 172.17.0.5)
2017-05-04 01:20:43 ambassador 0.8.0 INFO: ambassador found restarter PID 6
Traceback (most recent call last):
  File "/application/ambassador.py", line 489, in <module>
    main()
  File "/application/ambassador.py", line 475, in main
    new_config(envoy_restarter_pid = -1)    # Don't automagically signal here.
  File "/application/ambassador.py", line 303, in new_config
    num_mappings = len(rc.mappings)
TypeError: object of type 'NoneType' has no len()

Ambassador needs CI

...including, of course, automated tests.

fancier way to get the external IP

No clue how cross brittle or crossplatform the tail/awk stuff is, but at least limiting to just the ambassador service would kill some noise, especially on a busy cluster.

export AMBASSADORIP=$(kubectl get services ambassador | tail -1 | awk '{print $3}')

Users need SSL support

At minimum, they need to be able to specify a cert for the listener, and whether to use SSL to the back end.

Ambassador needs a cleaner build process

The current build process relies on hand-set environment variables, making sure you manually run bump2version, doesn't really support dev builds, etc. Yuck.

A dev build - which should be any build not run under CI - needs to figure out its own version number based on the most recent tag.

CI builds should increment the version number automagically.

Dev builds should default to using the datawire-dev DockerHub org, but that must be overridable.

Users want access to Istio's more complex routing logic

Istio is capable of a rich set of routing behaviors that Ambassador should provide access to, for example:

• A/B testing ("Split evenly between these backends")
• Canary deployment ("1% of traffic goes to this new backend -- OK, now 2%. Now 10%...")
• Service A goes to a cluster of 10 hosts. Service B goes to a cluster of 15. However, 5 of the hosts for B are the same as for A...

Much of these can be managed by weighted round-robin, but some will require intelligence in how Ambassador constructs Envoy clusters.

Users can use TLS client-cert authentication

Envoy already supports this.

DoD:

• users can enable the TLS-cert auth filter
• users have some way to configure valid principals

Istio Feature Check

Check that Istio features work as expected with Amabassdor. The big ticket item are that session identifiers propagate all the way to the edge and HTTP/2 works correctly.

Users want Ambassador to be able to route gRPC services.

This should Just Work in Envoy, but Ambassador may need to know more than it does about gRPC.

• need a way to specify that gRPC is active
• interop testing
• multiple-Ambassador testing

url rewrite would be nice

Users want to not need to manually trigger a reconfiguration.

A Better Way is to automagically reconfigure on POST.

Users would like an easy way to get a cert from LetsEncrypt

We can probably fire up certbot easily...

Users don't want Ambassador to expose port 8001

That's the Envoy admin port. We should gateway it through Ambassador, if it's exposed in any way at all.

External Authentication supports Json Web Tokens (JWT)

Ambassador needs to support Json Web Token ("JWT") an an authentication mechanism.

Information we need to collect:

• Which algorithms?
• Does it need to handle authorization as well as authentication (e.g. checking that the requested resource can be accessed with the claims attached to the JWT)

Outputs:

• There is an External Authentication Service implementation with JWT
• There is documentation for installing and configuring the JWT External Authentication Service.

Don't hint at support for using an existing Postgres

Always deploy ambassador-store instead, which will be used only for us.

Later we'll very likely be changing the storage model, so, better to isolate it.

Write proposal for Ambassador to Manage Istio Ingress

Currently Ambassador maintains a second Ingress rather than just managing the Istio Ingress. It would be ideal to remove this extra hop in each request routing by letting Ambassador manage the Istio Ingress directly. It would also simplify the deployment of Ambassador so that there are not two Ingresses to think about.

Outputs:

• Draft proposal is written

SDS needs a watchdog

If SDS or other such services die, they need to be restarted.

Ambassador SDS breaks on unnamed ports

If you create a service where a ports entry has no name, kaboom.

ambassador-sds uses the port name to match against the service name, in case a service advertises multiple ports that aren't really the same functionality. Should be OK to assume that a nameless port is a match.

Users need Ambassador not to crash if there are no defined services!

Sigh.

Launch needs a clear problem statement

This needs to be written down and agreed upon by all stakeholders ASAP. It will end up being a part of the Quick Start.

Ambassador should provide a way for skunkworks to pull stats

...since skunkworks should be showing nice donuts...

redeploying a kubernetes service results in temporary outage

when I redeploy a kubernetes service sitting behind a given ambassador route, there seem to be 5-10 seconds of downtime... this doesn't happen if I'm connecting directly to the external ip of the service

Ambassador needs to support Graphite statsd sink

Graphite is likely the simplest way we can show some stats.

Users don't actually need to specify a port. Drop that.

It's pointless; SDS handles that.

Ambassador needs to support an authentication service

As it stands, anyone in the world can do anything. That's obviously unacceptable.

The most straightforward way to handle this is to take request headers and hand them off to an auth service (which, naturally, will be vectored through Ambassador).

DoD:

• There exists an Envoy auth filter can call a REST service to get a go/no-go decision (GH #59)
• Datawire CI publishes a Docker image with an Envoy that includes the auth filter
• There exists a reference auth service that the auth filter can talk to
• Datawire CI publishes a Docker image with the auth service
• Ambassador can be configured via REST call to use the auth filter, talking to a named auth service
• Ambassador knows how to configure Envoy according to the configuration call above
• Ambassador configuration is documented (#67)
• REST API is documented (#67)

Ambassador should not use SDS for initial launch

To limit scope for this project, let's just use config rewriting and point services at the Kubernetes DNS entry for the service, rather than using the SDS to query Kubernetes endpoints.

Ambassador needs to migrate to the V2 GRPC Envoy API

The V1 config files are going to go the way of the dinosaur. We need to migrate to the V2 APIs.

Users would like to run more than one Ambassador

In theory this will Just Work™ now. It needs to be tested, though.

Users can use Auth0 / OAuth / OIDC for authentication

This might be covered by the definition of #9...