(dumb) proxy agent for proxy management

[istio]

(dumb) proxy agent for proxy management

[kyessenov]

We will need a channel to report on the status of the update. This channel depends on the mechanism. We can go through Kube API and have a dedicated TRP endpoints or a field in proxyconfig, or expose a special endpoint for the proxy status in the Manager.

Here is a simple proposal for the config delivery to the proxy:

• Each proxy container get allocated a configmap ID and mounts it as a volume /proxy/config/
• Manager pushes artifacts to the volume: proxy config file and any referenced files (TLS secrets, API IDL file,...)
• (Dumb) agent watches for file system changes under /proxy/config, initiates a hot restart with a new config (backing up old config directory using symlinks)
• If config is OK and ready to be served, post OK status to an API endpoint (on Kubernetes or Istio Manager)
• If config is NOT OK, post error details to the status endpoint
• Manager waits till all proxies report back and marks the update operation as a success or as a failure

This seems like a good compromise in terms of simplicity and features.
We need to elaborate on the distributed proxy problems - two-step publish/commit/undo/redo issues, but let's get at least this workflow done.

Thoughts? @rshriram, Enrico

[rshriram]

In general, this sounds like a fine idea. I have not played around with the Config Map being mounted as a volume and watching it for changes.

A few comments on the dumb agent doing hot restart: the hot restart step might be very proxy specific. For e.g., in nginx, weights can be adjusted in config files, followed by hot reload. In Envoy, weights for various clusters need to be specified in files in a filesystem tree, followed by an atomic symlink swap in the directory that Envoy is watching for changes (CDS api does not cover this). Various other configurations in Envoy are also modifiable via the file system.

Option 1: generate the complete configuration for all proxies and force hot reload always (not optimal).

Option 2: proxy agent has 3 modes for nudging the proxy: hot reload/symlink swap/post-to-Rest-endpoint. The exact mode to be used will be configured when launching the proxy agent.

[kyessenov]

In the interest of simplicity I ruled out config maps and a new endpoint for the proxy config on the manager server.

Instead, an agent subscribes to Kubernetes API and generates a new config an every update.
If the config is different, it triggers a proxy hot reload. The agent binary is built from manager sources and shares controller code with the server. If we need to pass more data from the manager server to the proxy agent, we need to define a TPR.

See PR #32 for details.

1. As for the weights, we can address that in a separate commit (once we have configs for the weights.) You are right, we will need atomic file system updates with symlink swaps.

2. There is no status information being propagated back. We need Proxy registration TPR to track failed config updates. @enricoschiattarella

[ayj]

Looks like this be closed and further proxy agent enhancements tracked with new issues?

[kyessenov]

Yes, most of this is already addressed, and further enhancements are awaiting Envoy PRs.