vmware-archive/gangway

An application that can be used to easily enable authentication flows via OIDC for a kubernetes cluster.

gangway

VMware has ended active development of this project, this repository will no longer be updated.

(noun): An opening in the bulwark of the ship to allow passengers to board or leave the ship.

An application that can be used to easily enable authentication flows via OIDC for a kubernetes cluster. Kubernetes supports OpenID Connect Tokens as a way to identify users who access the cluster. Gangway allows users to self-configure their kubectl configuration in a few short steps.

Deployment

Instructions for deploying gangway for common cloud providers can be found here.

How It Works

Kubernetes supports OpenID Connect (OIDC) as a user authentication mechanism. OIDC is an authentication protocol that allows servers to verify the identity of a user by way of an ID Token.

When using OIDC to authenticate with Kubernetes, the client (e.g. kubectl) sends the ID token alongside all requests to the API server. On the server side, the Kubernetes API server verifies the token to ensure it is valid and has not expired. Once verified, the API server extracts username and group membership information from the token, and continues processing the request.

In order to obtain the ID token, the user must go through the OIDC authentication process. This is where Gangway comes in. Gangway is a web application that enables the OIDC authentication flow which results in the minting of the ID Token.

Gangway is configured as a client of an upstream Identity Service that speaks OIDC. To obtain the ID token, the user accesses Gangway, initiates the OIDC flow by clicking the "Log In" button, and completes the flow by authenticating with the upstream Identity Service. The user's credentials are never shared with Gangway.

Once the authentication flow is complete, the user is redirected to a Gangway page that provides instructions on how to configure kubectl to use the ID token.

The following sequence diagram details the authentication flow:

API-Server flags

gangway requires that the Kubernetes API server is configured for OIDC:

https://kubernetes.io/docs/admin/authentication/#configuring-the-api-server

kube-apiserver
...
--oidc-issuer-url="https://example.auth0.com/"
--oidc-client-id=3YM4ue8MoXgBkvCIHh00000000000
--oidc-username-claim=email
--oidc-groups-claim=groups

Build

Requirements for building

• Go (built with version >= 1.12)
• esc for static resources.

A Makefile is provided for building tasks. The options are as follows

Getting started is as simple as:

go get -u github.com/heptiolabs/gangway
cd $GOPATH/src/github.com/heptiolabs/gangway
make setup
make

gangway's Issues

Incorrect Kubeconfig provided when multiple replicas used

What we're seeing

If there are multiple replicas of Gangway running there is a chance that a use will be provided another users' kubeconfig.

What we expected to see

A user is provided the correct kubeconfig every time.

Notes

This is caused by the Gangway created kubeconfigs being written to disk with the same name every time. Ovbiously this is a major security risk

Proposed solutions

• Have new instances of Gangway check to see if another version is running and initiate a leader election, creating an active/passive configuration for Gangway
• Generate kubeconfigs for a user using a unique ID that can be referenced when a user downloads their kubeconfig.

Example Deployment Config incorrect.

File in question:
docs/yaml/03-deployment.yaml

Error:
Environment "GANGWAY_SECURITY_SESSION_KEY" should be "SECURITY_SESSION_KEY"

securecookie: the value is not valid error

Hi, I'm running Gangway 3.0.0 authenticating from Keycloak and seeing an occasional error.

The steps we think are needed to reproduce is as follows:

1. Visit gangway
2. Click Sign in
3. Successfully get redirected to Keycloak
4. Successfully authenticate with Keycloak
5. Get redirected back to gangway
6. All is well, the user can get their kubectl config etc. No problems.

Now if they don't use gangway or kubernetes for a period (a week in my test instance) and then visit the Gangway page, as soon as they click "sign in" they receive the error:

securecookie: the value is not valid

When this occurred this morning I checked the cookies and noticed 3 from gangway:

• gangway
• gangway_id_token
• gangway_refresh_token

The first two were created Tue, 05 Mar 2019 at 09:55:55, the last a little earlier at 09:55:20.

When I revisited the gangway site and saw the error was approximately Tue, 12 Mar 2019 13:14:10.

Deleting all three cookies, and reloading the gangway home page allowed me to sign in successfully and get my kubectl config.

Add screenshots

We should add screenshots/animation of the service running to the README.

Build and publish images

Need to wire this up to Jenkins and have a full CI/CD build and publish of artifacts.

Any chance of a new release?

We are having to run a custom build from master because of a fix that went in 27/11/2018, I was wondering if there is any chance of a release version anytime soon? I notice the last official release was back in October.

Gangway logs configuration at startup, which includes session security key

At startup, gangway logs the effective configuration including the session security key.

We should either stop logging the config, or blank out the secrets that we don't want to log to stdout.

Replace go-bindata requirement

Replace go-bindata requirement as it is no longer maintained.

Bindata regen issues

There isn't a check if the bindata.go file is out of date which can cause bad deployments.

We should:

1. Put in a check to validate bindata.go
2. Have bindata.go regenerate automatically for each build in the Dockerfile

Make the default namespace configurable

Hey there, thanks for the nifty project!

That would be great if we could configure the default namespace. Without namespace in the kubeconfig, users are placed in default, which they can have no access to depending on the Kubernetes cluster setup.

Cleanup templating

Right now the two templates that are used have a bunch of repetitive html. Make go-bindata work such that there is nested template support.

User and context name are not uniquie any more.

We use Gangway for two clusters and we recently upgraded to v3.0.0.
We use the commands to create a users contexts so they are created in a single file.
After the upgrade the contexts it creates are being overwritten when the second one is created because the same email address is used for authentication for each cluster.

I fixed this issue in #75 and #79.
At some point the file I patched stopped being used and the changes were not merged into the commandline template.

As you can see Email is used for the name of the user.
https://github.com/heptiolabs/gangway/blob/master/templates/commandline.tmpl#L60

Audience and scopes are not apparently implemented as env vars

https://github.com/heptiolabs/gangway/blob/master/cmd/gangway/config.go#L35

Writing a helm chart and really trying to avoid a config map since env vars are easier to manage in helm.

Make user config key in kubeconfig configurable

We manage multiple kubernetes clusters and use gangway for onboarding.

The default template sets the username to be the email in commandline.tmpl. This is overwriting the user once we use another instance of gangway to generate a config for a second cluster.

The option to, for example, set it to {{ .ClusterName }}-{{ .Email }} without forking the project would be great.

Startup logs

Should log the IP:Port as well as that the controller started up to assist with debugging.

Support Multi-cluster

Currently, Gangway is engineered to work on a single cluster. It would be great to allow it to authenticate multiple clusters.

Relocate kubeconfig download button

Personally, I feel that the button to download the kubeconfig should be located at the top of the browser (see below).

cc @stevesloka

Post /token: x509: certificate signed by unknown authority

Hi, I'm having the same problem as #74 but can't get to solve it with the information available there.

My current configMap and deployment are:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gangway
  namespace: auth-system
data:
  gangway.yaml: |
    clusterName: "K8S"
    apiServerURL: "https://10.64.160.231:6443"
    authorizeURL: "https://auth.kubernetes.domain.net/auth"
    tokenURL: "https://auth.kubernetes.domain.net/token"
    clientID: "oidc-auth-client"
    clientSecret: "*******************"
    redirectURL: "https://kubectl.kubernetes.domain.net/callback"
    scopes: ["openid", "profile", "email", "offline_access"]
    usernameClaim: "email"
    emailClaim: "email"
    trustedCAPath: "/cacerts/tls.crt"

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: gangway
  namespace: auth-system
  labels:
    app: gangway
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gangway
  strategy:
  template:
    metadata:
      labels:
        app: gangway
        revision: "1"
    spec:
      containers:
        - name: gangway
          image: gcr.io/heptio-images/gangway:v2.0.0
          imagePullPolicy: Always
          command: ["gangway", "-config", "/gangway/gangway.yaml"]
          env:
            - name: GANGWAY_SESSION_SECURITY_KEY
              valueFrom:
                secretKeyRef:
                  name: gangway-key
                  key: sesssionkey
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            requests:
              cpu: "100m"
              memory: "100Mi"
            limits:
              cpu: "100m"
              memory: "100Mi"
          volumeMounts:
            - name: gangway
              mountPath: /gangway/
            - name: ca
              mountPath: /cacerts/
          livenessProbe:
            httpGet:
              path: /
              port: 8080
            initialDelaySeconds: 20
            timeoutSeconds: 1
            periodSeconds: 60
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            timeoutSeconds: 1
            periodSeconds: 10
            failureThreshold: 3
      volumes:
        - name: gangway
          configMap:
            name: gangway
        - name: ca
          secret:
            secretName: rootca

And the secret rootca was creating using:

For the root certificate:
openssl genrsa -des3 -out rootCA.key 4096

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt

kubectl create secret tls rootCA --key rootCA.key --cert rootCA.crt --namespace auth-system

For creating the certificates I followed the following steps:

For the certificate used when loading gangway:

openssl genrsa -out auth.kubernetes.domain.net.key 2048

openssl req -new -sha256 -key auth.kubernetes.domain.net -subj "/C=US/ST=CA/O=MyOrg, Inc./CN=auth.kubernetes.domain.net" -out auth.kubernetes.domain.net.csr

openssl x509 -req -in auth.kubernetes.domain.net.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out auth.kubernetes.domain.net.crt -days 500 -sha256

kubectl create secret tls auth.kubernetes.domain.net --key auth.kubernetes.domain.net.key --cert auth.kubernetes.domain.net.crt

Which is used in the following daemonset (I have a nginx daemonset in the nodes so I don't use Ingress):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - image: nginx:1.15.3-alpine
        name: nginx
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        volumeMounts:
        - name: "config"
          mountPath: "/etc/nginx"
        - name: secret-volume
          mountPath: /etc/nginx/ssl
      volumes:
      - name: config
        configMap:
          name: nginx-conf
      - name: secret-volume
        secret:
           secretName: auth.kubernetes.domain.net

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    worker_processes 1;
    error_log /dev/stdout info;

    events {
      worker_connections 10;
    }

    http {
      access_log /dev/stdout;
     
      server {
        listen 443 ssl;
        server_name auth.kubernetes.domain.net;
        ssl_certificate /etc/nginx/ssl/tls.crt;
        ssl_certificate_key /etc/nginx/ssl/tls.key;

        location / {
          proxy_pass http://dex.auth-system.svc.cluster.local:5556;
        }
      }
      server {
        server_name kubectl.kubernetes.domain.net
        ssl_certificate /etc/nginx/ssl/tls.crt;
        ssl_certificate_key /etc/nginx/ssl/tls.key;
        listen 443 ssl;
        #listen 80;
        location / {
          proxy_pass http://gangway-svc.auth-system.svc.cluster.local:80;
        }
      }
   }

Update go version

Update go version in both travis and Dockerfile.

APIServerURL is a required configuration field

We should explicitly fail if APIServerURL is not set.

Add Google integration docs

Add docs and example config for getting authed with Google identity services.

Consider moving to viper for config

Viper isn't perfect but makes it easy to do things like live reload of configs and pull overrides optionally form env variables.

Support rendering custom templates

Add support for rendering custom templates provided by the user, instead of the default ones that come out of the box.

Make it possible to pass in oath client-secret via env variable

This means that the bulk of the config file can be managed as a config map and just the secret is managed out of band.

Support for path based routing

Hi, is there anyway to set the base directory (root context) for gangway?
I have a single DNS entry for my cluster and I use path-based routing to the various UIs in my Ingress resources i.e. path: /gangway

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gangway
  namespace: gangway
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: mycluster.example.com
    http:
      paths:
      - backend:
          serviceName: gangwaysvc
          servicePort: http
        path: /gangway
  tls:
  - hosts:
    - mycluster.example.com
    secretName: my-tls

The problem is the gangway SIGN IN button routes to https://mycluster.example.com/login/ instead of https://mycluster.example.com/gangway/login

Click Sign In and you hit the default backend https://mycluster.example.com/login:

x509: certificate is valid for x, y, z, not a

At first, thanks for this great project! It works fine for me, except for this one tiny issue that the created kubeconfig is valid for three internal IP addresses of the cluster and not for the external one. I am not sure if I am making some mistake here or if this already configurable. But it would be cool if I could add another valid IP for the created certificate. Or maybe the address that is set in apiServerURL?

Thanks for any advise!

Parse token error ignored

Currently, the parse token method (https://github.com/heptiolabs/gangway/blob/master/cmd/gangway/handlers.go#L190) which takes an id_token and processes swallows the returned error.

Need to look into how to properly handle this error and not ignore. For exampe, if you pass a token foo it won't return an error.

should username be generated from `emailClaim` or `userClaim@clusterName` from config files?

According to the Documentation in the deployment manifests config-map.yaml the config setting emailClaim should be used to name the "user" part of the config.

However recent changes 1 2 has it being built by userClaim@clusterName from the config.

What is the intention? If the recent changes are the way to move forward then we can update the config and pull the emailClaim out completely, otherwise we should revert back to using the emailClaim.

Update documentation for v3.0

Documentation references a 2.3 container image version. Update the documentation and example config docs.

user in kubeconfig and commandline is still email address

followup from #124

i reconfigured our test cluster so it does not set emailClaim in the configmap but the user is still my email address.

This is what the commandline output looks like:

kubectl config set-credentials [email protected]  \
[snip]
kubectl config set-context APP-Dev --cluster=APP-Dev [email protected]

The resulting kubeconfig looks the same.

Refresh token with Google

Hello,

By default, the token is only valid for 1 hour.

Is it planned to manage the refresh token ?
imho, it's quite constraining for the user.

Thanks

Add kubeconfig context example

Right now gangway shows the user how to set credentials for their user, but does not add examples of creating a context/setting the default context. Add these so that it is even simpler for a user to get going.

Add a way to download the configured kubeconfig file

Currently, gangway displays a set of commands the user must run to produce a kubeconfig file after authentication.

From a UX perspective, it would be nice to be able to hit a button and download a fully configured kubeconfig file.

Big cookies do not work

We came across this issue when having gangway use UAA which gave us quite large tokens.

When the securecookie is bigger then 4k (which seems to be the maximum cookie size on all major browsers) then the cookie is not stored. When /callback then redirects to /commandline the loginRequired middleware "fails" and redirects back to /, we never hit /commandline.
If we switch from the session store from CookieStore to FilesystemStore the whole flow seems to work and we end up hitting the commandlineHandler.

Things we could do to help here:

• when the cookie is ~4k have gangway log a warning
• make the session store configurable (let users switch between CookieSession and FilesystemSession and potentially others)

Add command line instructions for users running on Windows

The commands displayed by gangway assume that the user is running a UNIX-based system.

Add instructions for downloading/configuring kubectl that work on Windows.

Migrate away from go-bindata

It is abandoned and not maintained :(

[session/session.go] Should session information be stored or ignored?

Encountered the following error in my browser after killing a Gangway pod and letting it restart:
securecookie: the value is not valid

Problem went away when attempting to reach Gangway via a private window

It looks like the gorilla/session code is storing data in a way that is sub-optimal per this older bug:
gorilla/sessions#16

I'd like to investigate this issue.

self-sign certificate?

How to solve a problem with self-sign certificate?
Post https://dex.k8s.example.com/token: x509: certificate signed by unknown authority

client secret

I'm trying to understand the security ramifications of gangway. As I understand it, in oidc, a web client's secret should never be given to a user. But the example screenshots in the repo do just that. How does this work?

Session Salt should be configurable

RIght now the salt is hard-coded, should be configurable so users could supply their own: https://github.com/heptiolabs/gangway/blob/master/internal/session/session.go#L25

Add CI for testing

Need to add Travis CI to do run tests on each PR and perform validation.

Display JWT claims after successfully signing in

The JWT claims are used by the kubernetes authentication system to determine a user's username and group membership.

Currently, knowing the username and groups of the user logging in is cumbersome. You have to copy the ID token and decode using something like jwt.io.

Instead, gangway could show the JWT claims on the post-login page. I can think of two user stories for this:

• As a cluster operator configuring gangway, I would like gangway to display the user claims in the post-login page so that configuring and troubleshooting gangway is easier.

• As a user of gangway, I would like the claims to be displayed in the post-login page so that I know what my username is, and which groups I belong to.

Open question: Are there any security concerns with displaying the claims in the browser?

Create secret command in README is incorred

should be kubectl create secret generic gangway --from-file=gangway.yaml=gangway-config.yaml

Add a line to the "kubectl config" script that removes the CA after configuration is done

The CA file is embedded in the kubeconfig file after running through the configuration commands.

Clean up the user's local dir by adding a line to the script that removes the CA file.

docs: Describe configuration file

Need to outline the configuration file noting what each parameter does, expected values and if they are required or not.

Add thorough docs

Need docs on how to use this with kube-apiserver (via kubeadm, kops, etc).

Needs an easy deployment manifest and notes of various configuration options.

Also need docs for various ID providers (auth0, dex, others?)

Bad comment in commandline.tmpl is being rendered in the browser

The commandline.tmpl file has a bad HTML comment, which shows up at the top of the rendered page.

embed CA cert

Currently the CA cert that is copy/pasted to setup kubeconfig is written to disk, would be nice if it was just embedded into the kubeconfig file.

Downloaded kubeconfig is missing data.

It looks like the downloaded kubeconfig is missing some configuration (see below)

apiVersion: v1
clusters:
- cluster:
    certificate-authority: 

It should be ...

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: 

Basically including the -data at the end of certificate-authority.

Adding this manually fixes the issue.

Use sane defaults on the http server

We should implement the changes mentioned here: https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/