deis / router Goto Github PK

View Code? Open in Web Editor NEW

80.0 11.0 57.0 418 KB

Edge router for Deis Workflow

Home Page: https://deis.com

License: MIT License

Makefile 4.26% Go 94.91% Shell 0.77% HTML 0.06%

deis-workflow nginx kubernetes k8s unsupported

router's Issues

Support configurable log level

This is one of just a few remaining features that alpha requires to achieve v1.x parity sans SSL support.

Examples

I have the router up and running now. Was working through the examples. Does the router understand the nodeport or is it configured to look at the container port?

My question is after setting up the hosts in /etc/hosts (e.g. nginx, apache, etc) and accessing via http://nginx.example.com: I got a 404 back from nginx.

Where does the config get written to in the container? I wanted to validate that it actually picked up the example services.

[25/Feb/2016:17:59:43 +0000] - 10.2.12.0 - - - 404 - "GET / HTTP/1.1" - 406 - "-" - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36" - "_" - - - nginx.example.com:31605 - - - 0.000

This post capturing-5xx-errors-debug-server shows how is possible to use a custom upstream to forward errors and obtain more details about the reason. This also enables the customization of the default and error pages (deis/deis#1082) without the need of a custom image

Add support for affinityArg

v1.x supported this. v2 should as well.

Add automatic SSL certificate registration through letsencrypt

Currently, the SSL options provided by deis/router support manual configuration of ssl certs. There's also support for a wildcard cert. It would be ideal if deis/router provided integration with Let's Encrypt so that an ssl cert is automatically generated (and maintained overtime) for each routable service's domains.

There's an issue over in the deis/deis queue which begins to address this functionality (deis/deis#4681) but given the functionality should probably be integrated directly into router, I'm creating an issue here in this queue, so that other people (like myself) can find it!

I don't have any immediate plans to begin implementing this functionality, but I'll keep this issue up to date when I do finally begin.

Routable label should be "namespaced"

Currently, the router finds "routable" apps by querying the k8s API for services with the label routable: "true".

Ideally, these labels should probably change to router.deis.io/routable: "true"

cc @helgi: Workflow would need a complementary change.

Add "production considerations" section to documentation

Since helm charts are really aimed at getting folks up and running fast and a proper deployment might require some customizations to the charts, it would be good to have checklist of things people should be sure are accounted for when running in production. (v1.x docs had a similar section.)

Off the top of my head:

Define the router's default domain (sometime aka "platform domain). If it's not defined vhosts all match on a regex. It's better to be explicit.
Provide a unique dhparam for Diffie Hellman key exchange.

Run docker build on all PR builds

Similar to what builder does in its .travis.yml file, router should run make ... docker-build to ensure that the Dockerfile is correct

Add support for "configuration extensions"

In the back of my head, I want to make alternate implementations of router possible, so all along I've intended to avoid exposing configuration options that are nginx-specific. This is easier said than done because, frankly, even with more generic names, the bulk of the router configuration is nginx-specific. Getting creative in coming up with more generic names for things only ends up obfuscating the function of each option.

To work around this, I propose adding support for implementation-specific "extensions" to the router configuration. So, for instance, anything that is undeniably nginx specific and meaningless in the context of another hypothetical implementation (such as HAProxy), might be moved into an "extensions": { "nginx": { ... } } section of the configuration.

add a model_test.go

#23 introduces non-trivial functionality, so we should have tests for it

Add CI using travis

Add containerized style checks

Add functional tests

The main barrier to this from the onset is that router requires a k8s apiserver to talk to...

I'm thinking of using a containerized, single node k8s to fulfill that requirement. See http://kubernetes.github.io/docs/getting-started-guides/docker/

Router config is driven off of annotations on the router rc, application services, and secrets. It's probably possible to see the containerized k8s with some of these things and then assert that router-generated nginx config appears as expected.

The obvious sticky point here might be that the containerized k8s needs to run privileged.

Router doesn't seem to be routing to controller

I seem to be able to get to NGINX, but not get routed through it to the controller. I don't see an issue in the queue covering it.

First, figure out how to hit the router:

⇒  ky --namespace=deis get po deis-registry-gmr1h
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"deis","name":"deis-registry","uid":"60a31f81-a29b-11e5-8beb-0800279dd272","apiVersion":"v1","resourceVersion":"161"}}
  creationTimestamp: 2015-12-14T19:46:32Z
  generateName: deis-registry-
  labels:
    name: deis-registry
  name: deis-registry-gmr1h
  namespace: deis
  resourceVersion: "226"
  selfLink: /api/v1/namespaces/deis/pods/deis-registry-gmr1h
  uid: 60a50b8e-a29b-11e5-8beb-0800279dd272
spec:
  containers:
  - env:
    - name: REGISTRY_STORAGE_DELETE_ENABLED
      value: "true"
    - name: REGISTRY_LOG_LEVEL
      value: info
    image: quay.io/deisci/registry:v2-alpha
    imagePullPolicy: Always
    name: deis-registry
    ports:
    - containerPort: 5000
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/lib/registry
      name: registry-storage
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-9nood
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: 10.245.1.3
  restartPolicy: Always
  serviceAccount: default
  terminationGracePeriodSeconds: 30
  volumes:
  - emptyDir: {}
    name: registry-storage
  - name: default-token-9nood
    secret:
      secretName: default-token-9nood
status:
  conditions:
  - status: "True"
    type: Ready
  containerStatuses:
  - containerID: docker://c495fae507ef4eb7449ece84ae012a2bd49383258bde0b970592cb238b58c33d
    image: quay.io/deisci/registry:v2-alpha
    imageID: docker://5dc9de767f4d89af5246fe38386aac2224b89c007938867afec345776c9af376
    lastState: {}
    name: deis-registry
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2015-12-14T19:49:45Z
  hostIP: 10.245.1.3
  phase: Running
  podIP: 10.246.93.12
  startTime: 2015-12-14T19:46:33Z

Test with curl:

⇒  curl http://10.245.1.3
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.9.6</center>
</body>
</html>

Test with xip.io:

⇒  curl http://deis.10.245.1.3.xip.io                                                             6 ↵
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.9.6</center>
</body>
</html>

But then if I use the deis client, I get this:

⇒  deis login http://deis.10.245.1.3.xip.io
Error: http://deis.10.245.1.3.xip.io does not appear to be a valid Deis controller.

SSL example requires a correction

This line is wrong:

https://github.com/deis/router/blame/master/README.md#L343

The cert's name (per the previous example) should be www-example-com-cert

Add stats endpoint back to router

It would be awesome if we could add this back but bind it to localhost.

Router pod hanges in Pending state indefinitely

I was attempting to install deis v2-alpha for the first time on a local vagrant coreos-kubernetes single node and the router pod seems to hang in "Pending" state indefinitely.

~/code/deis$ helm -v
helm version 0.3.1+d4c0fa8
~/code/deis$ kubectl version
Client Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.7", GitCommit:"6234d6a0abd3323cd08c52602e4a91e47fc9491c", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.7", GitCommit:"6234d6a0abd3323cd08c52602e4a91e47fc9491c", GitTreeState:"clean"}
~/code/deis$ kubectl get events
FIRSTSEEN   LASTSEEN   COUNT     NAME      KIND      SUBOBJECT   REASON    SOURCE    MESSAGE
~/code/deis$ kubectl get events --namespace=deis
FIRSTSEEN                         LASTSEEN                          COUNT     NAME                        KIND                    SUBOBJECT                              REASON             SOURCE                      MESSAGE
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-builder-vc0xg          Pod                     implicitly required container POD      started            {kubelet 172.17.4.99}       Started with docker id 2c17701859c0
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-etcd-discovery         ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-etcd-discovery-unf2g
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-builder                ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-builder-vc0xg
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-database-3rwhk         Pod                                                            scheduled          {scheduler }                Successfully assigned deis-database-3rwhk to 172.17.4.99
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-database-3rwhk         Pod                     implicitly required container POD      pulled             {kubelet 172.17.4.99}       Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-database-3rwhk         Pod                     implicitly required container POD      created            {kubelet 172.17.4.99}       Created with docker id 072caafe0524
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-database-3rwhk         Pod                     implicitly required container POD      started            {kubelet 172.17.4.99}       Started with docker id 072caafe0524
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-builder-vc0xg          Pod                                                            scheduled          {scheduler }                Successfully assigned deis-builder-vc0xg to 172.17.4.99
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-builder-vc0xg          Pod                     implicitly required container POD      pulled             {kubelet 172.17.4.99}       Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-database               ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-database-3rwhk
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-etcd-discovery-unf2g   Pod                                                            scheduled          {scheduler }                Successfully assigned deis-etcd-discovery-unf2g to 172.17.4.99
Tue, 19 Jan 2016 23:35:47 -0800   Tue, 19 Jan 2016 23:35:47 -0800   1         deis-builder-vc0xg          Pod                     implicitly required container POD      created            {kubelet 172.17.4.99}       Created with docker id 2c17701859c0
Tue, 19 Jan 2016 23:35:48 -0800   Tue, 19 Jan 2016 23:35:48 -0800   1         deis-etcd-1                 ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-etcd-1-7asrh
Tue, 19 Jan 2016 23:35:48 -0800   Tue, 19 Jan 2016 23:35:48 -0800   1         deis-etcd-discovery-unf2g   Pod                     implicitly required container POD      started            {kubelet 172.17.4.99}       Started with docker id e67f1295ba7f
Tue, 19 Jan 2016 23:35:48 -0800   Tue, 19 Jan 2016 23:35:48 -0800   1         deis-etcd-1-7asrh           Pod                                                            scheduled          {scheduler }                Successfully assigned deis-etcd-1-7asrh to 172.17.4.99
Tue, 19 Jan 2016 23:35:48 -0800   Tue, 19 Jan 2016 23:35:48 -0800   1         deis-etcd-discovery-unf2g   Pod                     implicitly required container POD      created            {kubelet 172.17.4.99}       Created with docker id e67f1295ba7f
Tue, 19 Jan 2016 23:35:48 -0800   Tue, 19 Jan 2016 23:35:48 -0800   1         deis-etcd-1-33zlc           Pod                                                            scheduled          {scheduler }                Successfully assigned deis-etcd-1-33zlc to 172.17.4.99
Tue, 19 Jan 2016 23:35:48 -0800   Tue, 19 Jan 2016 23:35:48 -0800   1         deis-etcd-discovery-unf2g   Pod                     implicitly required container POD      pulled             {kubelet 172.17.4.99}       Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:48 -0800   Tue, 19 Jan 2016 23:35:48 -0800   1         deis-etcd-1                 ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-etcd-1-33zlc
Tue, 19 Jan 2016 23:35:48 -0800   Tue, 19 Jan 2016 23:35:48 -0800   1         deis-etcd-1                 ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-etcd-1-a8d98
Tue, 19 Jan 2016 23:35:48 -0800   Tue, 19 Jan 2016 23:35:48 -0800   1         deis-etcd-1-a8d98           Pod                                                            scheduled          {scheduler }                Successfully assigned deis-etcd-1-a8d98 to 172.17.4.99
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-etcd-1-7asrh           Pod                     implicitly required container POD      created            {kubelet 172.17.4.99}       Created with docker id e8a7282684c9
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-minio-35irj            Pod                                                            scheduled          {scheduler }                Successfully assigned deis-minio-35irj to 172.17.4.99
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-registry               ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-registry-x40ur
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-minio                  ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-minio-35irj
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-etcd-1-7asrh           Pod                     implicitly required container POD      pulled             {kubelet 172.17.4.99}       Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-router                 ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-router-3pgpt
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-etcd-1-33zlc           Pod                     implicitly required container POD      pulled             {kubelet 172.17.4.99}       Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-etcd-1-33zlc           Pod                     implicitly required container POD      created            {kubelet 172.17.4.99}       Created with docker id 8d56e3becc0d
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-etcd-1-33zlc           Pod                     implicitly required container POD      started            {kubelet 172.17.4.99}       Started with docker id 8d56e3becc0d
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:35:49 -0800   1         deis-registry-x40ur         Pod                                                            scheduled          {scheduler }                Successfully assigned deis-registry-x40ur to 172.17.4.99
Tue, 19 Jan 2016 23:35:50 -0800   Tue, 19 Jan 2016 23:35:50 -0800   1         deis-registry-x40ur         Pod                     implicitly required container POD      started            {kubelet 172.17.4.99}       Started with docker id 62e3c011dc90
Tue, 19 Jan 2016 23:35:50 -0800   Tue, 19 Jan 2016 23:35:50 -0800   1         deis-workflow-ojpi2         Pod                                                            scheduled          {scheduler }                Successfully assigned deis-workflow-ojpi2 to 172.17.4.99
Tue, 19 Jan 2016 23:35:50 -0800   Tue, 19 Jan 2016 23:35:50 -0800   1         deis-registry-x40ur         Pod                     implicitly required container POD      created            {kubelet 172.17.4.99}       Created with docker id 62e3c011dc90
Tue, 19 Jan 2016 23:35:50 -0800   Tue, 19 Jan 2016 23:35:50 -0800   1         deis-etcd-1-7asrh           Pod                     implicitly required container POD      started            {kubelet 172.17.4.99}       Started with docker id e8a7282684c9
Tue, 19 Jan 2016 23:35:50 -0800   Tue, 19 Jan 2016 23:35:50 -0800   1         deis-registry-x40ur         Pod                     implicitly required container POD      pulled             {kubelet 172.17.4.99}       Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:50 -0800   Tue, 19 Jan 2016 23:35:50 -0800   1         deis-etcd-1-a8d98           Pod                     implicitly required container POD      pulled             {kubelet 172.17.4.99}       Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:50 -0800   Tue, 19 Jan 2016 23:35:50 -0800   1         deis-etcd-1-a8d98           Pod                     implicitly required container POD      created            {kubelet 172.17.4.99}       Created with docker id 016dad3a94c3
Tue, 19 Jan 2016 23:35:50 -0800   Tue, 19 Jan 2016 23:35:50 -0800   1         deis-etcd-1-a8d98           Pod                     implicitly required container POD      started            {kubelet 172.17.4.99}       Started with docker id 016dad3a94c3
Tue, 19 Jan 2016 23:35:50 -0800   Tue, 19 Jan 2016 23:35:50 -0800   1         deis-workflow               ReplicationController                                          successfulCreate   {replication-controller }   Created pod: deis-workflow-ojpi2
Tue, 19 Jan 2016 23:35:51 -0800   Tue, 19 Jan 2016 23:35:51 -0800   2         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           failed             {kubelet 172.17.4.99}       Failed to create docker container with error: no such image
Tue, 19 Jan 2016 23:35:52 -0800   Tue, 19 Jan 2016 23:35:52 -0800   1         deis-minio-35irj            Pod                     implicitly required container POD      started            {kubelet 172.17.4.99}       Started with docker id 6d55db862512
Tue, 19 Jan 2016 23:35:52 -0800   Tue, 19 Jan 2016 23:35:52 -0800   1         deis-minio-35irj            Pod                     implicitly required container POD      created            {kubelet 172.17.4.99}       Created with docker id 6d55db862512
Tue, 19 Jan 2016 23:35:52 -0800   Tue, 19 Jan 2016 23:35:52 -0800   1         deis-workflow-ojpi2         Pod                     implicitly required container POD      pulled             {kubelet 172.17.4.99}       Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:52 -0800   Tue, 19 Jan 2016 23:35:52 -0800   1         deis-workflow-ojpi2         Pod                     implicitly required container POD      started            {kubelet 172.17.4.99}       Started with docker id e11edbcf8edb
Tue, 19 Jan 2016 23:35:52 -0800   Tue, 19 Jan 2016 23:35:52 -0800   1         deis-workflow-ojpi2         Pod                     implicitly required container POD      created            {kubelet 172.17.4.99}       Created with docker id e11edbcf8edb
Tue, 19 Jan 2016 23:35:52 -0800   Tue, 19 Jan 2016 23:35:52 -0800   1         deis-minio-35irj            Pod                     implicitly required container POD      pulled             {kubelet 172.17.4.99}       Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:36:58 -0800   Tue, 19 Jan 2016 23:36:58 -0800   1         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id 71d22054cf12
Tue, 19 Jan 2016 23:36:58 -0800   Tue, 19 Jan 2016 23:36:58 -0800   1         deis-etcd-1-33zlc           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id b35e0b0e83cf
Tue, 19 Jan 2016 23:36:58 -0800   Tue, 19 Jan 2016 23:36:58 -0800   1         deis-etcd-1-33zlc           Pod                     spec.containers{deis-etcd-1}           pulled             {kubelet 172.17.4.99}       Successfully pulled image "quay.io/deis/etcd:2.0.0-alpha"
Tue, 19 Jan 2016 23:36:58 -0800   Tue, 19 Jan 2016 23:36:58 -0800   1         deis-etcd-1-a8d98           Pod                     spec.containers{deis-etcd-1}           pulled             {kubelet 172.17.4.99}       Successfully pulled image "quay.io/deis/etcd:2.0.0-alpha"
Tue, 19 Jan 2016 23:36:58 -0800   Tue, 19 Jan 2016 23:36:58 -0800   1         deis-etcd-1-a8d98           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id 8efd01257eaa
Tue, 19 Jan 2016 23:36:58 -0800   Tue, 19 Jan 2016 23:36:58 -0800   1         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id 71d22054cf12
Tue, 19 Jan 2016 23:36:58 -0800   Tue, 19 Jan 2016 23:36:58 -0800   1         deis-etcd-discovery-unf2g   Pod                     spec.containers{deis-etcd-discovery}   created            {kubelet 172.17.4.99}       Created with docker id 92dca6015865
Tue, 19 Jan 2016 23:36:58 -0800   Tue, 19 Jan 2016 23:36:58 -0800   1         deis-etcd-discovery-unf2g   Pod                     spec.containers{deis-etcd-discovery}   pulled             {kubelet 172.17.4.99}       Successfully pulled image "quay.io/deis/etcd:2.0.0-alpha"
Tue, 19 Jan 2016 23:35:51 -0800   Tue, 19 Jan 2016 23:36:58 -0800   3         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           pulled             {kubelet 172.17.4.99}       Successfully pulled image "quay.io/deis/etcd:2.0.0-alpha"
Tue, 19 Jan 2016 23:36:59 -0800   Tue, 19 Jan 2016 23:36:59 -0800   1         deis-etcd-discovery-unf2g   Pod                     spec.containers{deis-etcd-discovery}   started            {kubelet 172.17.4.99}       Started with docker id 92dca6015865
Tue, 19 Jan 2016 23:36:59 -0800   Tue, 19 Jan 2016 23:36:59 -0800   1         deis-etcd-1-a8d98           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id 8efd01257eaa
Tue, 19 Jan 2016 23:36:59 -0800   Tue, 19 Jan 2016 23:36:59 -0800   1         deis-etcd-1-33zlc           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id b35e0b0e83cf
Tue, 19 Jan 2016 23:36:59 -0800   Tue, 19 Jan 2016 23:36:59 -0800   1         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id 25c8e7a8026b
Tue, 19 Jan 2016 23:36:59 -0800   Tue, 19 Jan 2016 23:36:59 -0800   1         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id 25c8e7a8026b
Tue, 19 Jan 2016 23:37:00 -0800   Tue, 19 Jan 2016 23:37:00 -0800   1         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id 55077cb60778
Tue, 19 Jan 2016 23:37:00 -0800   Tue, 19 Jan 2016 23:37:00 -0800   1         deis-etcd-1-33zlc           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id 1b2c4809d343
Tue, 19 Jan 2016 23:37:00 -0800   Tue, 19 Jan 2016 23:37:00 -0800   1         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id 55077cb60778
Tue, 19 Jan 2016 23:37:00 -0800   Tue, 19 Jan 2016 23:37:00 -0800   1         deis-etcd-1-a8d98           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id c5942ad1fb6d
Tue, 19 Jan 2016 23:37:00 -0800   Tue, 19 Jan 2016 23:37:00 -0800   1         deis-etcd-1-a8d98           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id c5942ad1fb6d
Tue, 19 Jan 2016 23:37:01 -0800   Tue, 19 Jan 2016 23:37:01 -0800   1         deis-etcd-1-33zlc           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id 1b2c4809d343
Tue, 19 Jan 2016 23:37:10 -0800   Tue, 19 Jan 2016 23:37:10 -0800   1         deis-etcd-1-33zlc           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id 52e43cf0db66
Tue, 19 Jan 2016 23:37:10 -0800   Tue, 19 Jan 2016 23:37:10 -0800   1         deis-etcd-1-33zlc           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id 52e43cf0db66
Tue, 19 Jan 2016 23:37:10 -0800   Tue, 19 Jan 2016 23:37:10 -0800   1         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id 739a5a27bfeb
Tue, 19 Jan 2016 23:37:10 -0800   Tue, 19 Jan 2016 23:37:10 -0800   1         deis-etcd-1-a8d98           Pod                     spec.containers{deis-etcd-1}           created            {kubelet 172.17.4.99}       Created with docker id 061026e257b5
Tue, 19 Jan 2016 23:37:11 -0800   Tue, 19 Jan 2016 23:37:11 -0800   1         deis-etcd-1-a8d98           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id 061026e257b5
Tue, 19 Jan 2016 23:37:11 -0800   Tue, 19 Jan 2016 23:37:11 -0800   1         deis-etcd-1-7asrh           Pod                     spec.containers{deis-etcd-1}           started            {kubelet 172.17.4.99}       Started with docker id 739a5a27bfeb
Tue, 19 Jan 2016 23:38:01 -0800   Tue, 19 Jan 2016 23:38:01 -0800   1         deis-registry-x40ur         Pod                     spec.containers{deis-registry}         created            {kubelet 172.17.4.99}       Created with docker id b567282af91f
Tue, 19 Jan 2016 23:38:01 -0800   Tue, 19 Jan 2016 23:38:01 -0800   1         deis-registry-x40ur         Pod                     spec.containers{deis-registry}         started            {kubelet 172.17.4.99}       Started with docker id b567282af91f
Tue, 19 Jan 2016 23:38:01 -0800   Tue, 19 Jan 2016 23:38:01 -0800   1         deis-registry-x40ur         Pod                     spec.containers{deis-registry}         pulled             {kubelet 172.17.4.99}       Successfully pulled image "quay.io/deis/registry:2.0.0-alpha"
Tue, 19 Jan 2016 23:39:20 -0800   Tue, 19 Jan 2016 23:39:20 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         pulled             {kubelet 172.17.4.99}       Successfully pulled image "quay.io/deis/workflow:2.0.0-alpha"
Tue, 19 Jan 2016 23:39:20 -0800   Tue, 19 Jan 2016 23:39:20 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         created            {kubelet 172.17.4.99}       Created with docker id aba00a6696a1
Tue, 19 Jan 2016 23:39:20 -0800   Tue, 19 Jan 2016 23:39:20 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         started            {kubelet 172.17.4.99}       Started with docker id aba00a6696a1
Tue, 19 Jan 2016 23:39:52 -0800   Tue, 19 Jan 2016 23:39:52 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         created            {kubelet 172.17.4.99}       Created with docker id 920004ef72a9
Tue, 19 Jan 2016 23:39:52 -0800   Tue, 19 Jan 2016 23:39:52 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         started            {kubelet 172.17.4.99}       Started with docker id 920004ef72a9
Tue, 19 Jan 2016 23:40:21 -0800   Tue, 19 Jan 2016 23:40:21 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         created            {kubelet 172.17.4.99}       Created with docker id 9a1e0f032687
Tue, 19 Jan 2016 23:40:21 -0800   Tue, 19 Jan 2016 23:40:21 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         started            {kubelet 172.17.4.99}       Started with docker id 9a1e0f032687
Tue, 19 Jan 2016 23:40:40 -0800   Tue, 19 Jan 2016 23:40:40 -0800   1         deis-builder-vc0xg          Pod                     spec.containers{deis-builder}          pulled             {kubelet 172.17.4.99}       Successfully pulled image "quay.io/deis/builder:2.0.0-alpha"
Tue, 19 Jan 2016 23:40:41 -0800   Tue, 19 Jan 2016 23:40:41 -0800   1         deis-builder-vc0xg          Pod                     spec.containers{deis-builder}          created            {kubelet 172.17.4.99}       Created with docker id 0e9770e7b592
Tue, 19 Jan 2016 23:40:41 -0800   Tue, 19 Jan 2016 23:40:41 -0800   1         deis-builder-vc0xg          Pod                     spec.containers{deis-builder}          started            {kubelet 172.17.4.99}       Started with docker id 0e9770e7b592
Tue, 19 Jan 2016 23:40:51 -0800   Tue, 19 Jan 2016 23:40:51 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         started            {kubelet 172.17.4.99}       Started with docker id 2cd7e24b4baf
Tue, 19 Jan 2016 23:40:51 -0800   Tue, 19 Jan 2016 23:40:51 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         created            {kubelet 172.17.4.99}       Created with docker id 2cd7e24b4baf
Tue, 19 Jan 2016 23:41:03 -0800   Tue, 19 Jan 2016 23:41:03 -0800   1         deis-minio-35irj            Pod                     spec.containers{deis-minio}            created            {kubelet 172.17.4.99}       Created with docker id 2783dd751d72
Tue, 19 Jan 2016 23:41:03 -0800   Tue, 19 Jan 2016 23:41:03 -0800   1         deis-minio-35irj            Pod                     spec.containers{deis-minio}            started            {kubelet 172.17.4.99}       Started with docker id 2783dd751d72
Tue, 19 Jan 2016 23:41:03 -0800   Tue, 19 Jan 2016 23:41:03 -0800   1         deis-minio-35irj            Pod                     spec.containers{deis-minio}            pulled             {kubelet 172.17.4.99}       Successfully pulled image "quay.io/deis/minio:2.0.0-alpha"
Tue, 19 Jan 2016 23:41:21 -0800   Tue, 19 Jan 2016 23:41:21 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         started            {kubelet 172.17.4.99}       Started with docker id 922bebaa64fa
Tue, 19 Jan 2016 23:41:21 -0800   Tue, 19 Jan 2016 23:41:21 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         created            {kubelet 172.17.4.99}       Created with docker id 922bebaa64fa
Tue, 19 Jan 2016 23:41:52 -0800   Tue, 19 Jan 2016 23:41:52 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         created            {kubelet 172.17.4.99}       Created with docker id d6612db94e33
Tue, 19 Jan 2016 23:41:52 -0800   Tue, 19 Jan 2016 23:41:52 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         started            {kubelet 172.17.4.99}       Started with docker id d6612db94e33
Tue, 19 Jan 2016 23:42:22 -0800   Tue, 19 Jan 2016 23:42:22 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         started            {kubelet 172.17.4.99}       Started with docker id 9815d3320ba5
Tue, 19 Jan 2016 23:42:22 -0800   Tue, 19 Jan 2016 23:42:22 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         created            {kubelet 172.17.4.99}       Created with docker id 9815d3320ba5
Tue, 19 Jan 2016 23:42:52 -0800   Tue, 19 Jan 2016 23:42:52 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         created            {kubelet 172.17.4.99}       Created with docker id 922baffcd389
Tue, 19 Jan 2016 23:42:52 -0800   Tue, 19 Jan 2016 23:42:52 -0800   1         deis-workflow-ojpi2         Pod                     spec.containers{deis-workflow}         started            {kubelet 172.17.4.99}       Started with docker id 922baffcd389
Tue, 19 Jan 2016 23:43:09 -0800   Tue, 19 Jan 2016 23:43:09 -0800   1         deis-database-3rwhk         Pod                     spec.containers{deis-database}         created            {kubelet 172.17.4.99}       Created with docker id 004f951dbb67
Tue, 19 Jan 2016 23:43:09 -0800   Tue, 19 Jan 2016 23:43:09 -0800   1         deis-database-3rwhk         Pod                     spec.containers{deis-database}         started            {kubelet 172.17.4.99}       Started with docker id 004f951dbb67
Tue, 19 Jan 2016 23:43:09 -0800   Tue, 19 Jan 2016 23:43:09 -0800   1         deis-database-3rwhk         Pod                     spec.containers{deis-database}         pulled             {kubelet 172.17.4.99}       Successfully pulled image "quay.io/deis/postgres:2.0.0-alpha"
Tue, 19 Jan 2016 23:35:49 -0800   Tue, 19 Jan 2016 23:50:42 -0800   55        deis-router-3pgpt           Pod                                                            failedScheduling   {scheduler }                Failed for reason PodFitsPorts and possibly others
~/code/deis$ kubectl get pods --namespace=deis
NAME                        READY     STATUS    RESTARTS   AGE
deis-builder-vc0xg          1/1       Running   0          19m
deis-database-3rwhk         1/1       Running   0          19m
deis-etcd-1-33zlc           1/1       Running   2          19m
deis-etcd-1-7asrh           1/1       Running   2          19m
deis-etcd-1-a8d98           1/1       Running   2          19m
deis-etcd-discovery-unf2g   1/1       Running   0          19m
deis-minio-35irj            1/1       Running   0          19m
deis-registry-x40ur         1/1       Running   0          19m
deis-router-3pgpt           0/1       Pending   0          19m
deis-workflow-ojpi2         1/1       Running   2          19m

shared cert seemingly removed

Consider the following scenario:

two domains are added to an app, domainA and domainB
a previously added cert is attached to both domains
the cert is detached from only domainA

This scenario, in the form of the e2e test seen here, leads to the cert seemingly also detached from the other domainB (or perhaps the cert is removed altogether) even though certs:info shows the attachment to domainB still exists.

The current behavior after the steps above is a 404 is returned when curling domainB's SSL endpoint, whereas one would expect a 200 still be returned - as is the case when both domains are still attached to the cert.

See deis/workflow#492 for related scenario/logic.

Error building model; not modifying configuration

$ kubectl --namespace=deis logs deis-router-egcwv
2015/12/18 21:07:06 INFO: Starting nginx...
2015/12/18 21:07:06 INFO: nginx started.
2015/12/18 21:07:06 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:07:16 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: read tcp 10.244.33.8:54458->10.100.0.1:443: read: connection reset by peer.
2015/12/18 21:07:26 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:07:36 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:07:46 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:07:56 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:08:06 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.

Proposal: Configuration extensions

Replaces #36

Router is very configurable. Many of these configuration options, however, are really, at their core, nginx configuration options. Try as I might, it's very difficult to hide that fact, as many of the options exposed may not even have an analog in a hypothetical, alternative implementation (based on HAProxy, for instance). Although an alternative implementation is only hypothetical at the moment, it's very likely to happen sooner or later, even if only within the community.

To that end, I would like to propose a modest re-organization of the router's configuration options and the accompanying documentation to promote a clear separation between configuration options that are "spec" and therefore non-negotiable and a secondary class of configuration options that are "extensions;" that is to say-- implementation specific.

I would propose that this is to be achieved through a re-namespacing of all configuration options and careful re-organization of the configuration documentation.

By way of example, consider the following configuration options that can be set via annotations on the router's replication controller:

router.deis.io/workerProcesses
router.deis.io/workerConnections
router.deis.io/serverNameHashMaxSize
router.deis.io/gzip.disable
(And many others)

These could be renamed as follows:

router.deis.io/nginx.workerProcesses
router.deis.io/nginx.workerConnections
router.deis.io/nginx.serverNameHashMaxSize
router.deis.io/nginx.gzip.disable

Again by example, certain other annotations on the router's replication controller, such as the following, may not initially appear to be implementation-specific:

router.deis.io/defaultDomain

That is highly-speculative, however, because it's conceivable that other implementations might not need such information. (Even the current implementation considers it optional.) Taking this into consideration, there's no harm in also re-namespacing annotations such as these to:

router.deis.io/nginx.defaultDomain

In contrast to all of the above, all annotations that form the contract between the router and workflow should be considered "spec" and non-negotiable. Any alternative implementation of the router would have to respect these. These encompass all the annotations that workflow sets on "routable" application services, such as:

router.deis.io/domains
router.deis.io/certificates
router.deis.io/whitelist

For improved clarity, these could also be re-namespaced as well:

router.deis.io/app.domains
router.deis.io/app.certificates
router.deis.io/app.whitelist

In the end, I believe, with the right documentation, it becomes clear that the "app" annotations are spec and non-negotiable, whilst the "nginx" annotations are specific to the current router implementation.

I'd like to invite questions or concerns re: this approach.

Proposal: containerize the development environment

This is to lower the barrier to entry for hacking on Deis while also standardizing the development environment for all contributors.

Proposal: drop naxsi application firewall from alpha reqs

Generally, we've been shooting for v1.x feature parity with alpha (minus SSL support). @helgi made a case for also dropping naxsi from alpha and then perhaps slotting in something more fully baked, post alpha. I'll invite him to elaborate...

Allow validation of configuration values

Many configuration values one might set via annotations ultimately wind up in a dynamically generated nginx.conf. It might be nice to detect invalid values for each field, log a warning, and fallback to a default value. This way, Nginx itself never receives invalid configuration and logs will be more descriptive in terms of what configuration errors might exist.

router fails to come up with `FailedScheduling`

when using helm install from the HEAD of master on the deis/charts repo, here's the output of kd describe (kd is an alias for kubectl --namespace=deis):

ENG000656:builder aaronschlesinger$ kd describe pod deis-router-s9nd6
Name:               deis-router-s9nd6
Namespace:          deis
Image(s):           quay.io/deisci/router:v2-alpha
Node:               /
Labels:             app=deis-router
Status:             Pending
Reason:             
Message:            
IP:             
Replication Controllers:    deis-router (1/1 replicas created)
Containers:
  deis-router:
    Container ID:   
    Image:      quay.io/deisci/router:v2-alpha
    Image ID:       
    State:      Waiting
    Ready:      False
    Restart Count:  0
    Environment Variables:
      POD_NAMESPACE:    deis (v1:metadata.namespace)
Volumes:
  deis-token-2vm78:
    Type:   Secret (a secret that should populate this volume)
    SecretName: deis-token-2vm78
Events:
  FirstSeen LastSeen    Count   From        SubobjectPath   Reason          Message
  ─────────   ────────    ───── ────        ───────────── ──────          ───────
  13m       32s     32  {scheduler }            FailedScheduling    Failed for reason PodFitsHostPorts and possibly others

Default cert should not be used for "custom" domains

This is similar to, but different from #136. The two issues should not be confused.

Currently, if you do provide a platform-wide wildcard cert, that cert gets used for securing "custom" (fully-qualified) domains associated to an app-- and that's wrong because the cert is not going to match that domain. Every "custom" domain should only support SSL if a cert is explicitly associated to it.

Router pods fail to completely start

Installation of the stock deis-dev chart results in a router pod starting and going into the Running status, but with no containers running. get pod output looks like this:

deis-router-r7ey9   0/1       Running   1          7s

This is happening on a GKE cluster. See below for additional information:

Replication Controller YAML output

ENG000656:example-go aaronschlesinger$ kd get -o yaml rc deis-router
apiVersion: v1
kind: ReplicationController
metadata:
  annotations:
    chart.helm.sh/description: For testing only!
    chart.helm.sh/file: /Users/aaronschlesinger/.helm/workspace/charts/deis-dev/manifests/deis-router-rc.yaml
    chart.helm.sh/name: deis-dev
    chart.helm.sh/version: v2-beta
  creationTimestamp: 2016-02-19T00:10:02Z
  generation: 1
  labels:
    heritage: deis
  name: deis-router
  namespace: deis
  resourceVersion: "1680306"
  selfLink: /api/v1/namespaces/deis/replicationcontrollers/deis-router
  uid: 1f179601-d69d-11e5-9503-42010af00071
spec:
  replicas: 1
  selector:
    app: deis-router
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: deis-router
    spec:
      containers:
      - env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/deisci/router:v2-beta
        imagePullPolicy: Always
        livenessProbe:
          httpGet:
            path: /healthz
            port: 9090
            scheme: HTTP
          initialDelaySeconds: 1
          timeoutSeconds: 1
        name: deis-router
        ports:
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          protocol: TCP
        - containerPort: 2222
          hostPort: 2222
          protocol: TCP
        - containerPort: 9090
          hostPort: 9090
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /healthz
            port: 9090
            scheme: HTTP
          initialDelaySeconds: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      serviceAccount: deis-router
      serviceAccountName: deis-router
      terminationGracePeriodSeconds: 30
status:
  observedGeneration: 1
  replicas: 1

Pod YAML Output

ENG000656:example-go aaronschlesinger$ kd get -o yaml pod deis-router-r7ey9
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"deis","name":"deis-router","uid":"1f179601-d69d-11e5-9503-42010af00071","apiVersion":"v1","resourceVersion":"1680212"}}
  creationTimestamp: 2016-02-19T00:32:20Z
  generateName: deis-router-
  labels:
    app: deis-router
  name: deis-router-r7ey9
  namespace: deis
  resourceVersion: "1680572"
  selfLink: /api/v1/namespaces/deis/pods/deis-router-r7ey9
  uid: 3cacee5a-d6a0-11e5-9503-42010af00071
spec:
  containers:
  - env:
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    image: quay.io/deisci/router:v2-beta
    imagePullPolicy: Always
    livenessProbe:
      httpGet:
        path: /healthz
        port: 9090
        scheme: HTTP
      initialDelaySeconds: 1
      timeoutSeconds: 1
    name: deis-router
    ports:
    - containerPort: 80
      hostPort: 80
      protocol: TCP
    - containerPort: 443
      hostPort: 443
      protocol: TCP
    - containerPort: 2222
      hostPort: 2222
      protocol: TCP
    - containerPort: 9090
      hostPort: 9090
      protocol: TCP
    readinessProbe:
      httpGet:
        path: /healthz
        port: 9090
        scheme: HTTP
      initialDelaySeconds: 1
      timeoutSeconds: 1
    resources: {}
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: deis-router-token-lc88l
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: gke-aaron-e39d4573-node-hecx
  restartPolicy: Always
  serviceAccount: deis-router
  serviceAccountName: deis-router
  terminationGracePeriodSeconds: 30
  volumes:
  - name: deis-router-token-lc88l
    secret:
      secretName: deis-router-token-lc88l
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: null
    status: "False"
    type: Ready
  containerStatuses:
  - containerID: docker://e21451b0cbf512ce6745677b2ac3c96d8e1a465367619b04eb1478ea490beec2
    image: quay.io/deisci/router:v2-beta
    imageID: docker://22f57e21e69c33e97133b67f1fb26254712637e90241a42e08a060f6198a85a7
    lastState:
      terminated:
        containerID: docker://f14df2200bf3034c6c934e6a5a7434696995e3b8c219332de3431eb431ba07a5
        exitCode: 2
        finishedAt: 2016-02-19T00:40:46Z
        reason: Error
        startedAt: 2016-02-19T00:40:37Z
    name: deis-router
    ready: false
    restartCount: 51
    state:
      running:
        startedAt: 2016-02-19T00:40:47Z
  hostIP: 10.240.0.3
  phase: Running
  podIP: 10.168.1.38
  startTime: 2016-02-19T00:32:20Z

Describe Pod Output

ENG000656:builder aaronschlesinger$ kd describe pod deis-router-r7ey9
Name:               deis-router-r7ey9
Namespace:          deis
Image(s):           quay.io/deisci/router:v2-beta
Node:               gke-aaron-e39d4573-node-hecx/10.240.0.3
Start Time:         Thu, 18 Feb 2016 16:32:20 -0800
Labels:             app=deis-router
Status:             Running
Reason:             
Message:            
IP:             10.168.1.38
Replication Controllers:    deis-router (1/1 replicas created)
Containers:
  deis-router:
    Container ID:       docker://b5c27512a02dd210bc031342fdd06a81777730f4f99baac6348c4754d5cc7c42
    Image:          quay.io/deisci/router:v2-beta
    Image ID:           docker://22f57e21e69c33e97133b67f1fb26254712637e90241a42e08a060f6198a85a7
    State:          Running
      Started:          Thu, 18 Feb 2016 16:52:37 -0800
    Last Termination State: Terminated
      Reason:           Error
      Exit Code:        2
      Started:          Thu, 18 Feb 2016 16:52:27 -0800
      Finished:         Thu, 18 Feb 2016 16:52:36 -0800
    Ready:          False
    Restart Count:      122
    Environment Variables:
      POD_NAMESPACE:    deis (v1:metadata.namespace)
Conditions:
  Type      Status
  Ready     False 
Volumes:
  deis-router-token-lc88l:
    Type:   Secret (a secret that should populate this volume)
    SecretName: deis-router-token-lc88l
Events:
  FirstSeen LastSeen    Count   From                    SubobjectPath               Reason      Message
  ─────────   ────────    ───── ────                    ─────────────             ──────      ───────
  20m       20m     1   {kubelet gke-aaron-e39d4573-node-hecx}  implicitly required container POD   Pulled      Container image "gcr.io/google_containers/pause:0.8.0" already present on machine
  20m       20m     1   {kubelet gke-aaron-e39d4573-node-hecx}  implicitly required container POD   Created     Created with docker id dea7fa3830e7
  20m       20m     1   {scheduler }                                    Scheduled   Successfully assigned deis-router-r7ey9 to gke-aaron-e39d4573-node-hecx
  20m       20m     1   {kubelet gke-aaron-e39d4573-node-hecx}  implicitly required container POD   Started     Started with docker id dea7fa3830e7
  20m       20m     1   {kubelet gke-aaron-e39d4573-node-hecx}  spec.containers{deis-router}        Started     Started with docker id 43461bab895e
  20m       20m     1   {kubelet gke-aaron-e39d4573-node-hecx}  spec.containers{deis-router}        Created     Created with docker id 43461bab895e
  20m       20m     1   {kubelet gke-aaron-e39d4573-node-hecx}  spec.containers{deis-router}        Killing     Killing with docker id 43461bab895e
  20m       20m     1   {kubelet gke-aaron-e39d4573-node-hecx}  spec.containers{deis-router}        Started     Started with docker id d64c66841fca
  20m       20m     1   {kubelet gke-aaron-e39d4573-node-hecx}  spec.containers{deis-router}        Created     Created with docker id d64c66841fca
...snip...

@krancour - please let me know what other information you need from me. I'll keep the cluster running.

Document cloud LoadBalancer tweaks required for router

Refs deis/workflow#156

We should document the specific timeout requirements for the router component, since many long-running operations are pretty much guaranteed during deploy.

Proposal: use ConfigMap instead of annotations (when available)

ConfigMap (scheduled for 1.2)

Router starts but throws errors on some k8s installations

2015/12/21 18:21:08 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: read tcp 10.244.97.6:56526->10.100.0.1:443: read: connection reset by peer.
2015/12/21 18:21:18 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.

It also has a blank config

bash-4.3# cat /opt/nginx/conf/nginx.conf
user nginx;
daemon off;
events {}
http {}

Document how to manually configure an external load balancer

This is important for the cases where k8s is running on infrastructure where the k8s "cloud provider" uses a no-op for services of type: LoadBalancer.

Add support for additional SSL options

The following should be configurable as in v1.x:

sslCiphers
sslDhparam
sslProtocols
sslSessionCache
sslSessionTickets
sslSessionTimeout
sslBufferSize

Metrics

v2 beta requirements include operational metrics from the router. It's not yet clear what specific metrics are needed. This issue is just a placeholder for discussion and to ensure we don't lose track of this requirement.

May need to be more explicit about service account secret creation

See deis/workflow#314

If a cluster does not have the ServiceAccount admission control enabled, the deis service account on which router depends may not have had the secret it requires automatically generated.

This is a probably a pretty low priority, as I believe it's uncommon for this admission control to not be enabled.

How to do CI with such large dependency graph?

Because of this project's dependency on Kubernetes' API, glide up (which I have abstracted behind make bootstrap) resolves a massive graph of 102 dependencies (most of which are probably not needed). If running from scratch, this can take 15 minutes. That's not so bad for a developer. Even if glide up is re-run periodically, in theory, running from scratch is a one-time thing.

But how can we handle this in CI so that tests don't take forevvvvvvvar?

Note that this question first popped into my head after looking at this PR:

deis/etcd#11

Ping @mboersma @sgoings @technosophos for thoughts.

Pod goes into `CrashLoopBackoff` repeatedly

Installing deis-router through the deis helm chart leaves me with deis-router continually flapping:

$ kubectl get pods
NAME                        READY     STATUS              RESTARTS   AGE
deis-builder-lknfs          0/1       CrashLoopBackOff    16         1h
deis-database-yxntz         1/1       Running             0          1h
deis-etcd-1-ekw9s           1/1       Running             0          1h
deis-etcd-1-hrqyk           1/1       Running             0          1h
deis-etcd-1-pd1mu           1/1       Running             1          1h
deis-etcd-discovery-vqlzu   1/1       Running             0          1h
deis-minio-fsmqx            0/1       RunContainerError   0          1h
deis-registry-rarvy         1/1       Running             0          1h
deis-router-2iwxb           0/1       CrashLoopBackOff    15         1h
deis-workflow-tr7eb         1/1       Running             1          1h

Add liveness and readiness probes in manifests

As in deis/charts#60

Run container as non-root user

Right now the container is run as root.

Support timeouts configurable at app-level

This is one of just a few remaining features that alpha requires to achieve v1.x parity sans SSL support.

Strictly speaking, v1.x didn't do this, but these were configurable for the route to the controller in v1.x.

The v2 router treats the "controller" (workflow) as "just another app," which means that this needs to be configurable at the app level.

Workflow does not need to expose the ability to modify this setting. Router just needs to honor it if it is set. This way, the deis.io/routerConfig annotation in the service definition for workflow can be tweaked in the official deis helm chart to raise the timeouts. (This is to support some of the legitimately longer-running requests made to workflow.)

Support router-wide whitelist

This is one of just a few remaining features that alpha requires to achieve v1.x parity sans SSL support.

There is currently no way to diasble gzip compression

When we were using JSON embedded in router rc annotations to configure the router, one could disable gzip compression completely by nilling out the gzipConfig object in that JSON.

Since #72 we are no longer using JSON for router configuration and the ability to disable gzip compression was accidentally lost in the process. It should be restored.

hostname doesn't resolve for 3 mins

Did a git push for workflow and then tried to hit the app when it said it worked.

Couldn't for full 3 minutes

To ssh://[email protected]:2222/madcap-ricochet.git
   • [new branch]      master -> master
     core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
     curl: (6) Couldn't resolve host 'madcap-ricochet.10.3.0.161.xip.io'
     core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
     curl: (6) Couldn't resolve host 'madcap-ricochet.10.3.0.161.xip.io'
     core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
     curl: (6) Couldn't resolve host 'madcap-ricochet.10.3.0.161.xip.io'
     core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
     curl: (6) Couldn't resolve host 'madcap-ricochet.10.3.0.161.xip.io'
     core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
     Powered by Deis
     Release v2 on madcap-ricochet-v2-web-d771p

However looking at the router logs you can see it reloaded on time (confirmed the timings)

2016/01/13 01:25:09 INFO: Router configuration has changed in k8s.
2016/01/13 01:25:09 INFO: Reloading nginx...
2016/01/13 01:25:09 INFO: nginx reloaded.
2016/01/13 01:25:09 [notice] 12#0: signal process started
[13/Jan/2016:01:27:53 +0000] - 10.2.22.1 - - - 200 - "GET / HTTP/1.1" - 221 - "-" - "curl/7.43.0" - "~^madcap-ricochet\x5C.(?<domain>.+)$" - 10.3.0.37:80 - madcap-ricochet.10.3.0.161.xip.io - 0.001 - 0.001

Move some of README into the official docs

Can we move some of the README stuff into the official docs in deis/workflow?

rename deis endpoint to workflow

related to deis/deis#4952. Since the project is being renamed from deis to workflow, the router endpoint should now read workflow.mydomain.com instead of deis.mydomain.com.

Router SSL configuration: allow specification of SSL session ticket key

If we want to scale the NGINX instance to more than one instance and use SSL session tickets, each instance must have the same key.

Could you add that to the router annotation configuration?

Or even better could Deis automatically rotate the keys?
https://blog.twitter.com/2013/forward-secrecy-at-twitter-0

Fix the enforceWhitelists option; add support for default whitelist

Remove the enforceWhitelists option and simply enforce a whitelist for each app if one is defined, and otherwise, behave permissively.

generate router

I'm playing around with the router in a plain vanilla k8s cluster. When I do the helm generate router, it says that it ran 0 generators. Is this normal?

When I do deploy I get a permission denied error related to the service account. I'm trying to figure out if I'm not getting the template built correctly or if I've got more SELinux issues.

certs: a cert attached to a particular domain appears on another

Consider the following case of two domains for a given app, but only attaching a cert to one:

$ deis domains:add domain.one
Adding domain.one to app... done
$ deis domains:add domain.two
Adding domain.two to app... done
$ deis certs:add cert-one server.crt server.key
Adding SSL endpoint... done
$ deis certs:list
    Name   | Common Name | SubjectAltName |        Expires         |   Fingerprint   | Domains |  Updated   |  Created
+----------+-------------+----------------+------------------------+-----------------+---------+------------+------------+
  cert-one | foo.com     |                | 1 Mar 2017 (in 1 year) | 25:64[...]0E:59 |         | 4 Mar 2016 | 4 Mar 2016
$ deis certs:attach cert-one domain.one
Attaching certificate cert-one to domain domain.one... done
$ deis certs:list
    Name   | Common Name | SubjectAltName |        Expires         |   Fingerprint   |  Domains   |  Updated   |  Created
+----------+-------------+----------------+------------------------+-----------------+------------+------------+------------+
  cert-one | foo.com     |                | 1 Mar 2017 (in 1 year) | 25:64[...]0E:59 | domain.one | 4 Mar 2016 | 4 Mar 2016

Now, when we curl both domains, we appear to see the same cert (both return 200 OK as well):

$ curl -kvI -sD - -H "Host: domain.one" https://app.130.211.155.186.xip.io
* Rebuilt URL to: https://app.130.211.155.186.xip.io/
*   Trying 130.211.155.186...
* Connected to app.130.211.155.186.xip.io (130.211.155.186) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: foo.com
> HEAD / HTTP/1.1
> Host: domain.one
> User-Agent: curl/7.47.1
> Accept: */*

$ curl -kvI -sD - -H "Host: domain.two" https://app.130.211.155.186.xip.io
* Rebuilt URL to: https://app.130.211.155.186.xip.io/
*   Trying 130.211.155.186...
* Connected to app.130.211.155.186.xip.io (130.211.155.186) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: foo.com
> HEAD / HTTP/1.1
> Host: domain.two
> User-Agent: curl/7.47.1
> Accept: */*

cc @helgi @krancour

Support app-level whitelists

This is one of just a few remaining features that alpha requires to achieve v1.x parity sans SSL support.

Can we not use JSON in annotations for manually configured things?

Currently we have to manually edit annotations, but annotations are in YAML in JSON even though they are just simple key/value lists:

⇒  kdy get rc deis-router
apiVersion: v1
kind: ReplicationController
metadata:
  annotations:
    chart.helm.sh/description: For testing only!
    chart.helm.sh/file: /Users/mattbutcher/Code/helm_home/workspace/charts/deis/manifests/deis-router-rc.yaml
    chart.helm.sh/name: deis
    chart.helm.sh/version: 2.0.0-pre-alpha
    deis.io/routerConfig: |
      {
        "domain": "10.254.1.3.xip.io",
        "useProxyProtocol": false
      }

Is there a reason why we can't simply make these separate annotations?

They are error prone to edit, require additional work to parse, and require additional tooling for things like Helm.

Determine client IP when PROXY protocol not used

Port deis/deis#4877 to this project.

deis / router Goto Github PK

router's Issues

Replication Controller YAML output

Pod YAML Output

Describe Pod Output

Recommend Projects

Recommend Topics

Recommend Org