deis / router Goto Github PK
View Code? Open in Web Editor NEWEdge router for Deis Workflow
Home Page: https://deis.com
License: MIT License
Edge router for Deis Workflow
Home Page: https://deis.com
License: MIT License
This is one of just a few remaining features that alpha requires to achieve v1.x parity sans SSL support.
I have the router up and running now. Was working through the examples. Does the router understand the nodeport or is it configured to look at the container port?
My question is after setting up the hosts in /etc/hosts (e.g. nginx, apache, etc) and accessing via http://nginx.example.com: I got a 404 back from nginx.
Where does the config get written to in the container? I wanted to validate that it actually picked up the example services.
[25/Feb/2016:17:59:43 +0000] - 10.2.12.0 - - - 404 - "GET / HTTP/1.1" - 406 - "-" - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36" - "_" - - - nginx.example.com:31605 - - - 0.000
This post capturing-5xx-errors-debug-server shows how is possible to use a custom upstream to forward errors and obtain more details about the reason. This also enables the customization of the default and error pages (deis/deis#1082) without the need of a custom image
v1.x supported this. v2 should as well.
Currently, the SSL options provided by deis/router support manual configuration of ssl certs. There's also support for a wildcard cert. It would be ideal if deis/router provided integration with Let's Encrypt so that an ssl cert is automatically generated (and maintained overtime) for each routable service's domains.
There's an issue over in the deis/deis queue which begins to address this functionality (deis/deis#4681) but given the functionality should probably be integrated directly into router, I'm creating an issue here in this queue, so that other people (like myself) can find it!
I don't have any immediate plans to begin implementing this functionality, but I'll keep this issue up to date when I do finally begin.
Currently, the router finds "routable" apps by querying the k8s API for services with the label routable: "true"
.
Ideally, these labels should probably change to router.deis.io/routable: "true"
cc @helgi: Workflow would need a complementary change.
Since helm charts are really aimed at getting folks up and running fast and a proper deployment might require some customizations to the charts, it would be good to have checklist of things people should be sure are accounted for when running in production. (v1.x docs had a similar section.)
Off the top of my head:
Similar to what builder does in its .travis.yml file, router should run make ... docker-build
to ensure that the Dockerfile is correct
In the back of my head, I want to make alternate implementations of router possible, so all along I've intended to avoid exposing configuration options that are nginx-specific. This is easier said than done because, frankly, even with more generic names, the bulk of the router configuration is nginx-specific. Getting creative in coming up with more generic names for things only ends up obfuscating the function of each option.
To work around this, I propose adding support for implementation-specific "extensions" to the router configuration. So, for instance, anything that is undeniably nginx specific and meaningless in the context of another hypothetical implementation (such as HAProxy), might be moved into an "extensions": { "nginx": { ... } }
section of the configuration.
#23 introduces non-trivial functionality, so we should have tests for it
The main barrier to this from the onset is that router requires a k8s apiserver to talk to...
I'm thinking of using a containerized, single node k8s to fulfill that requirement. See http://kubernetes.github.io/docs/getting-started-guides/docker/
Router config is driven off of annotations on the router rc, application services, and secrets. It's probably possible to see the containerized k8s with some of these things and then assert that router-generated nginx config appears as expected.
The obvious sticky point here might be that the containerized k8s needs to run privileged.
I seem to be able to get to NGINX, but not get routed through it to the controller. I don't see an issue in the queue covering it.
First, figure out how to hit the router:
⇒ ky --namespace=deis get po deis-registry-gmr1h
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/created-by: |
{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"deis","name":"deis-registry","uid":"60a31f81-a29b-11e5-8beb-0800279dd272","apiVersion":"v1","resourceVersion":"161"}}
creationTimestamp: 2015-12-14T19:46:32Z
generateName: deis-registry-
labels:
name: deis-registry
name: deis-registry-gmr1h
namespace: deis
resourceVersion: "226"
selfLink: /api/v1/namespaces/deis/pods/deis-registry-gmr1h
uid: 60a50b8e-a29b-11e5-8beb-0800279dd272
spec:
containers:
- env:
- name: REGISTRY_STORAGE_DELETE_ENABLED
value: "true"
- name: REGISTRY_LOG_LEVEL
value: info
image: quay.io/deisci/registry:v2-alpha
imagePullPolicy: Always
name: deis-registry
ports:
- containerPort: 5000
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /var/lib/registry
name: registry-storage
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-9nood
readOnly: true
dnsPolicy: ClusterFirst
nodeName: 10.245.1.3
restartPolicy: Always
serviceAccount: default
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: registry-storage
- name: default-token-9nood
secret:
secretName: default-token-9nood
status:
conditions:
- status: "True"
type: Ready
containerStatuses:
- containerID: docker://c495fae507ef4eb7449ece84ae012a2bd49383258bde0b970592cb238b58c33d
image: quay.io/deisci/registry:v2-alpha
imageID: docker://5dc9de767f4d89af5246fe38386aac2224b89c007938867afec345776c9af376
lastState: {}
name: deis-registry
ready: true
restartCount: 0
state:
running:
startedAt: 2015-12-14T19:49:45Z
hostIP: 10.245.1.3
phase: Running
podIP: 10.246.93.12
startTime: 2015-12-14T19:46:33Z
Test with curl:
⇒ curl http://10.245.1.3
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.9.6</center>
</body>
</html>
Test with xip.io:
⇒ curl http://deis.10.245.1.3.xip.io 6 ↵
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.9.6</center>
</body>
</html>
But then if I use the deis
client, I get this:
⇒ deis login http://deis.10.245.1.3.xip.io
Error: http://deis.10.245.1.3.xip.io does not appear to be a valid Deis controller.
This line is wrong:
https://github.com/deis/router/blame/master/README.md#L343
The cert's name (per the previous example) should be www-example-com-cert
It would be awesome if we could add this back but bind it to localhost.
I was attempting to install deis v2-alpha for the first time on a local vagrant coreos-kubernetes single node and the router pod seems to hang in "Pending" state indefinitely.
~/code/deis$ helm -v
helm version 0.3.1+d4c0fa8
~/code/deis$ kubectl version
Client Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.7", GitCommit:"6234d6a0abd3323cd08c52602e4a91e47fc9491c", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.7", GitCommit:"6234d6a0abd3323cd08c52602e4a91e47fc9491c", GitTreeState:"clean"}
~/code/deis$ kubectl get events
FIRSTSEEN LASTSEEN COUNT NAME KIND SUBOBJECT REASON SOURCE MESSAGE
~/code/deis$ kubectl get events --namespace=deis
FIRSTSEEN LASTSEEN COUNT NAME KIND SUBOBJECT REASON SOURCE MESSAGE
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-builder-vc0xg Pod implicitly required container POD started {kubelet 172.17.4.99} Started with docker id 2c17701859c0
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-etcd-discovery ReplicationController successfulCreate {replication-controller } Created pod: deis-etcd-discovery-unf2g
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-builder ReplicationController successfulCreate {replication-controller } Created pod: deis-builder-vc0xg
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-database-3rwhk Pod scheduled {scheduler } Successfully assigned deis-database-3rwhk to 172.17.4.99
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-database-3rwhk Pod implicitly required container POD pulled {kubelet 172.17.4.99} Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-database-3rwhk Pod implicitly required container POD created {kubelet 172.17.4.99} Created with docker id 072caafe0524
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-database-3rwhk Pod implicitly required container POD started {kubelet 172.17.4.99} Started with docker id 072caafe0524
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-builder-vc0xg Pod scheduled {scheduler } Successfully assigned deis-builder-vc0xg to 172.17.4.99
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-builder-vc0xg Pod implicitly required container POD pulled {kubelet 172.17.4.99} Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-database ReplicationController successfulCreate {replication-controller } Created pod: deis-database-3rwhk
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-etcd-discovery-unf2g Pod scheduled {scheduler } Successfully assigned deis-etcd-discovery-unf2g to 172.17.4.99
Tue, 19 Jan 2016 23:35:47 -0800 Tue, 19 Jan 2016 23:35:47 -0800 1 deis-builder-vc0xg Pod implicitly required container POD created {kubelet 172.17.4.99} Created with docker id 2c17701859c0
Tue, 19 Jan 2016 23:35:48 -0800 Tue, 19 Jan 2016 23:35:48 -0800 1 deis-etcd-1 ReplicationController successfulCreate {replication-controller } Created pod: deis-etcd-1-7asrh
Tue, 19 Jan 2016 23:35:48 -0800 Tue, 19 Jan 2016 23:35:48 -0800 1 deis-etcd-discovery-unf2g Pod implicitly required container POD started {kubelet 172.17.4.99} Started with docker id e67f1295ba7f
Tue, 19 Jan 2016 23:35:48 -0800 Tue, 19 Jan 2016 23:35:48 -0800 1 deis-etcd-1-7asrh Pod scheduled {scheduler } Successfully assigned deis-etcd-1-7asrh to 172.17.4.99
Tue, 19 Jan 2016 23:35:48 -0800 Tue, 19 Jan 2016 23:35:48 -0800 1 deis-etcd-discovery-unf2g Pod implicitly required container POD created {kubelet 172.17.4.99} Created with docker id e67f1295ba7f
Tue, 19 Jan 2016 23:35:48 -0800 Tue, 19 Jan 2016 23:35:48 -0800 1 deis-etcd-1-33zlc Pod scheduled {scheduler } Successfully assigned deis-etcd-1-33zlc to 172.17.4.99
Tue, 19 Jan 2016 23:35:48 -0800 Tue, 19 Jan 2016 23:35:48 -0800 1 deis-etcd-discovery-unf2g Pod implicitly required container POD pulled {kubelet 172.17.4.99} Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:48 -0800 Tue, 19 Jan 2016 23:35:48 -0800 1 deis-etcd-1 ReplicationController successfulCreate {replication-controller } Created pod: deis-etcd-1-33zlc
Tue, 19 Jan 2016 23:35:48 -0800 Tue, 19 Jan 2016 23:35:48 -0800 1 deis-etcd-1 ReplicationController successfulCreate {replication-controller } Created pod: deis-etcd-1-a8d98
Tue, 19 Jan 2016 23:35:48 -0800 Tue, 19 Jan 2016 23:35:48 -0800 1 deis-etcd-1-a8d98 Pod scheduled {scheduler } Successfully assigned deis-etcd-1-a8d98 to 172.17.4.99
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-etcd-1-7asrh Pod implicitly required container POD created {kubelet 172.17.4.99} Created with docker id e8a7282684c9
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-minio-35irj Pod scheduled {scheduler } Successfully assigned deis-minio-35irj to 172.17.4.99
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-registry ReplicationController successfulCreate {replication-controller } Created pod: deis-registry-x40ur
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-minio ReplicationController successfulCreate {replication-controller } Created pod: deis-minio-35irj
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-etcd-1-7asrh Pod implicitly required container POD pulled {kubelet 172.17.4.99} Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-router ReplicationController successfulCreate {replication-controller } Created pod: deis-router-3pgpt
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-etcd-1-33zlc Pod implicitly required container POD pulled {kubelet 172.17.4.99} Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-etcd-1-33zlc Pod implicitly required container POD created {kubelet 172.17.4.99} Created with docker id 8d56e3becc0d
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-etcd-1-33zlc Pod implicitly required container POD started {kubelet 172.17.4.99} Started with docker id 8d56e3becc0d
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:35:49 -0800 1 deis-registry-x40ur Pod scheduled {scheduler } Successfully assigned deis-registry-x40ur to 172.17.4.99
Tue, 19 Jan 2016 23:35:50 -0800 Tue, 19 Jan 2016 23:35:50 -0800 1 deis-registry-x40ur Pod implicitly required container POD started {kubelet 172.17.4.99} Started with docker id 62e3c011dc90
Tue, 19 Jan 2016 23:35:50 -0800 Tue, 19 Jan 2016 23:35:50 -0800 1 deis-workflow-ojpi2 Pod scheduled {scheduler } Successfully assigned deis-workflow-ojpi2 to 172.17.4.99
Tue, 19 Jan 2016 23:35:50 -0800 Tue, 19 Jan 2016 23:35:50 -0800 1 deis-registry-x40ur Pod implicitly required container POD created {kubelet 172.17.4.99} Created with docker id 62e3c011dc90
Tue, 19 Jan 2016 23:35:50 -0800 Tue, 19 Jan 2016 23:35:50 -0800 1 deis-etcd-1-7asrh Pod implicitly required container POD started {kubelet 172.17.4.99} Started with docker id e8a7282684c9
Tue, 19 Jan 2016 23:35:50 -0800 Tue, 19 Jan 2016 23:35:50 -0800 1 deis-registry-x40ur Pod implicitly required container POD pulled {kubelet 172.17.4.99} Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:50 -0800 Tue, 19 Jan 2016 23:35:50 -0800 1 deis-etcd-1-a8d98 Pod implicitly required container POD pulled {kubelet 172.17.4.99} Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:50 -0800 Tue, 19 Jan 2016 23:35:50 -0800 1 deis-etcd-1-a8d98 Pod implicitly required container POD created {kubelet 172.17.4.99} Created with docker id 016dad3a94c3
Tue, 19 Jan 2016 23:35:50 -0800 Tue, 19 Jan 2016 23:35:50 -0800 1 deis-etcd-1-a8d98 Pod implicitly required container POD started {kubelet 172.17.4.99} Started with docker id 016dad3a94c3
Tue, 19 Jan 2016 23:35:50 -0800 Tue, 19 Jan 2016 23:35:50 -0800 1 deis-workflow ReplicationController successfulCreate {replication-controller } Created pod: deis-workflow-ojpi2
Tue, 19 Jan 2016 23:35:51 -0800 Tue, 19 Jan 2016 23:35:51 -0800 2 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} failed {kubelet 172.17.4.99} Failed to create docker container with error: no such image
Tue, 19 Jan 2016 23:35:52 -0800 Tue, 19 Jan 2016 23:35:52 -0800 1 deis-minio-35irj Pod implicitly required container POD started {kubelet 172.17.4.99} Started with docker id 6d55db862512
Tue, 19 Jan 2016 23:35:52 -0800 Tue, 19 Jan 2016 23:35:52 -0800 1 deis-minio-35irj Pod implicitly required container POD created {kubelet 172.17.4.99} Created with docker id 6d55db862512
Tue, 19 Jan 2016 23:35:52 -0800 Tue, 19 Jan 2016 23:35:52 -0800 1 deis-workflow-ojpi2 Pod implicitly required container POD pulled {kubelet 172.17.4.99} Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:35:52 -0800 Tue, 19 Jan 2016 23:35:52 -0800 1 deis-workflow-ojpi2 Pod implicitly required container POD started {kubelet 172.17.4.99} Started with docker id e11edbcf8edb
Tue, 19 Jan 2016 23:35:52 -0800 Tue, 19 Jan 2016 23:35:52 -0800 1 deis-workflow-ojpi2 Pod implicitly required container POD created {kubelet 172.17.4.99} Created with docker id e11edbcf8edb
Tue, 19 Jan 2016 23:35:52 -0800 Tue, 19 Jan 2016 23:35:52 -0800 1 deis-minio-35irj Pod implicitly required container POD pulled {kubelet 172.17.4.99} Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Tue, 19 Jan 2016 23:36:58 -0800 Tue, 19 Jan 2016 23:36:58 -0800 1 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id 71d22054cf12
Tue, 19 Jan 2016 23:36:58 -0800 Tue, 19 Jan 2016 23:36:58 -0800 1 deis-etcd-1-33zlc Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id b35e0b0e83cf
Tue, 19 Jan 2016 23:36:58 -0800 Tue, 19 Jan 2016 23:36:58 -0800 1 deis-etcd-1-33zlc Pod spec.containers{deis-etcd-1} pulled {kubelet 172.17.4.99} Successfully pulled image "quay.io/deis/etcd:2.0.0-alpha"
Tue, 19 Jan 2016 23:36:58 -0800 Tue, 19 Jan 2016 23:36:58 -0800 1 deis-etcd-1-a8d98 Pod spec.containers{deis-etcd-1} pulled {kubelet 172.17.4.99} Successfully pulled image "quay.io/deis/etcd:2.0.0-alpha"
Tue, 19 Jan 2016 23:36:58 -0800 Tue, 19 Jan 2016 23:36:58 -0800 1 deis-etcd-1-a8d98 Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id 8efd01257eaa
Tue, 19 Jan 2016 23:36:58 -0800 Tue, 19 Jan 2016 23:36:58 -0800 1 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id 71d22054cf12
Tue, 19 Jan 2016 23:36:58 -0800 Tue, 19 Jan 2016 23:36:58 -0800 1 deis-etcd-discovery-unf2g Pod spec.containers{deis-etcd-discovery} created {kubelet 172.17.4.99} Created with docker id 92dca6015865
Tue, 19 Jan 2016 23:36:58 -0800 Tue, 19 Jan 2016 23:36:58 -0800 1 deis-etcd-discovery-unf2g Pod spec.containers{deis-etcd-discovery} pulled {kubelet 172.17.4.99} Successfully pulled image "quay.io/deis/etcd:2.0.0-alpha"
Tue, 19 Jan 2016 23:35:51 -0800 Tue, 19 Jan 2016 23:36:58 -0800 3 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} pulled {kubelet 172.17.4.99} Successfully pulled image "quay.io/deis/etcd:2.0.0-alpha"
Tue, 19 Jan 2016 23:36:59 -0800 Tue, 19 Jan 2016 23:36:59 -0800 1 deis-etcd-discovery-unf2g Pod spec.containers{deis-etcd-discovery} started {kubelet 172.17.4.99} Started with docker id 92dca6015865
Tue, 19 Jan 2016 23:36:59 -0800 Tue, 19 Jan 2016 23:36:59 -0800 1 deis-etcd-1-a8d98 Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id 8efd01257eaa
Tue, 19 Jan 2016 23:36:59 -0800 Tue, 19 Jan 2016 23:36:59 -0800 1 deis-etcd-1-33zlc Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id b35e0b0e83cf
Tue, 19 Jan 2016 23:36:59 -0800 Tue, 19 Jan 2016 23:36:59 -0800 1 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id 25c8e7a8026b
Tue, 19 Jan 2016 23:36:59 -0800 Tue, 19 Jan 2016 23:36:59 -0800 1 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id 25c8e7a8026b
Tue, 19 Jan 2016 23:37:00 -0800 Tue, 19 Jan 2016 23:37:00 -0800 1 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id 55077cb60778
Tue, 19 Jan 2016 23:37:00 -0800 Tue, 19 Jan 2016 23:37:00 -0800 1 deis-etcd-1-33zlc Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id 1b2c4809d343
Tue, 19 Jan 2016 23:37:00 -0800 Tue, 19 Jan 2016 23:37:00 -0800 1 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id 55077cb60778
Tue, 19 Jan 2016 23:37:00 -0800 Tue, 19 Jan 2016 23:37:00 -0800 1 deis-etcd-1-a8d98 Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id c5942ad1fb6d
Tue, 19 Jan 2016 23:37:00 -0800 Tue, 19 Jan 2016 23:37:00 -0800 1 deis-etcd-1-a8d98 Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id c5942ad1fb6d
Tue, 19 Jan 2016 23:37:01 -0800 Tue, 19 Jan 2016 23:37:01 -0800 1 deis-etcd-1-33zlc Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id 1b2c4809d343
Tue, 19 Jan 2016 23:37:10 -0800 Tue, 19 Jan 2016 23:37:10 -0800 1 deis-etcd-1-33zlc Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id 52e43cf0db66
Tue, 19 Jan 2016 23:37:10 -0800 Tue, 19 Jan 2016 23:37:10 -0800 1 deis-etcd-1-33zlc Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id 52e43cf0db66
Tue, 19 Jan 2016 23:37:10 -0800 Tue, 19 Jan 2016 23:37:10 -0800 1 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id 739a5a27bfeb
Tue, 19 Jan 2016 23:37:10 -0800 Tue, 19 Jan 2016 23:37:10 -0800 1 deis-etcd-1-a8d98 Pod spec.containers{deis-etcd-1} created {kubelet 172.17.4.99} Created with docker id 061026e257b5
Tue, 19 Jan 2016 23:37:11 -0800 Tue, 19 Jan 2016 23:37:11 -0800 1 deis-etcd-1-a8d98 Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id 061026e257b5
Tue, 19 Jan 2016 23:37:11 -0800 Tue, 19 Jan 2016 23:37:11 -0800 1 deis-etcd-1-7asrh Pod spec.containers{deis-etcd-1} started {kubelet 172.17.4.99} Started with docker id 739a5a27bfeb
Tue, 19 Jan 2016 23:38:01 -0800 Tue, 19 Jan 2016 23:38:01 -0800 1 deis-registry-x40ur Pod spec.containers{deis-registry} created {kubelet 172.17.4.99} Created with docker id b567282af91f
Tue, 19 Jan 2016 23:38:01 -0800 Tue, 19 Jan 2016 23:38:01 -0800 1 deis-registry-x40ur Pod spec.containers{deis-registry} started {kubelet 172.17.4.99} Started with docker id b567282af91f
Tue, 19 Jan 2016 23:38:01 -0800 Tue, 19 Jan 2016 23:38:01 -0800 1 deis-registry-x40ur Pod spec.containers{deis-registry} pulled {kubelet 172.17.4.99} Successfully pulled image "quay.io/deis/registry:2.0.0-alpha"
Tue, 19 Jan 2016 23:39:20 -0800 Tue, 19 Jan 2016 23:39:20 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} pulled {kubelet 172.17.4.99} Successfully pulled image "quay.io/deis/workflow:2.0.0-alpha"
Tue, 19 Jan 2016 23:39:20 -0800 Tue, 19 Jan 2016 23:39:20 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} created {kubelet 172.17.4.99} Created with docker id aba00a6696a1
Tue, 19 Jan 2016 23:39:20 -0800 Tue, 19 Jan 2016 23:39:20 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} started {kubelet 172.17.4.99} Started with docker id aba00a6696a1
Tue, 19 Jan 2016 23:39:52 -0800 Tue, 19 Jan 2016 23:39:52 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} created {kubelet 172.17.4.99} Created with docker id 920004ef72a9
Tue, 19 Jan 2016 23:39:52 -0800 Tue, 19 Jan 2016 23:39:52 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} started {kubelet 172.17.4.99} Started with docker id 920004ef72a9
Tue, 19 Jan 2016 23:40:21 -0800 Tue, 19 Jan 2016 23:40:21 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} created {kubelet 172.17.4.99} Created with docker id 9a1e0f032687
Tue, 19 Jan 2016 23:40:21 -0800 Tue, 19 Jan 2016 23:40:21 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} started {kubelet 172.17.4.99} Started with docker id 9a1e0f032687
Tue, 19 Jan 2016 23:40:40 -0800 Tue, 19 Jan 2016 23:40:40 -0800 1 deis-builder-vc0xg Pod spec.containers{deis-builder} pulled {kubelet 172.17.4.99} Successfully pulled image "quay.io/deis/builder:2.0.0-alpha"
Tue, 19 Jan 2016 23:40:41 -0800 Tue, 19 Jan 2016 23:40:41 -0800 1 deis-builder-vc0xg Pod spec.containers{deis-builder} created {kubelet 172.17.4.99} Created with docker id 0e9770e7b592
Tue, 19 Jan 2016 23:40:41 -0800 Tue, 19 Jan 2016 23:40:41 -0800 1 deis-builder-vc0xg Pod spec.containers{deis-builder} started {kubelet 172.17.4.99} Started with docker id 0e9770e7b592
Tue, 19 Jan 2016 23:40:51 -0800 Tue, 19 Jan 2016 23:40:51 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} started {kubelet 172.17.4.99} Started with docker id 2cd7e24b4baf
Tue, 19 Jan 2016 23:40:51 -0800 Tue, 19 Jan 2016 23:40:51 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} created {kubelet 172.17.4.99} Created with docker id 2cd7e24b4baf
Tue, 19 Jan 2016 23:41:03 -0800 Tue, 19 Jan 2016 23:41:03 -0800 1 deis-minio-35irj Pod spec.containers{deis-minio} created {kubelet 172.17.4.99} Created with docker id 2783dd751d72
Tue, 19 Jan 2016 23:41:03 -0800 Tue, 19 Jan 2016 23:41:03 -0800 1 deis-minio-35irj Pod spec.containers{deis-minio} started {kubelet 172.17.4.99} Started with docker id 2783dd751d72
Tue, 19 Jan 2016 23:41:03 -0800 Tue, 19 Jan 2016 23:41:03 -0800 1 deis-minio-35irj Pod spec.containers{deis-minio} pulled {kubelet 172.17.4.99} Successfully pulled image "quay.io/deis/minio:2.0.0-alpha"
Tue, 19 Jan 2016 23:41:21 -0800 Tue, 19 Jan 2016 23:41:21 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} started {kubelet 172.17.4.99} Started with docker id 922bebaa64fa
Tue, 19 Jan 2016 23:41:21 -0800 Tue, 19 Jan 2016 23:41:21 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} created {kubelet 172.17.4.99} Created with docker id 922bebaa64fa
Tue, 19 Jan 2016 23:41:52 -0800 Tue, 19 Jan 2016 23:41:52 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} created {kubelet 172.17.4.99} Created with docker id d6612db94e33
Tue, 19 Jan 2016 23:41:52 -0800 Tue, 19 Jan 2016 23:41:52 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} started {kubelet 172.17.4.99} Started with docker id d6612db94e33
Tue, 19 Jan 2016 23:42:22 -0800 Tue, 19 Jan 2016 23:42:22 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} started {kubelet 172.17.4.99} Started with docker id 9815d3320ba5
Tue, 19 Jan 2016 23:42:22 -0800 Tue, 19 Jan 2016 23:42:22 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} created {kubelet 172.17.4.99} Created with docker id 9815d3320ba5
Tue, 19 Jan 2016 23:42:52 -0800 Tue, 19 Jan 2016 23:42:52 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} created {kubelet 172.17.4.99} Created with docker id 922baffcd389
Tue, 19 Jan 2016 23:42:52 -0800 Tue, 19 Jan 2016 23:42:52 -0800 1 deis-workflow-ojpi2 Pod spec.containers{deis-workflow} started {kubelet 172.17.4.99} Started with docker id 922baffcd389
Tue, 19 Jan 2016 23:43:09 -0800 Tue, 19 Jan 2016 23:43:09 -0800 1 deis-database-3rwhk Pod spec.containers{deis-database} created {kubelet 172.17.4.99} Created with docker id 004f951dbb67
Tue, 19 Jan 2016 23:43:09 -0800 Tue, 19 Jan 2016 23:43:09 -0800 1 deis-database-3rwhk Pod spec.containers{deis-database} started {kubelet 172.17.4.99} Started with docker id 004f951dbb67
Tue, 19 Jan 2016 23:43:09 -0800 Tue, 19 Jan 2016 23:43:09 -0800 1 deis-database-3rwhk Pod spec.containers{deis-database} pulled {kubelet 172.17.4.99} Successfully pulled image "quay.io/deis/postgres:2.0.0-alpha"
Tue, 19 Jan 2016 23:35:49 -0800 Tue, 19 Jan 2016 23:50:42 -0800 55 deis-router-3pgpt Pod failedScheduling {scheduler } Failed for reason PodFitsPorts and possibly others
~/code/deis$ kubectl get pods --namespace=deis
NAME READY STATUS RESTARTS AGE
deis-builder-vc0xg 1/1 Running 0 19m
deis-database-3rwhk 1/1 Running 0 19m
deis-etcd-1-33zlc 1/1 Running 2 19m
deis-etcd-1-7asrh 1/1 Running 2 19m
deis-etcd-1-a8d98 1/1 Running 2 19m
deis-etcd-discovery-unf2g 1/1 Running 0 19m
deis-minio-35irj 1/1 Running 0 19m
deis-registry-x40ur 1/1 Running 0 19m
deis-router-3pgpt 0/1 Pending 0 19m
deis-workflow-ojpi2 1/1 Running 2 19m
Consider the following scenario:
two domains are added to an app, domainA and domainB
a previously added cert is attached to both domains
the cert is detached from only domainA
This scenario, in the form of the e2e test seen here, leads to the cert seemingly also detached from the other domainB (or perhaps the cert is removed altogether) even though certs:info
shows the attachment to domainB still exists.
The current behavior after the steps above is a 404
is returned when curl
ing domainB's SSL endpoint, whereas one would expect a 200
still be returned - as is the case when both domains are still attached to the cert.
See deis/workflow#492 for related scenario/logic.
$ kubectl --namespace=deis logs deis-router-egcwv
2015/12/18 21:07:06 INFO: Starting nginx...
2015/12/18 21:07:06 INFO: nginx started.
2015/12/18 21:07:06 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:07:16 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: read tcp 10.244.33.8:54458->10.100.0.1:443: read: connection reset by peer.
2015/12/18 21:07:26 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:07:36 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:07:46 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:07:56 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/18 21:08:06 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
Replaces #36
Router is very configurable. Many of these configuration options, however, are really, at their core, nginx configuration options. Try as I might, it's very difficult to hide that fact, as many of the options exposed may not even have an analog in a hypothetical, alternative implementation (based on HAProxy, for instance). Although an alternative implementation is only hypothetical at the moment, it's very likely to happen sooner or later, even if only within the community.
To that end, I would like to propose a modest re-organization of the router's configuration options and the accompanying documentation to promote a clear separation between configuration options that are "spec" and therefore non-negotiable and a secondary class of configuration options that are "extensions;" that is to say-- implementation specific.
I would propose that this is to be achieved through a re-namespacing of all configuration options and careful re-organization of the configuration documentation.
By way of example, consider the following configuration options that can be set via annotations on the router's replication controller:
These could be renamed as follows:
Again by example, certain other annotations on the router's replication controller, such as the following, may not initially appear to be implementation-specific:
That is highly-speculative, however, because it's conceivable that other implementations might not need such information. (Even the current implementation considers it optional.) Taking this into consideration, there's no harm in also re-namespacing annotations such as these to:
In contrast to all of the above, all annotations that form the contract between the router and workflow should be considered "spec" and non-negotiable. Any alternative implementation of the router would have to respect these. These encompass all the annotations that workflow sets on "routable" application services, such as:
For improved clarity, these could also be re-namespaced as well:
In the end, I believe, with the right documentation, it becomes clear that the "app" annotations are spec and non-negotiable, whilst the "nginx" annotations are specific to the current router implementation.
I'd like to invite questions or concerns re: this approach.
This is to lower the barrier to entry for hacking on Deis while also standardizing the development environment for all contributors.
Generally, we've been shooting for v1.x feature parity with alpha (minus SSL support). @helgi made a case for also dropping naxsi from alpha and then perhaps slotting in something more fully baked, post alpha. I'll invite him to elaborate...
Many configuration values one might set via annotations ultimately wind up in a dynamically generated nginx.conf
. It might be nice to detect invalid values for each field, log a warning, and fallback to a default value. This way, Nginx itself never receives invalid configuration and logs will be more descriptive in terms of what configuration errors might exist.
when using helm install
from the HEAD
of master
on the deis/charts repo, here's the output of kd describe
(kd
is an alias for kubectl --namespace=deis
):
ENG000656:builder aaronschlesinger$ kd describe pod deis-router-s9nd6
Name: deis-router-s9nd6
Namespace: deis
Image(s): quay.io/deisci/router:v2-alpha
Node: /
Labels: app=deis-router
Status: Pending
Reason:
Message:
IP:
Replication Controllers: deis-router (1/1 replicas created)
Containers:
deis-router:
Container ID:
Image: quay.io/deisci/router:v2-alpha
Image ID:
State: Waiting
Ready: False
Restart Count: 0
Environment Variables:
POD_NAMESPACE: deis (v1:metadata.namespace)
Volumes:
deis-token-2vm78:
Type: Secret (a secret that should populate this volume)
SecretName: deis-token-2vm78
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
13m 32s 32 {scheduler } FailedScheduling Failed for reason PodFitsHostPorts and possibly others
This is similar to, but different from #136. The two issues should not be confused.
Currently, if you do provide a platform-wide wildcard cert, that cert gets used for securing "custom" (fully-qualified) domains associated to an app-- and that's wrong because the cert is not going to match that domain. Every "custom" domain should only support SSL if a cert is explicitly associated to it.
Installation of the stock deis-dev chart results in a router pod starting and going into the Running
status, but with no containers running. get pod
output looks like this:
deis-router-r7ey9 0/1 Running 1 7s
This is happening on a GKE cluster. See below for additional information:
ENG000656:example-go aaronschlesinger$ kd get -o yaml rc deis-router
apiVersion: v1
kind: ReplicationController
metadata:
annotations:
chart.helm.sh/description: For testing only!
chart.helm.sh/file: /Users/aaronschlesinger/.helm/workspace/charts/deis-dev/manifests/deis-router-rc.yaml
chart.helm.sh/name: deis-dev
chart.helm.sh/version: v2-beta
creationTimestamp: 2016-02-19T00:10:02Z
generation: 1
labels:
heritage: deis
name: deis-router
namespace: deis
resourceVersion: "1680306"
selfLink: /api/v1/namespaces/deis/replicationcontrollers/deis-router
uid: 1f179601-d69d-11e5-9503-42010af00071
spec:
replicas: 1
selector:
app: deis-router
template:
metadata:
creationTimestamp: null
labels:
app: deis-router
spec:
containers:
- env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/deisci/router:v2-beta
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /healthz
port: 9090
scheme: HTTP
initialDelaySeconds: 1
timeoutSeconds: 1
name: deis-router
ports:
- containerPort: 80
hostPort: 80
protocol: TCP
- containerPort: 443
hostPort: 443
protocol: TCP
- containerPort: 2222
hostPort: 2222
protocol: TCP
- containerPort: 9090
hostPort: 9090
protocol: TCP
readinessProbe:
httpGet:
path: /healthz
port: 9090
scheme: HTTP
initialDelaySeconds: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
dnsPolicy: ClusterFirst
restartPolicy: Always
serviceAccount: deis-router
serviceAccountName: deis-router
terminationGracePeriodSeconds: 30
status:
observedGeneration: 1
replicas: 1
ENG000656:example-go aaronschlesinger$ kd get -o yaml pod deis-router-r7ey9
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/created-by: |
{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"deis","name":"deis-router","uid":"1f179601-d69d-11e5-9503-42010af00071","apiVersion":"v1","resourceVersion":"1680212"}}
creationTimestamp: 2016-02-19T00:32:20Z
generateName: deis-router-
labels:
app: deis-router
name: deis-router-r7ey9
namespace: deis
resourceVersion: "1680572"
selfLink: /api/v1/namespaces/deis/pods/deis-router-r7ey9
uid: 3cacee5a-d6a0-11e5-9503-42010af00071
spec:
containers:
- env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/deisci/router:v2-beta
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /healthz
port: 9090
scheme: HTTP
initialDelaySeconds: 1
timeoutSeconds: 1
name: deis-router
ports:
- containerPort: 80
hostPort: 80
protocol: TCP
- containerPort: 443
hostPort: 443
protocol: TCP
- containerPort: 2222
hostPort: 2222
protocol: TCP
- containerPort: 9090
hostPort: 9090
protocol: TCP
readinessProbe:
httpGet:
path: /healthz
port: 9090
scheme: HTTP
initialDelaySeconds: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: deis-router-token-lc88l
readOnly: true
dnsPolicy: ClusterFirst
nodeName: gke-aaron-e39d4573-node-hecx
restartPolicy: Always
serviceAccount: deis-router
serviceAccountName: deis-router
terminationGracePeriodSeconds: 30
volumes:
- name: deis-router-token-lc88l
secret:
secretName: deis-router-token-lc88l
status:
conditions:
- lastProbeTime: null
lastTransitionTime: null
status: "False"
type: Ready
containerStatuses:
- containerID: docker://e21451b0cbf512ce6745677b2ac3c96d8e1a465367619b04eb1478ea490beec2
image: quay.io/deisci/router:v2-beta
imageID: docker://22f57e21e69c33e97133b67f1fb26254712637e90241a42e08a060f6198a85a7
lastState:
terminated:
containerID: docker://f14df2200bf3034c6c934e6a5a7434696995e3b8c219332de3431eb431ba07a5
exitCode: 2
finishedAt: 2016-02-19T00:40:46Z
reason: Error
startedAt: 2016-02-19T00:40:37Z
name: deis-router
ready: false
restartCount: 51
state:
running:
startedAt: 2016-02-19T00:40:47Z
hostIP: 10.240.0.3
phase: Running
podIP: 10.168.1.38
startTime: 2016-02-19T00:32:20Z
ENG000656:builder aaronschlesinger$ kd describe pod deis-router-r7ey9
Name: deis-router-r7ey9
Namespace: deis
Image(s): quay.io/deisci/router:v2-beta
Node: gke-aaron-e39d4573-node-hecx/10.240.0.3
Start Time: Thu, 18 Feb 2016 16:32:20 -0800
Labels: app=deis-router
Status: Running
Reason:
Message:
IP: 10.168.1.38
Replication Controllers: deis-router (1/1 replicas created)
Containers:
deis-router:
Container ID: docker://b5c27512a02dd210bc031342fdd06a81777730f4f99baac6348c4754d5cc7c42
Image: quay.io/deisci/router:v2-beta
Image ID: docker://22f57e21e69c33e97133b67f1fb26254712637e90241a42e08a060f6198a85a7
State: Running
Started: Thu, 18 Feb 2016 16:52:37 -0800
Last Termination State: Terminated
Reason: Error
Exit Code: 2
Started: Thu, 18 Feb 2016 16:52:27 -0800
Finished: Thu, 18 Feb 2016 16:52:36 -0800
Ready: False
Restart Count: 122
Environment Variables:
POD_NAMESPACE: deis (v1:metadata.namespace)
Conditions:
Type Status
Ready False
Volumes:
deis-router-token-lc88l:
Type: Secret (a secret that should populate this volume)
SecretName: deis-router-token-lc88l
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
20m 20m 1 {kubelet gke-aaron-e39d4573-node-hecx} implicitly required container POD Pulled Container image "gcr.io/google_containers/pause:0.8.0" already present on machine
20m 20m 1 {kubelet gke-aaron-e39d4573-node-hecx} implicitly required container POD Created Created with docker id dea7fa3830e7
20m 20m 1 {scheduler } Scheduled Successfully assigned deis-router-r7ey9 to gke-aaron-e39d4573-node-hecx
20m 20m 1 {kubelet gke-aaron-e39d4573-node-hecx} implicitly required container POD Started Started with docker id dea7fa3830e7
20m 20m 1 {kubelet gke-aaron-e39d4573-node-hecx} spec.containers{deis-router} Started Started with docker id 43461bab895e
20m 20m 1 {kubelet gke-aaron-e39d4573-node-hecx} spec.containers{deis-router} Created Created with docker id 43461bab895e
20m 20m 1 {kubelet gke-aaron-e39d4573-node-hecx} spec.containers{deis-router} Killing Killing with docker id 43461bab895e
20m 20m 1 {kubelet gke-aaron-e39d4573-node-hecx} spec.containers{deis-router} Started Started with docker id d64c66841fca
20m 20m 1 {kubelet gke-aaron-e39d4573-node-hecx} spec.containers{deis-router} Created Created with docker id d64c66841fca
...snip...
@krancour - please let me know what other information you need from me. I'll keep the cluster running.
Refs deis/workflow#156
We should document the specific timeout requirements for the router component, since many long-running operations are pretty much guaranteed during deploy.
ConfigMap (scheduled for 1.2)
2015/12/21 18:21:08 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: read tcp 10.244.97.6:56526->10.100.0.1:443: read: connection reset by peer.
2015/12/21 18:21:18 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
It also has a blank config
bash-4.3# cat /opt/nginx/conf/nginx.conf
user nginx;
daemon off;
events {}
http {}
This is important for the cases where k8s is running on infrastructure where the k8s "cloud provider" uses a no-op for services of type: LoadBalancer
.
The following should be configurable as in v1.x:
v2 beta requirements include operational metrics from the router. It's not yet clear what specific metrics are needed. This issue is just a placeholder for discussion and to ensure we don't lose track of this requirement.
If a cluster does not have the ServiceAccount admission control enabled, the deis
service account on which router depends may not have had the secret it requires automatically generated.
This is a probably a pretty low priority, as I believe it's uncommon for this admission control to not be enabled.
Because of this project's dependency on Kubernetes' API, glide up
(which I have abstracted behind make bootstrap
) resolves a massive graph of 102 dependencies (most of which are probably not needed). If running from scratch, this can take 15 minutes. That's not so bad for a developer. Even if glide up
is re-run periodically, in theory, running from scratch is a one-time thing.
But how can we handle this in CI so that tests don't take forevvvvvvvar?
Note that this question first popped into my head after looking at this PR:
Ping @mboersma @sgoings @technosophos for thoughts.
Installing deis-router through the deis helm chart leaves me with deis-router continually flapping:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deis-builder-lknfs 0/1 CrashLoopBackOff 16 1h
deis-database-yxntz 1/1 Running 0 1h
deis-etcd-1-ekw9s 1/1 Running 0 1h
deis-etcd-1-hrqyk 1/1 Running 0 1h
deis-etcd-1-pd1mu 1/1 Running 1 1h
deis-etcd-discovery-vqlzu 1/1 Running 0 1h
deis-minio-fsmqx 0/1 RunContainerError 0 1h
deis-registry-rarvy 1/1 Running 0 1h
deis-router-2iwxb 0/1 CrashLoopBackOff 15 1h
deis-workflow-tr7eb 1/1 Running 1 1h
As in deis/charts#60
Right now the container is run as root.
This is one of just a few remaining features that alpha requires to achieve v1.x parity sans SSL support.
Strictly speaking, v1.x didn't do this, but these were configurable for the route to the controller in v1.x.
The v2 router treats the "controller" (workflow) as "just another app," which means that this needs to be configurable at the app level.
Workflow does not need to expose the ability to modify this setting. Router just needs to honor it if it is set. This way, the deis.io/routerConfig
annotation in the service definition for workflow can be tweaked in the official deis helm chart to raise the timeouts. (This is to support some of the legitimately longer-running requests made to workflow.)
This is one of just a few remaining features that alpha requires to achieve v1.x parity sans SSL support.
When we were using JSON embedded in router rc annotations to configure the router, one could disable gzip compression completely by nilling out the gzipConfig
object in that JSON.
Since #72 we are no longer using JSON for router configuration and the ability to disable gzip compression was accidentally lost in the process. It should be restored.
Did a git push
for workflow and then tried to hit the app when it said it worked.
Couldn't for full 3 minutes
To ssh://[email protected]:2222/madcap-ricochet.git
• [new branch] master -> master
core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
curl: (6) Couldn't resolve host 'madcap-ricochet.10.3.0.161.xip.io'
core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
curl: (6) Couldn't resolve host 'madcap-ricochet.10.3.0.161.xip.io'
core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
curl: (6) Couldn't resolve host 'madcap-ricochet.10.3.0.161.xip.io'
core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
curl: (6) Couldn't resolve host 'madcap-ricochet.10.3.0.161.xip.io'
core@micro-kube ~/example-go $ curl madcap-ricochet.10.3.0.161.xip.io
Powered by Deis
Release v2 on madcap-ricochet-v2-web-d771p
However looking at the router logs you can see it reloaded on time (confirmed the timings)
2016/01/13 01:25:09 INFO: Router configuration has changed in k8s.
2016/01/13 01:25:09 INFO: Reloading nginx...
2016/01/13 01:25:09 INFO: nginx reloaded.
2016/01/13 01:25:09 [notice] 12#0: signal process started
[13/Jan/2016:01:27:53 +0000] - 10.2.22.1 - - - 200 - "GET / HTTP/1.1" - 221 - "-" - "curl/7.43.0" - "~^madcap-ricochet\x5C.(?<domain>.+)$" - 10.3.0.37:80 - madcap-ricochet.10.3.0.161.xip.io - 0.001 - 0.001
Can we move some of the README stuff into the official docs in deis/workflow
?
related to deis/deis#4952. Since the project is being renamed from deis
to workflow
, the router endpoint should now read workflow.mydomain.com
instead of deis.mydomain.com
.
If we want to scale the NGINX instance to more than one instance and use SSL session tickets, each instance must have the same key.
Could you add that to the router annotation configuration?
Or even better could Deis automatically rotate the keys?
https://blog.twitter.com/2013/forward-secrecy-at-twitter-0
Remove the enforceWhitelists option and simply enforce a whitelist for each app if one is defined, and otherwise, behave permissively.
I'm playing around with the router in a plain vanilla k8s cluster. When I do the helm generate router
, it says that it ran 0 generators. Is this normal?
When I do deploy I get a permission denied error related to the service account. I'm trying to figure out if I'm not getting the template built correctly or if I've got more SELinux issues.
Consider the following case of two domains for a given app, but only attaching a cert to one:
$ deis domains:add domain.one
Adding domain.one to app... done
$ deis domains:add domain.two
Adding domain.two to app... done
$ deis certs:add cert-one server.crt server.key
Adding SSL endpoint... done
$ deis certs:list
Name | Common Name | SubjectAltName | Expires | Fingerprint | Domains | Updated | Created
+----------+-------------+----------------+------------------------+-----------------+---------+------------+------------+
cert-one | foo.com | | 1 Mar 2017 (in 1 year) | 25:64[...]0E:59 | | 4 Mar 2016 | 4 Mar 2016
$ deis certs:attach cert-one domain.one
Attaching certificate cert-one to domain domain.one... done
$ deis certs:list
Name | Common Name | SubjectAltName | Expires | Fingerprint | Domains | Updated | Created
+----------+-------------+----------------+------------------------+-----------------+------------+------------+------------+
cert-one | foo.com | | 1 Mar 2017 (in 1 year) | 25:64[...]0E:59 | domain.one | 4 Mar 2016 | 4 Mar 2016
Now, when we curl both domains, we appear to see the same cert (both return 200 OK as well):
$ curl -kvI -sD - -H "Host: domain.one" https://app.130.211.155.186.xip.io
* Rebuilt URL to: https://app.130.211.155.186.xip.io/
* Trying 130.211.155.186...
* Connected to app.130.211.155.186.xip.io (130.211.155.186) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: foo.com
> HEAD / HTTP/1.1
> Host: domain.one
> User-Agent: curl/7.47.1
> Accept: */*
$ curl -kvI -sD - -H "Host: domain.two" https://app.130.211.155.186.xip.io
* Rebuilt URL to: https://app.130.211.155.186.xip.io/
* Trying 130.211.155.186...
* Connected to app.130.211.155.186.xip.io (130.211.155.186) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: foo.com
> HEAD / HTTP/1.1
> Host: domain.two
> User-Agent: curl/7.47.1
> Accept: */*
This is one of just a few remaining features that alpha requires to achieve v1.x parity sans SSL support.
Currently we have to manually edit annotations, but annotations are in YAML in JSON even though they are just simple key/value lists:
⇒ kdy get rc deis-router
apiVersion: v1
kind: ReplicationController
metadata:
annotations:
chart.helm.sh/description: For testing only!
chart.helm.sh/file: /Users/mattbutcher/Code/helm_home/workspace/charts/deis/manifests/deis-router-rc.yaml
chart.helm.sh/name: deis
chart.helm.sh/version: 2.0.0-pre-alpha
deis.io/routerConfig: |
{
"domain": "10.254.1.3.xip.io",
"useProxyProtocol": false
}
Is there a reason why we can't simply make these separate annotations?
They are error prone to edit, require additional work to parse, and require additional tooling for things like Helm.
Port deis/deis#4877 to this project.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.