error updating pangeo.pydata.org

[pangeo-data]

error updating pangeo.pydata.org

[minrk]

Yes, I have run into this. Try making extraConfig a dict instsead of a string:

hub:
  extraConfig:
    customPodHook: |
      from kubernetes import client
      ...

I think both a scalar string and a dict/map are supposed to work, but I've found using a string results in this warning. I think it's just a warning, not an error, though.

The error about not matching labels I think is due to recent label changes, which requires you to use --force to upgrade (which does delete & replace if patch fails). We recently dealt with this on Binder.

[jacobtomlinson]

Hmm this is odd. I think it is related to jupyterhub/zero-to-jupyterhub-k8s#653.

Maybe @yuvipanda has some thoughts?

[rabernat]

I can google my way to relevant-sounding issues
helm/helm#1844
helm/helm#2437

But I don't know if they are actually relevant.

$ helm version
Client: &version.Version{SemVer:"v2.9.0", GitCommit:"f6025bb9ee7daf9fee0026541c90a6f557a3e0bc", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.0", GitCommit:"f6025bb9ee7daf9fee0026541c90a6f557a3e0bc", GitTreeState:"clean"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.3-gke.0", GitCommit:"a7b719f7d3463eb5431cf8a3caf5d485827b4210", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:01Z", GoVersion:"go1.9.2b4", Compiler:"gc", Platform:"linux/amd64"}

[rabernat]

My crash course in helm / kubernetes has left me with the strong impression that this whole thing is a house of cards. Everything I try gives me a cryptic error. How can we make this process work better for cloud-mortals like myself?

[rabernat]

Does anyone have any idea how to move forward from this error? Delete the deployment and re-install?

[mrocklin]

I had an odd experience using newer versions of kubernetes on GCP recently. Maybe downgrade to the default version?

[mrocklin]

Alternatively, maybe we need to become more accustomed to asking for help from upstream in the Helm project itself. Perhaps someone should raise a github issue?

[mrocklin]

OK, I've moved one step forward (I think) with #22 but am now running into another issue:

mrocklin@carbon:~/workspace/pangeo/gce$ helm upgrade jupyter ~/workspace/helm-chart/pangeo/ -f jupyter-config.yaml -f copy-secret-config.yaml
Error: UPGRADE FAILED: no ConfigMap with the name "nginx-proxy-config" found

Perhaps someone here has thoughts

[tjcrone]

what happens if you try installing the deployed chart as described here: https://github.com/pangeo-data/helm-chart?

[mrocklin]

That would not include my change in #22 so instead I'm deploying the chart from a local version that has been modified

…

On Wed, May 2, 2018 at 10:16 AM, Tim Crone ***@***.***> wrote: what happens if you try installing the deployed chart as described here: https://github.com/pangeo-data/helm-chart? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#21 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszA_5e5mztB1ViGJCoV7I5ez0qpXJks5tub-rgaJpZM4TuFLn> .

[tjcrone]

I see that there is no autohttps template in that version of jupyterhub. Strange. What happens if you try v0.7-a5c532d?

[mrocklin]

Unfortunately that version still defines extraConfig as a mapping

$ helm inspect jupyterhub/jupyterhub --version v0.7-a5c532d | grep extraConfig:
extraConfig: {}

[rabernat]

I feel like some input from the jupyterhub team could potentially save us hours of trial and error here. Should we consider reaching out more directly?

[mrocklin]

Sure. @yuvipanda is already pinged. Lets expand to @choldgraf

[tjcrone]

I managed to deploy the pangeo helm chart on a fresh gce cluster using v0.7-fd73c61, the current config file in the notebook-image branch (changed the loadbalancer ip), and a stripped-down secrets file that does include https, and was unable to replicate the ConfigMap error.

secret-config.yaml:

jupyterhub:
  proxy:
    secretToken: "074fd8fb655f412996012342d2014a278e49rts0f05bcca0396f0122752597327"
    https:
      enabled: true
      hosts:
        - "test.org"
      type: letsencrypt
      letsencrypt:
        contactEmail: "[email protected]"

[tjcrone]

quick side question, did you reserve your load balancer external ip address on GCP?

[rabernat]

What I did yesterday was to upgrade the existing ephemeral IP to a static IP. So now I believe it will persist.

…

Sent from my iPhone

On May 2, 2018, at 12:23 PM, Tim Crone ***@***.***> wrote: quick side question, did you reserve your load balancer external ip address on GCP? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jgerardsimcock]

I have not tried this yet but it seems like this may be a way forward. helm/helm#3933

[tjcrone]

I was able to replicate these warnings by rolling forward to v0.7-82bed4a

2018/05/02 12:44:56 warning: cannot overwrite table with non table for extraConfig (map[])
2018/05/02 12:44:56 warning: cannot overwrite table with non table for extraConfig (map[])

I needed to do a helm dependency update in order for my changes to be recognized. Also when upgrading, I usually set the --force and --recreate-pods flags.

[tjcrone]

It would seem to me that helm is not registering the change in location for the nginx-configmap.yaml file, which gets moved to a subdirectory in more recent versions of the jupyterlab helm chart. Using the --debug flag during upgrade and grepping for nginx-config will show you where it is looking for the file. I'm thinking that the --reset-values and/or --recreate-pods and/or --force flags could help with this situation. I would try --reset-values first.

[choldgraf]

also pinging @minrk here, as he may have experience with this.

[mrocklin]

I've walked back to the version of JupyterHub that we were using before and changed extraConfig to be a map in pangeo-data/pangeo#235

This appears to be functioning well. Thank you @minrk