googleclouddataproc/jupyterhub-dataprocspawner

jupyterhub-dataprocspawner's Issues

Dataproc Hub - Multiple user for a Dataproc cluster

Is it possible for multiple user to use JupyterLab that is hosted on a Dataproc cluster?

Getting code 500 when trying to login

Another issue detected while logging in to JH.

Here are logs from docker container:

[I 2020-08-26 12:38:20.673 JupyterHub base:742] User logged in: admin
[E 2020-08-26 12:38:20.676 JupyterHub web:1792] Uncaught exception POST /hub/login?next= (::ffff:213.110.141.242)
    HTTPServerRequest(protocol='http', host='35.222.13.252:8000', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='::ffff:213.110.141.242')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1703, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/login.py", line 150, in post
        self.redirect(self.get_next_url(user))
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 649, in get_next_url
        if user.spawner.active:
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 340, in spawner
        return self.spawners['']
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 187, in __getitem__
        self[key] = self.spawner_factory(key)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 333, in _new_spawner
        spawner = spawner_class(**spawn_kwargs)
      File "/usr/local/lib/python3.6/dist-packages/dataprocspawner/spawner.py", line 237, in __init__
        address=f'{self.region}-dataproc.googleapis.com:443'))
    TypeError: __init__() got an unexpected keyword argument 'address'

[E 2020-08-26 12:38:20.684 JupyterHub log:173] {
      "X-Forwarded-Host": "35.222.13.252:8000",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "::ffff:213.110.141.242",
      "Accept-Language": "en-US,en;q=0.9,ru;q=0.8",
      "Accept-Encoding": "gzip, deflate",
      "Referer": "http://35.222.13.252:8000/hub/login?next=",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36",
      "Content-Type": "application/x-www-form-urlencoded",
      "Origin": "http://35.222.13.252:8000",
      "Upgrade-Insecure-Requests": "1",
      "Cache-Control": "max-age=0",
      "Content-Length": "24",
      "Connection": "close",
      "Host": "35.222.13.252:8000"
    }
[E 2020-08-26 12:38:20.688 JupyterHub log:181] 500 POST /hub/login?next= (admin@::ffff:213.110.141.242) 16.77ms

This still blocks me from testing the code of my PR

Dataproc clusters from users with capitalization in their names fail GCP validation

If a user has capitalization in their name, they will not be able to spawn Dataproc clusters due to the cluster naming scheme including the username. We need to conform to Dataproc's cluster naming convention of:

(?:[a-z](?:[-a-z0-9]{0,49}[a-z0-9])?)))

If a user has capitalization in their name, they will receive an error like below when attempting to create a cluster:

Error: HTTP 500: Internal Server Error (Error in Authenticator.pre_spawn_start: HTTPError HTTP 500: error (Cluster name 'hub-dp-env-1-Mike-Tango' must match pattern (?:[a-z](?:[-a-z0-9]{0,49}[a-z0-9])?)))

Jupiter is failed to connect to the DataprocSpawner

I got a lot of the following messages in the logs of my JupiterHub with dataprocspawner.

[W 2020-01-13 12:05:07.661 JupyterHub utils:226] Failed to connect to http://<user_name>-jupyterhub-m.us-central1-a.c.<project_name>.internal:12345/user/ju/ ([Errno -2] Name or service not known)

How to reproduce:

1. Deploy & run JupiterHub manually by instructions https://github.com/GoogleCloudPlatform/dataprocspawner#quick-start
2. Click "Start new server"
3. You will get this issue #4 (comment)
4. Click "Stop my server"
   JupiterHub will not stop your server, and you will see a lot of errors in the logs.

In the end, I got the timeout error: didn't respond in 900 seconds .

Any ideas?

Fix the odd formatting in the README

The bullet points under "Manual Docker Image" are bolded and not normal text.

Add DataprocSpawner to Jupyterhub's Spawner wiki

https://github.com/jupyterhub/jupyterhub/wiki/Spawners

Bad config encountered during initialization

Now for some reasons the service inside the container cannot be started due to the following error:
... [C 2020-08-12 10:50:53.531 JupyterHub application:90] Bad config encountered during initialization: [C 2020-08-12 10:50:53.532 JupyterHub application:91] The 'spawner_class' trait of <jupyterhub.app.JupyterHub object at 0x7fb719523748> instance must be a type, but 'dataprocspawner.DataprocSpawner' could not be imported ...
The code changes now can't be tested, please fix.

Dataproc Hub User Spawned cluster creation - Unable to disable component gateway in the new cluster created

While creating a "User spawned cluster" from Dataproc Hub JupyterHub (based off a custom YAML config), unable to override the default value and set to "False", so that it does not show up in the UI of the Dataproc cluster

The below lines force the Dataproc spawner to enable the component gateway. Is there a way to set to false from the YAML configuration or through some flag.

Reason for this ask: In a team setup anyone can use this backdoor entry and get access to others notebooks, which they are not supposed to.

Link to code

jupyterhub-dataprocspawner/dataprocspawner/spawner.py

Line 1468 in 71b8b3c

cluster_data['config']['endpoint_config']['enable_http_port_access'] = True

 # Forces Component Gateway
    cluster_data['config'].setdefault('endpoint_config', {})
    cluster_data['config']['endpoint_config']['enable_http_port_access'] = True

Version missmatch and Redirect loop detected

It seems the version pulled on dockerfile "1.0.1dev" is missmatching with the dataproc clusters version "1.0"

Furthermore, the latest version appears to have some kind of template issues..
I need to add the following to the jupyterhub_config.py to successfully load the login page:

c.JupyterHub.data_files_path = '/usr/local/share/jupyterhub'

Still can't manage to make it work though, because after the cluster is provisioned and running, I keep getting:

500 : Internal Server Error
Redirect loop detected.

Any ideas?

Notebook created in GCS directory is not getting saved

Notebook created in GCS directory is not getting saved

Only checkpoint getting saved but in bucket location not the actual notebook

Docker build should exclude `.git` and `tests/` directories

We should exclude as much as possible in a .dockerignore file. This helps avoid unnecessary build steps.

User-managed notebooks on dataproc workbench fail

See issue tracker https://issuetracker.google.com/issues/220190970

I'm not 100% sure this repo is involved but I wanted to raise an issue in case it is. Essentially, both existing and new user-managed notebooks are failing to start Jupyter with the message Failed to enable unit: Unit file jupyter.service does not exist.