Make it a standalone service

[jupyterhub]

Make it a standalone service

[betatim]

This issue is about making nbserverproxy ready to be a standalone service, so I assumed you wanted to contribute to that effort with the script you posted. Having random bits of code strewn around the issues increases the chances that they will become dead code that doesn't work/isn't maintained. So it should go in the documentation, examples or other bit of the repo somewhere.

https://github.com/BerserkerTroll/tornado_proxy — a (yet another) standalone Tornado proxy, a rewrite of the nbserverproxy without notebook dependencies and with flexibility in mind.
https://github.com/BerserkerTroll/nbserverproxy/tree/standalone — nbserverproxy on top of the standalone tornado_proxy.

See also https://github.com/BerserkerTroll/nbxpra/tree/nbserverproxy (HTML5 remote desktop notebook extension similar to nbvnc supporting multiple desktops, see pic below) which demonstrates how to proxy unix sockets instead of TCP ports (inspired by https://gist.github.com/bdarnell/8641880). (Xpra or Xorg is able to discover and self-assign a free screen number, which corresponds to unix socket file name, thus making random TCP port allocation unnecessary).
https://github.com/BerserkerTroll/tornado_proxy/blob/5d8550e9559c57749c18a39751b5a055084bcfdf/tornado_proxy.py#L245 shows that one does not need the AddSlashHandler class because it is easily possible to add slash using tornado.web.RedirectHandler.

Do not forget to add the --process-dependency-links option when installing packages with pip.

[betatim]

For those who want to follow: https://github.com/BerserkerTroll/nbxpra/issues/1 is the new issue for nbxpra discussion.

@minrk this seem to work for me (tested against SimpleHTTPServer)

#!/usr/bin/python3


import json
import pprint
import os
import re
import sys
from urllib.parse import urlparse

import requests
from tornado import web, httpserver, ioloop

from jupyterhub.services.auth import HubOAuthenticated, HubOAuthCallbackHandler

import nbserverproxy


class APIHandler(web.RequestHandler):
    """Relay API requests to the Hub's API using the service's API token."""
    def get(self, path):
        api_token = os.environ['JUPYTERHUB_API_TOKEN']
        api_url = os.environ['JUPYTERHUB_API_URL']
        r = requests.get(api_url + '/' + path,
            headers={'Authorization': 'token %s' % api_token},
        )
        r.raise_for_status()
        self.set_header('Content-Type', 'application/json')
        self.write(r.text)


class EchoHandler(HubOAuthenticated, web.RequestHandler):
    """Reply to an HTTP request with the path of the request."""
    @web.authenticated
    def get(self):
        self.write(self.request.path)


def main():
    pprint.pprint(dict(os.environ), stream=sys.stderr)

    StandaloneLocalProxyHandler = type(
        'StandaloneLocalProxyHandler',
        (HubOAuthenticated, nbserverproxy.handlers.WebSocketHandlerMixin, web.RequestHandler),
        dict(nbserverproxy.handlers.LocalProxyHandler.__dict__)
    )

    app = web.Application([
        (r'.*/api/(.*)', APIHandler),
        (os.getenv('JUPYTERHUB_OAUTH_CALLBACK_URL'), HubOAuthCallbackHandler),
        (os.getenv('JUPYTERHUB_SERVICE_PREFIX') + r'proxy/(\d+)(.*)', StandaloneLocalProxyHandler),
        (r'.*', EchoHandler),
    ], cookie_secret=os.urandom(32))

    r = re.compile('--port=(\d+)')
    port = next( (r.match(p).group(1) for p in sys.argv if r.match(p)), '8888')

    server = httpserver.HTTPServer(app)
    server.listen(port)
    try:
        ioloop.IOLoop.instance().start()
    except KeyboardInterrupt:
        print('\nInterrupted')


if __name__ == '__main__':
    main()

(inspired by jupyterhub/tests/mockservice.py)
nbserverproxy.handlers.LocalProxyHandler also works, it just doesn't oauthenticate against the hub.

[betatim]

Would you like to create a PR to contribute the above code?

@betatim ehm... Contribute where? Everyone may get the above script, put it into a container with, for example, Xpra and spawn desktops instead of Jupyter notebooks in their k8s cloud (if their Hub is configured to use KubeSpawner).

I don't think the short script posted above is related to this repo in the sense that it should be contributed into it.

This issue is about making nbserverproxy ready to be a standalone service, so I assumed you wanted to contribute to that effort with the script you posted.

It all depends on what to understand under "standalone". As you can see, I do not run [SingleUser]NotebookApp so I'm not using LocalProxyHandler

as a notebook extension.

(See the OP message).

In this sense it is already standalone, no action is required. The issue can be closed right now.

---

It is quite another matter if by "standalone" one means "doesn't inherit from IPythonHandler" (and does not depend on notebook at all). Then probably the projects should be split into 2: generic websocket-aware tornado proxy and notebook extension which uses this proxy.

(But the jupyterhub package is required for oauthentication and it depends on the notebook package, so I don't think getting rid of this dependency is worth the time and effort)

There are also several things I'm not happy about: the URL pattern the proxy handles, the way it creates websocket connection to the back-end, the fact that it instantiates httpclient instead of receiving it in the constructor (I would like to provide mine one with custom resolver able to return unix sockets instead of TCP ones).

[betatim]

Thanks for your contribution.

Ok, StandaloneLocalProxyHandler isn't completely functional. http works, proxying websocket fails because self.log is undefined.

@minrk It would be great to have jupyterhub.services.auth a standalone package, because getting the whole jupyterhub package just for auth is a bit too much.

[ryanlovett]

Cool! Really excited to try this out.

[ryanlovett]

@BerserkerTroll I'm testing nbxpra on mybinder but it isn't quite working for me. xpra, Xorg, and xterm are all running but the new tab stays on the "pending" message for a while, then closes. .xsession-errors has no errors. The Xorg log is full of

[ 37663.659] (EE) dbus-core: error connecting to system bus: org.freedesktop.DBus.Error.FileNotFound (Failed to connect tosocket /var/run/dbus/system_bus_socket: No such file or directory)

Am I missing anything in my test environment?

@ryanlovett I'm starting xpra with --dbus-launch=no option. According to the Xpra help this should prevent it from starting dbus. Until some version it was working this way. But after some version it seems that Xpra started to ignore command line options and rely on config files in /etc. I haven't digged much into this yet. :(

But more likely the problem here is not in Xpra but int the X server trying to connect to dbus.

@ryanlovett I was expecting to focus on the standalonization here. The link to nbxpra was given to reveal the reason why tornado_proxy initialize method parameters are what they are: to enable socket files proxying.

[ryanlovett]

Good point, I'll copy my comments to nbxpra.

I've extracted jupyterhub.services.auth as a standalone project https://github.com/BerserkerTroll/jupyterhub_services_auth
https://gist.github.com/BerserkerTroll/5fa14f7edd82295fb8936ecfc028f4af — the code from #1 (comment) implemented using tornado_proxy and jupyterhub_services_auth.

tornado_proxy depends only on tornado, jupyterhub_services_auth — on tornado and traitlets. No need in notebook or jupyterhub heavy dependencies to implement a proxy authenticating against a hub.

[manics]

This sounds like https://github.com/ideonate/jhsingle-native-proxy 😄

There's a lot of duplicated code, though there's also several custom additions specific to ContainDS Dashboards. Should we aim to merge the standalone features added by jhsingle-native-proxy back into jupyter-server-proxy, or should we keep them as separate projects and close this issue?