dit4c/dit4c.github.io

DIT4C GitHub Pages website

DIT4C - Data Intensive Tools for the Cloud

DIT4C is a scalable platform for providing containerized web-based programming and data analysis environments to researchers.

• Client == Modern web browser
• No local credentials: use your GitHub or AAF account
• Based on rkt - run ACIs or Docker images on bare metal or cloud computing

All authentication is via federated identity providers - all a user needs is a modern web browser.

Some of the current environments available are:

• Base - web-based TTY sessions and basic file management
• Jupyter Notebook + Base
• NLTK + Jupyter Notebook + Base
• OpenRefine + Base
• RStudio + Base
• Apache Zeppelin + Base
• X11 - Base + X11 sessions via HTML5 VNC client
• Octave + X11
• QGIS + Jupyter + RStudio + X11 - QGIS with supporting Python & R environments

Motivation

DIT4C is focused on meeting two needs:

• Training sessions - having a working install right from the beginning means training participants start programming sooner, and do so in a consistent environment.
• Reproducible research - container sharing and export allows complete working environments to be exchanged and archived.

Architecture

DIT4C separates the portal environment which manages user access and containers from the compute nodes that provide them.

Core services:

• portal - user-facing UI and scheduler coordination
• scheduler - manages compute clusters and schedules containers on individual nodes

Additional services:

• image server - necessary for saving and exporting instances
  • dit4c-imageserver-filesystem - store images on a filesystem (possibly mounted via NFS)
  • dit4c-imageserver-swift - store images in OpenStack Swift object storage

Auxiliary "helper" container images:

• dit4c-helper-listener-*
  • dit4c-helper-listener-ngrok2 (DEFAULT) - development image that exposes containers via ngrok.com (don't use this in production)
  • dit4c-helper-listener-ssh - expose containers using SSH reverse port forwarding and reverse-proxy server. When used with dit4c-routingserver-ssh it supports HTTP & HTTPS.
  • dit4c-helper-listener-ngrok1 (DEPRECATED) - expose containers via your own ngrok1 servers. Supports HTTP, HTTPS & individual TCP ports.
• dit4c-helper-auth-portal - proxies container services behind portal-provided auth
• dit4c-helper-upload-webdav - uploads saved images to a webdav server

See https://dit4c.github.io/ for further archtecture details.

Security

All container instances are issued an OpenPGP key prior to starting which is convertible to a JSON Web Key (JWK) or SSH key. This allows container helpers to independently contact the portal to update and retrieve information using a signed JSON Web Token (JWT).

The portal also provides keys via a public registry, which will allow future helpers to authenticate independently to other services or retrieve encrypted content. This is still a work in progress.

Installation Requirements

Servers

While DIT4C could be run on a single server for development purposes, a secure installation will attempt to segregate the portal, scheduler and compute onto separate physical or virtual machines.

A valid minimal configuration would include:

• portal/image server (public ports exposed: HTTPS)
  • nghttpx/nginx as HTTPS reverse-proxy
  • portal
  • image server
  • Cassandra database for portal
• scheduler (public ports exposed: none)
  • scheduler
  • Cassandra database for scheduler
• compute node (public ports exposed: none)
  • no installed software required
  • CoreOS recommended
  • SSH port must be accessible by scheduler

TLS Certificates

HTTPS should be used with the portal, image server (recommended, but optional) and routing server. Let's Encrypt is sufficient for the portal and image server, but all HTTPS routing server implementations require a wildcard certificate. While highly discouraged, a self-signed certificate can be used for the routing server if necessary (presumably while a valid wildcard certificate is being sourced).

Installing

See https://dit4c.github.io/ for installation instructions.

dit4c.github.io's Issues

No resolver issue

2014/09/18 05:05:22 [error] 15#0: *3039 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/static/components/codemirror/addon/mode/overlay.js?v=d8c2d47f490ce2737d63567d72cd6cbd HTTP/1.1", host: "giuliasavo.dit4c.metadata.net", referrer: "https://giuliasavo.dit4c.metadata.net/ipynb/notebooks/Untitled0.ipynb"
2014/09/18 05:05:34 [error] 15#0: *3519 auth request unexpected status: 500 while sending to client, client: 130.220.209.65, server: _, request: "GET /ipynb/static/widgets/js/widget_image.js HTTP/1.1", host: "parisafarzanehfar.dit4c.metadata.net", referrer: "https://parisafarzanehfar.dit4c.metadata.net/ipynb/notebooks/Untitled1.ipynb"
2014/09/18 05:05:34 [error] 15#0: *3519 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/static/widgets/js/widget_image.js HTTP/1.1", host: "parisafarzanehfar.dit4c.metadata.net", referrer: "https://parisafarzanehfar.dit4c.metadata.net/ipynb/notebooks/Untitled1.ipynb"
2014/09/18 05:05:47 [error] 15#0: *3667 auth request unexpected status: 500 while sending to client, client: 130.220.209.65, server: _, request: "GET /ipynb/static/components/bootstrap/bootstrap/js/bootstrap.min.js?v=d700a93337122b390b90bbfe21e64f71 HTTP/1.1", host: "mcaleb.dit4c.metadata.net", referrer: "https://mcaleb.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:05:47 [error] 15#0: *3667 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/static/components/bootstrap/bootstrap/js/bootstrap.min.js?v=d700a93337122b390b90bbfe21e64f71 HTTP/1.1", host: "mcaleb.dit4c.metadata.net", referrer: "https://mcaleb.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:05:47 [error] 15#0: *3670 auth request unexpected status: 500 while sending to client, client: 130.220.209.65, server: _, request: "GET /ipynb/static/base/js/namespace.js?v=76addbdaa966b406064a39ab8e364d01 HTTP/1.1", host: "mcaleb.dit4c.metadata.net", referrer: "https://mcaleb.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:05:47 [error] 15#0: *3670 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/static/base/js/namespace.js?v=76addbdaa966b406064a39ab8e364d01 HTTP/1.1", host: "mcaleb.dit4c.metadata.net", referrer: "https://mcaleb.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:05:48 [error] 15#0: *3692 auth request unexpected status: 500 while sending to client, client: 130.220.209.65, server: _, request: "GET /ipynb/static/tree/js/clusterlist.js?v=b8bd2d2d3e02af27c46ae339f6cabd9b HTTP/1.1", host: "mcaleb.dit4c.metadata.net", referrer: "https://mcaleb.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:05:48 [error] 15#0: *3692 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/static/tree/js/clusterlist.js?v=b8bd2d2d3e02af27c46ae339f6cabd9b HTTP/1.1", host: "mcaleb.dit4c.metadata.net", referrer: "https://mcaleb.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:05:54 [error] 15#0: *3766 auth request unexpected status: 500 while sending to client, client: 130.220.209.65, server: _, request: "GET /ipynb/static/base/js/dialog.js?v=85f27bfb6d7862a1ecac24b754f32a88 HTTP/1.1", host: "sbhandari.dit4c.metadata.net", referrer: "https://sbhandari.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:05:54 [error] 15#0: *3766 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/static/base/js/dialog.js?v=85f27bfb6d7862a1ecac24b754f32a88 HTTP/1.1", host: "sbhandari.dit4c.metadata.net", referrer: "https://sbhandari.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:05:54 [error] 15#0: *3775 auth request unexpected status: 500 while sending to client, client: 130.220.209.65, server: _, request: "GET /ipynb/static/base/images/ipynblogo.png?v=5892b775169dd3ddb0c9ec906f4183de HTTP/1.1", host: "sbhandari.dit4c.metadata.net", referrer: "https://sbhandari.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:05:54 [error] 15#0: *3775 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/static/base/images/ipynblogo.png?v=5892b775169dd3ddb0c9ec906f4183de HTTP/1.1", host: "sbhandari.dit4c.metadata.net", referrer: "https://sbhandari.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:07:04 [error] 15#0: *4098 auth request unexpected status: 500 while sending to client, client: 130.220.209.65, server: _, request: "GET /ipynb/clusters?_=1411013218162 HTTP/1.1", host: "sbhandari.dit4c.metadata.net", referrer: "https://sbhandari.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:07:04 [error] 15#0: *4098 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/clusters?_=1411013218162 HTTP/1.1", host: "sbhandari.dit4c.metadata.net", referrer: "https://sbhandari.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:07:04 [error] 15#0: *4104 auth request unexpected status: 500 while sending to client, client: 130.220.209.65, server: _, request: "GET /ipynb/static/components/codemirror/mode/rst/rst.js?v=9c14ec4fdc261c89d6b0c49251452572 HTTP/1.1", host: "mcaleb.dit4c.metadata.net", referrer: "https://mcaleb.dit4c.metadata.net/ipynb/notebooks/Untitled2.ipynb"2014/09/18 05:07:04 [error] 15#0: *4104 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/static/components/codemirror/mode/rst/rst.js?v=9c14ec4fdc261c89d6b0c49251452572 HTTP/1.1", host: "mcaleb.dit4c.metadata.net", referrer: "https://mcaleb.dit4c.metadata.net/ipynb/notebooks/Untitled2.ipynb"
2014/09/18 05:07:26 [error] 15#0: *4300 auth request unexpected status: 500 while sending to client, client: 130.220.209.65, server: _, request: "GET /ipynb/static/custom/custom.css?v=900035aa0c126bb85df55c5b3e51b6f1 HTTP/1.1", host: "sbhandari.dit4c.metadata.net", referrer: "https://sbhandari.dit4c.metadata.net/ipynb/tree"
2014/09/18 05:07:26 [error] 15#0: *4300 no resolver defined to resolve dit4c.metadata.net, client: 130.220.209.65, server: _, request: "GET /ipynb/static/custom/custom.css?v=900035aa0c126bb85df55c5b3e51b6f1 HTTP/1.1", host: "sbhandari.dit4c.metadata.net", referrer: "https://sbhandari.dit4c.metadata.net/ipynb/tree"

Work out how portal reverse proxy will send traffic to the right VM

Presumably this is going to be another nginx_auth_request solution. In any case, before we can have separate nodes we'll need this.

How does a name get proxied to a port number?

Have to research Docker API and work out how a name (probably part of the hostname) can be translated into the port number for reverse proxying.

Work out if RStudio requires alteration to use on DIT4C platform

RStudio may be able to run as a single user without authentication and under a path. This needs confirmation, as it affects if a cloud platform can be used for teaching R in late September.

Handle expired keys in cookies better

At the moment you can get redirected back to the portal without having your expired cookie replaced.

Consider replacing TTY.js

Butterfly runs on Python and Tornado, and uses the entire screen.

It would require a bit to get working though, as while nothing precludes running it under a path, it hasn't actually been written to support that.

Convert project images from Fedora 20 to CentOS 7

Fedora really isn't a stable foundation for long-term use. Now that RHEL 7 is out, we should be using it instead (or switch to Ubuntu).

Should be able to zoom terminal independently of other tools on the the container.

Zooming terminals shouldn't make iPython/RStudio UI unusable.

Combine dit4c software into a single repository

Increasingly there's some room for creating shared libraries that more than one part of DIT4C use. A multi-project build would make this much easier, and also allow version numbers to be synced between the software components.

Track down why term.js returns 502 occasionally

Looks like one of the JS files failed with an error during the first bootcamp:

128.250.0.86 - - [15/Sep/2014:05:06:07 +0100] "GET /tty/term.js HTTP/1.1" 502 574 "https://parisafarzanehfar.dit4c.metadata.net/tty/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36" "-"

Establish feasibility for cookie-based authentication of websocket-capable reverse-proxy

We want to do our own authentication rather than directly integrating with iPython notebook. Reverse-proxying traffic after authentication sounds good, but it'll mean we need to be able to reverse-proxy websockets as well.

Revise UI to handle multiple compute nodes

tty-lean.js default port should not take precedence over config

Issue with the way config variables are specified at the command line - default port from command line overrides the config file port.

Shell options: TTY.js?

Need to confirm how a shell will be provided to the user.

Create placeholder/redirect for website

Explore Octave options

Due to lack of web interface, remote desktop is most likely required.

X11vnc -create -xdummy + websockify + NoVNC + octave --force-gui?

Conversion of highcommand UI to AngularJS

EmberJS was a mistake. Development is a pain - it needs to go.

Allow multiple project images

At the moment "dit4c/python" is hard-coded in dit4c-highcommand. That needs to change - it needs to be possible to select a project template.

Get iPython Notebook running inside Docker

We want to isolate projects as much as possible, and the best way to do that is probably Docker.

RStudio requests sometimes 504 timeout

Nginx (either at compute-node or container level) probably has a lower timeout threshold than it should.

Haven't caught the case where a user tries to create a new container without a name

Need to check and ensure that both the UI and API don't allow it.

dit4c-machineshop should authenticate non-GET requests

It's not safe to allow dit4c-machineshop to talk to receive requests over public networks, but to scale out nodes we need it to be.

dit4c-gatehouse is using public key files, so there's no reason dit4c-machineshop can't receive signed requests. Those requests will need timestamps or unique IDs to prevent relay attacks.

iPython should be in a virtualenv

sudo to install PIP packages is clunky, and easily avoided if we don't use the system python directly. The iPython Docker image should use a virtualenv Python instead.

User identity mechanism

Which identity provider should be used for the production service? How would that work when developing?

Change DIT4C "Projects" to be called "Containers"

"Project" just isn't a good name, and it's used by RStudio. Before we get too deep in, it's time to change the name. "Container" isn't great as a name, but it's at least consistent with Docker.

IPython Notebook seems to hang more often than it should

There are also more 101 codes in the Nginx logs than there really should be, indicating that websocket connections might be cut off.

Users should be able to download containers

There's no reason why users should need our platform to run a container, so why not let them download the container so they continue elsewhere?

What do we use as a file store?

We probably don't want the file store to be tied to the execution environment - especially early on when we're not sure how sustainable the service is going to be.

So, what do we use as a back-end filestore? It has to appear like one on a console though, so something that mounts with FUSE is probably the way to go.

dit4c-gatehouse shouldn't require local key files

Sure, you can load the key locally. Cycling the keys isn't possible in that situation though, and it makes creating new Gatehouse instances just that little bit harder.

Ideally we'd take a URL on start which would be monitored.

Get Greg to give usability feedback

DIT4C needs better documentation

There isn't even a README.md for dit4c/dit4c.

Terminals cease to exist sometimes

[tty.js] 23 Created pty (id: /dev/pts/4, master: 16, pid: 527).
[tty.js] 22 Client disconnected.
[tty.js] 22 Killing all pty's.
[tty.js] 22 Closed pty (/dev/pts/2): 15.
[tty.js] 22 Closed pty (/dev/pts/0): 11.
[tty.js] 22 Closed pty (/dev/pts/3): 12.
[tty.js] 24 Session 24 created.
[tty.js] 24 Client attempting to write to a non-existent terminal. (id:     /dev/pts/4)
[tty.js] 24 Client attempting to write to a non-existent terminal. (id:     /dev/pts/4)
[tty.js] 25 Session 25 created.
[tty.js] 25 Created pty (id: /dev/pts/1, master: 14, pid: 559).
[tty.js] 23 Client disconnected.
[tty.js] 23 Killing all pty's.
[tty.js] 23 Closed pty (/dev/pts/4): 16.
[tty.js] 26 Session 26 created.
[tty.js] 26 Client attempting to write to a non-existent terminal. (id:     /dev/pts/1)
[tty.js] 26 Client attempting to write to a non-existent terminal. (id:     /dev/pts/1)
[tty.js] 24 Client disconnected.
[tty.js] 24 Killing all pty's.
[tty.js] 26 Created pty (id: /dev/pts/0, master: 11, pid: 591).
[tty.js] 25 Client disconnected.
[tty.js] 25 Killing all pty's.
[tty.js] 25 Closed pty (/dev/pts/1): 14.

Solve 502 websocket responses for TTY.js

TTY.js connections are currently flaky for some reason, with 502 errors being returned from TTY.js on occasion. As this is unacceptable behaviour for demos, it's a high priority to fix.