leetz/beakerx_demo

BeakerX: Beaker extensions for Jupyter

BeakerX is a collection of JVM kernels and interactive widgets for plotting, tables, autotranslation, and other extensions to Jupyter Notebook. BeakerX is in beta and under active development.

The documentation consists of tutorial notebooks on GitHub. You can try it in the cloud for free with Binder.

BeakerX is the successor to the Beaker Notebook (source code archive). It comes from Two Sigma Open Source. Yes we are hiring.

This README is for developers. Users should see the documentation on the homepage for how to install and run BeakerX.

Dependencies:

• conda
• yarn >= 1.0.1

Build and Install

conda create -y -n beakerx 'python>=3' nodejs pandas openjdk maven
source activate beakerx
conda config --env --add pinned_packages 'openjdk>8.0.121'
conda install -y -c conda-forge ipywidgets
(cd beakerx; pip install -e . --verbose)
beakerx install

If you are using Ubuntu 17+ and you see an error while yarn install. Try:

sudo apt remove cmdtest

In order to run demo files for data catalog you need the following libraries in your conda env:

pip install quandl matplotlib

Screenshot for Data Browser

Here is a screenshot of open data browser, if correctly installed.

Build and Install for Jupyter Lab

conda create -y -n labx 'python>=3' nodejs pandas openjdk maven
source activate labx
conda config --env --add pinned_packages 'openjdk>8.0.121'
conda install -y -c conda-forge jupyterlab
(cd beakerx; pip install -e . --verbose)
beakerx install
jupyter labextension install @jupyter-widgets/jupyterlab-manager
(cd js/lab; jupyter labextension install .)

Running with Docker

docker run -p 8888:8888 beakerx/beakerx

Update after Java change

The kernels are installed to run out of the repo, so just a local build should suffice:

(cd kernel; ./gradlew build)

Update after JS change

The notebook extensions are installed to run out of the repo, so just a local build should suffice:

(cd js/notebook; yarn install)

Run Tests

The Java and TypeScript unit tests are run with every build. See [test/README.md] for how to run the e2e tests.

Groovy with Interactive Plotting and Tables:

Autotranslation from Python to JavaScript

Interactive Tables

Architecture and Code Overview

BeakerX is a collection of kernels and extensions for Jupyter. The code is organized into subdirectories as follows:

• beakerx The Python packages. The main beakerx package has:

  • a customized KernelSpec to allow BeakerX to configure the JVMs started to run the kernels,

  • a server extension for the javadoc, settings, and version endpoints,

  • the beakerx command line program, which has the bkr2ipynb converter as well as install and uninstall functions,

  • the Python API for the runtime (tables, plots, easyform), including automatically installing a displayer for pandas tables, and autotranslation;

  • the webpack (compiled JavaScript, TypeScript, CSS, fonts, images); and

  • the compiled Java JARs.

  There is a separate python package (beakerx_magics) for the %%groovy magic so it can always be loaded without loading the regular beakerx package (which would turn on display of pandas tables with our table widget).

• doc Documentation consisting of executable tutorial notebooks. StartHere.ipynb at the top level links to these and is the intended way to navigate them. There is a subdirectory for each language.

• docker configuration files for creating the Docker image. There is a subdirectory docker/base for an image with BeakerX's dependencies (the Ubuntu and conda packages). The main image is built by compiling BeakerX and installing BeakerX in the base image.

• js There are two subdirectories of JavaScript and TypeScript, js/lab and js/notebook. New code is being written in TypeScript.

  The lab subdirectory has the extension for Jupyter Lab (distributed by npm). Notebook has two extensions, one for the widgets (which are included in Lab as well, and are also separately distributed with npm for embedded applications such as nbviewer), and one for the notebook application. This adds a tab to the tree view with our options panel.

  And for regular notebook pages the extension handles: running initialization cells, publication, autotranslation, the getCodeCells and runByTag APIs, callbacks for table and plot actions, UI customizations such as changing the fonts, allowing wide code cells, and disabling autosave.

• kernel The Java implementation of the kernels is here. The main directory is kernel/base which has generic code for all the languages. The base kernel has classes for Jupyter's Comm protocol (a layer over ZMQ), magics, the classpath (including loading from maven), and the kernel parts of the widget APIs.

  There is also a subdirectory for each language which has the evaluator for that language. Scala has wrappers for the widgets so they have native types.

• test The e2e tests, which are made with webdriver (selenium, chromedriver, jasmine).

Contributing

See CONTRIBUTING.md.

Releasing

See RELEASE.md.

Attribution

Beaker contains and depends on many projects including:

The kernel is originally derived from lappsgrid, but has been rewritten in Java and refactored and expanded.

The Java support uses Adrian Witas' org.abstractmeta.toolbox.

ANTLR Copyright (c) 2012 Terence Parr and Sam Harwell

d3 Copyright (c) 2010-2015, Michael Bostock

IPython Copyright (c) 2008-2014, IPython Development Team Copyright (c) 2001-2007, Fernando Perez Copyright (c) 2001, Janko Hauser Copyright (c) 2001, Nathaniel Gray

Scala Copyright (c) 2002-2015 EPFL Copyright (c) 2011-2015 Typesafe, Inc.

Guava Copyright (C) 2012 The Guava Authors

Apache Spark Copyright (C) 2014 and onwards The Apache Software Foundation.

H2 database engine This software contains unmodified binary redistributions for H2 database engine (http://www.h2database.com/), which is dual licensed and available under the MPL 2.0 (Mozilla Public License) or under the EPL 1.0 (Eclipse Public License). An original copy of the license agreement can be found at: http://www.h2database.com/html/license.html