This repository contains source code for Persist (PyPi) extension.
Persist is a JupyterLab extension to enable persistent interactive outputs in JupyterLab notebooks. Check out the introductory video below.
Persist.Introduction.mp4
Persist is developed as part of a publication and will appear in EuroVis 2024.
Supplementary material including example notebooks, walkthrough notebooks, notebooks used in the study (including participant notebooks) and the analysis notebooks can be accessed here.
Computational notebooks, such as Jupyter, support rich data visualization. However, even when visualizations in notebooks are interactive, they still are a dead end: Interactive data manipulations, such as selections, applying labels, filters, categorizations, or fixes to column or cell values, could be efficiently apply in interactive visual components, but interactive components typically cannot manipulate Python data structures. Furthermore, actions performed in interactive plots are volatile, i.e., they are lost as soon as the cell is re-run, prohibiting reusability and reproducibility. To remedy this, we introduce Persist, a family of techniques to capture and apply interaction provenance to enable persistence of interactions. When interactions manipulate data, we make the transformed data available in dataframes that can be accessed in downstream code cells. We implement our approach as a JupyterLab extension that supports tracking interactions in Vega-Altair plots and in a data table view. Persist can re-execute the interaction provenance when a notebook or a cell is re-executed enabling reproducibility and re-use.
We evaluated Persist in a user study targeting data manipulations with 11 participants skilled in Python and Pandas, comparing it to traditional code-based approaches. Participants were consistently faster with Persist, were able to correctly complete more tasks, and expressed a strong preference for Persist.
Persist works with Vega-Altair charts directly for the most part. Vega-Altair and Vega-Lite offer multiple ways to write a specification. However Persist has certain requirements that need to be fulfilled.
-
The selection parameters in the chart should be named. Vega-Altair's default behavior is to generate a name of selection parameter with auto-incremented numeric suffix. The value of the generated selection parameter keeps incrementing on subsequent re-executions of the cell. Persist relies on consistent names to replay the interactions, and passing the name parameter fixes allows Persist to work reliably.
-
The point selections should have at least the fields attribute specified. Vega-Altair supports selections without fields by using the auto-generated indices to define selections. The indices are generated with the default order of rows in the source dataset. Using the indices directly for selection can cause Persist to operate on incorrect rows if the source dataset order changes.
-
Dealing with datetime in Pandas is challenging. To standardize the way datetime conversion takes place within VegaLite and within Pandas when using Vega-Altair, the TimeUnit transforms and encodings must be specified in UTC. e.g
month(Date)should beutcmonth(Date).
- JupyterLab >= 4.0.0 or Jupyter Notebook >= 7.0.0
- pandas >= 0.25
- altair >= 5
- ipywidgets
- anywidgetTo install the extension, execute:
pip install persist_extIf the Jupyter server is running, you might have to reload the browser page and restart the kernel.
To remove the extension, execute:
pip uninstall persist_extPersist uses hatch to manage the development, build and publish workflows. You can install hatch using pipx, pip or Homebrew (on MacOS or Unix).
Install hatch globally in isolated environment. We recommend this way.
pipx install hatchInstall hatch in the current Python environment.
WARNING: This may change the system Python installation.
pip install hatchpip install hatchJupyter extensions use a custom version of yarn package manager called jlpm. When any relevant command is run, hatch should automatically install and setup up jlpm.
After installing hatch with your preferred method follow instructions below for workflow you want. We prefix all commands with hatch run to ensure they are run in proper environments.
Run the setup script from package.json:
hatch run jlpm setupWhen setup is completed, open three terminal windows and run the follow per terminal.
Setup vite dev server to build the widgets
hatch run watch_widgetsStart dev server to watch and build the extension
hatch run watch_extensionRun JupyterLab server with minimize flag set to false, which gives better stack traces aqnd debugging experience.
hatch run run_labTo build the extension as a standalone Python package, run:
hatch run build_extensionTo publish the extension, first we create a proper version. We can run any of the following
hatch version patch # x.x.1
hatch version minor # x.1.x
hatch version major # 1.x.xYou can also append release candidate label:
hatch version rcFinally you can directly specify the exact version:
hatch version "1.3.0"Once the proper version is set, build the extension using the build workflow.
When the build is successful, you can publish the extension if you have proper authorization:
hatch publishThe widget architecture of Persist is created using anywidget projects.
The interactive visualizations used by Persist are based on the excellent, Vega-Lite and Vega-Altair projects. Specifially the implementation of JupyterChart class in Vega-Altair was of great help in understanding how Vega-Altair chart can be turned into a widget. We gratefully acknowledge funding from the National Science Foundation (IIS 1751238 and CNS 213756).
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent