tweag/jupyenv

Declarative and reproducible Jupyter environments - powered by Nix

Jupyter ❤️ Nix

This repository provides a Nix-based framework for the definition of declarative and reproducible Jupyter environments. These environments include JupyterLab - configurable with extensions - the classic notebook, and configurable Jupyter kernels.

Getting started

The following snippet will create a new project directory, initialize the project with a flake template, enable a Python kernel, and start the JupyterLab environment.

mkdir my-project
cd my-project
nix flake init --template github:tweag/jupyenv
nix run

After some time, you should see the following in your terminal. Your default web browser should open and enter the JupyterLab environment. If it does not, use one of the suggested URLs.

...
[I 2022-10-11 18:47:30.346 ServerApp] Jupyter Server 1.17.1 is running at:
[I 2022-10-11 18:47:30.346 ServerApp] http://localhost:8888/lab?token=8f2261a45601848bd79eda97d8d39c3d0f4978bc61fbe346
[I 2022-10-11 18:47:30.346 ServerApp]  or http://127.0.0.1:8888/lab?token=8f2261a45601848bd79eda97d8d39c3d0f4978bc61fbe346
[I 2022-10-11 18:47:30.346 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2022-10-11 18:47:30.359 ServerApp] 
    
    To access the server, open this file in a browser:
        file:///home/bakerdn/.local/share/jupyter/runtime/jpserver-286971-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/lab?token=8f2261a45601848bd79eda97d8d39c3d0f4978bc61fbe346
     or http://127.0.0.1:8888/lab?token=8f2261a45601848bd79eda97d8d39c3d0f4978bc61fbe346

Documentation

See the website for further instructions about the environment, kernels, and extensions.

Contributing

PRs are welcome!

This project provides a development shell which you can enter with nix develop. Please run pre-commit run -all before submitting your pull request for review. This will run a nix formatter for consistency.

If you are new to contributing to open source, this guide helps explain why, what, and how to successfully get involved.

License

This project is licensed under the MIT License. See the LICENSE file for details.

jupyenv's Issues

Make env overridable?

Thanks for this project! I've been finding it very useful.

I recently ran into an issue when I wanted to add jupyterWith to an existing project's shell.nix, which already contained several dependencies external to the Python environment.

My first thought was to add something like

pkgs.python3.buildEnv.override { extraLibs = [ (jupyterlabWith { ... }) ]; }

to buildInputs in the existing mkShell call, but then I was missing additional deps (e.g. generateDirectory) and env vars defined in (jupyterlabWith { ... }).env.

I'm relatively new to Nix, so I may be missing a standard way to do this, but I was eventually able to solve the problem by making env overridable, and then replacing mkShell in my existing expression with (jupyterlabWith { ... }).env.override { ... }. Here is my proposed change:

https://github.com/mcwitt/jupyterWith/commit/bea6658d3686f5fb7cc57b0129adaf0902600f0a

Does this seem like a reasonable way to allow composition of jupyterWith with existing environments?

Merging in Nixpkgs

This issue will track the progress of the work for merging in Nixpkgs, announced here. I will do this here so that there is less noise in the Nixpkgs repo.

mkDirectoryWith can invisibly fail to produce static

it seems to be possible that the impure mkDirectory can fail to build and nix still produces the derivation. In that case, only a true garbage collect of this derivation helps to force nix to rebuild it.

This is another hint that we should probably have a prebuild default jupyterLab with some extensions to get a truly pure derivation in the end. Maybe we can do this once jupyterlab 1.0 is out.

Iscala kernel

Hello!

I use nixOS a lot and it would be nice to have a Scala kernel for Jupyter.

Thank you very much!

Contribute jupyterlab as a service to nixpkgs?

Would this be suitable to be contributed as a service to nixpkgs similar to the existing services.jupyter? Sorry if this is already described somewhere, but what does this do differently, aside from being an environment for nix-shell rather than a service?

(I'm asking because I've been using a deployment of the existing jupyter service on a remote server and wondering how this would fit into the broader picture. I wanted to substitute jupyter with jupyterlab to see what that is like.)

Find a way to generate a Jupyter folder from a single file

We want to be able to eventually merge this repository into Nixpkgs. Therefore, we must find a way to make compatible #72 (default set of extensions) and the fact that Nixpkgs will not accept to have neither the full JupyterLab folder in the repository, neither a link given by us pointing to this folder uploaded somewhere.

Our current best pointer is to use the yarn.lock generated by Jupyter during the installation of extensions to reverse-engineer the Jupyter folder. The difficulties are:

• It is not completely clear how to generate the Jupyter folder using just the node_modules folder. There are Javascript files that mention explicitly the names of installed extensions.

• When installing extensions, Jupyter downloads a version it considers good, and keep it inside a local folder. The lock files does not refer to a particular version of the extension, but to this local copy of the extension.

This issue will follow the progress towards this goal.

features.h missing in xeus-cling

the xeus-cling kernel fails on docker and nixos with:

[I 20:53:13.375 LabApp] KernelRestarter: restarting kernel (3/5), new random ports
In file included from input_line_1:1:
In file included from /nix/store/4ga86h16l157r7bas9hcwxgl9d3r32s6-gcc-7.4.0/lib/gcc/x86_64-unknown-linux-gnu/7.4.0/../../../../include/c++/7.4.0/new:39:
In file included from /nix/store/4ga86h16l157r7bas9hcwxgl9d3r32s6-gcc-7.4.0/lib/gcc/x86_64-unknown-linux-gnu/7.4.0/../../../../include/c++/7.4.0/x86_64-unknown-linux-gnu/bits/c++config.h:533:
/nix/store/4ga86h16l157r7bas9hcwxgl9d3r32s6-gcc-7.4.0/lib/gcc/x86_64-unknown-linux-gnu/7.4.0/../../../../include/c++/7.4.0/x86_64-unknown-linux-gnu/bits/os_defines.h:39:10: fatal error: 'features.h' file not found
#include <features.h>
         ^~~~~~~~~~~~
/nix/store/lalaj49lwiaf7gwdxjqbz63h3jw24s3z-xeusCling/bin/xeusCling: line 4:    29 Segmentation fault      (core dumped) /nix/store/si3szb318jkvm1xsmwhjypjh351rmyd1-xeusCling/bin/xeus-cling "$@"

it works with the nix package manager on ubuntu

Fix dataHaskell example

Changes to the default package sets

Hi

A Overlay instructions

Trying google with no success - So I will appreciate help here.
It tried to run nix-buid in the folder nix in order to install the overlays without success.
I am new with nix and jupyterWith
Can you provide the specific instructions to install specific python/haskell/R librairies with nix/python-overlays.nix || nix/haskell-overlays.nix?
Within the python-overlays, an example with fetchgit and one with fetch from local would be more thab welcome.

B nix-shell with example provided failed.

I did:

nix-shell example/Haskell/bayesMonad/shell.nix

I got this error.

npm ERR! /build/.npm/_logs/2019-09-15T10_19_30_306Z-debug.log

Errored, use --debug for full output:
ValueError: "jupyterlab-ihaskell" is not a valid npm package
builder for '/nix/store/5x3wj8zijkffvwrrd7f39c88p5k5b8gg-jupyterlab-extended.drv' failed with exit code 1
building '/nix/store/kxy7grqvjn87zdviyz2n8bs6241r12pn-ieee754-0.8.0.drv'...
cannot build derivation '/nix/store/kz3nf9jl8b93pmxaainb8177akw8d42v-python3.7-jupyterlab-0.35.4.drv':
1 dependencies couldn't be built
error: build of '/nix/store/k953hyqsj02rwbn4ayx9kymm1978jh3h-ihaskell-kernel.drv', '/nix/store/kz3nf9jl8b93pmxaainb8177akw8d42v-python3.7-jupyterlab-0.35.4.drv' failed

Christophe

Module not found in Python example

I run the following:

git clone https://github.com/tweag/jupyterWith
cd jupyterWith/example/Python/datascience
nix-shell --command "jupyter lab"

This opens JupyterLab in my browser. I then create a new Python 3 notebook with a single cell containing

import pandas

Running this cell yields the following error:

ModuleNotFoundError                 Traceback (most recent call last)
<ipython-input-1-38d4b0363d82> in <module>
----> 1 import pandas

ModuleNotFoundError: No module named 'pandas'

override nixpkgs haskellPackages with custom ihaskell

The kernel fails with:

[I 16:01:15.348 LabApp] KernelRestarter: restarting kernel (3/5), new random ports
ihaskell: 
Ran commands: 
ghc-pkg check
which ghc-pkg

Exception: user error (shelly did not find ghc-pkg in the PATH:

probably ghc-pkg is not properly propagated to jupyterlab?

Rust kernel: evcxr support via evcxr_jupyter

Is there any interest in adding support for evcxr_jupyter?
It looks like it has some quirks, but I think it would be a nice addition to this project.

evcxr is in tree already, though evcxr_jupyter --install needs to be run after evcxr is installed.

Host for Remote Access

Hi!

I'd like to host the jupyterwith on my server for remote access.
When I "just" try to access it, i get 403 and the shell shows me, that I should set
"NotebookApp.allow_remote_access".

That is inside the jupyter.config and should be a oneliner, using the official pgk this would be
services.jupyter.notebookConfig = ["c.NotebookApp.allow_remote_access=true"]

sadly I am new to nix and my naive-attempts didn't work out.
In the end, the NotebookApp.allow_remote_access must stand in the jupyter_config.py.

Any clues where to put this?

expose commands to start the environment shells such as ghci or ipython

it would be nice if we could use cabal build and ghci or python in the kernel environments. Not from Jupyterlab but from the nix-shell.

Checkout Jupyter 2.1.0

The new release has an embedded extension manager, that may or may not improve our life. If the only thing it does is manage extensions on the JupyterLab folder from the frontend, the situation doesn't change. Otherwise, maybe we can make some things simpler.

Link: https://jupyterlab.readthedocs.io/en/stable/getting_started/changelog.html#v2-1-0

add more extensions

Jupyterhub?

This overlay is an extremely important 'path forward' for us; we are evaluating nixos to simplify our data science stack, and jupyter + extensions has been one of the pain points. While jupyterWith solves this, we actually need a system wide multi-user implementation utilizing jupyterhub. Since we are still very much newbies, we are wondering if there is any way we can utilize the maturity of jupyterWith, but using it to deploy a 'batteries included' jupyterhub configuration?

"mv: cannot overwrite non-directory" on prooted nix environment

-sh-4.2$ nix-shell --command "jupyter lab"
building '/nix/store/i97fwk0x0p0882g8jsbbcabr0rndimv9-source.drv'...

trying https://github.com/gibiansky/IHaskell/archive/c070adf8828dad378bb0048483c16f2640a339b5.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 160 0 160 0 0 1052 0 --:--:-- --:--:-- --:--:-- 1052
100 743k 0 743k 0 0 1773k 0 --:--:-- --:--:-- --:--:-- 1773k
unpacking source archive /tmp/nix-build-source.drv-0/c070adf8828dad378bb0048483c16f2640a339b5.tar.gz
mv: cannot overwrite non-directory '/nix/store/p1cl8mnghbdzpqffb4gqv63aadldbkf2-source' with directory '/tmp/nix-build-source.drv-0/unpack/IHaskell-c070adf8828dad378bb0048483c1
6f2640a339b5'
builder for '/nix/store/i97fwk0x0p0882g8jsbbcabr0rndimv9-source.drv' failed with exit code 1
cannot build derivation '/nix/store/w4n7ny2lc1m3lfq1lgvi6c559jhpnvhp-cabal2nix-ihaskell.drv': 1 dependencies couldn't be built
error: build of '/nix/store/w4n7ny2lc1m3lfq1lgvi6c559jhpnvhp-cabal2nix-ihaskell.drv' failed
(use '--show-trace' to show detailed location information)

ihaskell kernel fails to build through shell.nix

Warning:
    This package indirectly depends on multiple versions of the same package. This is very likely to cause a compile failure.
      package haskell-src-meta (haskell-src-meta-0.8.0.3-A0INOjY46DeCpbNyU3ew6W) requires haskell-src-exts-1.20.3-KjyAxm84ddk16DoDOnTGLG
      package ihaskell (ihaskell-0.9.1.0) requires haskell-src-exts-1.21.0-Av9ahj0Fv1J61UrpP0Lbr9
      package hlint (hlint-2.1.12-8LPlcWvLdL3CSLtI1Da8GJ) requires haskell-src-exts-1.21.0-Av9ahj0Fv1J61UrpP0Lbr9
      package haskell-src-exts-util (haskell-src-exts-util-0.2.4-FQcQZkmkCIXDClqeYxzo5G) requires haskell-src-exts-1.21.0-Av9ahj0Fv1J61UrpP0Lbr9

...

*** abort because of serious configure-time warning from Cabal
builder for '/nix/store/jmsmfx8kbr77dwnczwjnnx7av6nrd5wa-ihaskell-0.9.1.0.drv' failed with exit code 1
cannot build derivation '/nix/store/gkaxhkv29gqjq60b0kqn5s0b7k3f3bxl-ihaskell-kernel.drv': 1 dependencies couldn't be built
error: build of '/nix/store/25f9s06z75069w2l3rhvidfm2gq2a9dz-ipython-kernel.drv', '/nix/store/gkaxhkv29gqjq60b0kqn5s0b7k3f3bxl-ihaskell-kernel.drv', '/nix/store/qpsmflkrs0n34brx63b8kran7bfbfpwh-python3.7-notebook-5.7.2.drv' failed

jupyterlab builds fine, ihaskell fails ...

can't install joblib

here my output

=================================== FAILURES ===================================
_____________________________ test_lz4_compression _____________________________

tmpdir = local('/build/pytest-of-nixbld/pytest-0/test_lz4_compression0')

    @with_lz4
    def test_lz4_compression(tmpdir):
        # Check that lz4 can be used when dependency is available.
        import lz4.frame
        compressor = 'lz4'
        assert compressor in _COMPRESSORS
        assert _COMPRESSORS[compressor].fileobj_factory == lz4.frame.LZ4FrameFile

        fname = tmpdir.join('test.pkl').strpath
        data = 'test data'
>       numpy_pickle.dump(data, fname, compress=compressor)

joblib/test/test_numpy_pickle.py:1047:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
joblib/numpy_pickle.py:498: in dump
    compress_level)) as f:
joblib/numpy_pickle_utils.py:192: in _write_fileobject
    filename, compresslevel=compresslevel)
joblib/compressor.py:234: in compressor_file
    self._check_versions()
joblib/compressor.py:229: in _check_versions
    if lz4 is None or LooseVersion(lz4.__version__) < LooseVersion('0.19'):
/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/lib/python3.7/distutils/version.py:52: in __lt__
    c = self._cmp(other)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = LooseVersion ('v2.1.2'), other = LooseVersion ('0.19')

    def _cmp (self, other):
        if isinstance(other, str):
            other = LooseVersion(other)

        if self.version == other.version:
            return 0
>       if self.version < other.version:
E       TypeError: '<' not supported between instances of 'str' and 'int'

/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/lib/python3.7/distutils/version.py:337: TypeError
============== 1 failed, 1012 passed, 67 skipped in 67.06 seconds ==============
[INFO:MainProcess:MainThread] process shutting down
[DEBUG:MainProcess:MainThread] running all "atexit" finalizers with priority >= 0
[DEBUG:MainProcess:MainThread] Interpreter shutting down. Waking up queue_manager_threads [(<Thread(QueueManagerThread, started daemon 140736364254976)>, <joblib.externals.loky.process_executor._ThreadWakeup object at 0x7fffed463a20>)]
[DEBUG:MainProcess:QueueManagerThread] queue management thread shutting down
[DEBUG:MainProcess:QueueManagerThread] closing call_queue
[DEBUG:MainProcess:QueueManagerThread] telling queue thread to quit
[DEBUG:MainProcess:QueueManagerThread] joining processes
[DEBUG:MainProcess:QueueFeederThread] feeder thread got sentinel -- exiting
[DEBUG:MainProcess:QueueManagerThread] queue management thread clean shutdown of worker processes: []
[DEBUG:MainProcess:MainThread] running the remaining "atexit" finalizers
builder for '/nix/store/zhrfqgpp5a4yg4k2r7w6hfkimzf5s1vy-python3.7-joblib-0.12.4.drv' failed with exit code 1
cannot build derivation '/nix/store/0z9166ghjdzxxkpcwzg6s10vjkxf7s7v-python3-3.7.2-env.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/bm8x35wkl1y5l5yc9sgxhrmwxwg5z1cj-ipython-python.drv': 1 dependencies couldn't be built
error: build of '/nix/store/bm8x35wkl1y5l5yc9sgxhrmwxwg5z1cj-ipython-python.drv', '/nix/store/jhv8vm2fsyz29ypmlvyxm6366l9sxhv1-python3.7-jupyterlab-0.35.4.drv' failed

overlay.nix missing

I am getting this:

$ nix-build
error: getting status of '.../projects/jupyter/jupyter-with/overlay.nix': No such file or directory

I think the overlay.nix was just not added. You have it @guaraqe , right?

Establish the Jupyter extensions to be shipped by default

We want to have a default Jupyter Lab folder that is well-suited to most applications to be the default ones. This amounts to choosing a set of extension for which we will generate a folder and upload somewhere. The goal of this issue is to determine this set.

A pointer is the following link, showing the set of Jupyter extensions sorted by popularity: https://www.npmjs.com/search?q=keywords%3Ajupyterlab-extension&ranking=popularity

Another is the "awesome" repo for Jupyter Lab extensions: https://github.com/mauhai/awesome-jupyterlab

An aspect that should also be taken into account are extensions that are necessary (or very useful) for using some of the kernels we have, such as IHaskell.

add pythonPackages to the environment

Strange MKL dependency + allowUnfree error

Ok I'm baffled...I have { allowUnfree = true; } in my ~/.config/nixpkgs/config.nix and furthermore, I do not use MKL in any python overlays as it breaks scikit-learn.

I have two nearly identical shells other than one using vanilla jupyter and the other jupyterWith:
https://gist.github.com/tbenst/c88da1138628a6be1d5b88d3a9337f2a

The former builds fine, but the latter gives:

error: Package ‘mkl-2019.5.281’ in /nix/store/mi8xmwmybbxa8w7062j1b00q6x42x6gc-source/pkgs/development/libraries/science/math/mkl/default.nix:110 has an unfree license (‘issl’), refusing to evaluate.

Note that nix-shell -p mkl also works fine. Any idea what could be happening?

Integrating jupyterlab extensions

I'm going to spend some time in the next few weeks to see how I can get a pure nix derivation for extensions. I will focus on the following extensions:

• ipywidgets
• jupyterlab-git

Including a local Cabal project via overlays

I want to import a module from a local Haskell-Cabal package via cabal-2-nix.
The package is not part of nixpkgs.
I tried to use the overlay mechanism but had no success.
Could you please add an example or some documentation on how to do this.

Document how to add additional packages to the environment, such as pandoc

I use jupyterWith through this shell.nix:

let
  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) { };

  iPython = jupyter.kernels.iPythonWith {
    name = "numpy";
    packages = p: with p; [
      numpy
      bokeh
    ];
  };

  jupyterEnvironment = jupyter.jupyterlabWith {
    kernels = [ iPython ];
    directory = jupyter.mkDirectoryWith {
      extensions = [
        "jupyterlab_bokeh"
      ];
    };
  };

in
  jupyterEnvironment.env

and activate the environment with nix-shell --option sandbox false --command "jupyter lab".

When I try to export a notebook to a PDF, my browser shows an error that pandoc is not installed.

I can't figure out how to get pandoc into my jupyterWith environment. This seems like a common task, perhaps suitable for documenting in the readme?

iHaskell kernel doesn't build

Describe the bug
The iHaskell kernel does not build. It fails with a 404 error when trying to fetch https://gitlab.haskell.org/ghc/head.hackage/raw/master/patches/haskell-src-meta-0.8.3.patch:

building '/nix/store/fv2p3zlabwlc8ia7f3shgrqxpcq5d4yv-haskell-src-meta-0.8.3.patch.drv'...

trying https://gitlab.haskell.org/ghc/head.hackage/raw/master/patches/haskell-src-meta-0.8.3.patch
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404 
error: cannot download haskell-src-meta-0.8.3.patch from any mirror
builder for '/nix/store/fv2p3zlabwlc8ia7f3shgrqxpcq5d4yv-haskell-src-meta-0.8.3.patch.drv' failed with exit code 1
cannot build derivation '/nix/store/ryzcpipwlcxnvv8mddikky9dncvghf8z-haskell-src-meta-0.8.3.drv': 1 dependencies couldn't be built
building '/nix/store/rx9dk62f951k3spmwpwd79i85y10n0sq-hg-git-0.8.12.tar.gz.drv'...
cannot build derivation '/nix/store/3l7f2gpziv72wmz0incpzv2phh7y9khj-interpolate-0.2.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/m6lji6hkpasnfdny8s41kcqjgmqk519k-hpack-0.33.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/31hci3yhl19xnxyfp97n2azsi2q28a7z-cabal2nix-2.15.0.drv': 1 dependencies couldn't be built
building '/nix/store/5v89dr0g4p0vzad9xrcf64a18f4xyb7l-unzip60.tar.gz.drv'...
cannot build derivation '/nix/store/bpzjns6f98y4hs3i5jxa95f2c7xmh23b-cabal2nix-ihaskell.drv': 1 dependencies couldn't be built
error: build of '/nix/store/bpzjns6f98y4hs3i5jxa95f2c7xmh23b-cabal2nix-ihaskell.drv' failed
(use '--show-trace' to show detailed location information)

To Reproduce
nix-build the following:

let
  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) {};

  iHaskell = jupyter.kernels.iHaskellWith {
    name = "haskell";
    # packages = p: with p; [ hvega formatting ];
  };
in
  iHaskell

Expected behavior
I expected iHaskell to be built.

Environment

• OS name + version: Ubuntu 19.10
• Version of the code: the example seems to pull the most recent version from master

Additional context
Nix 2.2.2, nixpkgs 19.03

Editable (development mode) Python packages?

Is it possible to add editable Python packages (development mode) to the Python kernel environment? That is, as if the package was installed as pip install -e. Nixpkgs supports this to some extent: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#development-mode. But I don't see how to combine that approach with the approach of jupyterWith.

Add a Cachix cache

This would make the setup faster both for users and for the test suite, since it has to rebuild everything everytime.

Juniper kernel doesn't show plots in Jupyter Lab

this seems to be a problem of the kernel and this is the related issue:

JuniperKernel/JuniperKernel#6

Understanding jupyterWith nixpkgs `pkgs` version used

Hey, it's not a bug report but rather questions from a Nix newbie.

Looking at the repo, I can see that jupyterWith uses by default a specific version of nixpkgs

https://github.com/tweag/jupyterWith/blob/7929eceafe2f59257c089707d2130570cf22a8dc/default.nix#L2

https://github.com/tweag/jupyterWith/blob/7929eceafe2f59257c089707d2130570cf22a8dc/nix/default.nix#L10

https://github.com/tweag/jupyterWith/blob/7929eceafe2f59257c089707d2130570cf22a8dc/nix/nixpkgs.nix#L2

So if I want to use jupyterWith with the most recent Python packages (from nixpkgs master branch for instance) I would have to call

jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) { inherits pkgs; };

by specifying inherits pkgs;. However when doing that, I also need to use the overlays defined here

https://github.com/tweag/jupyterWith/blob/7929eceafe2f59257c089707d2130570cf22a8dc/nix/default.nix#L4

On the pkgs that I pass down.

Am I understanding correctly ?

If I am correct, how can I import those overlays into my .nix (I am using home-manager) ?

Thanks

Fix tensorflow Haskell example

Progression towards a stable release

This is a action plan towards having a stable JupyterWith. The ideal goal of this plan is to have:

• JupyterWith as a part of Nixpkgs. Having JupyterWith as an external package makes it more flexible, but creates a continuous mismatch between Nixpkgs and us. Having this as the default interface for Nixpkgs would make it more accessible, and maintenance more progressive.

• Extensions customizable at will. Using JupyterWith should not be a barrier to any use of Jupyter. All extensions should be accessible, even if through the use of an escape hatch.

• A ready-to-use package set for data science. In Python and maybe Haskell. The main public of Jupyter Notebooks are data scientists, so there should be as little friction as possible for this use case.

The third point is kinda orthogonal to other points, and is covered by issue #62. For the other two points, the proposed steps are:

1. Create a standard set of lab and server extensions. This amounts to make a generate-folder with adequate parameters, and to stock this folder somewhere. The README should make clear that using only the default extensions lead to the most stable behavior. This should solve the problem of most users opening issues in this repository. #72

2. Setup the standard set of extensions using Nix. When generating the JupyterLab folder, a yarn.lock file is created. We should be able to use yarn2nix to generate an appropriate Jupyter folder using only this file. This would open the door for adding JupyterWith to Nixpkgs, since we would have no external dependencies. #73

3. Make the procedure above accessible to users. It should be easy for users to reproduce the procedure above, and to be able to setup their own JupyterLab distribution using a yarn.lock generated by JupyterLab.

4. Declarative setup of extensions. This would allow to use any extension declaratively from a Nix expression. This could take two forms. The first is having some kind of Jupyter LTS package set of extensions. The second is collaborating with Jupyter folks so that Nix can be used as a determinist package manager, and to be able to fix versions of all packages easily.

This issue is here so we can follow progress, the actual work will happen in other issues.

Python package strange behavior.

I have file called jupyter.nix as follows:

{ jupyterWith # path obtained from fetchFromGitHub
, haskellPackages
, haskellPkg ? (_: [])
, pythonPkg ? (_: [])
, pythonOverridesList ? []
}:

let
  jupyterSrc = jupyterWith + "/nix";
  inherit (import jupyterSrc {}) pkgs lib;
  jupyterPython3 = (import jupyterSrc {}).python3;
  jupyter = import jupyterWith {};

  iPython =jupyter.kernels.iPythonWith {
    python3 =
      jupyterPython3.override (old: with lib; {
        packageOverrides =
          foldr composeExtensions old.packageOverrides
            pythonOverridesList;
      });
    name = "iPython";
    packages = pythonPkg;
  };

  iHaskellWithPackages = jupyter.kernels.iHaskellWith {
    inherit haskellPackages;
    name = "iHaskell";
    packages = haskellPkg;
  };
in {
  # Jupyter Lab and Kernels
  jupyterLabWithKernels =
    jupyter.jupyterlabWith {
      kernels =
        [
          iHaskellWithPackages
          iPython
        ];
    };
}

This supposed to be called within my project's shell which looks like this:

shellToolOverrides = ghc: super:
        let
          pytorch-src = getDeps  ./deps/python/pytorch;
          haskellPackages = ghc;
          pythonPkg  =
            p: with p;
              [ pytorch numpy matplotlib pandas seaborn scikitlearn umap-learn ];
          haskellPkg =
            p: with p;
              [ hvega statistics vector ihaskell-hvega aeson aeson-pretty foldl
                hasktorch libtorch-ffi hasktorch-codegen HUnit random-shuffle
              ];
          jupyter = myApp.jupyter;
          # pythonOverridesList can have multiple style on how python package installed.
          pythonOverridesList = [
            (self: super:
              {
                pytorch =
                  (import  (pytorch-src + "/release.nix") {}).pytorchFull;
            })
          ];
        in {
          inherit (jupyter haskellPackages haskellPkg pythonPkg pythonOverridesList) jupyterLabWithKernels;
          ghc-mod = null;
        };

All haskell packages works fine. Python package works fine except pytorch (I'm getting it from stites's pytorch-world.

The strange behavior is in my first attempt, the following code:

import torch
torch.__version__

works under Python3 - iPython kernel ( I need to change the kernels first to make it work).

When I leave my nix-shell environment, and try to run jupyter lab again, it caused ModuleNotFoundError. But numpy works fine, and when I look into /nix/store/...-python3-3.7.5-env/lib/python3.7/site-packages/ there is no torch.

Another thing that I try is building jupyterWith using python36. But apparently, jupyterWith force us to use python3Packages here, while in my opinion, we can have python36Packages and so on. Having python3Packages caused another error, which ipykernel and jupyter build using python 3.7.5, instead of our custom python package set (in this case, python36).

Installing package with --extra-index-url

I have a package that takes an external repo index URL as an argument to install - pip3 install _ --extra-index-url _ --user. Do you have any suggestions about how this can be implemented in jupyterWith?

How can I fix npm fetch SSLCertVerificationError ?

SSLCertVerificationError

Platform: osx
test env : https://github.com/GTrunSec/analysis-arg-lab/blob/master/osx-env.nix
ihaskell_labextension for Jupyterlab version 2.0 later.

Make libraries accessible outside of Jupyter

It would be nice to define a JupyterWith environment and be able to:

• have libraries accessible on Jupyter
• have libraries accessible on the shell, with python or ghci, for example.

This should not be too hard in the Python and Haskell cases.

One question that arises is how to manage a shell where there are multiple concurrent kernels for the same language. The solution I see for the moment is to create different wrapping executables called python-$KERNEL_NAME or ghci-$KERNEL_NAME, which access the correct environments.

unable to use the suggested shell.nix (as git too old?)

I have nix-2.2.1 (also seen with nix-2.1.3) and with the README-suggested shell.nix (shown below) I get the following failure:

% nix-shell
nix-shell
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
error: program 'git' failed with exit code 129

My guess is that it's using the system git which is not particularly modern (and I can't easily update):

% git --version
git version 1.8.3.1

I assume there's a way for me to tell nix to install its own git and use that, but I am a nix neophyte and have no idea how to do this, so any help would be greatly appreciated (I also understand that this is not a jupyterWith problem directly).

For reference, the shell.nix I used is:

% cat shell.nix
let
  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) {};

  iPython = jupyter.kernels.iPythonWith {
    name = "python";
    packages = p: with p; [ numpy ];
  };

  iHaskell = jupyter.kernels.iHaskellWith {
    name = "haskell";
    packages = p: with p; [ hvega formatting ];
  };

  jupyterEnvironment =
    jupyter.jupyterlabWith {
      kernels = [ iPython iHaskell ];
    };
in
  jupyterEnvironment.env

Update Jupyter to version 1.x

There is work in progress in the Nixpkgs repository here:

• NixOS/nixpkgs#65586
• NixOS/nixpkgs#65625

a PR in progress: NixOS/nixpkgs#67423

Create a standard data science overlay

We should have a default overlay containing a sort of LTS for standard data science libraries: tensorflow, pytorch, numpy, pandas, etc...

Many of these libraries are not always trivial to install, so this has an added value by itself, besides the convenience for the explorative data scientist who whishes to use JupyterWith, who does not have to be concerned with package setup.

Errors loading jupytext with `jupyter lab`

I'm trying to enable jupytext as the content manager for my notebooks. However, it seems the python module isn't available to the loader during start-up with jupyter lab (error being ModuleNotFoundError: No module named 'jupytext')

I made a gist of my shell.nix file and the output of jupyter lab here:

(https://gist.github.com/mdtisdall/3446714f86373697609961b8b5d593bb)

jupyterlab terminal doesn't work in docker environment

when clicking on "terminal" in the jupyterlab launcher in the docker environment, an exception is thrown:

[E 22:45:19.596 LabApp] Uncaught exception POST /api/terminals?1551307519574 (172.17.0.1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='POST', uri='/api/terminals?1551307519574', version='HTTP/1.1', remote_ip='172.17.0.1')
    Traceback (most recent call last):
      File "/nix/store/wfnwy4byyfgdymzl98rx65w6vps25r1q-python3.7-tornado-5.1/lib/python3.7/site-packages/tornado/web.py", line 1590, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/nix/store/wfnwy4byyfgdymzl98rx65w6vps25r1q-python3.7-tornado-5.1/lib/python3.7/site-packages/tornado/web.py", line 3006, in wrapper
        return method(self, *args, **kwargs)
      File "/nix/store/shr9l7bl21y9i4ibsvx5mqb8y1jk5jy8-python3.7-notebook-5.7.2/lib/python3.7/site-packages/notebook/terminal/api_handlers.py", line 16, in post
        name, _ = self.terminal_manager.new_named_terminal()
      File "/nix/store/siq6c0n6mllbq5jlf50hgwv17dkpdh7x-python3.7-terminado-0.8.1/lib/python3.7/site-packages/terminado/management.py", line 319, in new_named_terminal
        term = self.new_terminal()
      File "/nix/store/siq6c0n6mllbq5jlf50hgwv17dkpdh7x-python3.7-terminado-0.8.1/lib/python3.7/site-packages/terminado/management.py", line 171, in new_terminal
        pty = PtyProcessUnicode.spawn(argv, env=env, cwd=options.get('cwd', None))
      File "/nix/store/mmca1wgjind71kqqypc0c3riqbas0ahp-python3.7-ptyprocess-0.6.0/lib/python3.7/site-packages/ptyprocess/ptyprocess.py", line 209, in spawn
        command_with_path = which(command)
      File "/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/lib/python3.7/shutil.py", line 1135, in which
        if os.path.dirname(cmd):
      File "/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/lib/python3.7/posixpath.py", line 156, in dirname
        p = os.fspath(p)
    TypeError: expected str, bytes or os.PathLike object, not NoneType
[W 22:45:19.600 LabApp] Unhandled error

we should probably install a shell.

ipykernel tests fail on macOS Mojave

I use this shell.nix:

let
  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) {};

  iPython = jupyter.kernels.iPythonWith {
    name = "python";
    packages = p: with p; [ numpy ];
  };

  jupyterEnvironment =
    jupyter.jupyterlabWith {
      kernels = [ iPython ];
    };
in
  jupyterEnvironment.env

The errors generally look like this:

self = <jupyter_client.kernelspec.KernelSpecManager object at 0x111ebb1d0>
kernel_name = 'python3'

    def _find_spec_directory(self, kernel_name):
        """Find the resource directory of a named kernel spec"""
        for kernel_dir in self.kernel_dirs:
            try:
>               files = os.listdir(kernel_dir)
E               PermissionError: [Errno 1] Operation not permitted: '/usr/local/share/jupyter/kernels'

That path exists on my machine.

IHaskell kernel keeps dying

I'm getting an error when trying to follow the README instructions with nix-shell and the provided shell.nix:

> /nix/store/3k8h7k93fgxa2x2lcj4asa1k3vsxfy7r-nodejs-8.12.0/bin/npm pack jupyterlab-ihaskell
npm ERR! code ENOTFOUND
npm ERR! errno ENOTFOUND
npm ERR! network request to https://registry.npmjs.org/jupyterlab-ihaskell failed, reason: getaddrinfo ENOTFOUND registry.npmjs.org registry.npmjs.org:443
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

npm ERR! A complete log of this run can be found in:
npm ERR!     /build/.npm/_logs/2019-01-23T15_11_59_983Z-debug.log

Errored, use --debug for full output:
ValueError: "jupyterlab-ihaskell" is not a valid npm package
builder for '/nix/store/f0m8jmyhsnxzrpq7r4f27pb8h84gb5hs-jupyterlab-extended.drv' failed with exit code 1
cannot build derivation '/nix/store/vqhfiixs6i859r1sa0qq17ndnkdf7cy3-python3.7-jupyterlab-0.35.4.drv': 1 dependencies couldn't be built
error: build of '/nix/store/vqhfiixs6i859r1sa0qq17ndnkdf7cy3-python3.7-jupyterlab-0.35.4.drv' failed

Is this something I need to fix on my end or is something else wrong?

Cachix not up-to-date

It looks like jupyterwith.cachix.org does not contain a cached build for the latest master commit, 7929ece.

I am using the jupyterwith Cachix cache. Opening a nix-shell using https://github.com/tweag/blog-resources/blob/master/monad-bayes-series/shell.nix still takes forever, and it is clearly building lots of Jupyter-related things.

Related: #12

kernels could be completely isolated from jupyterlab

otherwise the jupyterlab derivation has to be recomputed each time ...

Make sure JupyterWith works on Mac

This should make part of the stable release process.

integrate into the nixpkgs jupyter/default.nix derivation

iPython kernel doesn't build

Describe the bug
The iPython kernel does not build. It failes with a 404 error when trying to fetch https://github.com/makepost/ninja/commit/567815df38a2ff54ad7478a90bd75c91e434236a.patch:

building '/nix/store/nirp2j1niqjl9342n0ypcxf67nzi0swm-fix-issue-1510.patch.drv'...

trying https://github.com/makepost/ninja/commit/567815df38a2ff54ad7478a90bd75c91e434236a.patch
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found
error: cannot download fix-issue-1510.patch from any mirror
builder for '/nix/store/nirp2j1niqjl9342n0ypcxf67nzi0swm-fix-issue-1510.patch.drv' failed with exit code 1
building '/nix/store/kx57xwcy2s755h8p12sbaf9whpwwpwn4-fontconfig-2.12.6.tar.bz2.drv'...
cannot build derivation '/nix/store/7cf4mz62qwbq73jhs8sx0j5hf5g4v7z8-ninja-1.9.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/cyibx1z0fr879vjix2h19zibbymkyk3b-glib-2.62.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/29gn6rharj5gl3ibkz32xv74miv0icg9-systemd-243.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/zwl0fwsid5yr05dmrn085xddpqy98gkv-xorgproto-2019.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/wxg68f4v8gyw4nmm69vjvdd69fwalk9v-cups-2.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/n9grs3n58ifx7ny7qk42924dy4dbl2q3-fontforge-20190413.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/y70200nwy60x3bx5i42w2mavxgzrx8dc-libX11-1.6.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/2w5fy1b7dxpm545h10cc0xypazli757q-libXft-2.3.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5igx3spskq034jd8bzbr97q53ll19mc5-python3-3.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/09xnw7b1a9lglr18lqd84s6xqp76as4v-xlibs-wrapper.drv': 1 dependencies couldn't be built
building '/nix/store/scj72wlnxjn1sznjkhlvd3kzcxhbnydf-itstool-2.0.6.tar.bz2.drv'...
cannot build derivation '/nix/store/fv14ibbm9l9n2gpyx71zqdmsms5l1c8p-python3.7-matplotlib-3.1.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/i73wzdddaxckg88a48xi48rab4qvmvkx-python3.7-tkinter-3.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ybpa6432pnsf0ym0vwz5l9b47bk6wx03-python3-3.7.5-env.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/1baqkncq6gxj5nyrsvp1z2z4myzv5sc7-ipython-python.drv': 1 dependencies couldn't be built
error: build of '/nix/store/1baqkncq6gxj5nyrsvp1z2z4myzv5sc7-ipython-python.drv' failed

To Reproduce
nix-build the following:

let
  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) {};

  iPython = jupyter.kernels.iPythonWith {
    name = "python";
    packages = p: with p; [ numpy scipy matplotlib ];
  };
in
  iPython

Expected behavior
I expected iPython to be built.

Environment

• OS name + version: Ubuntu 19.10
• Version of the code: the example seems to pull the most recent version from master

Additional context
Nix 2.2.2, nixpkgs 19.03