davidleoni/jupman

A template for online books made with Jupyter notebooks

jupman's Issues

Unicode characters do not appear in PDF

Unicode characters such as ✪ don't display in PDFs

To fix it, tried to add this in conf.py without success:
Taken inspiration from this

#: To fix @ issue, copied from here: https://github.com/sphinx-doc/sphinx/issues/4136#issuecomment-335956163
latex_engine = 'xelatex'
latex_elements = {
    'papersize': 'a4paper',    
    'preamble' : r"""\usepackage[T1]{fontenc} 
\usepackage[utf8]{inputenc}
"""
}

Cell output should be scrollable

Add toggable solution elements for exercises

It would be nice to have toggable solution divs for exercises.

NOTE: there are various options, will try to collect some in jupman-tests.ipynb

generate exercises from solutions

It would be cool to automatically generate exercises from solutions when building .To do so, we need to distinguish two types of exercises.

Type of exercises

There can be two kinds of exercises, exercises in python files and exercises in jupyter files.

Since Jupman 0.8, it is possible to automatically generate an exercise from a solution file by stripping text marked with special tags. It is possible to inspect generated files in _build/jupman/ directory

Note: in the zips for the students containing solutions, the tag comments are automatically removed from the solution files as well.

Exercises in python files

See example: exercises/python-intro/python-intro.ipynb

In this type of exercises, typically you have a jupyter file (like python-intro.ipynb) that describes the exercise and then the actual exercises are in python files.

If there is a solution file eding in _solution.py but no file ending in _exercise.py like this:

• python_intro1_solution.py
• python_intro1_test.py

then Jupman will try to generate one from a _solution.py file. To do so, it will look for tags to strip inside the solution file.

If there is already an exercise file like this:

• python_intro2_exercise.py
• python_intro2_solution.py
• python_intro2_test.py

Jupman will just copy the existing file.

Exercises in jupyter files

See example: exercises/jupyter-intro/jupyter-intro-solution.ipynb

This type of exercises stay in a jupyter notebook itself. In this case, the following applies:

If there is a notebook ending in -solution.ipynb, the following applies (WARNING: for ipynb files we use dash -, not the underscore _):

• the notebook must contain a title as markdown like # bla bla solution

  Note text must contain solution text, which can be customized in conf.py (you might need to translate it)

• the notebook must contain tags to strip

Tags to strip

Start tags begin with a # while end tags begin with a #\

jupman-raise

Replaces code inside with an Exception (text is customizable in conf.py). Be careful to position the comment exactly with the indentation yuoi want the raise to appear. For example:

def add(x,y):   
    #jupman-raise
    return x + y
    #/jupman-raise

becomes

def add(x,y):   
    raise Exception('TODO IMPLEMENT ME !')

jupman-strip

Just strips code inside

def f(x):
    print(x)

#jupman-strip
def help_func(x,y):
    return x - y
#/jupman-strip

def g(y):
    return y

becomes

def f(x):
    print(x)

def g(y):
    return y
    

write solution here

This special tag for python code erases whatever is found afterwards the # write solution here comment

• you can put how many spaces you want in the comment
• phrase can be customized in conf.py

w = 5

#  write  solution here

x = 5 + w
y = 2 + x

becomes

w = 5

#  write  solution here

Links in zipped exercises should point to website

Relative links in notebooks in zips don't make much sense, as target might be missing from the distribution.

Instead, they should be automatically made to point to the website.

Generate decent PDF of exams

Default Jupyter PDF export is ugly, adds lots of useless numberings.

There are better Latex templates around, we could use them

Python Tutor does not work in JupyterLab

On Ubuntu 16.04, jupyterlab-2.1.4 jupyterlab-server-1.1.5 attempting to run Python Tutor I get:

See also Switch to JupyterLab #13

Go Multilingual

i18n topic is a bit complex, will start with minimal changes

• #164

Include jupman.js on ReadTheDocs

Currently jupman.js and jupman.css are included on ReadTheDocs only when initial cell with jupman.init does not have "nbsphinx": "hidden" metadata.

To include javascript on RTD we should use standard Sphinx mechanism instead:

http://www.sphinx-doc.org/en/stable/extdev/appapi.html#sphinx.application.Sphinx.add_javascript

Embedding website as autoresizable iframe

Would be nice to allow a container website to embed generated website.

We need to allow some message passing between host and iframe to overcome CORS issues.

Python Tutor doesn't overflow in local build

Probably this is due to library loading order, see layout warning

Remove extra line from Python Tutor code

Just to remind myself of this oddity

I had to artificially add an extra line at the beginning of shown code in Python tutor otherwise with small input (1 or two lines) was giving strange errors in the browser.

can't have subfolders for Python

Can't have notebooks in subfolders, as to import jupman module

Possible solutions:

a. add stuff like sys.path.append('../') to all files
b. using symlinks to link parent algolab
c. add some new default kernel that includes the paths I want
d. use Jupyter magic like %run ../../lib.py

all solutions which I don't quite like

See also

• Stack overflow
• This blog for Jupyter magic

Attempts with d. Jupyter magics

I tried putting this:

  %run -i ../../jupman
  jupman_init()

Passing parameters like

--toc

and

--root=../../

Some thoughts:

• Tried static classes but thought they might look confusing to users
• Since %run executes the script, to prevent namespace pollution all functions start with 'jupman_'
• Also note that with %run, last command output is not shown in Jupyter, for that you have to explictly call jupman_init() function in Jupyter
• tried importing .ipynb, but discoverd it is not possible to pass parameters

Decision: sys.path.append

Finally adopted this, it's less weird and just works

  import sys
  sys.path.append('../)
  import jupman
  jupman.init()

automatically remove empty cells

It's quite easy to forget empty cells, it would be nice to automatically remove them.

Provide theming example

By default we use sphinx-rtd-theme, but we can how to customize theme by providing some extra css

Switch to JupyterLab

JupyterLab is certainly the future , but as August 2018 is still v0.33 and NBSphinx does not officially supports it.

PDF generation tends to hang

Currently using plain pdflatex *.tex twice, instead, we should use build commands such as readthedocs ones:

python /home/docs/checkouts/readthedocs.org/user_builds/softpython/envs/latest/bin/sphinx-build -b latex -D language=it -d _build/doctrees . _build/latex

cd _builds/latex
latexmk -r latexmkrc -pdf -f -dvi- -ps- -jobname=softpython -interaction=nonstopmode

Extract tests as questions

It would be cool to parse for tests (assert and unittests) and display them in HTML as question "What is this function supposed to output with this input? User should be able to provide an executable answer to check against the expected result. This might also alleviate the technical burden of having to understand asserts in order to solve some exercises

Linked files are not copied locally

This is weird, files linked from a notebook are not copied when I run sphinx locally, but they copied indeed on ReadTheDocs!

See also nbsphinx documentation

Workaround: put stuff in overlay/_static like overlay/_static/trial.pdf, and refer to them like [My trial](_static/trial.pdf) although this will cause warnings during building, like:

/home/da/Da/prj/jupman/prj/jupman-tests.ipynb: WARNING: file not found: '_static/trial.pdf'

Shorten filenames

We need to support more patterns and simplify file names.

See also: Restructure folders #33

Current situation

text notebook + python solution

- stacks.ipynb
- stack_solution.py
- stack_test.py

Generates

- stacks.ipynb
- stack_solution.py
- stack_exercise.py
- stack_test.py

notebook solution

- lists_solution.ipynb

Generates

- lists_solution.ipynb
- lists_exercise.ipynb

notebook solution + python solution

(This doesn't actually work, I just use it manually for exams)

- exam_solution.ipynb
- stack_solution.py
- stack_test.py

Generates

- exam_solution.ipynb
- exam_exercise.ipynb
- stack_solution.py
- stack_exercise.py
- stack_test.py

Desired patterns

Desired: text notebook + python solution

- stacks.ipynb
- stack_solution.py
- stack_test.py

Generates

- stacks.ipynb
- stack.py
- stack_test.py
- stack_sol.py

Desired: notebook solution

- lists_sol.ipynb

Generates

- lists_sol.ipynb
- lists.ipynb

Desired: notebook solution + python solution

- exam_sol.ipynb
- stack_sol.py
- stack_test.py

Generates (pre-publication):

- exam.ipynb
- stack.py
- stack_test.py

Generates (post-publication):

- exam.ipynb
- exam_sol.ipynb
- stack.py
- stack_sol.py
- stack_test.py

Run tests when building

Now that we have fixed scheme for file names, it would be nice to run tests automatically (in conf.py)

If code is from exercises without implementation, we should just check the file compiles.

use proper logging

Current custom logging works but is embarassing

Breadcrumbs don't work with sections

When there are multiple sections, breadcrumbs always show only the first one, making them useless

As a temporary workaround, we can disable them

Docs » FIRST SECTION» PAGE FROM OTHER SECTION

Embed ipywidgets

I would like ipywidgets to show in generated htmls, tried with softpython project without success.

Apparently requirejs is missing, see this issue:

spatialaudio/nbsphinx#84

Apparently parallel builds cause problems:

spatialaudio/nbsphinx#414

MathJax does not work

After setting a local MathJax (so it doesn't try to fetch it from cdn ), website now display this:

This is infinity: \(\infty\)

In PDF shows correctly.

Improve Python Tutor

Python tutor should work offline and sometimes needs overflowing

Managed to do it this way:

• took code from v5 dev of Python Tutor, didn't need to install extra modules
• merged everything in jupman.py using modulize script
• using dynamic js linking (uncompressed js is ~800kb), works also in Jupyter 6
• seems to not work in Jupyter Lab 1.2 on Windows 10 (apparently fetches js as text), need more testing

Restructure folders

Currently we have this folder structure which gets replicated into zips:

jupman.py
img
  - common.png
exercises
  - lists
     - img
        - list-chart.png
        - lists_exercise.py
        - lists_solution.py
exams
  - 2019-10-14
     - stack_solution.py

A better alternative would be:

- etc
    - jupman.py
    - img 
      - common.png
- lists
    - img
      - list-chart.png
    - lists.py
    - lists_sol.py
    - lists_test.py
- exams
   - 2019-10-14-exam
      - stack_sol.py

When zipped, I would like to bring into exercise folders common files, like so:

- lists
  - etc
     - jupman.py
     - img
        - common.png                         
  - img
     - list-chart.png
  - lists_exercise.py
  - lists_solution.py

This would require to fix paths in :

• .py imports
• .ipynb :
  • python imports
  • image markdown (both ![]() and <img src format)
  • (possibly) HTML links (<a href)

replace index with a page as home

Currently we have a huge TOC as home, but I would like a proper page, but it is not so easy (as always...):

• Seems like it is not possible to include index headers on the left menu :
  sphinx-doc/sphinx#4602
• This guy managed to do something about it
• Markdown: index as markdown is supported only if you include rst with eval_rst
• Jupyter notebook : if you try to include toc in a jupyter notebook, Sphinx doesn't put index page paragraphs in the left sidebar
• Select other theme: if you change from RTD theme, you don't get sidebar links on every page

2.0.0 solution

• use index.ipynb as home
• put contents in toc.rst
• toc-page.rst as master_doc. Only needed to show the toc in a separate page.

Disadvantages:

• if you click on the home (with the icon) in the upper left corner you will be redirected to toc-page.html, so in toc-page.rst I put a redirect to index.html: fixed in 3.3 with jupman.js #71
• Index writing shows at the bottom of the toc instead of the top: fixed in 3.3 with this obscene hack

0.8 solution

Discarded, didn't like the redirect

• from now on you need home.ipynb file, because replacing index.rst is a nightmare!
• new index.rst is just a placeholder which simply redirects to home.html. Do not modify it.
• put the toctree in toc.rst

Can't include svg images

In notebooks they work fine, but in latex I get this error:

! LaTeX Error: Unknown graphics extension: .svg.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.383 \noindent\sphinxincludegraphics{{cc-by}.svg}
                                                  
? 
! LaTeX Error: Unknown graphics extension: .svg.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.383 \noindent\sphinxincludegraphics{{cc-by}.svg}

publishing exams strips jupman-raise

... but its pointless

Also, it shouldn't copy generated exercises ...

global cc-by.png is not copied in zips

mmm what a mistery

Switch to Github Actions

Would like to reduce dependency on ReadTheDocs by using Github Actions

Still, to get a full ReadTheDocs-like experience, you can optionally add 'readthedocs_ext.readthedocs' in conf.py extensions, see https://github.com/readthedocs/readthedocs-sphinx-ext/issues/27#issuecomment-451181542

create admin/test directory

should also create a directory like private/2019-02-13-admin/test for testing exam results

Sections in toc

I would like to have a sidebar like

- home
- page1
- page2
- Section A
    - page3
    - page4
- Section B
    - page5
    - page6

Obviously, it's more complicated then expected.

So far, I used the dirty trick of putting an unholy fake space separator :

.. toctree::
    :caption: Index
    :maxdepth: 2
    :glob:


    Home <index.ipynb>
    usage.md
    jupman-tests.ipynb

|space|
==========

.. |space| unicode:: U+0020 .. space


Part B
============

.. toctree::
    :caption: Exercise examples
    :maxdepth: 2
    :glob:

    exercises/python-intro/python-intro.ipynb
    Jupyter introduction <exercises/jupyter-intro/jupyter-intro-solution.ipynb>

Part C
============


.. toctree::
    :caption: Templates
    :maxdepth: 2
    :glob:


    past-exams.ipynb
    exam-project.ipynb
    project-ideas.ipynb
    jm-templates/project-NAME-SURNAME-ID/markdown.ipynb
    jm-templates/project-NAME-SURNAME-ID/requirements.txt
    changelog.md


Index
-----------

It's almost impossible to find RST errors

Notebooks are converted to RST, but converted RST files are not saved so it's impossible to trace where errors are, see https://github.com/spatialaudio/nbsphinx/issues/83

Notice this might not be an issue anymore when switch to recommonmark is implemented:
is https://github.com/spatialaudio/nbsphinx/issues/36

Add Google analytics

RTD already provides it, but if we build it on GitHub Action elsewhere we need an extra Sphinx plugin and pass it as a secret

There is an extension sphinxcontrib-googleanalytics but as of July 2020 but doesn't work, see https://github.com/sphinx-contrib/googleanalytics/issues/2. Resolved by copy-pasting code.

Improve menu navigation

Expand icons in the menu in non-current sections are not visible. To expand entries you are forced to click on them which triggers loading of TOC page. This is cumbersome.

Instead, we should always visualize + signs and clicking should just expand the section entry without going to TOC

notebooks in templates/ directory are excluded

For jupman projects, I chose to have a directory I named templates where to store i.e. exam templates.

Looking at conf.py, I saw a property templates_path=['_templates']. According to Sphinx docs it should keep html templates, but without understanding it much I had the bad idea to remove the underscore and change it to templates_path=['templates']. This made notebooks inside templates automatically excluded during the build :-/

So to avoid confusion we should rename templates directory to jm-templates and restore templates_path to ['_templates']

Note to myself: if I use jm-templates and link exam tepmlate from Jupman tests notebook, it's fine. But if I include everything by putting jm-templates/**/* in index.rst I get this error:

/home/da/Da/prj/jupman/prj/index.rst:24: WARNING: toctree contains reference to document 'jm-templates/exam/theory/README' that doesn't have a title: no link will be generated
/home/da/Da/prj/jupman/prj/jm-templates/exam/exam-yyyy-mm-dd.ipynb:78: WARNING: undefined label: intro.ipynb#commandments (if the link has no caption the label must precede a section header)
/home/da/Da/prj/jupman/prj/past-exams/2000-12-31/exam-2000-12-31.ipynb:78: WARNING: undefined label: intro.ipynb#commandments (if the link has no caption the label must precede a section header)
Process Process-6:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/da/.local/lib/python3.5/site-packages/sphinx/util/parallel.py", line 94, in _process
    pipe.send((failed, collector.logs, ret))
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/usr/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'setup.<locals>.<lambda>'
Process Process-7:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/da/.local/lib/python3.5/site-packages/sphinx/util/parallel.py", line 94, in _process
    pipe.send((failed, collector.logs, ret))
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/usr/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'setup.<locals>.<lambda>'
Process Process-8:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/da/.local/lib/python3.5/site-packages/sphinx/util/parallel.py", line 94, in _process
    pipe.send((failed, collector.logs, ret))
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/usr/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'setup.<locals>.<lambda>'

^C
interrupted!
Traceback (most recent call last):
  File "./build.py", line 261, in <module>
    run_sphinx(manuals, formats)
  File "./build.py", line 130, in run_sphinx
    res = run(cmd)
  File "./build.py", line 99, in run
    cwd=cwd
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 695, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.5/subprocess.py", line 1059, in communicate
    stdout = self.stdout.read()
KeyboardInterrupt

Support OpenGraph

Maybe one day RTD will support OpenGraph, see issue https://github.com/sphinx-doc/sphinx/issues/2645

But if we don't have RTD, we can use this: https://github.com/wpilibsuite/sphinxext-opengraph

Remove 'solutions' from headers

Currently you are required to append 'solutions' to header names, but it's redundant, especially considering we are more and more putting theory inside the notebooks. On derived exercise notebooks we would still append 'exercise' to distinguish

Solution cells should be hidden by default

Ideally, the processor should recognize solutions from text and transform output accordingly to have hideable panes. Obviously it shouldn't overwrite existing files on user directory.

Currently, there seems to be no standard way to hide cells in Jupyter notebook - with jupyter lab situation seems better.

See also #6 and #14

Generated exercises should be browsable

Currently, exercises generated from solutions are stored in a zip, but I would like them to be generated in the same folder as solutions, without write them in source directory (don't want to accidentally overwrite existing files)

Create example exam while RTD building

Example exam 2000-12-31 should be created during ReadTheDocs build. Note now it is gitignored.

To do so, it would probably be better convert the bash script create_example_exam.sh into a Python script, this way it would also become more portable

Generated PDF sucks

Generated PDF is ugly on my laptop, it's also missing the toc for some reason.

Good news is it looks fine on ReadTheDocs

Check for non-executed cells

Supposing it is user job on its own computer to run the notebooks before submitting them to readthedocs, when build.py is launched it could be useful to check if a notebook has unexecuted cells, and it that case fail the build.

Another option would be to automatically run the notebook and save it (we proper messages), but I'm worried this might overwrite files currently open in Jupyter.

pending_xref error in markdown files

apparently system is not able to resolve correct links from .md files to existing .ipynb files

• sphinx: 2.3.1
• nbsphinx: 0.7.1

In particular, nbsphinx stops execution with error:

         NotImplementedError: Unknown node: pending_xref

Display a 'last modified' date on top of pages

Review tag-based element filtering

There is a new very interesting feature in nbconverter 5.3.0 for filtering cells based on metadata tags I should try when I find the time. You can filter cell input, output and both. More here:

• spatialaudio/nbsphinx#15 (comment)
• https://groups.google.com/forum/#!starred/jupyter/W2M_nLbboj4
• See also toggable cells issue #6

As everything, we really need to make sure it works both in HTML and PDF.

Currently I'm using

> jupyter notebook --version
4.4.1
> nbconvert --version
5.1.1

Check for nbsphinx hidden in first cell

if first cell contains import jupman but it is not hidden, a warning should be generated

Create package

jupman.py should be redistributed to users as it is meant to be import in notebooks, but jupman_tools.py and tests could stay in a proper separate package

It would be appealing to ship student zips with a jupman.py file containing inside all the required files, If files then need to be physically present on disk, the code could automatically generate them.

See also

• #160