coderefinery/sphinx-lesson

Sphinx extension for creating CodeRefinery lessons

Sphinx test lesson

This is a Sphinx extension for software-carpentry style lessons. It is designed as a replacement for the Jekyll-based software templates.

• View the documentation (and example) on github pages.
• Template repository, to copy from.

Features

• Sphinx, including power from all of its extensions.
• ReST
• Markdown via the myst_parser parser, so has access to all Sphinx directives natively
• Jupyter as a source format, including executing the notebook (via myst_nb).
• Automatically building via Github Actions and automatic deployment to Github Pages. Included workflow file builds all branches, so you can also preview pull requests.
• Directives for exercises/prereq/etc, works in both ReST and md.
• The Sphinx part can be separated into a separately installable and versionable Python package, so we don't need git sub-modules.
• Execute code cells in markdown (via myst_nb).
• Consists of sub-extensions for substitutions. Adding sphinx_lesson as an extension will bring in these:
  • sphinx_lesson.directives (the core directives)
  • sphinx_lesson.md_transforms (reprocess some other markdown format into myst_nb format)
  • myst_nb (not developed by us)

Host Site Locally for Development

1. Create a virtual python environment:

   python -m venv venv
   

2. Activate the virtual environment:

   source activate venv/bin/activate
   

3. Install python packages:

   pip install -r requirements.txt
   

4. Build local files (this can also be used for deployment):

   make html
   # Output in _build/html/
   make clean html             # clean + full rebuild
   

5. Or, start a live-compiled service for your compiled site for local development:

   make livehtml
   

   Then view created site in your browser at http://localhost:8000 (follow the link in your console).

Status

In beta use by CodeRefinery and active development. External users would be fine (but let us know so we know to keep things stable).

sphinx-lesson's Issues

Wrong link in PyP

In https://pypi.org/project/sphinx-lesson/

Homepage says
https://github.com/coderefinery/sphinx_lesson (notice underscore)
Should be
https://github.com/coderefinery/sphinx-lesson

error locally after upgrading sphinx_lesson

after upgrading sphinx_lesson from 0.8.7 to 0.8.10 i get this error locally:

Extension error (sphinx_lesson.exerciselist):
Handler <function find_exerciselist_nodes at 0x1099034c0> for event 'env-check-consistency' threw an exception (exception: No such config value: root_doc)

This happens also in a repo where it worked with 0.8.7

@rkdarst

Prompt (e.g. `$` is copied with sphinx-copybutton)

Example: https://coderefinery.github.io/git-intro/basics/#recording-a-snapshot-with-git
copying the text excludes the $, but if you click the copy button, the $ are copied.

Can we color boxes differently?

Now: "challenge" and "discussion" and "callout" and ... are all blue.

Should this repo be a template repo?

Hello and thanks for sharing this repo! I am wondering what is the best way of using it for developing new lessons. I can fork it and rename it, but the forking relationship is not exactly "correct": most of the contents will be removed or reworked in an incompatible manner. Importing it is also an option, I guess, but it seems templates automatize this process somewhat. What are your suggestions?

How to make the side menu collapsible?

The exercise list doesn't work with links in the exercises

At least I see one error if you try to put a link in there:

[...]
  File "/home/rkdarst/git/testing/venv/lib/python3.9/site-packages/docutils/nodes.py", line 227, in walkabout
    visitor.dispatch_departure(self)
  File "/home/rkdarst/git/testing/venv/lib/python3.9/site-packages/sphinx/util/docutils.py", line 520, in dispatch_departure
    super().dispatch_departure(node)
  File "/home/rkdarst/git/testing/venv/lib/python3.9/site-packages/docutils/nodes.py", line 2008, in dispatch_departure
    return method(node)
  File "/home/rkdarst/git/testing/venv/lib/python3.9/site-packages/docutils/nodes.py", line 2030, in unknown_departure
    raise NotImplementedError(
NotImplementedError: <class 'sphinx.writers.html5.HTML5Translator'> departing unknown node type: pending_xref

No solution yet, not sure what the problem is. I guess it is somehow related to duplicating the admonition nodes and that's somehow messing up the HTML generation.

problem rendering math

hey!
I'm trying to fix this math-rendering problem for jupyter notebooks as source for lesson episodes. See the latex code under https://enccs.github.io/NordIQuEst-workshop/E1_qc-1/#quantum-circuits for an example of the problem.

When building locally, math renders correctly with an old messy conda environment that's been updated once in a while. This working environment has the following packages:

$ conda list | grep 'myst\|sphinx\|jupyter'

jupyter-cache             0.4.3                    pypi_0    pypi
jupyter-client            7.1.2                    pypi_0    pypi
jupyter-core              4.9.2                    pypi_0    pypi
jupyter-server            1.13.5                   pypi_0    pypi
jupyter-server-mathjax    0.2.5                    pypi_0    pypi
jupyter-sphinx            0.3.2                    pypi_0    pypi
jupyterlab-pygments       0.1.2                    pypi_0    pypi
jupyterlab-widgets        1.0.2                    pypi_0    pypi
myst-nb                   0.13.2                   pypi_0    pypi
myst-parser               0.15.2                   pypi_0    pypi
pydata-sphinx-theme       0.8.1                    pypi_0    pypi
sphinx                    4.4.0                    pypi_0    pypi
sphinx-autobuild          2021.3.14                pypi_0    pypi
sphinx-book-theme         0.3.2                    pypi_0    pypi
sphinx-coderefinery-branding 0.1.0                    pypi_0    pypi
sphinx-copybutton         0.5.0                    pypi_0    pypi
sphinx-lesson             0.8.10                   pypi_0    pypi
sphinx-minipres           0.2.1                    pypi_0    pypi
sphinx-rtd-theme          1.0.0                    pypi_0    pypi
sphinx-rtd-theme-ext-color-contrast 0.3.0                    pypi_0    pypi
sphinx-tabs               3.3.1                    pypi_0    pypi
sphinx-thebe              0.1.2                    pypi_0    pypi
sphinx-togglebutton       0.3.0                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-bibtex      2.4.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi

A new local environment with the dependencies i thought were sufficient does not work however. This non-working env has this:

$ conda list | grep 'myst\|sphinx\|jupyter'

jupyter-cache             0.5.0                    pypi_0    pypi
jupyter-client            7.3.1                    pypi_0    pypi
jupyter-core              4.10.0                   pypi_0    pypi
jupyter-server            1.17.0                   pypi_0    pypi
jupyter-server-mathjax    0.2.5                    pypi_0    pypi
jupyter-sphinx            0.3.2                    pypi_0    pypi
jupyterlab-pygments       0.2.2                    pypi_0    pypi
jupyterlab-widgets        1.1.0                    pypi_0    pypi
myst-nb                   0.15.0                   pypi_0    pypi
myst-parser               0.17.2                   pypi_0    pypi
sphinx                    4.5.0                    pypi_0    pypi
sphinx-autobuild          2021.3.14                pypi_0    pypi
sphinx-copybutton         0.5.0                    pypi_0    pypi
sphinx-lesson             0.8.12                   pypi_0    pypi
sphinx-minipres           0.2.1                    pypi_0    pypi
sphinx-rtd-theme          1.0.0                    pypi_0    pypi
sphinx-rtd-theme-ext-color-contrast 0.3.0                    pypi_0    pypi
sphinx-tabs               3.3.1                    pypi_0    pypi
sphinx-togglebutton       0.3.1                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi

Any ideas what's missing?

Copy button on code blocks broken in main page

It seems that the copy button is broken (i.e. not rendering correctly), but only for code blocks on the main page:
https://enccs.github.io/intermediate-mpi/#graphical-and-text-conventions

should `discussion` be included in the exercise list directive?

reasons for (+) and against (-)

• (-) They usually don't require preparation by exercise leaders
• (+) seeing them helps to understand the overall flow of the lesson
• (+) they often have a solution after, which would be included in the exercise list anyway. This would be confusing.
• (-) the page could start to get very long.

Automatic counters for challenges

This is my usage scenario:

1. I am delivering the workshop.
2. Learners have cloned the lesson repository to follow the exercises.
3. Exercises are organized by day in a content/code/day-N folder.
4. Each day-N folder has subfolders with one exercise each. The folder are prefixed with a number, for easier reference. The subfolder contains an incomplete scaffold and its solution. For example:

content/code/day-2/
└── 21_automata-cxx
    ├── external
    ├── solution
    └── src

Can there be automatic counters for the challenge boxes in the lesson? Such that it's easily discoverable into which folder to navigate when one is following along on the lesson webpage. I am not even sure this is possible in Sphinx, to be honest 😅

sphinx-book-theme consideration

https://coderefinery.github.io/sphinx-lesson/branch/sphinx-book-theme/index.html

This is a sample built using sphinx-book-theme. My plan is to leave the branch and keep rebasing forward so we have a demo of what it looks like (or maybe simply deleting the branch...)

Add .jupyter_cache/ to .gitignore

sphinx-lesson doesn't have structured page metadata

in the jekyll theme, each page has metadata at the top that gets inserted into the HTML template, as well as the index page. This was one of my annoyances (you can't change to arbitrary themes, it can only support html)

In sphinx-lesson, we have a admonition block at the top/bottom with this kind of data in it. It can be different on the pages, and there is nothing to force it to be the same in every episode. Since this metadata is in a admonition block with a certain class, someone could make an extension that extracts them out, but that would be extra work but doesn't exist yet.

Equally, the table of contents is organized completely manually.

Are these differences acceptable for sphinx-lesson? Is this metadata used for any purpose, or is it this way just because jekyll makes it natural?

exerciselist directive sometimes only shows the solution

seems to happen for all discussion directives, e.g. https://coderefinery.github.io/git-intro/exercises/#motivation

and also some exercises, e.g. the "Test your understanding" exercise on https://coderefinery.github.io/git-intro/basics/

new directives to complement 'challenge'

From @bast: I would find "type along" "try on your own" and "watch as demo" more useful than "challenge".

I agree. I would add exercise, demo and type-along - the main reason I haven't is that current directives are based on CR jekyll-common (and thus I guess carpentries?) - do we want to maintain semantic compatibility with that, or has that time sort of gone past?

(perhaps they had a reason for their choices, too, which we should consider)

release on PyPI

Right now development is very fast, so this is what is in requirements.txt:

https://github.com/coderefinery/sphinx-lesson/archive/master.zip

But we should probably release on PyPI.

• What is our version numbering strategy - do we have many small releases?
• Right now development is so fast it's useful to say "everyone work from master". Maybe, for our own lessons for the next few months, we can keep doing that anyway?

Build problems on Python 3.9.2

I am getting jupyter-packaging 0.9.1 requires setuptools>=49.4.0, but you'll have setuptools 49.2.1 which is incompatible with the following requirements.txt (which is used in 2 lessons):

sphinx-lesson
Sphinx
sphinx_rtd_theme
sphinx_rtd_theme_ext_color_contrast
myst_nb

I have used the following steps:

python -m venv venv
source venv/bin/activate
python -m pip install -r requirements.txt

Review documentation

I've tried to add some documentation, but it isn't that well refined yet: https://coderefinery.github.io/sphinx-lesson/

• how much rst/markdown basics do we duplicate?
• how much do we consider advanced use cases?
• Is there enough to get new people started quickly?
• is there enough that people can start new lessons? (lower priority)

Dropdown boxes shown by default

Is there a way to have a dropdown box whose content is shown by default?

From the code here: https://github.com/coderefinery/sphinx-lesson/blob/master/sphinx_lesson/directives.py#L70-L71 it seems I can either have the dropdown, with the contents initially hidden, or the contents shown, but no button to fold them.

Create a "prototype page"

Carpentries lessons are very standardized, with a fixed format. CR doesn't exactly have that, but we do have conventions. Perhaps we should think about what the typical conventions are, and how it differs from Cs.

Here is a prototype page I've made so far:
https://coderefinery.github.io/sphinx-lesson/sample-episode-rst/
We should all take a look and make sure it looks as we expect a typical page to look, and adopt any conventions early.

Examples:

• embed solution admonition block within the exercise block itself
• conventions for exercise labeling/titles

Unknown directive type "exercise".

The .. exercise:: directive does not work for me.

Steps to reproduce:

1. sphinx-quickstart
2. pip install -r requirements.txt
   requirements.txt
3. change conf.py

project = 'test-sphinx-lesson'
copyright = '2021, Kevin Sawade'
author = 'Kevin Sawade'


# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx_lesson',
        'sphinx_lesson.directives',
        'sphinx.ext.autodoc',
        'sphinx.ext.napoleon',
        'sphinx.ext.todo',
        'sphinx.ext.coverage',
        'sphinx.ext.githubpages',
        'sphinxcontrib.yt',
        'sphinx.ext.mathjax',
        'sphinx_copybutton',
        'sphinx.ext.intersphinx',
        'sphinx_rtd_theme_ext_color_contrast',
        'sphinx.ext.githubpages'
]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

4. run make html

Error:

Running Sphinx v3.5.4
making output directory... done
myst v0.13.7: MdParserConfig(renderer='sphinx', commonmark_only=False, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, update_mathjax=True, enable_extensions=['dollarmath'], disable_syntax=[], url_schemes=None, heading_anchors=None, html_meta=[], footnote_transition=True, substitutions=[], sub_delimiters=['{', '}'])
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index
D:\sphinx-lesson-test\index.rst:14: WARNING: Unknown directive type "exercise".

.. exercise::

   This is an exercise
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done

Tested systems:
Ubuntu 18.04, python3.8, conda
Windows 10, python3.8, conda

cssname method now missing

I believe this change (d7ff66e#diff-de1821ead97ff0f20a56e28dadb88a529ada12b5fb085a3b98e202274f64ddceR38) broke downstream code like at https://github.com/ENCCS/openmp-gpu/blob/7672ede569a08d0a4429c99e5cdb75b97ae41682/content/conf.py#L128

Can we re-introduce a cssname method that calls get_cssname so that we don't have to find and fix all such repos, please? Or should we be handling directives differently?

Are the symbols emojis? Or something else?

I'm trying to change the symbols in some admonitions and am using emojis. They are however not rendered, so I am wondering whether the content is something other than an emoji: https://github.com/coderefinery/sphinx-lesson/blob/master/sphinx_lesson/_static/sphinx_lesson.css#L19

Is it possible to make the notebooks executable?

I would like to be able to launch the notebooks on MyBinder from within the lesson à la Executable Book: https://jupyterbook.org/interactive/launchbuttons.html Is it possible? If not, could it be possible?

local development server

Do we have a convenient solution to preview/test our development locally?
I consider adding a development server wrapper in python. But I rarely develop in python so I might miss something obvious. What is the most practicable setup for you?

Can't link directly to exercises

In sphinx, admonition blocks don't generate HTML <hX> links and anchors which can be directly linked to. This has annoyed me, because I can't link straight to exercise blocks. This can be done in the jekyll theme because of how it works (interestingly, because the exercise block isn't structured into a unique type but uses a div + class).

My proposed solution would be to recommend each exercise is in a short section labeled "Exercises: topic", possibly with some intro text about what these exercises mean, and then one exercise admonition for each exercise, with a title describing it.

What do you think?

"git checkout s-l-t/master -- conf.py" is outdated

I can fix this but first want to learn a bit more about how it works.

Settle on what our ideal template repository is

How do we recommend people to get started?
a) We should have some example that is easy to copy and modify to start fresh.
b) when converting an existing lesson, there should be some minimal skeleton you can merge --allow-unrelated-histories to bring in all the files you need, but no more.

What we have so far:

• https://github.com/coderefinery/sphinx-lesson-template-empty
  This empty template is my latest "use this to start off". It has no content in it, which means it is easier to merge into an existing project and do some renames.

• https://github.com/coderefinery/sphinx-lesson-template
  This started as a fork of this very repository, then it split off as a sample lesson that could be used to start off. I think this doesn't really have a purpose anymore.

• Regarding the above two: should the template repo have sample files, or be a bare skeleton (and you copy from the primary sphinx-lesson repo, for example the sample page)? Or do we have one template of both types? Less is probably better.

• We also have python-for-scicomp and github-without-command-line that can serve as templates.