executablebooks/meta

A community dedicated to supporting tools for technical and scientific communication and interactive computing

meta's Issues

MyST vs myst

Three times in the last 2 days now I've had people pronounce MyST as My ess tee as opposed to myst. I asked about it and they said they thought this is naturally how it was pronounced because of the capitalization.

How do folks feel about this? Are we OK with a lot of people assuming it is My ess tee?

And if not, how do we feel about using all-lowercase in our spelling, logos, etc?

Javascript implementation of MyST markdown

I can't remember if we've created an issue to track this or not, so I figured we should start one here to keep track of relevant information etc.

Background - currently MyST markdown exists as a collection of extensions to a Python markdown parser that we've developed called markdown-it-py. This was ported from the popular and extensible Javascript MD parser called markdown-it.

The goal behind this was to have a Python implementation of markdown-it that was structured very similarly, in the hopes that it would be easy to port syntax extensions back and forth between JS and Python. This might also be a path towards getting MyST markdown support in JupyterLab, if it also adopts markdown-it for its parser.

ToDo: At a minimum, we could:

• Write a set of markdown-it extensions that mimic the extra syntax added by MyST-markdown. I believe those elements are in the myst-blocks and myst-role folders.
• Write the writers that will take the tokens generated from the initial parse and output HTML elements (and whatever else needs to be done to get the "output" side of the markdown-it extensions working).
• Write up some kind of javascripty wrapper that uses markdown-it to parse markdown and outputs rendered tokens. This could be the "MyST Markdown javascript library"
• Think about ways to incorporate that library into other javascript applications, such as:
  • A Jupyter Notebook extension that over-rides the markdown parser with our parser
  • A Jupyter lab extension that over-rides the markdown parser with our parser
  • Similar extensions for other tools (e.g. iooxa?)

Note that some of this has already been done in the MyST markdown vscode extension here: https://github.com/executablebooks/myst-language-support/blob/master/src/md_it_plugin.ts?rgh-link-date=2020-06-24T01%3A07%3A08Z

Assess `deathbeds/pidgy` project

deathbeds/pidgy as reported by @choldgraf.

From a brief scan of the documentation; it was as clear as mud what its actually trying to achieve lol.

Looking at one of these intro.md.ipynb files though, it seems as though you write all your Markdown in code cells!?

Plan for Chris' upcoming baby leave

Hey all - I think I've told you all, but just to make sure: I'm expecting a baby girl (our first) in early August! (if this is the first you've heard of this...then surprise!!!!)

What to expect

I plan on taking roughly 1-2 months of "Chris is totally unreliable". Maybe more, maybe less depending on how we are doing, but this is what we should plan on. I'll be very rarely checking emails etc so I can jump in if there's an emergency, but day-to-day duties I probably shouldn't be relied on for anything.

I think that the current status of the Executable Books stack is in good shape. Our lower-level repositories have started to solidify nicely and we've had fewer and fewer major changes, which is a good thing. Folks seem to really like MyST markdown, which is awesome. They're also intrigued by MyST Notebooks, though I detect some confusion as to how "MyST-NB" (the sphinx package) and "MyST Notebooks" (the notebook format) relate to each other. Jupyter Book has also seen a few big-ish API changes, but hopefully those will solidify over time as well.

Some things others could keep watch over for a bit

Here are a few things that I want to highlight that others may wish to spend time on / think about in my absence:

• We have an ongoing conversation with Eric from ReadTheDocs about getting Jupyter Book in readthedocs and/or MyST markdown adopted as an "official" recommendation from RTD.
• We still need to "announce" a new Jupyter Book (we've made a few releases, but no big splash yet). I think it'd be good to do this before JupyterCon so that we can use the conference to build more momentum.
• I'm going to submit a JupyterCon talk for October. I think this has a high chance of being accepted. I'm happy to put in the time to give the presentation (it is pre-recorded so timing is hopefully not as much of an issue) but would love assistance here or in the subsequent Q&A sessions).
• Jupyter Book and sphinx-book-theme could use more 👀 on it to help with issue triage, merges, etc. As more people are using it, more and more issues are being created and I'm not able to keep up right now, much less when I'm on leave. If you want a little dashboardy idea of what's happening across our repos, check out this executablebooks dashboard page
• We have a potential collaboration with Iooxa brewing that I want to make sure we don't drop, as this could result in some interesting new developments on the web/javascript/authoring side of things. It's something we could potentially use some of the Berkeley grant $$ on.

Not all of these need to be equally prioritized, but I wanted to put it all out here for transparency's sake and so we can get an idea of what to focus on.

Things I hope to get done before I go on leave

• Merge some of the outstanding PRs in jupyter book, sphinx-book-theme, etc
• Get another minor release of Jupyter Book out (7.2)
• Submit jupytercon talk
• Anything else???

Lemme know if you have any thoughts / comments / or questions. I'd also be happy to have a meeting to re-sync with folks (either as a group or via 1-on-1s) in the next week or two.

Numbered/labeled theorems, propositions and lemmas, as well as a proof environment.

This is one of the core deliverables from the proposal, to make our output on par with LaTeX. A typical PDF example is shown below. Further comments are given after the image.

Regarding how this is rendered --- which I guess depends on the theme, hence CCing @DrDrij --- for PDF this looks great but probably not for HTML. (For starters, italics rarely look nice with sans serif fonts.) To me, for HTML, this style looks better:

JupyterLab extension to browse jupyter books from the interface

It would be really cool if we had some lightweight JupyterLab interface elements that would let people step through notebooks the same way that they do a Jupyter Book. I can think of two things to add:

1. Add a "previous" and "next" button for each page using the _toc.yml file. Similar to these buttons:

   

2. Add a toc.yml browser to the left sidebar that behaves just like the file browser, but where the files are ordered according to toc.yml, not the naming of the notebooks.

Utilize the tufte style for latex as well?

I noticed that there is also a tufte latex package. The tufte style is a heavy influence for the layout of jupyter book. I wonder if we could utilize this package (maybe optionally) for some consistency in the outout look and feel https://github.com/Tufte-LaTeX/Tufte-LaTeX

ping @mmcky and @rossbar who are both interested in PDF output

Grab `executablebooks.org`?

I just noticed that ExecutableBooks.org is available for ~$13 a year. Any interest in using that for documentation instead of executablebookproject.github.io?

As an aside, I also grabbed https://github.com/executablebooks just in case we like that name more long-term. I do think it's more memorable than "ExecutableBookProject" but I don't feel strongly about it.

(the reason that I'm bringing this up now is that I realized if we want this org to be a long-term home for our various projects, even beyond the life of the grant, then we should think about memorability and such)

Draft of a Jupyter Book rewrite blog post

Hey all - I put together the first blog post introducing our new stack. Its goal is to focus on users, give them an idea of what is new in Jupyter Book, and what is different under the hood.

I imagine cross-posting it to the Jupyter Blog, as well as the EBP "updates" section. In the future, I think we could write a follow-up post that dives into the other projects in our stack in more detail.

Would love to hear thoughts, comments, suggestions, and edits (feel free to do so in the hackmd directly):

https://hackmd.io/TAgkZbtiRKmT-4sIGlndaA

Support JOSS with a letter for their Sloan grant

Hey all - I spoke recently with Arfon Smith who is one of the leaders of the Journal of Open Source Software (part of "The Open Journals"). We had a productive conversation about Jupyter Book and how it could integrate into some of their publishing pipelines.

JOSS is applying for a grant to Sloan to fund work aimed at generalizing their JOSS publishing infrastructure to be useful to other contexts. It could also potentially work on integrations with open source publishing tools, such as those built by the Executable Books Project.

I've written up a draft letter of support for their application to Sloan. Would others in this team be willing to add their name to the bottom? (comments etc are also welcome!)

https://docs.google.com/document/d/1C_729R5Ws-W8GtNgpYp_V0nP3avZ1laWjIxrWcPbxaE/edit?usp=sharing

If anybody is not OK with me giving this letter of support to Arfon to include in their application, please let me know as well. I believe we need to send this to Arfon by early next week.

cc @jstac and @gregcaporaso since it'll be particularly helpful to have the other PIs on the grant, but anybody is welcome to add their name if they wish.

Upload python3 wheels too

Is your feature request related to a problem? Please describe.

I want to be able to install myst-parser from wheels, without having to use the source distribution. However, pypi doesn't have python3 wheels, only python2 wheels.

Describe the solution you'd like

The release process includes uploading python3 wheels.

Describe alternatives you've considered

Be ok with using source install, since pip does that automatically anyway.

Additional context

I was playing with installign this into PyIodide, which seems to require wheels

Move my sphinx-copybutton repository here?

Hey all - how do folks feel about me moving my sphinx-copybutton repository to the EBP organization? I think this is a nice longer-term home for sphinx-copybutton rather than my personal github repo. I can standardize the testing infrastructure so that it is more similar to the other EBP projects.

https://github.com/choldgraf/sphinx-copybutton

Adopt a code of conduct

Code of Conducts are helpful ways to define the expectations about how people should interact with one another in our public spaces (e.g. github repositories). They are also important for making projects more inclusive and to standardize community norms. How do folks feel about:

1. Adopting a code of conduct for the whole EBP project, that applies to any repository under executablebooks/
2. At least as an interrim, say that all EBP repositories need to match the jupyter code of conduct

This is something that will be come increasingly important as our repositories start to be used and developed by other people. I don't think it's a crucial thing to have now, but will become moreso over time (if we are successful!)

Have a show-and-tell meeting?

I think it'd be fun and interesting to do a quick show-and-tell of our various build systems. Are folks interested in having a one-or-so-hour meeting where we each get 30 minutes to give a brief overview of our respective build systems? E.g., design philosophy, what technical choices are made, pros/cons, etc?

Review/document existing online texts that we find attractive or interesting

• https://quantum.country/qcvc

ruamel or pyyaml?

I seem to remember in a conversation with @chrisjsewell that he really liked ruamel.yaml. However I noticed that pyyaml is used quite a lot in our various repositories. I've used both before, and think ruamel is much more fully-featured, though a bit more complex to use. I'm curious if folks have thoughts on this. It would probably be best if we just used a single YAML package...

Add a "team" page to executablebooks.org

How do people feel about us adding a team page to executablebooks.org? I imagine that we can use the list of people here:

https://github.com/orgs/executablebooks/people

and programmatically generate a table with EBP members like here: https://jupyterhub-team-compass.readthedocs.io/en/latest/team.html#jupyterhub-team (maybe minus the emojis, unless people would like them)

are folks +1 on that? In general, I'd like to find more ways to surface and recognize the work that the team is doing across these projects, also I'd like a way for us to add new members of the team that weren't a part of the original grant

notebook css styling for nbconvert and ebp

Quick question(s) -- I'd appreciate suggestions/guidance on writing css styles that will work with both ebp and nbconvert for notebook cells/widgets/pages. Does ebp plan to port the notebook less files to scss for bootstrap4 compatibility (and am I correct that jupyterlab has moved to straight css with no preprocessor)?

Improve/Standardize branding

1. Themes should be consistent across EBP repos documentation. This should in part be achieved by #54, and also it would be good to have a logo for EBP and each repo as discussed in executablebooks/jupyter-cache#16 (comment)

2. Visitors to the repos/documentation should get a good first impression that theseare well-supported endeavours. In lieu of having the repos within jupyter org (as discussed in executablebooks/cli/issues/87) it would be good if the front pages contained some indication that they are "officially endorsed" by jupyter org. Also noting that they are backed by the sloan grant would be good. Including the logos might be a good start:
   
   

#56 is also a related issue

Make some pre-releases?

In our last meeting, @jstac suggested that we could release the myst-parser and myst-nb projects in a "public beta" fashion, relatively soon. I think it could be a good idea to get some of these tools out there sooner than later. What do people think about:

• In the next few days, start telling twitter, open-source colleagues, etc about the MyST-parser for Sphinx
• After @AakashGfude's PR lands in MyST-NB and we have a week-or-so to test it out, do the same thing for MyST-NB

What do folks think about that?

And on the second point, we should also reach out to the nbsphinx project and see if they have any ideas for how we could dovetail our stacks into one another.

The one question I have here, is whether @chrisjsewell thinks that recent conversations about markdown-it-py should delay the "announcement" of MyST and the MyST parser. Chris, what do you think?

Blog post for the Jupyter blog?

Hey all - how do we feel about making a blog post for the Jupyter blog? (https://blog.jupyter.org/)

I think that this project is in-scope enough that's it's reasonable to post to this blog. However, I think the biggest question is how we can use a post like this to drive attention to the project.

For example, if we wish, should we use it as an opportunity to announce early prototypes of tools? Or should we keep it abstract and at the project level (such as https://ebp.jupyterbook.org/en/latest/updates/02-25-2020.html).

And more generally, how do folks feel about us posting updates about our tools in general? I have held off on making any major announcements about it, beyond reaching out to our close collaborators. But at some point we want to drive more attention to these tools in the hopes that they gain adoption and development.

plans to add css for @media print?

We're using some simple templates with nbconvert 6 to add page breaks and page counters that work with headless chrome, and would like to make it as easy as possible to move these over to ebp. Are there plans for css3 @page styling? For example here's something that works with counters and page breaks. We also needed to borrow this mathjax2 config from R pagedown to get headless chrome to render correctly on multiple pages.

[short notice] Give a talk at JupyterCon

This is very very short notice - so I apologize for not catching it sooner - but the JupyterCon 2020 CFP for talks closes Monday night (Pacific time). I think that we could use this conference as an opportunity to give a talk about the ExecutableBooks project, the new Jupyter Book re-write, etc.

I am happy to submit a proposal on our behalf. I think that if we frame it as a "next evolution of Jupyter Book" kinda thing, then it'll have a very high probability of being accepted (I'd probably submit it under the "Jupyter Tools" track).

I've started off with a very rough title/outline at this doc link:

https://docs.google.com/document/d/1S-piiM9gjqq886BKEGKzKdXRJBuEMC6EGEFhnvAaPkQ/edit?usp=sharing

I'd love any comments, edits, suggestions etc, as well as a signal as to whether you think this is or isn't a good idea! If I don't hear any strong negative responses I'll plan to submit the application in ~36 hours (Monday evening California time)

Let me know what you all think!

Keep track of PRs to other projects

Just a thought - over time we may make more PRs to other projects (e.g. if we upstream part of jupyter-sphinx or Mistletoe). If we do so, perhaps we want to keep track of the URLs for these PRs so that we can list them among our project's accomplishments. I think it would send a good message about the nature of the project, and also make some of these contributions less-invisible.

What do folks think? And if so, any idea for a good way to do this?

Move anything still relevant over from proposal GH repo

Review issues in

https://github.com/jstac/publishing_tool_application

and shift over anything still relevant.

Discussion for beta release

Originally in #44, moving here so we can keep track of it:

TO-DO LIST MOVED TO: https://github.com/orgs/executablebooks/projects/1

Sphinx 3 is out

Sphinx just released their public 3.0 release:

https://pypi.org/project/Sphinx/3.0.0/

We should make sure that things work with Sphinx 3.0...I think many of the sub-repositories are pinned to sphinx<3.0

Create a `sphinx-thebelab` extension?

A few projects now have used Thebelab in Sphinx (e.g., jupyter-sphinx, sphinx-gallery, and now myst-nb). Each one has slightly different approaches to config etc, and in some cases the thebelab logic is a bit complex under the hood (e.g. jupyter-sphinx has its own thebelab nodes that replace regular cell nodes).

I wonder if it would be beneficial to have a single sphinx-thebelab extension that others could use. This extension could be used either

• as a user-facing tool. e.g. provide a .. thebelab-button:: directive along with letting users configure the class used to mark a thebelab-ready cell (similar to how sphinx-copybutton works).
• as a dev-facing tool. e.g., a theme developer could use thebelab buttons in their theme that are provided by this extension

I think for the most basic Thebelab functionality, this may be overkill, but as features are added we could benefit from a single thebelab package (e.g. if somebody wanted a spinning or a loading widget while waiting for the kernel).

What do folks think? Would it be useful? Unnecessary?

cc @akhmerov who may not be pinged by these issues

Evaluate potential collaboration and management tools

There has already been some discussion around possible collaboration tools.

• trello for issue tracking -> decision: use github repository for issues, wiki pages etc.
• zulip for conversations -> decision: won't use an additional tool. most already have slack if needed

It may be nice to evaluate some more project tools for collaborative work on technical specifications and documents:

• Confluence

free for small teams (not open source, but open-source license requests are available)

• Bookstack

Appears geared towards collaborative document writing, requires self hosting

Implement Git Commits Convention (w/ Emoji)

Just a bit of fun, but I quite like the idea of consolidating our commit "protocol" to use an emoji based system:
(a) its just more visually appealing, but also (b) it would standardise commits in a way that could allow for automated changelog creation

https://github.com/carloscuesta/gitmoji has lots of stars, but actually I quite like https://github.com/ahmadawais/Emoji-Log for its simplicity. Although basically we just agree on some commit categories + emojis and stick to them across the organisation.

Thoughts?

Create a multi-book theme?

Many people have asked how they can share a collection of books with Jupyter Book (e.g. books that live across multiple repositories).

One easy-ish way to do this would be to use IFrames. So I wonder if we could build a very lightweight Sphinx theme that would be designed to have book links along the top, and a wide content section that would simply embed a jupyter book as an IFrame inside.

Discuss challenges with using Sphinx

From a recent conversation with @stefanv, @rossbar, and others, I wanted to bring up a few criticisms that I have commonly heard about Sphinx, so that we can discuss how we can deal with them, and how much of a problem they'd be. I'd also be curious to hear what @chrisjsewell and @mmcky think about this.

Sphinx is brittle and hard to debug

Something I have commonly heard is that Sphinx is too brittle, and often becomes a burden on the developer who has a hard time figuring out why something isn't working. E.g., error messages are hard to connect with changes that need to be made in the code, and diving into the Sphinx internals can quickly become a day's effort because of the complexity of Sphinx and because it is poorly-documented.

Sphinx can be slow

I've also heard complaints that building sites with Sphinx can become extremely slow. For example, I believe that building the Matplotlib documentation took on the order of 8-10 minutes. Some of this is probably because of running code, examples, etc, but I wonder if we can run a profiler on Sphinx to see where the bottlenecks are. For example, if we added a step that converted documents into rST before going into Sphinx, would that significantly increase the build overhead? If we have a document with lots of cross-references, what does the function of added time look like (e.g. $O^N$ or $O(N)$)

Docutils is difficult to debug and to contribute to

If the problem lies with Docutils, and not Sphinx, then a new set of challenges arise. Docutils is hosted on Sourceforge, and uses a mailing list for all conversations as well as code patches etc. This can make contributing upstream improvements tedious and time-consuming.

Those are the main ones that I can think of. I think the main question is whether these are insurmountable problems, or if they are problems that we can improve through upstream patches, or through engineering the backend in a way that avoids common pitfalls that users run into with Sphinx. It'd be great to hear what folks think.

Early adopters / projects to support / target

It won't be very hard to convert some existing texts built using bookdown over to jupyter-book, after switching their Rmd to MyST.

For example, @najuzilu is currently studying from https://introtcs.org/public/, which is an introductory CS course at Harvard. I don't think it would be too time consuming to switch it over. The first obvious benefit for the students would be downloadable jupyter notebooks corresponding to each lecture.

Roadmaps and goals for next phase

Hey all - I think that everybody has done a fantastic job of getting all of our tools out the door (and in a surprisingly short amount of time!). MyST is generally getting a great reception from the developer community (https://twitter.com/paulweveritt/status/1264181873619730432) and people are really positive about the pre-release for Jupyter Book! 🎉

As more people use the tools, and also potentially become interested in contributing, it'll be helpful for both us and them if we have an idea of where we wanna take things next. What are the directions in which we'd like the project to improve, evolve, etc? What are the major new features, or new bugs, that we'd like to work on?

So this issue is just a place to collect what you all think would be the biggest improvements over what we have right now. Eventually, we can convert this into one-or-more "roadmaps" or "wishlists" that we can use to signal the direction of the project.

I'll try to think about this myself over the next few days, and will add them below! It would be great to see your thoughts as well!

Jupyter-book demo project

We aim to complete and deploy a relatively fully featured demo project exploiting EBP tools and built using the CLI.

This will supplement existing repos https://github.com/ExecutableBookProject/myst-nb.example-project and https://github.com/ExecutableBookProject/myst-parser.example-project.

The CLI documentation will show in simple steps how to

• Download the project (perhaps w/o assuming familiarity with git)
• Build the project locally
• View the html build locally (and perhaps the PDF, with comments that high quality PDF is coming)
• Edit the source files and rebuild (edit-compile-view loop)
• Give suggestions on how to publish

The project should contain

• Code, both inputs and outputs
• inline and displayed maths
• citations
• cross-refs within and between pages, including numbered equations
• figures with captions and cross-refs
• examples of glue functionality in MyST-NB

This relates to #40, where quickstart is mentioned. In my view, it will remove the need to add quickstart functionality beyond create.

This is one of the key components needing action from the discussion in #44.

We agree that #39 (blog) can proceed once this issue is addressed.

Some markup features used by book authors

I wanted to list out a couple of the markup features that I really appreciate from using Jupinx/Weave/etc. and perhaps tie them to Rmd/bookdown. Most of these are things I currently use, although some of them are things I wish I had.

I think the goal shouldn't just be about getting the functionality working, but rather making sure that the syntax is clean and easy to read/write for the end-users. In this case, the end-user I am thinking about is someone writing a serious book with PDF/HTML/Jupyter as output formats.

I will leave it up to you to see whether these map into a cell-based jupyter approach, but my intuition tells me that many of them do not. But they all match the semantics of the Rmd/bookdown language. If I ever say "jupyter" here, I am talking only about jupyter as an output type, not an editing front-end or intermediate format in a build pipeline.

1. Equation and reference numbering

Right now, jupinx allows equation numbering but it is a little ugly (on a syntactic level as well as the actual output when generating ipynb with numbering).

For the syntax, jupinx right now can add in a label to a full math environment

.. math::
    :label: la_se

    a x = b

Referencing :eq:`la_se`

But it is hard to write normal latex, especially if you want to have multiple numbered equations. Rmd/bookdown does this in as latex-centric way. Writing almost correct latex (except for having to escape the #, which is reasonable for python/julia/R)

\begin{align} 
c y &=d \notag \\
a x &= b (\#eq:la_se)
\end{align} 

A link to the numbered equation in (\#eq:la_se)

Being able to write almost proper latex in the documents would be very liberating!

One other note on the equation numbering in Jupyter: I think along the time-horizon of this grant, it should be considered whether requiring MathJax automatic numbering is accceptable (through either extension or updates to Jupyter front-ends themselves). If so, then the generated HTML that jupinx does for jupyter notebook output (which ends up being a layout mess, but where there seemed no other approach) could be replaced.

2. Multi-language display/typesetting (but not execution!)

There are times when you want multiple languages displayed in the same document - although only one of them would be executable in a jupyter output. But that means you still want nice and language specific syntax highlighting/etc. in the PDF and html output. Ideally you would also have beautiful syntax highlighting in jupyter outputs for languages outside of the main kernel, but we could live without it.

Classic places where you want pretty syntax highlighting in HTML/PDF outside different from your core language are for yaml, toml, and bash blocks. I don't believe this is currently in Jupinx. For example, in https://github.com/QuantEcon/lecture-source-jl/edit/master/source/rst/getting_started_julia/getting_started.rst we use

.. code-block:: none

    git clone https://github.com/quantecon/quantecon-notebooks-julia

where I would prefer a code-block:: bash but don't think it is implemented.

Rmd definetely has this. For example, look at https://github.com/rstudio/bookdown/blob/master/inst/examples/04-customization.Rmd which includes latex and yaml blocks and I suspect could handle a {bash, eval=false} as well.

3. Code blocks without any execution

In Jupinx this is

.. code-block:: julia
    :class: no-execute

In Rmd the chunk is

```{r, eval=false}

There are lots of places we would want to use this in designing online courses.

4. Generating output without code blocks

It is nice to be able to see output (usually figures) but where the code may not be displayed.

Currently, I don't think it is possible in jupinx but in Rmd it is done through

```{r, echo=FALSE}

For figures, this would have two parts. The first is that the images need to be generated and added, and the second is that the assets need to be managed for the Jupyter deployment process (i.e. jupyter notebooks linking to an embedded online image from the generated source). See the comments below on assets

5. Code which is executable and executes, but where the output is hidden

There are many cases where the output is too ugly to be displayed in PDF/latex/distributed notebooks, but where you want it to run. For example, we don't want the package manager outputs to run.

.. code-block:: julia
:class: hide-output

] add InstantiateFromURL

In Rmd I think this is done with

```{julia, results="hide"}
] add InstantiateFromURL
```

6. Literal includes

There are a lot of times when you want to have a literal inclusion of text markup into the document. For example, we need to have a header in each of our notebooks with a version number that we can bump easily. To do that, we have a file like https://github.com/QuantEcon/lecture-source-jl/blob/master/source/_static/includes/deps_generic.jl

and then include it with something like

.. literalinclude:: /_static/includes/deps_generic.jl
     :class: hide-output

My suspicion is that Rmd has better ways to deal with this. I think it is child-documents? Something like

```{julia, child = 'deps_generic.Rmd'}

If this was easier, I would use probably use it more often.

7. Unit/regression testing/coverage/etc.

Unit and regression testing is a little tricky with writing online books. You don't necessarily want to have a complete regression test on the layout, but you frequently want to have the code itself tested. That way, you can rearrange the layout but if someone submits a PR that breaks a calculation, you know about it.

In jupinx, we do this by having a special test class for code blocks. This is only displayed in the output on a test build. See https://github.com/QuantEcon/lecture-source-jl/edit/master/source/rst/dynamic_programming/mccall_model.rst for example

First, you might have setup code which conditionally runs

.. code-block:: julia
    :class: test

    using Test

and then embedded in the content you can have the actual code which runs during tests

.. code-block:: julia
    :class: test

    @testset "Reservation Wage Tests" begin
        @test compute_reservation_wage(mcm()) ≈ 47.316499766546215
        @test compute_reservation_wage_direct(mcm()) ≈ 47.31649975736077
    end

I don't think that having the test conditionally run is really needed... it was just the easiest way to implement the feature for jupinx.

With Rmd, I believe you would do this with a block which executes but shows neither the code nor the output. It would always run (which is fine by me) but never display in the output.

```{julia, echo=false, results = "hide"}
@testset "Reservation Wage Tests" begin
    @test compute_reservation_wage(mcm()) ≈ 47.316499766546215
    @test compute_reservation_wage_direct(mcm()) ≈ 47.31649975736077
end
```

The trick there is that Rmd chunks show errors by default, so if there was an assertion failure you would get that output. Then it is easy enough to write a CI tool to check for regressions by looking to see if an error occurs.

This is imperfect (e.g. tough to have regerssion tests for figures) but gets the job done.

I should point out that an alternative approach used in Julia's markdown is to have output that needs to be tested within the markdown itself in jldoctest.

For example, see https://juliadocs.github.io/Documenter.jl/v0.7/man/doctests.html#

To implement that feature in something like Rmd you could have a new chunk type which test which looks for an output and checks it. Then you wouldn't have special hidden code chunks but rather it could check existing chunks. For example, I could imagine something like the following:

```{julia}
x = 2 + 3  # some code you are executing
```
Some code to check tests against below:
```{julia, test=true}
x

# output

5
```

The behavior of the test=true chunk would be as follows: in normal builds, it would just drop anything in code chunks below # output completely. In a build tagged as test it would execute the code above the # output and compare it exactly to the output from the execution. It would throw an error if it failed.

An important feature for this feature, which the julia documenter implemented, is to automatically fill in the # output blocks from any jldoctest chunks. Basically, you can just create a bunch of code chunks as jldoctest, run some sort of update_tests utility on a file, and it adds or replaces the # output and output to match the current execution. This sort of functionality would make me incredibly happy and make testing much easier.

8. Generating a REPL session

There are times when you want to generate a display block which shows a REPL session rather than having everything in a single code block.

I believe that Rmd might do this with

```{julia, prompt-true}
x = 2 + 5
y = 6
```

becoming something like

> x = 2 + 5
7

> y = 6
6

9. Line-by-line generation of cells rather than splitting each up.

This is a variation on the previous one. I don't believe that Rmd has a distinction between code that should be executable in a notebook and ones that should just be displayed. So the following feature would require a new chunk option.

Lets say you had an option called cell=split or cell=single which would decide whether to run code line-by-line creating cells as it goes or doing the whole thing in the same cell. The default would be single. But if you did the split, then it would act like you had taken the code and executed a whole bunch of different cells for each.

e.g.

```{julia, cell=split}
x = 2 + 5
y = 6
x + y
```

Would be equivalent to

```{julia}
x = 2 + 5
```

```{julia}
y = 6
```

```{julia}
x + y
```

I would use this feature a lot as there are frequently times where you want to write the code together but would really want people using jupyter output to execute line by line.

10. Raw blocks conditional on a particular output type

Hopefully this is clear. But it would mean you can keep output-specific stuff inside of the files rather than messy post-processing.

11. All sorts of figure layout options

The issue here comes down to differences in formating of images/etc. for html/pdf/etc. There are a few options in jupinx, but Rmd figures have much more control of sizing in the output. This becomes especially important if you have the features described above of having blocks of code which run but where the code is not displayed in the output.

12. Management of image/figure assets in online buckets

For HTML this isn't really an issue since the assets are generated with paths relative to the generated files. No problems. But for Jupyter it is an issue since you need the assets to link somewhere online.

Right now, this is a little bit of a mess in RST since sphinx wasn't designed directly for Jupyter. Everything is related to a static folder and the generated links are fudged after the fact given conf.py (e.g. https://github.com/QuantEcon/lecture-source-jl/blob/master/conf.py). The rst block itself would still look locally, pre-fudging.

.. figure:: /_static/figures/julia_term_1.png
   :width: 100%

I don't know how this is done in Rmd. It may not be.

13. Index

I don't believe this is in jupinx. In bookdown it is beautiful and just comes from latex. i.e. put in Some text that includes \index{Markov Chains} would end up in the index .

It only generates the index in pdf, though (see https://bookdown.org/yihui/bookdown/latex-index.html)

14. Footnotes

From the sphinx docs, this is what it looks like in jupinx

Some text that requires a footnote [#f1]_ .

.. rubric:: Footnotes

.. [#f1] Text of the first footnote.

But the bookdown one is a lot easier. I think it is just

Some text that requires a footnote ^[This is a footnote.]

There is a way to reference footnotes as well, but that is rarely needed.

15. Conditional output for different types of jupyter notebooks

Sadly, colab, jupyterlab, and jupyter notebook might have different outputs to be "perfect". For colab, for example, we wante to generate notebooks with the colab package additions already setup, whereas we might not want to for a nbgitpuller/binderhub setup.

As an example, with our quantecon datascience lectures we want to have the following code generated for jupyter+colab output (but not executed, as should be clear from the eval=false

```{python, eval=false}
! pip install qeds fiona geopandas xgboost gensim folium pyLDAvis descartes
```

For non-colab jupyterhub/binderhub we do not want that line of code or even unexecuted cell visible.

16. Theorem and latex environments

Sadly, does not exist in Jupinx... and I wish it did!

See https://bookdown.org/yihui/bookdown/markdown-extensions-by-bookdown.html#theorems

```{theorem, pyth, name="Pythagorean theorem"}
For a right triangle, if $c$ denotes the length of the hypotenuse
and $a$ and $b$ denote the lengths of the other two sides, we have

$$a^2 + b^2 = c^2$$
```

See \@ref(prefix:pyth)

Again, it is very close to writing latex and in fact generates latex styling.

17. Chunk caching

This seems to be Rmd specific feature, and a very useful one - especially for things like Julia where code may take a long time to execute.

18. Custom blocks for layout

Jupinx doesn't allow us to define our own blocks for a book, per se, but the flexibility of the directives makes it possible to define new ones with the existing syntax.

Bookdown has a very clean way to extend more generally with custom blocks. See https://bookdown.org/yihui/bookdown/custom-blocks.html

Core design constraints and guiding principles

Hey all - I've found that it is helpful to determine two things to guide the design/development of projects:

• Core constraints . These are things that must be satisfied by the final output, and can help in narrowing the space of possibilities to focus on the most-impactful product.
• Guiding principles. These are general approaches to take when trying to decide between multiple options. They aren't hard rules but heuristics to follow.

(as a general rule, it's OK to have more principles, and we should try to minimize the number of core restraints)

Would folks be interested in adding their own thoughts / brainstorm to this list? I'll add a comment with some quick ones that come to mind from my perspective - these don't need to be strict rules if we wish, but at the least it's a way to see where we're coming from.

Slideshow Presentations

How/when/what/why?

One for the future, but since I'm just writing a presentation now I'll kick this off.

For sphinx, I haven't really come across any good implementations. In terms of Jupyter Notebooks, RISE I think is the most mature implementation, but is obviously very notebook centric.

Stepping back a bit; is there a good/simple Markdown orientated slide show generator? From a quick google, this provides some possible options: https://opensource.com/article/18/5/markdown-slide-generators.

With presentations, the "difficulty" is that you usually want to be able to have more specific control over the layout, which is not as easy with just text-based files. So it would be ideal if it had some kind of two-way relationship with a GUI. Also, obviously the behemoth in the room is Powerpoint, and ideally there would be at least a one-way export to this format.

Investigate ipypublish

@jstac recently mentioned that there's another tool out there in the Jupyter / Python ecosystem that's quite relevant. It's called ipypublish and seems like it might be useful as a part of the book publishing process. This is an issue to investigate it and discuss our takeaways.

It seems like ipypublish has a bunch of different build systems depending on the outputs that's requested. It uses Jupyter Notebooks as an input, and converts them to outputs using some combination of nbconvert, pandoc, and sphinx. They've also written a number of Pandoc filters that parse markdown content and that could be used independently of the ipypublish build process.

I'll try installing and running a few outputs etc on my machine - would love to hear thoughts from anybody else who's used it or looked through it as well.

Evaluate the full build process to find steps that can use modular tools

A few times we've spoken about how we'd like a modular collection of tools that can work together alone or with more lightweight coordinating functions (like a CLI etc).

This is an issue to discuss where are the major components of that process, and where we'd like to make immediate progress:

https://docs.google.com/presentation/d/1uooAicF2MPl32p2z-_z-behfXHi77tbhvgfKafJiKlY/edit#slide=id.p

That's just a starting point for conversation, happy to iterate there or here!

Move our documentation to the sphinx-book-theme

How do folks feel about moving our documentation to the sphinx-book-theme? That way we can dogfood our own tools and become more familiar with them, and hopefully also uncover bugs and think of new features more easily that way.

It will need to wait until pydata cuts a release, so that we can then depend on that release and cut our own pypi release of the book theme.

Create an EBP cookie-cutter repository

We should consider creating a cookie-cutter template package for a minimal book building project setup, using the ExecutableBookProject infrastructure. It might replicate some of the functionality of sphinx-quickstart a bit, but I think it's a better approach.

See here, for a good example of what one of these repositories looks like: https://github.com/audreyr/cookiecutter-pypackage

QuantEcon Python programming example

I've requested that @najuzilu help us by converting this source material

https://github.com/QuantEcon/lecture-python-programming/tree/master/source/rst

into myst-based source and then generating html output via the new jupyter-book CLI.

The idea is to implement a non-trivial example with a substantial amount of source material. Currently it includes embedded code + outputs, cross-refs across pages, footnotes and maths.

@najuzilu, if I remember correctly, the objective is to have this done by 30th this month. Please coordinate

• with @AakashGfude and @choldgraf regarding addition of execution layer and embedded outputs
• with @choldgraf about the documentation for the CLI and how it could be improved
• with me (@jstac) regarding content -- I want us to add in a few citations as well
• with @mtiley as required --- I think he's willing to lend a hand.

Please supply feedback as you go along and update us in this issue.

If you have time, please also investigate PDF output direct from the html. @phaustin mentions some options in #44 (comment)

Take over stewardship of `thebelab`?

Hey all - I am curious if folks here are interested in taking over stewardship of the thebelab project (soon to be re-named Thebe). It uses JupyterHub/Binder + JupyterLab APIs and a lightweight Javascript library to convert static websites into dynamic ones where you can run code. We use it in Jupyter Book, and we also have a little Sphinx extension for it here: https://sphinx-thebe.readthedocs.io/en/latest/

Thebelab is currently under @minrk's repository, but we've discussed that it would be better for maintainabiliy / visibility / growth / etc to move it into a community repository somewhere. We've discussed this a bit here: jupyter-book/thebe#214

The two most-reasonable places seemed to be jupyterhub/ or executablebooks/. I could see an argument for either option, but I'm curious what folks here think about the idea? I think conceptually it could make more sense in executablebooks/ (since it is primarily about adding interactivity to static pages for improving the reading experience). It should also be a relatively well-scoped project that isn't a ton of maintenance burden, as long as we have a basic understanding of javascript workflows.

We're also discussing bringing it into jupyterhub here: jupyterhub/team-compass#313

What do folks think? Of interest?

Project website

We've discussed having a website as a "meta" documentation for the project, so it seemed like the "meta" repository might be a good place for it :-)

I just added a barebones documentation site to this repository, it's hosted here:

https://executablebookproject.readthedocs.io/en/latest/

A few questions:

• +1 to hosting it in this repo?
• Any structural changes we should make to the site in general?
• When content is fleshed out, how about pointing this to ebp.jupyterbook.org?

Use Sphinx error classes?

In sphinx-gallery/sphinx-gallery#690 I learned that Sphinx has some of its own error classes https://www.sphinx-doc.org/en/master/extdev/appapi.html#exceptions

I'm curious if anybody has used these before (@chrisjsewell ? @mmcky ?) and if they think it'd be a good idea for us to use them in these projects. Perhaps it could be a way to expose less gobbledygook in the error messages that users get?

Have a sync meeting?

Hey all - in particular, @jstac , @mmcky , @chrisjsewell

It's been several weeks since the last time that we spoke together. There has been a ton of progress on our various tools (though still much more to do!)

Can we have a meeting to re-sync a little bit, set some expectations for what we'll have time to work on in the coming weeks, and make a plan for early releases? In particular, I'd like to plan a date by which we release early versions of these tools as "public-ready".

One reason I think this is timely is because a lot of courses are going to move online in the next month, over the summer, and into next semester. The earlier we can solidify the EBP stack, the higher chance that we will be providing a really crucial tool to help instructors move their work remote.

I'm generally available for most of this and next week...can we find a time that works? I am also happy to have two meetings if the timezones don't work out...

Things to discuss before Chris leaves

I'm leaving on Wednesday morning (flight @ 11am). We have a few more opportunities for discussions and brainstorming before I go.

What shall we prioritize? I think we are relatively clear on the high-level build picture. So that means we can dive in on one of the steps in-between if it'd be particularly helpful.

Some ideas from me:

• Discussing what system we'll use for notebook cacheing and running (maybe related to executablebooks/MyST-Parser#47)
  • Is the "cache" layer an on-disk collection of notebooks, or some kind of database representation of notebooks
    • The plan is to use a "Jupyter Cache" tool that can do either (and is backend-agnostic).
  • If so, does that mean we need to build a bunch of sphinx components that utilize a database instead of notebooks
    • Ultimately it'll probably be a Sphinx Parser that knows how to pull from a jupyter cache and put it into the Sphinx AST. This will probably live next to the "ipynb sphinx parser"
  • And if that's the case, shall we still build a proper ipynb sphinx parser in case others want to read notebooks in to Sphinx directly
    • Yes, chris h will give this a shot.
• How will the HTML/PDF builders use the Sphinx AST in their outputs? Maybe @mmcky can describe this process since I'll need to understand it for hooking up Jupyter Book CSS/JS
  • Parse the ipynb outputs into a "MimeNode" at the parsing stage so it's in the AST. Then build a post-transform that will loop through the MIMENodes and convert them to Sphinx AST objects depending on the "builder" type (this is where the right mimebundle gets chosen)
• What kind of meeting cadence shall we have in the next few weeks while @chrisjsewell is still in Australia?
  • Meet once a week and follow the JupyterHub meeting team practices to plan an execute the meeting.
• MyST-based notebook representation syntax

Standardize docstrings

we should probably standardize our docstrings

I have been using numpydoc style, which looks like this:

Parameters
----------
x : type
    Description of parameter `x`.
y
    Description of parameter `y` (with type not specified)

but I noticed @chrisjsewell has been using a different style (not sure exactly which, but it looks like the following)

:param text: the file text
:param code_directive: the name of the directive to search for containing code cells
:param raw_directive: the name of the directive to search for containing raw cells
:param add_source_map: add a `source_map` key to the notebook metadata,
 is a list of the starting source line number for each cell.
:raises MystMetadataParsingError if the metadata block is not valid JSON/YAML

We should just pick one and use that...

Create some user personas

I've found that writing out user personas is a useful way to think about who you're designing technology for. I've created a wiki page for us to brainstorm some of these personas. I'll try to add perspectives from some of the people that I work with here @ Berkeley, and will add to it over time.

https://github.com/ExecutableBookProject/meta/wiki/User-personas

I'd love for folks to add your own user personas, edit the ones that are there, or to discuss some of the ones that we put up there.