Migration of theme to modern responsive theme

[willingc]

Migration of theme to modern responsive theme

[pradyunsg]

I've made some more progress, and things are at a point that it's worth sharing.

The main thing is that I've settled down on the design/aesthetic I'm going for, as well as the typography and navigation design now. I'm pretty happy with how this is working now!

Screen.Recording.2021-11-21.at.20.39.54.mov

I also ended up writing a development workflow tool which is a PEP 517 backend as well. So... yea. That happened. 😅

The next things to do would be the rest of the scaffold (like the top bar, the footer, figuring out sidebar overflows etc), adding search + domain index + module index pages, and... stylizing everything that's in the metaphorical kitchen sink for Sphinx. And figuring out how to get Python's version selectors working with this theme. So... yea, baby steps.

[pradyunsg]

Okay, so... I've definitely been nerd-sniped by this. Here's an update/brain-dump on how that's going. 😅

There's a video at the end too!

I've started working on a Sphinx theme, for documentation sets with the size and structure of Python's documentation. The initial working name is Lutra, which is what I'll use in this comment, but it is definitely subject to change. Here's the documentation sites I'm currently using, to draw inspirations from (additional suggestions welcome):

• Furo (du'oh!)
• Ionic Framework documentation
• Just the Docs for Jekyll
• mkdocs-material for MkDocs
• Tailwind CSS documentation

My thought process for how things could work is: Lutra builds upon sphinx-basic-ng (more on this below) and becomes the base theme for python-docs-theme and pypa-docs-theme, with the python-docs-theme being only for Python's documentation. This model means we can get rid of documentation-related templates in CPython's source tree and specialise python-docs-theme to only be for Python, which may be super useful if we want to add build-time logic (in the Sphinx extension code for python-docs-theme), for generating the switchers (#4) or including the top bar from https://www.python.org/ (as I'd originally proposed in python/python-docs-theme#10).

(theme dependency tree -- status quo)

basic (built-in Sphinx theme)
╰─▶ default (built-in Sphinx theme)
    ╰─▶ python-docs-theme
        ╰─▶ pypa-docs-theme

(theme dependency tree -- what I'm currently thinking of)

basic (built-in Sphinx theme)
╰─▶ basic-ng
    ├─▶ [other themes]
    ╰─▶ lutra
        ├─▶ pypa-docs-theme
        ├─▶ python-docs-theme
        ╰─▶ [other themes]

As for sphinx-basic-ng -- that's something that will quite possibly be used by some other major themes in the Sphinx ecosystem (like sphinx-book-theme and pydata-sphinx-theme) -- I'm working closely with multiple theme authors/maintainers to figure out ways to reduce the repeated work, and have an actually thought out scaffold for this. See sphinx-basic-ng's repository/issue tracker if you're curious. I worked on it a bit last weekend and am pretty happy with where it is now, in terms of the actual implementation. It still needs a lot of documentation for theme authors, but slow-and-steady progress is the ongoing goal for all these things. And because video >> pictures >> words, here's a quick demo of sphinx-basic-ng, specifically the default behaviour of the fairly customisable scaffolding that comes with it:

Screen.Recording.2021-10-10.at.15.15.44.mov

[willingc]

I can take a look at the existing python-docs-theme along with Furo to determine if it makes more sense to add Python-specific stuff to Furo or vice versa. One of the nice things about Furo is it was designed with Sphinx 3.x in mind so it will bake in the advantage of updates from that, at least from a theme perspective. I've been using Furo in other projects and have been very happy with it in those.

[JulienPalard]

I just don't know how to test

I just made this very simple: the python-docs-theme now provide a cpython built documentation as a check artifact. It even has the version and language switchers, exactly like it's built on docs.python.org (with all the dirty tricks I'm not proud of, like editing HTML using sed).

Next step would be to temporarily (has to be defined...) host artifacts somewhere (has to be defined...) and make a bot to give the link in the PR?

[hugovk]

Next step would be to temporarily (has to be defined...) host artifacts somewhere (has to be defined...) and make a bot to give the link in the PR?

How about including some test docs that uses the theme, and deploy to Read the Docs? We could then enable previews from PRs, and it would automatically build and deploy using the theme to a test RTD site, making it easy to test the PR on desktop and mobile.

For example, python/devguide#679 has a preview at https://cpython-devguide--679.org.readthedocs.build/

[birkenfeld]

Can I just say that I love it?

[pradyunsg]

It's a new year! And I have made even more progress!

• There's a dark mode toggle now (auto/dark/light), and everything is stylized to respect this!

  Screen.Recording.2022-01-07.at.02.27.46.mov

• I've stylized most of the content that Sphinx can output. Tables are the last remaining piece, and I still need to decide what to do with API documentation. I'm quite happy with how the anchors / :target styling for various things has worked out though!

  Screen.Recording.2022-01-07.at.02.26.24.mov

As you can probably tell, there's two big pieces that need to be stylized still -- the header and footer. Along with that, I also wanna get with a few bits of functionality done, before we can start testing things out with the docsbuild-scripts (eg: highlighting right sidebar entries on scroll, showing "back to top" when scrolling up, "edit this page" buttons, "was this page helpful", ability to add stuff to the top of the right sidebar etc).

I was briefly worried that this is currently written as a Sphinx 4+ theme, but it looks like the current main on CPython is Sphinx 4 already: https://github.com/python/cpython/blob/c9137d4b638c0699b904011cafe68895d28dd80b/Doc/requirements.txt#L6

[CAM-Gerlach]

Hi! Let me play project manager here.

That was a good attempt, but I think your acting could use some more work. You clearly specified what you were asking, didn't use even a single buzzword and demonstrated far too much understanding of what's actually going on 😆

[pradyunsg]

For posterity, there was some discussion about the documentation theme in the Python Documentation Community Discord (starting around https://discord.com/channels/935215565872693329/935215566334079058/975795392027447387).

---

I've gotten Lutra to a point where it's building the Python documentation without crashes and most elements are stylised. Here's a link to CPython's current main rendered with Lutra: https://pradyunsg-cpython-lutra-testing.readthedocs.io/en/latest/index.html.

I'd like to gather folks' inputs/opinions/feedback, to figure out what else needs to be changed/improved about it. To avoid hijacking this issue, I've filed pradyunsg/lutra#22 which is where I'd request folks to post their feedback. I've also added some additional context there, for folks who are interested!

Fair warning: The theme is still WIP, so don't be surprised if you spot things that are borked/non-functional -- please let me know if/when you see that!

[JulienPalard]

@pradyunsg, quoting you to continue in this relevant thread:

If there's genuine interest in the modernization of the Python documentation and python-docs-theme (i.e. fairly major changes to look more like from-this-decade), I am very very interested in helping with that. :)

I think we all agree that the theme should evolve. Even from an SEO point of view (mobile-first ranking in SERPs) it's important.

As you understood, my front-end skill is absent, that's why I refrained to comment on python/python-docs-theme#55 for example (which I followed carefully). I also avoid to merge PRs in the theme without approbations of skilled front-end developers: I can test the visual quality, but not the implementation quality.

But if you need any help understanding how the language and version switchers work, or if they need to be changed, I'll gladly help here. I recently added a github workflow on python-docs-theme to build with the version and language switchers (html downloadable as a build artifact).

[pradyunsg]

I just flagged this discussion to @humitos, as he's one of the folks in RTD. :)

[JulienPalard]

For info, here's how the current language and version switchers in docs.python.org are working:

• The theme places an empty div <div class="language_switcher_placeholder"></div> so we can play around later.
• The theme have the CSS style for those switchers.
• The docsbuild script is the only source of truth about known versions and known languages.
• The docsbuild script creates a JS file with this list of languages and versions, what's needed to create the DOM elements, and the event handlers to do the switches.
• And now for the ugly part: the docsbuild script injects the JS on every pages.

Here are some ideas on how to enhance this: #4

[JulienPalard]

There's a video at the end too!

Haha, he's joking. /me read /me read /me read. Oh he wasn't.

I'm not able to help with CSS/UI/UX but you have my full support if you need anything docsbuild-scripts side. Once mature enough we can even test things at scale on docs.python.org/dev/.

[birkenfeld]

Still love it! If I was still an active Sphinx maintainer, I'd suggest that this become the new default 😁

[JulienPalard]

The collapsing sidebar

I personally don't like it, but I heard of people using it.

any suggestions for how to easily survey a bunch of folks?

Deploy the theme on docs.python.org/dev/ (I can help) and see if someone miss something? I used this strategy a lot at $DAYJOB, we called it WDD for Whine Driven Development: Push to production and raise an ear to catch if someone whines, (works best in big open spaces).

[JulienPalard]

There's two PRs opened on python-docs-theme to modernize the current theme:

• responsive
• dark theme

But I do not have the CSS-fu to help here, I can only test them.

Beware if we want to switch theme, they'll have to support our language and version switchers, which is currently probably too convoluted :

• cpython's tools/templates/layout.html add placeholders in the rootrellink block.
• docsbuild switch (which knows the list of versions and languages) injects a switchers.js right before </body>

In the other hand, rootrellink is from sphinx, not the theme, so it could by chance just work.

[Mariatta]

I admit that I also don't have front end JS/CSS skills. I've been following all the discussions in python-docs-theme repo as well. I. just don't know how to test or how to review those changes.

[pradyunsg]

language and version switchers

Yup yup! IIUC, those are the main CPython-docs-specific thing, and there's decent complexity in the build scripts due to this. Let me know if there's something else to think of when working on this. :)

I'll take a closer look at the build scripts over this long weekend! FWIW, I'm on the PSF Slack, so if it's needed at some point, it would be possible to have a fast-than-GitHub-comments chat there.

Also, If there's no "strict" time pressure, I'm working on https://github.com/pradyunsg/sphinx-basic-ng/, which might actually be a better base to build upon than Furo. That + Furo's content styling basically describes the other Sphinx theme I'm working on right now. :)

[JulienPalard]

I have no idea how incompatible our language and version switchers are with RTD though.

[willingc]

I believe that RTD has a pretty comprehensive version switcher. We should reach out to Eric Holscher and team.

[humitos]

Hi all! I don't know exactly how the build process for docs.python.org works, but IIRC what I saw once was a set of custom scripts (to generate the version selector) together with a sphinx-build call.

Unfortunately, Read the Docs does not officially support executing arbitrary commands (*). However, if those extra commands are only used to generate the version selector, they probably won't be needed if the docs are hosted at RTD since the version selector is automatically managed/created by RTD with all the active versions for the project.

When building on RTD with our own Sphinx theme (sphinx-rtd-theme) the version selector is nicely integrated into the theme (bottom left). On the other hand, when building with other themes, the version selector is shown as a floating sticky element (bottom right).

Examples:

• Spanish translation of Python official's documentation
• Python's official documentation on Read the Docs + sphinx-hoverxref extension installed

As you can see there, the version selector from CPython docs is not displayed but the RTD floating sticky element at the bottom right is shown instead.

That said, I'd add that we are working on a new flyout and footer API to allow Sphinx Theme's authors to manipulate the data that RTD uses to generate the version selector as they want and provide a better integration with their themes.

(*) you can by adding them in the conf.py as a hacky workaround, tho.

---

If you only want fast previews, I'd suggest creating a project on RTD, enabling PR builder, and discuss the changes over the rendered version. I think the only downside that this has is that your version selector won't be shown but still, it will work for 90% of the theme/style discussions I suppose.

If you eventually want to migrate your docs to RTD --which I think it's not your goal for now-- we should check that all the requirements that you currently have are supported on Read the Docs (e.g. I already known that multiple PDF outputs are not supported).

In any case, whatever you decide regarding RTD, feel free to ping me and let me know if I can help somehow. I'm happy to help here.

[pradyunsg]

(oh, and I tweaked the colors to be purple out of the box, but it's like, ~10 lines of code in a downstream theme/user's conf.py file to change that -- I'd imagine that python-docs-theme should change that to the cyan-ish colors from the video in the comment before)

[ssbarnea]

@pradyunsg It looks awesome! Do we have a RTD site where we can test the theme? I like it and I would like to make that the prefered choice for most python based projects, including lots of Ansible tools my team is managing.

It would be great if we could get an audit/review from an UX designer, especially around the accessibility areas. Making changes after adoption would be much harder so is better to do the accessibility checklist before.

I do have some previous experience with i18n/a11y but not enough to do it myself. Still, I will mention few bits

• able to use most screen sizes, portrait/landscape, mobile, different text size in browser, it should be ok even zoom at 125%
• color blindness
• display visited links differently than normal links (:hover, :visited)
• some level of branding
• Does it work well with RTD service?
• Is auto/light/dark mode preference saved using localStorage instead of cookies? Nobody likes them, or the mandatory prompts their presence require.

I do remember a nightmare related to Docker documentation which for 3+ years it was impossible to display in a narrow/split screen (even 4K/5K) as the sidebar was switching to mobile view and taking the whole page width.

[pradyunsg]

Do we have a RTD site where we can test the theme?

No. I'll put this up once I'm ready to put things up for testing. :)

It would be great if we could get an audit/review from an UX designer, especially around the accessibility areas.

I have spent a not-insignificant amount of time working on the accessibility aspects of Furo, so I'm not "flying blind" -- none the less, if we can get someone who has done this professionally to review these aspects, that would be great! :)

Is auto/light/dark mode preference saved using localStorage instead of cookies?

See https://github.com/pradyunsg/furo/blob/d6b217f8bb021f137928aa30f1b41e829e87d73f/src/furo/assets/scripts/furo.js#L57, which is what this is based off.

[pradyunsg]

Hmm... I was thinking about this a bunch today, and a good question came to mind: What do people find good about the existing docs.python.org theme? Not the content, not the fact that it's translated, not the fact that you can search things, not the fact that you can check different versions easily -- just the actual Sphinx theme. What do people like about that?

One potential answer that came to mind was: The collapsing sidebar. So... I've gone ahead and implemented that. And, also, yes, this is very much a prompt to tell me more such things. 😅

I can't really think of anything else personally, and short of sending an email to something like python-ideas, does anyone have any suggestions for how to easily survey a bunch of folks?

[webknjaz]

@pradyunsg add a banner/admonition to the docs like the one I've made for pip in the past, for the UX research? Our community/docs folks at Ansible do this as well putting links to surveys there.

[JulienPalard]

add a banner

Oh yes, we do use a "deprecation notice" banner on docs.python.org too, see: https://docs.python.org/2.7/.

[pradyunsg]

add a banner/admonition to the docs like the one I've made for pip in the past, for the UX research?

I'd prefer to instead allow adding a upvote/downvote thing, into a survey platform or whatever -- similar to the bottom of https://squidfunk.github.io/mkdocs-material/getting-started/#with-git or https://www.gatsbyjs.com/docs/quick-start.

Oh yes, we do use a "deprecation notice" banner on docs.python.org too, see: docs.python.org/2.7.

Yup! We'll treat that as an "announcement", similar to how Furo does things! :)

[encukou]

Hi! Let me play project manager here.

What's the status of this issue? With python/python-docs-theme#46 merged, is this “done”? If not, when will it be considered done? Should individual issues be filed for the remaining work?

[pradyunsg]

What's the status of this issue?

Still needs more work. :P

With python/python-docs-theme#46 merged, is this “done”?

I don't think so. When Python documentation came up in a Talk Python to Me podcast episode, one of the things explicitly pointed out was that docs.python.org vs something like https://tailwindcss.com/docs/ are completely different experiences -- with the latter being a lot more pleasing to look at.

If not, when will it be considered done?

I don't think this should be considered done unless we make some progress on improving the overall site design.

Should individual issues be filed for the remaining work?

I don't think we have a clear set of steps for what we want to do here TBH. :)

[pradyunsg]

So... that theme that I've been posting progress updates about here -- I've put the still-very-WIP code for it in a GitHub repository: https://github.com/pradyunsg/lutra/. The theme's own documentation is built using itself, and is available at https://pradyunsg.me/lutra/

Everything about this is very WIP but... hey, it is available publicly under MIT now! :)

PS: If you have feedback on this, use the theme's issue tracker rather than posting comments here.

[AA-Turner]

Is not having an index page intentional? I think also I'd prefer if languages was spelled out as the word, I was confused by the red glyph in the navigation bar and thought it was a rendering error initially.

Screenshot

A

[pradyunsg]

Waiting on the RTD build which will add a description for why that page is empty: pradyunsg/cpython@f41c09b

The red glyph is mostly a placeholder reminder for me, that I have to handle language and version selectors outside of RTD builds.

---

If there's more to discuss, let's do that over at pradyunsg/lutra#22! :)