compwa/compwa.github.io

Source code for the ComPWA Organization website

compwa.github.io's Issues

Finalize develop page

As the ComPWA repositories are developed for an academic environment, they are expected to evolve continuously. We therefore need to focus more on the developer than the user (one wonders whether there should be a distinction).

In an interactive set-up like that that, a more extensive contribute page would be a valuable resource, not only to standardise contribute conventions (it's current purpose), but also to have an inventory of what developer tools are out there and how to use them (didactic purpose).

Some examples:

• Working with Git and GitHub
• Writing/running unit tests
• Python developer tools (e.g. linting and formatters)
• Sphinx and reStructuredText syntax
• Continuous integration and local automatisation (e.g. pre-commit)

Of course, we don't need to write tutorials that are already out there: bundling the essentials and referring to helpful resources is already an improvement.

See also:

• ComPWA/tensorwaves#48 (comment)
• ComPWA/pycompwa#36

To-do list:

• How to build documentation (reST and sphinx)
• Creating a pull request
• Note about linear commit history and squashing
• pylint, flake8, black, mypy, and how our coding conventions are automatically enforced through their configs (ComPWA/tensorwaves#48 (comment))
• Explain how to install nbextension --> not necessary since Jupyter Lab v3.x
• Note about how linear commits serve as version patches
• How to checkout a PR or a branch from another remote

Provide specific developer instructions for this repository

This repository has some peculiarities that not explained by the general develop page. Some things that need explaining (potentially on the general page as well):

• What does environment.yml do, and where does it fetch (optional) pip dependencies and constraints
• What is the first notebook cell in each notebook
• Why are the pip install package==0.x.x statements needed
• How best to work with cell metadata
• How to 'freeze' cell output once a report is finished
• How to store embed images in the notebooks without committing them to the repository (see this tip).
• How to create GIF animations (see #129 (comment))
• ...

Fix pytest-notebook job

The ci-tests.yml and notebooks.yml Actions are currently not stable:
https://github.com/ComPWA/compwa-org/actions/runs/1881099598
The reason is that each notebook installs its own pinned dependencies, which somehow results in a ContextualVersionConflict. Ideally, these jobs should run over each notebook, install the dependencies, and run them without problems.

Improve and freeze P-vector reports

Some left-overs from the todo-list of #278.

To do

Beta Give feedback

1. Explain code, particularly DynamicsSymbolBuilder
2. Change color cycle?
3. Render fit process with CSVSummary
4. Widget for starting parameters
5. Parametrize Breit-Wigner with coupling g as well
6. Test coupled fit with resonance mass between threshold
7. Remove i from phase space factor
8. Freeze TR-030, TR-031, and TR-032
9. Improve ratios Argand plots
10. Fix title of TR-032 (coupled channel fit)
11. Generalize TR-032 so it can handle multiple poles

Allow running specific notebooks from notebook action

Allow running a selection of notebooks in the Run all notebooks job. See here for how to do it with workflow_dispatch arguments.

Write Technical Report on recursive kinematic variable expressions with SymPy

Currently, kinematic variables like helicity angles are computed in AmpForm with NumPy only. This is clumsy, because AmpForm should ideally only produce mathematical expressions. The only reason why this NumPy implementation is in AmpForm is that TensorWaves (which implements the back-ends) is kept as 'ignorant' about physics as possible.

The difficulty with expression kinematic variable computation with SymPy lies in the fact that we require Einstein summation, see e.g. implementation here:
https://github.com/ComPWA/ampform/blob/f38179483ac20664584014d9eb659866966681c2/src/ampform/data.py#L209-L213

This is difficult to be done with SymPy in such a way that the lambdified expression has the same form (at least when using NumPy as a back-end). Some implementation attempts were made in (local) branches for the AmpForm repo. These should be documented in a TR. They can also be extended with a similar solution presented in ComPWA/ampform#115 (which allows defining custom lambdification methods for certain nodes in the expression tree).

Additional notes

• EinsteinPy: docs | GitHub
• Einstein-Convention like implicit summation with sympy - Stack Overflow

System 12 3D drawing of θ₁

Theta_1 in system-12 should be the angle between vec p1, and -vec p3

from /report/999.html

Speed up documentation build on RTD by freezing TR-019

It takes more than ten minutes to run the Julia kernal notebook (TR-019). This is not really a problem locally or on GitHub Actions, because it can reuse the caches of jupyter-cache. RTD cannot reuse its cache, so it's better to freeze the notebook.

See also #207 (comment) about #201 though...

Add page about symbolic computations

Outline

Beta Give feedback

1. Brief intro about amplitude analysis software challenges
2. Explanation of new computational backends in Python like JAX
3. Introduction to CAS with SymPy
4. Benefits of a CAS (transparency, templating, flexibility)
5. Model serialization (see also ComPWA/RUB-EP1-AG-events#54
6. Generalization to libraries (ComPWA packages)

Should essentially be a non-academic summary of also https://github.com/ComPWA/RUB-EP1-AG/issues/61

Proofread

Beta Give feedback

1. @miriamfritsch
2. @shenvitor

Visualize parametric phase space boundary in TR-017

From #139 (comment)

TR-017 works out a parametrization in polar coordinates. It's easy to visualize this with Julia (#139 (comment)), but it would be nice if this parametrization is also visualized in the same (Python) TR.

Migrate organization websites

The pages for this repository are currently hosted on RTD. Its URL is a bit ugly though (compwa-org.rtfd.io), especially since the original ComPWA C++ project has been archived and ComPWA is to be considered an organization anyway.

Since the pages for compwa-org on RTD are not versioned, we could host them on GitHub Pages, particularly on the namespace for the ComPWA GitHub organization (compwa.github.io). That URL currently hosts pycompwa's documentation, but that could be moved to compwa.github.io/pycompwa. The only thing that's lost is the ability to have PR previews, but that could be kept by just letting compwa-org.rtfd.io redirect to compwa.github.io (see RTD redirects).

ComPWA ++ / pycompwa

Beta Give feedback

1. compwa.github.io → compwa.github.io/pycompwa
2. github.com/ComPWA/ComPWA → github.com/ComPWA/ComPWA-legacy
3. compwa.rtfd.io → compwa.github.io/legacy
4. compwa.github.io/ComPWA → compwa.github.io/legacy-doxygen

Organization page redirects

Beta Give feedback

1. github.com/ComPWA/compwa-org → github.com/ComPWA/compwa.github.io
2. Build PR previews on compwa.rtfd.io
3. DOC: migrate to compwa.github.io #234
   📝 Docs +1
   
4. Test redirects of entires pages on RTD including subpages. Optionally, use sphinx-reredirects (absolute redirects) or sphinxext-rediraffe (Git diffs)
5. compwa-org.rtfd.io → compwa.rtfd.io
6. compwa.rtfd.io → compwa.github.io, but keep PR preview on compwa.rtfd.io
7. Update links to pages, see "compwa-org.r" code search

Remove compwa.github.io pre-commit hooks from ComPWA/policy

Some hooks like pin-nb-requirements are only used by ComPWA/compwa.github.io.

Tasks

Beta Give feedback

1. BREAK: remove pin-nb-requirements hook policy#308
   ⚠️ Interface +1
   
2. DX: import pin-nb-requirements from ComPWA/policy #256
   🖱️ DX +1
   

Use `scipy.integrate.quad_vec()` in symbolic integral lambdification

TR-016 investigates how to implement the lambdification for a symbolic integral over vectorized, complex-valued input. It did not yet investigate the use of scipy.integrate.quad_vec(), which seems to work well with lambdas that take array intances from outside and can therefore be put inside generated lambdify code.

Also note the version of quadpy that is currently pinned in TR-016 is not available anymore (something with licences). It may therefore be worth entirely rewriting the report so that it only uses scipy.

Freeze notebooks

Currently, all notebooks are run every commit in order to generate output cells on the web pages. This has a few major disadvantages:

• ADRs and TRs are written for specific package versions: they are meant to illustrate a certain problem at a moment in time during development.
• CI will take longer the more notebooks are added.

Instead, we could/should:

• store cell output in the notebooks. Disadvantage is that the notebooks become larger and diffs are harder to read. But in principle, notebooks should not change once they have been committed to the master branch.
• only run CI on notebooks that are affected by a certain PR and check whether they run correctly.
• provide a means of freezing the dependencies for each notebook (probably just a Jupyter magic cell !pip install ...).

Investigate cut rotation by rotating meshgrid definitions themselves

Create installable `src` folder and move pre-commit definitions there

#256 created a .pre-commit folder for local pre-commit hooks. These hooks don't work when pre-commit is run from another working directory. To fix this, the project can be made installable with the source code for the local pre-commit hooks under the src directory.

See for example here.

`jaxlib`: could not find a version that satisfies

Bug description

the line

%pip install -q ampform==0.14.0 qrules==0.9.7 sympy==1.10.1 tensorwaves[jax,pwa]==0.4.5

causes the error

Note: you may need to restart the kernel to use updated packages.
ERROR: Could not find a version that satisfies the requirement jaxlib; extra == "jax" (from tensorwaves[jax,pwa]) (from versions: none)
ERROR: No matching distribution found for jaxlib; extra == "jax"

once I remove jax from the tensorwaves options, it works.

System info

I am on the e73542d commit of polarization-sensitivity branch running the notebook 017.ipynb

Bug resulted on the following system:

• OS: Windows 10
• Python version: 3.8.13
• Virtual environment: conda env create from the project folder

Dependency issue

What happened?

TR-000 run in Colab is not successful due to the following issues.

Relevant log output

Is it possible to reproduce the bug with a small code snippet?

See above

Additional steps to reproduce the bug

No

Which Python version were you using?

None

Python dependencies

No response

Write TR on amplitude analysis with zfit

Write the Amplitude analysis example in the form of a TR and do the fit and data generation with zfit (see ComPWA/tensorwaves#258). This might serve as feedback to zfit and possible evolve into a benchmark comparison.

Note that such a notebook may also evolve into ComPWA/.github#6.

Tip

See TR-020: Amplitude analysis with zfit.

Add commenting functionality to documentation pages

Install the sphinx-comments extension to allow commenting with Hypothesis.is and utterances.es.

Example

Speed up Binder builds

It currently takes a very long time to build a Binder image when launching a notebook. This load time might be reduced if installing only the doc requirements instead of all dev requirements.

Add short package descriptions for PyPI

ComPWA packages liste on PyPI (https://pypi.org/search/?q=compwa) do not have a description (only a long_description).

Publish ComPWA repositories under a DOI

Publish QRules, AmpForm, and TensorWaves on Zenodo. See:
https://guides.github.com/activities/citable-code

• Activate Zenodo on the GitHub organization
• Create new release
• Mint DOI (edit metadata on Zenodo)
• Add Zenodo badge
  ― QRules
  ― AmpForm
  ― TensorWaves

Set Ruff E402 as default for notebooks

Currently, the rule E402 (module-import-not-at-top-of-file) is deactivated for notebooks.

compwa.github.io/pyproject.toml

Line 256 in bde4df9

"E402",

We should activate this for notebooks and switch off that rule for the relevant notebooks through per-file-ignores.

cos θ should be flat, dN/dθ ~ sin θ

The distribution over the angle does not look as expected.

Can it be because of the boost to wrong direction? beta-> -beta

Pin requirements

QRules, AmpForm, TensorWaves, and PWA Pages have pinned requirements, which makes it easier to spot problems introduced by requirement upgrades. This would for instance be helpful in #49.

Pinned requirements also speed up Conda+pip, which improves the local DX and will speed up Binder as well. Compare for instance:
https://github.com/ComPWA/compwa-org/actions/runs/1195243261 (old: 3m14)
https://github.com/ComPWA/compwa-org/actions/runs/1254008065 (new: 2m35)

Embed intersphinx links in code snippets

Matplotlib's documentation has API-links in code blocks, see for instance here:

This would be an awesome upgrade to the docs, as it removes the need to provide links to the relevant classes and functions in the surrounding text.

Some solutions:

• https://sphinx-gallery.github.io/stable/configuration.html#link-to-documentation
• https://sphinx-gallery.github.io/stable/syntax.html#plain-rst
• https://sphinx-codeautolink.readthedocs.io

Improve developer instructions at compwa-org.rtfd.io

Collection of suggestion for the Help developing pages.

Tasks

Beta Give feedback

1. In zsh one has to write pip install ".[dev]" (with quotation marks)
2. Add link to download page for Miniconda

Formulate ComPWA organization ideals

Developer experience:

• Open source development for Partial Wave Analysis. Make it easy for anyone to contribute.
• Easy to setup for developers
• Code modularity and transparency. For example separation of the expertsystem and tensorwaves. First takes incorporates the physics and builds an amplitude model. Tensorwaves can use amplitude models and perform fits, but does not include any physics logic.
• Facilitate using industry techniques and tools in software development for PWA
• Accommodate both stable development and flexibility for ongoing analyses

User experience:

• Easy to install and use for analysis users

Note: ComPWA rather sacrifices functionality for design and developer experience related developments. The flexible branching model aims to make up for the functionality loss.

Describe GitHub Projects in issue managements explanation

See existing ZenHub description. Should be replaced with the new GitHub Issues.

Investigate effect of symbolic trigonometric simplification on numeric performance

• See discussion on PyHEP 2022 Slack.
• See also ComPWA/ampform-dpd#44

Concatenate 4-momentum objects from `vector` package

I did not find a way to do Concatenation in 4-momentum objects from vector package directly

Here is the temporary solution:

• Convert to tuple of numpy

• Concatenate via numpy for the tuple of numpy

• Convert back to tuple of vector objects

Just for the record.
Maybe we will be finding a better solution soon(?)

Use nbmake in all repos

ComPWA/tensorwaves#359 switched from pytest-notebook top nbmake. This has several advantages:

• Notebooks can be run again individually from the notebook (or with tox -e nb)
• attrs can be upgraded to >21.1 (pytest-notebook sets limit <21), which works much better with pyright and VSCode
  https://www.attrs.org/en/stable/types.html#pyright

Implement per-page changelog with `sphinx-git`

It would be useful to have an overview of which PRs affected a specific TR under compwa.github.io/reports.

• May need to make some adaptations to the sphinx-git impementation so that the env.docname is inserted.
• An option is to use MyST substitutions with env.docname, but it seems not possible to use recursive substitutions.

Plotly plots of Riemann sheets are not showing on ComPWA webpage

The 3D interactive plots of the Riemann sheets made using plotly are not being displayed on RTD, even though they are shown in Jupyter Lab.

Tasks

Beta Give feedback

1. Remove sphinx-needs plugin compwa-org#209
   8 of 8
   🔨 Maintenance 🖱️ DX +2
   
2. DOC: add Plotly example notebook compwa-org#208
   3 of 3
   📝 Docs 🖱️ DX +2
   

Add lambdification source code example for large expressions

Support Technical Report notebooks with Julia kernels

Would be nice to be able to experiment with Julia in the Technical Reports, so that the project is not necessarily tied to Python.

Some requirements:

• easy to set up the kernel (through Conda?)
• easy to open the notebook with this kernel in VSCode and Jupyter Lab
• code snippets should render correctly in static HTML pages
• possible to run the notebook during the Sphinx build with myst-nb
• (?) support Pluto notebooks

---

🎉 See result in TR-019.

Embed Zenodo metadata in repositories

The metadata of Zenodo DOIs currently need to be edited by hand after a release, because (for instance) Zenodo does not extract the package description from README.md.

• One improvement would be to embed a .zenodo.json file. For an example, see here (requires ComPWA membership), and then tab "Metadata".
  ComPWA/qrules#112
  ComPWA/ampform#165
  ComPWA/tensorwaves#329
• It needs to be checked whether certain meta-data like "version" can be left out. Zenodo already guesses this correctly from Git, so it does not make sense to also embed the version in .zenodo.json.
  --> Works ✔️ https://zenodo.org/record/5561166
• Additionally, a check could be added to the check-dev-files pre-commit hook so that the .zenodo.json content stays up to date with the rest of the package.

Create DOI for compwa-org

Publish pin_requirements script as GitHub Action

The pin_requirements.py is currently 'updated' in other repositories through pre-commit. This just means it's copied into the .constraints folder. Cleaner would be to publish this script as a GitHub Action, because it's only run on GitHub Actions.

Merge about page into main landing page

Should be done before #156

Formulate guidelines for release flow

Follow-up to ComPWA/expertsystem#68

Some ideas:

• How do/can releases tie in with linear commit history?
• Note on semantic versioning
• Format of a release draft on GitHub
• Tools for keeping Git tags and package version (under setup.py) in sync
• Decide on need to create a separate commit for the release itself (depends on what comes out of ComPWA/expertsystem#70)

For more ideas, see comments under ComPWA/expertsystem#68

Insert form factors in TR-013 (spin alignment)

TR-013 by #111 differs from 10.1155/2020/6674595 in two ways:

• It uses Breit-Wigners without form factors. Since ComPWA/ampform#251 however, it's possible to generate 'helicity-like coefficient names' for an amplitude model generated with formalism="canonical-helicity". This makes it possible to generate all LS-combinations and generate the coefficients from Table 1.
• It uses coefficients for the amplitude of the each decay chain, not helicity couplings. See ComPWA/ampform#254. Note that the difficulty here lies in correctly correlating the helicity couplings as described in Appendix D.

Binder does not launch due to hide_code

See https://mybinder.org/v2/gh/ComPWA/compwa-org/b8340de?urlpath=lab/tree/docs/report/000.ipynb (for latest commit b8340de)

Technical Reports overview is not a data table anymore

The searchable data table on the Technical Reports overview does not work anymore. The script is still there, but it does not convert the Markdown table created by _list_technical_reports.py into a data table.

Remove `sphinx-needs` plugin

Tasks

Beta Give feedback

1. Remove pr and issue roles (does not work well with jupyterlab-myst)
2. Remove sphinx_issues plugin
3. Replace all need roles with doc links
4. Convert all spec directives to card
5. Add links to related PRs and issues in card footer
6. Convert sphinx-needs tags to cell tags of the card cell. These will be used by the script that create a DataTable of reports.
7. Remove sphinx-needs
8. Convert needtable to data table generated by script. See here for inspiration

Original post

The problem seems to be caused by sphinx-needs, which we use to structure and interlink the technical reports (e.g. the table at compwa-org.rtfd.io/reports.html). Once the sphinx_needs extension has been disabled (564e827), Plotly figures are visible again (removing any need or spec directives from the notebook also does not fix the problem).

An alternative for organizing report notebooks may be sphinx-gallery, but this does not seem to work well with MyST-NB. Ideally, we want to have something like learn.astropy.org, where you can search for notebooks with filters etc.

Some alternatives:

• Write some script that creates a DataTable from the TR notebooks, just like EBP's feature vote.
• ablog.rtfd.io
  No longer actively maintained
• learn.astropy.org
  Looks best, but we would have to Gatsby, a whole different ecosystem than Sphinx
  Originally posted by @redeboer in #206 (comment)

Investigate difference between unbinned fit with PDFs versus normalization in the estimator

ComPWA (C++) and TensorWaves do their unbinned negative log likelihood fit with pure functions as opposed to working with Probability Density Functions (PDFs). The normalization in each fit step therefore takes place in the estimator and not in the function. Fit fractions are then computed by integrating at the end, not by reading off coefficients.

Some questions to answer:

• Does this approach result in the same observables, like fit fractions (in our experience: yes)?
• Is there a performance win or loss?

Pin all dependencies in the technical reports

Currently, TRs have a pip install statement where the packages that are required for the notebook itself. The packages are pinned to a specific version in an attempt to make the notebook reproducible, but indirect dependencies are not pinned. This results in bugs like #272.

Unfortunately, Jupyter notebooks do not support launching sub-kernels. That would be the ideal solution: you have a main developer environment for the repository that you can for instance use to start Jupyter Lab, and then each notebook launches a subkernel with it's own specific set of dependencies.

The closest we can get to an isolated kernel is by doing a pip install with --prefix and then using sys.path.insert to import from that directory. You can create a constraint file for all indirect dependencies with pip compile (or faster: uv pip compile). Providing a constraint file for each notebook then makes each technical report reproducible.

Disadvantages:

• The notebook cannot be run on its own: you need the constraint file. That works fine on Binder, but Google Colab (and Deepnote?) only have access to the notebook, not the whole repo.
• You need to pip compile the constraint files. Is that done through the notebook as well? How to provide a mechanism where the writer of the report can specify which direct dependencies are required and then compile a constraint file from that?
• ...