teddygroves/bibat

A batteries-included template for Bayesian data analysis projects

bibat's Issues

SBC

Add a step implementing model checking by simulation based calibration

Flow chart illustrating intended workflow

Anaconda compatibility

Automatic api documentation

Add example of automatically generating a report to the documentation

See comment here.

Sensible way to not run model configurations

Add a do_not_run flag to the model configuration object?

Check test coverage

pyOpenSci/software-submission#83 (comment)

• coverage tool
• badge
• check some more things

Optionally remove example analysis

The example analysis is useful for getting started with bibat, but it might be nice to have the option for experienced users to remove it, avoiding the need to manually delete or edit any example analysis files.

Basic plotting

There should be a script for drawing plots based on a finished fit. For example

• plot obervations vs predictive intervals
• plot marginal posteriors

Reconsider InferenceData storage format

I haven't looked at zarr before.

Also maybe netcdf is now working on arm macs.

Makefile target to install dependencies

Lovely plots

https://github.com/killiansheriff/LovelyPlots

Document dependencies

See pyOpenSci/software-submission#83 (comment)

These are the ones likely to cause problems (cmdstan is generally troublesome and quarto is the only dependency apart from python and make that bibat doesn't try to install):

• cmdstan
• quarto

Update pydantic code to work with version 2

CI is currently breaking due to the new pydantic release.

It shouldn't be too hard to switch to update for the new version...

reloo

It should be possible to use the arviz reloo function as well or instead of the standard compare one in order to cover cases where the loo log likelihood can't be reliably estimated for some observations.

Move docs to sphinx/readthedocs

The readme page is getting a bit long - a proper documentation website would be better I think.

Better __init__.py

Main thing to do is add an __all__ that indicates what belongs to bibat.

Fail fast for unsupported python versions

End to end test with github actions

Ensure that running cookiecutter and then make results produces some files on all platforms

Test all doc build options with github actions

Currently the workflow at .github/workflows/test_end_to_end.yml only tests that python -m pytest and make analysis work. It should also test that the docs build properly.

Merge makefiles

It feels strange to me to have two different makefiles, one specific for docs and the other for all other parts of the analysis. I think it adds an extra barrier to keeping docs up to date

Monitor and improve documentation page accessibility

Use an accessibility checker as recommended here.

Rationalise minimum python version

pyOpenSci/software-submission#83 (comment)

• Is it really necessary to have 3.9 as the minimum version?
• Make CI test all supported Python versions
• Make clear in documentation which versions are supported

Whether/how to not use make

See comments here pyOpenSci/software-submission#83 (comment): the current dependency on make is not ideal for windows users.

On the other hand, since cmdstanpy depends on make already maybe this is the least bad option.

Maybe snakemake and/or nox could be partly incorporated - worth looking at.

Versioning

Investigate possible sources of docs slowness

split documentation into different pages

see comments here.

json input data

There should be an option for input data to be in json format.

In this case the default data and some of the functions will need to be changed.

Code of conduct

Save prepared data as single json file

This is probably on the whole more convenient than keeping track of whole folders.

Better documentation in the target project

pyOpenSci/software-submission#83 (comment)

Funcitonal tests for more wizard choices

For example, if you choose not to have a tests directory, does the makefile turn out correct?

comments on the vignette

The link to data from the 2006 Major league season is not rendered correctly (6th paragraph in intro)

"After I answered the wizard’s questions bibat creted a new folder called baseball that looked like this:" answers should be in the vignette (save author names and email, or maybe even with those and using placeholder ones)

I don't understand the part about "Since this worked, I added a new makefile target for the raw data files:", the . at the beginning of the last line doesn't look right but more importantly, this is not used anywhere.

Naming of paths is inconsistent accross vignette. Given how there are multiple files with similar names, I would stick to paths relatives to the project home for everything. For example, right at the beginning, we start editing src/prepared_data.py, then we move onto data_preparation_functions.py (which lives inside src too) and right after to prepare_data.py (now in the project home).

"To check that all this works, we can run the script prepare_data.py manually or using make analysis." doesn't work with make analysis because tests still refer to example data. Moreover, when running python prepare_data.py I get a pydantic validation error.

stopped here for now

Richer example model

The example model is currently a simple linear regression. It might be nicer to make it a bit richer, perhaps a varying-intercept multilevel model.

Copy input information to output folder

This can be handy for debugging, we've found it very useful in Maud.

add cruft support

It would be helpful if bibat supported updating using cruft.

consider pydantic for model configurations

documentation: make analysis vs just running the python script

CI

possibly copy cookiecutter-cms

Add a pre-commit

pyOpenSci/software-submission#83 (comment)

Optional sphynx docs

Example here https://github.com/MolSSI/cookiecutter-cms

For making it optional, copy how the report bit is done in hooks/post_gen_project.py

Tutorial in readme

tests directory and examples