ast0815/remu

ReMU - Response Matrix Utilities

ReMU - Response Matrix Utilities

A framework for likelihood calculations and hypothesis testing using binned events and response matrices.

Setup

Installing development version from source

It is recommended to run this software in a virtual Python environment (virtualenv). This ensures that all required packages are present in the tested version and do not interfere with other packages installed on the system. The easiest way to do so, is to simply source setup.sh after checking out the source code:

$ . setup.sh

This will automatically do something along these lines:

$ # Create a new virtual environment
$ virtualenv ENV
$ # Activate the environment
$ . ENV/bin/activate
$ # Upgrade pip to the latest version (optional)
$ pip install --upgrade pip
$ # Install all required packages
$ pip install -r requirements.txt
$ # Install actual package
$ pip install -e .

It will only create a new virtual environment if one does not exist already. You can also specify which Python version to use by providing the respective virtualenv argument to setup.sh, e.g. . setup.sh -p python3. You might need to install additional system libraries to compile all packages.

ReMU requires Python >=3.8.

Installing official releases with pip

Alternatively you can install official releases directly with pip:

$ # Create a new virtual environment
$ virtualenv ENV
$ # Activate the environment
$ . ENV/bin/activate
$ # Upgrade pip to the latest version (optional)
$ pip install --upgrade pip
$ # Install remu and its dependencies
$ pip install remu==1.0.0

If you want to make sure the optional dependencies are also installed, use pip's 'Extras' syntax:

$ # install remu including all optional dependencies
$ pip install remu[mcmc]==1.0.0

Tests

Run all test cases of the framework:

$ pip install -r test-requirements.txt
$ ./run_tests.sh

Online documentation

Online documentation including examples can be found on the project's readthedocs page:

https://remu.readthedocs.io

Citing

If you use ReMU in a publication, please cite it as follows:

L. Koch, ReMU - Response Matrix Utilities, http://github.com/ast0815/remu, doi:10.5281/zenodo.1217572

Or just use the DOI and let your bibliography manager handle the rest for you. You can cite specific versions of the software too. Just follow the link behind the DOI badge and choose the DOI specific for the release.

remu's Issues

Consistently marginalise over subbinnings when plotting

Right now the subbinning marginalisation is handled separately from the marginalisation for plotting puposes. The plotting marginalisation functions should also be applied in one swoop to the subbinning entries, instead of marginalising the subbinnings first.

Add option to include priors as additional likelihood constraint when maximising

This could be used to include external measurements in the fit.

Implement p-value using Wilk's theorem

See above. Might be useful for approximate solutions.

Do calculations on GPU

We re doing a lot of matrix multiplying here. Running this on a GPU might speed things up. Possible packages to facilitate this:

• Numba
• CuPy
• cudamat
• PyTorch

Nuisance bin efficiencies behave unexpectedly

They make no sense.

log-normal distribution of weights?

Might it be better to model the weights of events in a matrix bin as log-normal distributed, rather than normal distributed?

Systematics example LikelihoodMachine building only works with non-sparse matrices

Should make it work in the general case, as the Builder produces sparse arrays by default.

Speed up filling by using parallelisation for bin number calculation

Could speed things up with a mapper.

Add a CategoricalBinning

Rather than (mis)using LinearBinnings, often it would be handy to just have category bins. E.g. spcific bins for reaction type enums, or text fields.

Make PyMC and matplotlib dependencies optional

Matplotlib should only be loaded when plotting and PyMC is only needed when actually doing an MCMC.

Automatically scale parameters for fitting

Fitting works much better when everything is scale ~1 and not e.g. ~1e-38. Would be convenient if the maximisers just handle this automatically.

Publication with ReMU

Someone needs to do an actual publication using ReMU. We will freeze version 1.0 for that first publication.

We need proper documentation

Docstrings are good, but a proper website (readthedocs?) would be better.

Building new REMU with each source!?

If I source remu from a new location I rebuild the virtual environment in that location.

Matplotlib warning during tests with Python 3

Test plotting matrices. ... No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
No handles with labels found to put in legend.
ok

Add method to get derivatives of predictors

This could speed up fitting quite a bit.

Implement Pandas input and output

Using Pandas as input and output handler would make ReMU support a lot of standard file formats.

Load root histograms diractly as response matrix instead of filling event by event

Useful for more compact data releases.

Flux uncertainty helper

We need a helper class to deal with flux uncertainties:

• User provides XSEC prediction for each neutrino energy bin.
• Helper class uses flux prediction and covariance matrix (and target mass uncertainty?) to generate randomly varied event rate predictions.
• Set of predictions is fed to LikelihoodMachine to calculate p-values etc.

Unclear subbinning method documentation

The documentation about the insertion and marginalisation of subbinnings is a bit unclear. It should be made more obvious that they do not change the object itself, but only return a modified clone.

Adding of Predictors

Simple hypotheses can be added, because they are just vectors of values, i.e. arrays. It could be useful to also add composite hypotheses. The result would be a new composite hypothesis with the parameters of both input hypotheses.

Support ROOT files as inputs

It might be a good idea to support ROOT files/trees as inputs to fill Binnings. A lot of particle physics experiments use root files as de-facto standard of data exchange.

Single-file format

It might be useful to have a "one file stores all" format for sharing matrices, instead of the current "the needed information is spread over multiple files" approach.

Add easy method to combine LinearBinnings into a RectilinearBinning.

Implement comfort function for Bayes factors

See above. Probably more useful than the Posterior Distribution of the Likelihood Ratio, according to statisticians.

Support of more flexible binning schemes

I.e. not just Cartesian products of 1D binnings.

Archival file format

To ensure a maximum data shelf life, we should support storing everything in a neat, self-explanatory text format.

Needs a better name

Likelihood Machine is an unwieldy name. This software deserves something better.

Ideas:

• Likelihound
• Smearfic

Model comparisons without data

It would be useful to create model comparisons without data, i.e. determine whether two models are distinguishable given a detector response.

Examples need revisiting

I have not looked at the examples in a while. Need to make sure they work and represent the intended workflow for v1.0.

Do we need PhaseSpaces?

So far they are not actually used for anything. Should maybe just be removed.

Test bin efficiency plot with truth limits with matplotlib > 2.0.0

Might be broken?

Add option for weighted average likelihood with toy matrices

Useful for cases where available matrices are not equally likely.

Make it easier to vary fitter parameter while keeping bounds with BasinHoppingMaximiser

Right now the options dictionary is completely replaced rather than updated.

fill and fill_up_truth documentation not good enough

Use histbook as backend to replace custom binnings?

Could make use of available functionality:

https://github.com/scikit-hep/histbook

Fix JeffreysPrior

Does not follow the general design at the moment. It treats the response matrix separate from the parameter translator, while now everything is a Predictor.

Documentation of `fix_parameters` is not good.

We need LikelihoodMachine plotting functions, to plot comparisons in reco space

Right now this is a mess.

MCMC trace plotting needs some love

Right now the MCMC trace plotting functions are only working for MCMCs where all truth bins are free parameters directly. This should be made usable for all sorts of CompositeHypotheses.

BasinHoppingMaximizer should choose reasonable step sizes

Large number of reco binning (2 variables with ~30,40 bins) lead to memory error when using the optimisation of truth binning

Hi ,

When i try to use the method for merging the bins of the truth binning (or equivalent the method to compute the squared mahalanobis distance) i got Memory Error if my number of reco bins are to high.

As this method need to generate a number of random matrix wich depend of the number of reco bins, maybe the issue is coming from this.

Best regards,

Refactoring heavy classes

Some of the classes are pretty heavy. Might want to think about refactoring some of the methods to live somewhere else. At the very least the plotting methods seems to be a prime candidate for this.