Need tests for jupyter notebooks

[clelange]

Need tests for jupyter notebooks

[clelange]

I looked into this in some more detail. In principle, there is a tool that nicely integrates with pytest and coverage: nbval---it checks the output cell by cell, and is also sensitive to different orders (e.g. for dictionaries in py2 vs. py3) and line breaks in our !ls statements. This can in principle be fixed by some regular expressions. Here is an example of the doc_sanitize.cfg I created while playing around with it:

[JupyROOT version]
regex: JupyROOT\s\d.\S*
replace: JupyROOT version

[file1]
regex: additional\_figure\_\d*\.yaml\s*
replace: file

[file2]
regex: signalEffVsMass\.png\s*
replace: file

[file3]
regex: submission\.yaml\s*
replace: file

[file4]
regex: thumb\_\s*
replace: file

[file5]
regex: Figure\_\d*-\D\.png\s*
replace: file

[file6]
regex: figure\_\d\D\.yaml\s*
replace: file

Example command for testing:

py.test --nbval examples/correlation.ipynb --sanitize-with doc_sanitize.cfg --current-env

The other option, already mentioned above, is papermill, maybe together with scrapbook. Then one could write tests such as the pyhf notebook tests, and basically just make sure that the notebook still runs and in addition e.g. just check for the output files created or even just run something like that:

papermill correlation.ipynb test_papermill/correlation.ipynb --kernel python3

@AndreasAlbert What's your take on that?

[kratsg]

/cc @matthewfeickert as he will have opinions here.

[matthewfeickert]

Hi @clelange, @AndreasAlbert. I would recommend papermill as @kratsg mentioned. We use it very successfully for pyhf with pytest, and I also use it to similarly validate my own work.

As you've already mentioned, scrapbook works as the compliment to papermill if you need feedback or to check notebook output.

Let me know if you have any questions.

[clelange]

Thanks for the feedback, @matthewfeickert and @kratsg - we'll probably go with scrapbook and papermill first and maybe later see whether we use nbval.