lanl/fiducia

An open source package for DANTE signal analysis using the cubic spline unfold method.

fiducia

FIDUCIA (C20073): Filtered Diode Unfolder (using) Cubic-spline Algorithm

This is a diode array signal deconvolver based on the cubic splines method. Given time-resolved diode data (e.g. Dante), this code can produce time-resolve x-ray spectra, and radiation temperatures. Functionality is described in Fiducia's online documentation.

Installation

Official releases of Fiducia are published to pypi.org and can simply be pip installed like so:

pip install fiducia

More detailed installation instructions can be found here.

License

Fiducia is released under a 3-clause BSD license.

Citing Fiducia

If you use Fiducia in your work, please follow the best practices for citing Fiducia, which can be found here.

Acknowledgements

Development of Fiducia was supported by the U.S. Department of Energy, and the NNSA.

fiducia's Issues

Add DOE Code citation for Fiducia

Here is the link to the Fiducia abstract: https://www.osti.gov/doecode/biblio/58624

Consider publishing Fiducia releases to Zenodo

Projects like PlasmaPy publish releases to Zenodo so that all contributors get recognition/citations. We should consider doing the same.

Track papers citing fiducia

Generate an informal list of papers citing Fiducia, just as a sample of Fiducia's capabilities/impact.

Solid angle typo

Remove the per steradian units when plotting/saving spectrum, when the solid angle of the detector is already accounted for. By default fiducia.loader.loadResponses accounts for the solid angle, so that the responseFrame is in units of V/GW instead of V/GW/sr. This then means that the spectrum is in units of GW/eV instead of GW/eV/sr.

Units need to be fixed for:

• fiducia.visualization.plotResponse
• fiducia.visualization.plotStreak
• fiducia.cspline.reconstructSpectrum
• fiducia.main.analyzeSpectrum
• fiducia.main.feelingLucky

Add methods for handling NIF Dante data

• Include flags to choose whether data is from NIF or OMEGA Dante
• Add function(s) for reading raw Dante data from NIF
• Add function(s) for reading response functions from NIF

When some of Fiducia's functionality gets moved to plasmapy, we should make it general enough for handling any calibrated diode array data with a corresponding set of response function curves.

Handling for unordered knots

Sometimes knot photon energies aren't in the same order as increasing channel number and this causes the spline function to integrate backwards in certain sections.
Best place to do this is probably in segmentsArr().

Create a logo

All good projects have a sweet logo - why not ours?

Add a changelog

Use this to track/record major changes between versions.

User specified background subtraction for hysteresis correction

Automatic peak finding is inefficient at detecting portions of the trace that are signal instead of background. User specified background ranges need to be added to account for all possible data shapes and to give easier to use hysteresis corrections.

Initial release of code

Release merge fiducia code into repo.

Intelligent plotting

Fix plotting flags to more intelligently display desired plots from feelingLucky() and similar functions.

Hysteresis corrections

Diode channels can sometimes have hysteresis, where the voltage after the x-ray signal does not drop back down to 0 V. Currently we handle this by fitting some cubic function to before and after signal, but this is prone to failure. Also there isn't a particular physical reason for using a cubic polynomial.

• Literature search on diode hysteresis. Physical reasons, and physically accurate model for correcting hysteresis.
• Robust method for separating signal peak from zero level and hysteresis level, as well as separating from oscilloscope noise.

Auto-handle low signal levels

Low signal levels on certain channels can yield highly oscillatory spectra. When these channels are excluded the spectra become non-oscillatory again.

• When and why does this happen?
• Why do spectra not just approach the limit as though the channel is not included in the analysis?
• Is there a sensible way to implement a method which catches and mitigates this oscillatory behavior (i.e. drop a channel when the signal becomes too low at that time step)?

Contributors file

Add a file to track contributors. Initial names:

• Pawel M. Kozlowski
• Dan Barnak
• Myles Brophy

Oscilloscope noise cleaner

Diodes often show shot noise or oscilloscope noise at leading edge or trailing edge of signal trace. This should be cleaned/corrected to enable better peak finding for hysteresis correction and signal alignment steps.

Code coverage

Generate pytest and hypothesis testing code coverage reports. Publish as badges under the README file.

Overview description for each submodule

Each submodule's .rst file for sphinx should include a description of what is the purpose of the submodule. This will help users identify where functions are located, and help contributors identify how to categorize code contributions into the various submodules.

Overview doc of cubic spline theory

Create a page that goes through the main points described in the Barnak et al paper on cubic spline unfolds.

• How cubic splines are defined by their knot points, and what moving these knot points does.
• Assumptions/boundary conditions.
• How cubic splines remove energy dependence and therefore results in an invertible matrix to solve for x-ray spectrum.
• How uncertainties are propagated, including covariance terms due to the various channels having overlapping responses in photon energy space.

Create CONTRIBUTING.md

Include guidelines on how to contribute to the project. Make it clear for new contributors to understand the structure of the package and which coding conventions we follow.

Forward propagate spectra into signal traces

Generate signal traces from inferred spectra to be overlaid with measured signal traces. This can then be used as a goodness of fit metric, aka self-consistency checker.

Create templates for issues and pull requests

Issues template should clearly outline what information is required to submit an issue.
PR template should provide structure on explaining which issue is being resolved and how it is being resolved by the PR.

Include citation

Include citation for Barnak et al paper on cubic splines analysis method. This should be included in the README and as a dunder method for the top-level fiducia module.

Open release example data

Dante data (offsets, attenuators, response function curves, and measured signals) to be used for testing and for examples included in documentation.

Negative elements for power array coming from `inferPower()`

fiducia.main.inferPower(energies, spectra) produces an array with the total radiated power for each time. Negative values of power are nonphysical, and later cause issues when trying to calculate the radiation temperature which results in those elements being NaN.

Update requirements file

The requirements.txt file for installation may be out of date. Last time we ran it, skimage was missing as a dependency. Current dependencies should be checked via a fresh install in a new virtual environment (perhaps using a Github CI).

Time-integrated spectrum

Need a function that integrates the dante voltages over time and solves the cubic spline for the time-integrated spectrum corresponding to the time integrated voltages.

Error propagation

Find and repair bugs in error propagation. 100% error on response function curves currently yields unrealistically small error bars on spectra and radiation temperature unfolds.

Survey Optimizations

Investigate some additional optimizations:

• Parallelize using multiprocessing Python module. Each time step in the streak unfold is independent of the other times steps, so this would be embarrassingly parallel.
• Remove the triple for-loop in the unfold in favor of a matrix operation using xarray. Need to be careful about how signal interpolation and error propagation across interpolation is handled here.

Move streaked spectra into xarray DataArray

Streaked spectra yield 3D info (spectrum vs photon energy vs time), this is currently handled by outputting a tuple of numpy arrays. We could handle it better by placing the data into an xarray dataArray.

Replace documentation references to Dante

Make the documentation more general by referring to data from "filtered diode arrays" instead of "Dante". This package is general enough to work on many diagnostics.

Detailed exception handling

Most exceptions currently throw an Exception, we should make these more specific by replacing them with built-ins like NotImplementedError, and our own custom error classes that inherit from the built-in Python Exception class. For example, create a MethodNotFound error.

Uncertainties warning

Add a warning message when running Fiducia analysis that propagation of uncertainties is a feature which is currently in development, and the resultant uncertainties should not currently be trusted.

Disentangling knotSolve and error and stats

knotSolve should give a spline solution without requiring statistical analysis of the spline array

GitHub Discussions

Add links to the contributing and help documentation to the GitHub Discussions page. We can use this space for Q&A and other discussions that don't neatly fall under an Issue.

HXRD unfolds

Hard x-ray detectors are another useful filtered diode diagnostic which is fielded at various HED facilities. Figure out how to produce a spectral unfold from HXRD data fielded on Omega-60.

Measurement Frame vs separate time and signal DataFrames

Both signalsAtTime() and signalInt() call for a monolithic DataFrame that has both time and signal information contained within. However, the native output for all loader/rawProcess functions are separate time and signal DataFrames. I suggest changing the loader/rawProcess functions to output a single measurementFrame that contains both time and signal.

Add MANIFEST.IN

Add a MANIFEST file for generating a packaged version of Fiducia for easier installation. Upload the package as an artifact and link to it for users to download directly.

Signal alignment

Some filtered diode arrays do not have a timing fiducial (e.g. Omega-60 Dante) and therefore require alignment of peak signals across multiple diode channels to unfold spectra. Currently Fiducia can only align Dante signals based on maximum point of a peak. We should add additional options.

• Align to timing fiducial (e.g. small signal from beam hitting a foil).
• Alignment based on NIF timing fiducial whenever we add capability for handling NIF data.

Dot product behavior in xarray differs from numpy

This is relevant to substituting xarray for certain multi-dimensional array operations instead of using numpy arrays with for-loops. Apparently dot product behavior of xarray is different from that of numpy.

detArrInv.dot(signalsy0), where signalsy0 is a 1D numpy array, returns a different result depending on if detArrInv is a 2D numpy.ndarray or a xarray.DataArray.
However, np.dot(detArrInv, signalsy0) returns the same result.
I found this while transitioning to xarray, it caused a lot of headache. Will look into this further, but on first pass of the documentation I can't find any indication that there should be a difference when dotting 1D arrays.

Old dataset handling

Datasets prior to the 2009 update to the Dante data software (C. Sorce et al.) cannot be read into Fiducia.

Github versioning

Come up with a process by which we release snapshot versions of the code.
Create separate branches for source code and main branch, where source code will be original underlying source code, and main branch will replace some source code with imports from plasmapy once some Fiducia functionality is moved to plasmapy.

Check boundary attribute in DataArray

Raise a warning when user tries to unfold on a dataset which used a different boundary. Boundary condition is used in multiple places, such as MCMC error propagation, and later unfolding. We need to a check to ensure consistency of boundary conditions across all of these operations during an analysis run.