fiduceo/fcdr_hirs

Development code for HIRS FCDR uncertainties

Welcome to the development code for the HIRS FCDR.

To get the full documentation, please enter the "doc" directory and type "make html" to get the HTML documentation, or read the reST source files inside doc/.

At a future date, the documentation will be available fully online but this depends on all dependencies being importable by readthedocs, which they currently aren't.

fcdr_hirs's Issues

Properly handle repeated scanlines within same file

Sometimes, there is a fold in a HIRS datafile, with scanline time jumping back, perhaps always scanline number too. Happens in particular with older HIRS near the end of the data file. Need to check that those are proper duplicate scanlines, in that case the best way to treat it is probably to discard the repeated scanlines. See for example /group_workspaces/cems2/fiduceo/Data/HIRS/tirosn_hirs_1978/10/30/NSS.HIRX.TN.D78303.S1928.E1948.B0024445.WI.gz

In : print(M[["hrs_scnlin", "time"]][-15:])
[(181, datetime.datetime(1978, 10, 30, 19, 47, 44, 224000))
 (182, datetime.datetime(1978, 10, 30, 19, 47, 50, 624000))
 (183, datetime.datetime(1978, 10, 30, 19, 47, 57, 24000))
 (184, datetime.datetime(1978, 10, 30, 19, 48, 3, 424000))
 (185, datetime.datetime(1978, 10, 30, 19, 48, 9, 824000))
 (186, datetime.datetime(1978, 10, 30, 19, 48, 16, 224000))
 (187, datetime.datetime(1978, 10, 30, 19, 48, 22, 624000))
 (188, datetime.datetime(1978, 10, 30, 19, 48, 29, 24000))
 (189, datetime.datetime(1978, 10, 30, 19, 48, 35, 424000))
 (183, datetime.datetime(1978, 10, 30, 19, 47, 57, 24000))
 (184, datetime.datetime(1978, 10, 30, 19, 48, 3, 424000))
 (185, datetime.datetime(1978, 10, 30, 19, 48, 9, 824000))
 (186, datetime.datetime(1978, 10, 30, 19, 48, 16, 224000))
 (187, datetime.datetime(1978, 10, 30, 19, 48, 22, 624000))
 (188, datetime.datetime(1978, 10, 30, 19, 48, 29, 24000))]

Calculate uncertainty due to wrongness of quadratic approximation

Calculate uncertainty due to wrongness of quadratic approximation.

Invalid entries in firstline db

I appear to sometimes have invalid entries in the firstline db. For example, the log below failed with ValueError: invalid literal for int() with base 10: b'\xa9\xec'.

/group_workspaces/cems2/fiduceo/Users/gholl/logs/2017/01/11/plot_hirs_noise/noaa09_1985/1985_12.lsf.err

noise correlation estimate exceeds 1

For december 2016 metopa HIRS I get noise correlations well above 1 (up to 20). This makes no sense, so there is a bug somewhere.

Consider smoothing/averaging noise level estimates

Noise estimates on HIRS vary more than they should, although some of this is consistent with limited sampling. Either way, we might consider using more than one calibration cycle to estimate the noise level, or combining the estimates from IWCT and space targets if they are sufficiently similar.

smarter disk memoisation

I need smarter disk memoisation when rereading granules. If we first read many fields from a granule, then reread the same granule with less fields or other flags, it cannot use the cache because the arguments are not identical. This is an issue with the best method for overlap filtering. Consider improving this by implementing caching on a lower level, storing the actual bytes?

Handle situation where space and IWCT noise estimates differ

Usually, noise estimates for space and IWCT are identical. However, in some cases they differ. This means the assumption that noise on space counts = noise on Earth counts = noise on IWCT counts no longer holds. The processing code needs to identify this and probably interpolate the noise estimate on Earth counts.

Estimate uncertainty in radiance-to-BT-conversion due to SRF

The SRF uncertainty is unique in that it adds a new error source in the conversion from radiance units to brightness temperature units. All other sources of uncertainty originate in earlier steps. I need to implement code, probably outside of the measurement equation formalism, to estimate this propagation.

Plot correlation matrix of noise correlations per scanpos for IWCT and space view

FCDR_HIRS.analysis.fieldmat contains code to plot scatter density matrices for anomalies between both scan positions and channels, but the correlation matrix is only plotted between channels. Need to plot correlation matrices between scan positions, along with their statistical significance, to verify that there is or is not such a correlation. If there is, also need to look at timeseries thereof.

Finalise method and code for SRF shift recovery

Need to finalise method and code for recovering shifts in Spectral Response Functions (SRFs)

Adapt estimate of magnitude of random component

The estimate of the random component of the uncertainty is based on the assumption that variability in the space or IWCT counts is entirely due to random effects. Correlations between scan positions show evidence for a periodic effect, this means the uncertainty estimate for the purely random effect is too large.

Don't throw away too much data

Sometimes a lot of data are flagged. Perhaps more can be salvaged than I currently do. Instead of throwing away all flagged data, should try to see what is wrong with it and see if we can salvage it and stick a large (and metrologically justified) uncertainty on it.

Calculate uncertainty due to Earthshine model and inputs

Calculate uncertainty due to Earthshine model and inputs and propagate this through (uncertainty nr. 13).

See also #18

Add check for case where variability in IWCT or space counts is too low (such as 0)

Sometimes the HIRS space view is always returning the same count value. Need to see whether those values are salvageable by estimating a calibration value using nearby OK calibrations (like the self-emission model) or not at all. The attached file shows an entire month where all space counts are constant, but the data are not flagged and my reading routine is not detecting the problem.

How to handle time sequence errors?

For example:

/group_workspaces/cems2/fiduceo/Data/HIRS/noaa19_hirs_2014/03/03/1457541023.NSS.HIRX.NP.D14062.S1558.E1728.B2611011.WI.gz

• scanline 239 has timestamp 2014-03-03T16:24:09.399
• scanline 240 is missing
• scanline 241 has timestamp 2014-03-03T16:23:16.151, quality indicator bits 30 (“time sequence error detected”), 28 (“insufficient data for calibration anomaly detected”), and 27 (“Earth not found”), and scan line quality flag bit 21 ("this record starts a sequence that is inconsistent with previous times"), 12 (“scan line was calibrated with marginal PRT data”), and 6 (“Earth location questionable because of questionable time code”).

Clearly, scanline 241 is problematic, but I need to do something with the time axis. If I resort the scanlines according to their reported timestamp, 11 scanlines change position. The most conservative response would be to reject those 11 scanlines. Other alternatives is to reject only the scanlines just before the one with the time jump until time is sequential again, or the ones just after... not sure what to do.

Consider correlations between error sources

The law of error propagation has a term for correlations between error sources. See GUM, equation 13. Some sources are clearly correlated; for example, both the gain (slope) calculation and the offset calculation are a function of temperature. How do we evaluate and estimate ρ between those effects?

Calculate uncertainty due to PRT counts noise

Calculate uncertainty due to PRT counts noise (effect number 6) and propagate this through to Earth radiance and BT.

Document, improve, quantify handling of LZA for HIRS/2

In HIRS/2, LZA is only given once per scanline. I should verify that my calculation for LZA for the entire scanline is correct, and perhaps estimate the associated uncertainty.

Implement self-emission estimate into measurement equation

Current measurement equation ignores self-emission. That is very bad. Needs to be implemented.

This goes into, or replaces, FCDR.interpolate_between_calibs.

Improve choice of scanlines in overlaps

When scanlines overlap, I need to choose the best one. Not good enough to choose the first one as I do now:

When asking for granule N, read N-1 and N+1, for the overlapping scanlines choose the best one. Can look at flags, sanity checks on PRT counts etc.

Write script to combine HIRS-HIRS matchups with telemetry information

For the harmonisation as performed by NPL, need to write a script that adds telemetry information to HIRS-HIRS matchups and applies the FIDUCEO measurement equation.

Estimate worst-case uncertainty due to periodic extraneous signal

Awaiting a proper fix for periodic signal (see #38), should make a conservative "on the safe side" estimate of the uncertainty it results in, and propagate this through. This is effect 17 but really effect 17 is two different effects…

Convert uncertainties from radiances units to BT units

For both the total uncertainty and all individual uncertainty components, we want to be able to calculate the uncertainty not only in radiance space, but also in BT space. Need to write a method to carry out this conversion. I believe this does not affect the correlation structure.

Some outliers are undetected

Despite filtering, some outliers are still slipping through. For example, sometimes counts=0. Sometimes this is valid, sometimes it is not. Need to implement better checks.

Consider smoothing/averaging correlation coefficients

Correlation coefficients vary from one calibration cycle to the next. Potentially this causes too much variation; should consider averaging between subsequent ones.

Propagate uncertainty in emissivity

Uncertainty in emissivity correction will come out of harmonisation. Need to code the propagation of this uncertainty to radiance and BT.

Implement non-linear component into measurement equation

Currently the measurement equation as implemented is linear. Need to implement non-linear component.

This also affects all sensitivity coefficients.

Implement systematic checks to confirm space view is really viewing space

Normally, there are 48 space views, the first 8 positions being ignored because the mirror is not in place yet. Sometimes it takes longer to get into place and we should ignore the first calibration positions.

Calculate uncertainty due to PRT counts to temperature conversion

Calculate uncertainty due to PRT counts to temperature conversion (effect 8).

Smarter automatic sensitivity determination

I should be able to code the sensitivities calculation smarter, such that when wanting the sensitivity to T_IWCT I automatically substitute until I reach T_IWCT, then no further or others. This is low priority.

Develop model to correct for periodic extraneous signal identified in MetOp-A and others

In MetOp-A HIRS and elsewhere, there is a clear periodic signal affecting space and IWCT counts equally for all channels. This in so periodic that it can potentially be corrected for. Try to see if I can reliably predict the phase in the IWCT (and ICCT where applicable) from the space views. If so, apply this model to determine the phase in the Earth views. Look for constant Earth scenes to see if I can identify it there as well. Correct for it and estimate the uncertainty for the correction.

Identify undesired electronics effects and quantity the resulting uncertainty

Sometimes, strange things happen. For example, in NOAA-18, the SW and LW channels sometimes take turn in being bad. This needs to be identified and quantified as an additional uncertainty source, and propagated through to radiances.

Estimate digitisation effects

Need to estimate the uncertainty due to digitisation of Earth counts, space counts, IWCT radiance counts, IWCT PRT counts, and self-emission model input counts. The latter are probably small.

Fix units for PRT counts to temperature effect, need to split?

The PRT counts to temperature effect is on d_PRTnk. When expressed symbolically, should this be d_PRTn(k) and be a vector? How should I represent this? It affects the units in the (codified) effects tables, as well as the sensitivity coefficients, which are going to be different depending on k.

Propagate uncertainty in non-linearity

Uncertainty in non-linearity will come out of harmonisation. Calculate how this propagates.

Calculate uncertainty due to PRT representation

Calculate uncertainty due to PRT representation and propagate this through.

Implement estimate of Earthshine into measurement equation

Estimate of Earthshine needs to be implemented into measurement equation.

How to code radiance-to-BT?

Radiance to BT conversion does not fit in measurement equation. How can I still code this one correctly? Will need some form of special case. But important.

Create simulated HIRS-HIRS matchups to aid harmonisation

Create simulated HIRS-HIRS matchups to aid in harmonisation. This allows NPL to estimate uncertainties in their estimated fundamental calibration parameters.

HIRS returns temp_ict even when there shouldn't be

In [82]: h19 = FCDR_HIRS.fcdr.HIRS4FCDR(name="noaa19", satname="noaa19")

In [81]: (head, lines) = h19.read("/group_workspaces/cems2/fiduceo/Data/HIRS/noaa19_hirs_2016/05/15/2369673413.NSS.HIRX.NP.D16136.S1029.E1215.B3745253.GC.gz", return_header=True)

In [84]: "temp_ict" in lines.dtype.names
Out[84]: True

More fully define attributes for xarray Dataset and NetCDF

More fully define attributes for xarray Dataset and NetCDF. For example, for the bit fields.

Add MetOp-B SRFs

Need to read MetOp-B SRFs, perhaps put them in ArtsXML format or adapt reading routine and format for all.

Calculate uncertainty due to self-emission model

Calculate and propagate uncertainty due to self-emission model (effect number 14)

See also #18

Combine effects with equal correlation structures into single variable

Currently I'm having one uncertainty variable per effects table in the FDCR, but the project has agreed to combine uncertainties if their correlation structures are identical. Need to systematically combine the Effect objects to consolidate them together.

Calculate uncertainty due to self-emission model parameters

Calculate and propagate uncertainty due to self-emission model parameters (effect 15)

Quantify correlations between channels in FCDR

In the HIRS FCDR, need to quantify correlations between channels. This needs to be done over an appropriate time scale, such as 6, 12, or 24 hours.

How to express dimensions/coordinates on attributes, such as for correlation matrix?

In the FCDR files, the channel correlations is a 19×19 matrix, with dimension [calibrated_channels, calibrated_channels]. But ideally it should be an variable attribute for whatever uncertainty it correspond to. Is it possible to have "data" as a variable attribute, with associated dimensions and coordinates? If not, how should this be done instead?

Propagate PRT Type B uncertainty

PRT have Type B uncertainty of 0.1K. Propagate this through.

Make sure I always identify when calibration views are not viewing calibrations

Make sure I always identify and properly account for THIS:

Design architecture for codification and calculation of effects

Need to design classes or other form of code architecture for the systematic codification and calculation of effects. That includes:

• Sensitivity coefficients: perhaps only to its “parent” effects should be enough, then by chaining all the way up to $$R_E$$
• Correlation structures
• Anything else