fiduceo/mms

Multisensor matchup system

MMS

Introduction

The Multisensor Matchup System is a set of software components that allows to detect simultaneous measurements of geolocated instrument data. Its main purpose is to support the validation of satellite based instruments by generating cross-sensor matches of either instruments from different platforms or between satellite and ground-based measurements.

mms's Issues

Triple Sensor scenario crash - MatchupTool

2017-08-08 07:24:46 - INFO: com.bc.fiduceo.matchup.strategy.AbstractMatchupStrategy - getSecondaryObservations - Received 0 secondary satellite observations of sensor type aatsr-en
2017-08-08 07:28:15 - SEVERE: com.bc.fiduceo.matchup.MatchupToolMain - main - 53 - null
java.lang.NullPointerException
at com.bc.fiduceo.matchup.strategy.SampleCollector.addSecondarySamples(SampleCollector.java:72)
at com.bc.fiduceo.matchup.strategy.InsituPolarOrbitingMatchupStrategy.createMatchupCollection(InsituPolarOrbitingMatchupStrategy.java:175)

Resolve NPE at this point. The tool shall skip this and continue with the next triple.

ERA5 post processing failure extracting 19x7 NWP profiles

From @rossmaidment the ERA5 post-processing tool is failing with the following error when trying to extract ERA5 NWP for one of the LST ceilometer to ATSR2 MD

+ input_dir=/gws/nopw/j04/nceo_uor/users/rmaidment/mms/data/mms/mmd/mmd36/ceilometer_atsr-e2_closest_match
+ start_date=1996-211
+ end_date=1996-213
+ job_config=usecase-36_atsr-e2-pp.xml
+ config_dir=/gws/nopw/j04/nceo_uor/users/rmaidment/mms/software/new_mms/config
++ date -u +%Y%m%d-%H%M%S
+ echo '20210428-212941 ingestion 1996-211 - 1996-213 ...'
+ /gws/nopw/j04/nceo_uor/users/rmaidment/mms/software/new_mms/bin/post-processing-tool.sh -i /gws/nopw/j04/nceo_uor/users/rmaidment/mms/data/mms/mmd/mmd36/ceilometer_atsr-e2_closest_match -start 1996-211 -end 1996-213 -j usecase-36_atsr-e2-pp.xml -c /gws/nopw/j04/nceo_uor/users/rmaidment/mms/software/new_mms/config
2021-04-28 22:29:45 - INFO: com.bc.fiduceo.post.PostProcessingTool - initializeContext - Loading configuration ...
2021-04-28 22:29:45 - INFO: com.bc.fiduceo.post.PostProcessingTool - initializeContext - Success loading configuration.
2021-04-28 22:29:46 - INFO: com.bc.fiduceo.post.PostProcessingTool - computeFile - Compute file 'mmd36_ceilometer_atsr-e2_1996-211_1996-213.nc'
.
.
.
2021-04-28 22:29:49 - SEVERE: com.bc.fiduceo.post.PostProcessingTool - computeFiles - 212 - Unable to execute post processing for matchup 'mmd36_ceilometer_atsr-e2_1996-211_1996-213.nc'
2021-04-28 22:29:49 - SEVERE: com.bc.fiduceo.post.PostProcessingTool - computeFiles - 213 - Cause: last (-2147483648) must be >= first (766)
ucar.ma2.InvalidRangeException: last (-2147483648) must be >= first (766)
	at ucar.ma2.Range.<init>(Range.java:129)
	at ucar.ma2.Range.<init>(Range.java:75)
	at ucar.ma2.Section.<init>(Section.java:94)
	at ucar.nc2.Variable.read(Variable.java:543)
	at com.bc.fiduceo.post.plugin.era5.SatelliteFields.readVariableData(SatelliteFields.java:150)
	at com.bc.fiduceo.post.plugin.era5.SatelliteFields.readSubset(SatelliteFields.java:34)
	at com.bc.fiduceo.post.plugin.era5.SatelliteFields.compute(SatelliteFields.java:243)
	at com.bc.fiduceo.post.plugin.era5.Era5PostProcessing.compute(Era5PostProcessing.java:189)
	at com.bc.fiduceo.post.PostProcessingTool.run(PostProcessingTool.java:285)
	at com.bc.fiduceo.post.PostProcessingTool.computeFile(PostProcessingTool.java:253)
	at com.bc.fiduceo.post.PostProcessingTool.computeFiles(PostProcessingTool.java:209)
	at com.bc.fiduceo.post.PostProcessingTool.runPostProcessing(PostProcessingTool.java:120)
	at com.bc.fiduceo.post.PostProcessingToolMain.main(PostProcessingToolMain.java:51)

From the log I'm guessing it has failed to calculate the upper bound of the NWP extraction window. This could be due to missing data in the satellite latitude/longitude (some of the matches were at the edge of swath) or the size of the NWP extraction window.

Further investigation showed that the error only occurs when the NWP extraction window is large (19x7 or more). From usecase-36_atsr-e2-pp.xml:

<satellite-fields>
  <x_dim name='nwp_x' length='19' />
  <y_dim name='nwp_y' length='7' />
  <z_dim name='nwp_z' length='137' />

NOTE - for usecase-36_atsr-e2-pp I think we can just set the size to 1x1. But we may need larger NWP sizes for future MDs

MODIS: HDFEOS to NetCDF conversion errors

The input MODIS files are in HDF-EOS format which includes several variables which combine multiple channels into a single array. When we convert these to netCDF we separate each channel or band into a different netCDF variable - this also requires us to distribute some of the associated metadata between different variables.

However in the case of the valid_range attribute for mod021km-te_EV_250_Aggr1km_* the whole array should be copied to each channel.
e.g. we currently have:

	mod021km-te_EV_250_Aggr1km_RefSB_ch01:valid_range = 0s ;
	mod021km-te_EV_250_Aggr1km_RefSB_ch02:valid_range = 32767s ;
	mod021km-te_EV_250_Aggr1km_RefSB_Uncert_Indexes_ch01:valid_range = 0b ;
	mod021km-te_EV_250_Aggr1km_RefSB_Uncert_Indexes_ch02:valid_range = 15b ;
	mod021km-te_EV_250_Aggr1km_RefSB_Samples_Used_ch01:valid_range = 0b ;
	mod021km-te_EV_250_Aggr1km_RefSB_Samples_Used_ch02:valid_range = 28b ;

This should have been:

	mod021km-te_EV_250_Aggr1km_RefSB_ch01:valid_range = 0s,  32767s;
	mod021km-te_EV_250_Aggr1km_RefSB_ch02:valid_range = 0s,  32767s ;
	mod021km-te_EV_250_Aggr1km_RefSB_Uncert_Indexes_ch01:valid_range = 0b, 15b ;
	mod021km-te_EV_250_Aggr1km_RefSB_Uncert_Indexes_ch02:valid_range = 0b, 15b ;
	mod021km-te_EV_250_Aggr1km_RefSB_Samples_Used_ch01:valid_range = 0b, 28b ;
	mod021km-te_EV_250_Aggr1km_RefSB_Samples_Used_ch02:valid_range = 0b, 28b ;