Loading SEVIRI channel data which are not available in (native format) file

[adybbroe]

Loading SEVIRI channel data which are not available in (native format) file

[djhoese]

@adybbroe I'm not sure I completely understand. Could you please provide a the full "ugly" traceback when the KeyError occurs?

[adybbroe]

I would rather suggest @sjoro to have a look at this again.

i'll try to provide an example this week.

here's an example. i have a native MSG SEVIRI file which does not have IR 9.7 channel available.

1. loading VIS 0.6 channel goes smoothly as expected
2. trying to load IR 9.7 channel throws the ugly KeyError traceback
3. loading VIS 0.8 channel after this succeeds, but prints and repeats the IR 9.7 channel KeyError traceback.

i can provide the native file in question for testing purposes.

hope this helps.

In [11]: scn1 = Scene(
   ....:           sensor="seviri",
   ....:           reader="native_msg",
   ....:           filenames=[path+fname]
   ....:           )

In [12]: scn1.load([0.6])
[DEBUG: 2017-04-21 11:11:10 : native_msg] Calibration time 0:00:00.090550

In [13]: scn1.load([9.7])
[ERROR: 2017-04-21 11:11:14 : satpy.readers.yaml_reader] Could not load dataset 
'DatasetID(name='IR_097', wavelength=(9.38, 9.66, 9.94), resolution=3000.403165817, 
polarization=None, calibration='brightness_temperature', modifiers=())': 'Channel IR_097 not available 
in the file'
Traceback (most recent call last):
  File "/tcenas/home/joro/pytroll/satpy/satpy/readers/yaml_reader.py", line 810, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **slice_kwargs)
  File "/tcenas/home/joro/pytroll/satpy/satpy/readers/yaml_reader.py", line 671, in _load_dataset_data
    proj = self._load_entire_dataset(dsid, ds_info, file_handlers)
  File "/tcenas/home/joro/pytroll/satpy/satpy/readers/yaml_reader.py", line 589, in  _load_entire_dataset
    projectable = fh.get_dataset(dsid, ds_info)
  File "/tcenas/home/joro/pytroll/satpy/satpy/readers/native_msg.py", line 262, in get_dataset
    raise KeyError('Channel % s not available in the file' % key.name)
KeyError: 'Channel IR_097 not available in the file'
[WARNING: 2017-04-21 11:11:14 : satpy.scene] The following datasets were not created: 
DatasetID(name='IR_097', wavelength=(9.38, 9.66, 9.94), resolution=3000.403165817,     
polarization=None, calibration='brightness_temperature', modifiers=())

In [14]: scn1.load([0.8])
[ERROR: 2017-04-21 11:11:18 : satpy.readers.yaml_reader] Could not load dataset     'DatasetID(name='IR_097', wavelength=(9.38, 9.66, 9.94), resolution=3000.403165817,     polarization=None, calibration='brightness_temperature', modifiers=())': 'Channel IR_097 not available in     the file'
Traceback (most recent call last):
  File "/tcenas/home/joro/pytroll/satpy/satpy/readers/yaml_reader.py", line 810, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **slice_kwargs)
  File "/tcenas/home/joro/pytroll/satpy/satpy/readers/yaml_reader.py", line 671, in _load_dataset_data
    proj = self._load_entire_dataset(dsid, ds_info, file_handlers)
  File "/tcenas/home/joro/pytroll/satpy/satpy/readers/yaml_reader.py", line 589, in _load_entire_dataset
    projectable = fh.get_dataset(dsid, ds_info)
  File "/tcenas/home/joro/pytroll/satpy/satpy/readers/native_msg.py", line 262, in get_dataset
    raise KeyError('Channel % s not available in the file' % key.name)
KeyError: 'Channel IR_097 not available in the file'
[DEBUG: 2017-04-21 11:11:18 : native_msg] Calibration time 0:00:00.091112
[WARNING: 2017-04-21 11:11:18 : satpy.scene] The following datasets were not created:     
DatasetID(name='IR_097', wavelength=(9.38, 9.66, 9.94), resolution=3000.403165817,     
polarization=None, calibration='brightness_temperature', modifiers=())

In [15]:

[djhoese]

If I'm understanding this correctly here is what is happening:

1. Load scene with seviri file that does not always have all datasets available
2. Ask for a dataset that could exist, but doesn't exist in the provided file
3. Dataset is not loaded and a KeyError is logged but not raised all the way to the user (this is expected behavior)
4. Ask for a different dataset that is present in the file
5. Reader gets asked for both the failed dataset and the newly requested dataset and see log message again

I honestly didn't code for this because I didn't think of the case where a file might not have a dataset it is "supposed" to have. I've tested this both with the viirs_sdr reader and the native_msg reader and the reader does indeed get asked multiple times for the failed dataset which is not desired. It was less noticeable in VIIRS SDR because it fails a little earlier in the process and only prints a single warning log message.

I'll fix this so if a dataset fails to load it isn't re-requested.

[djhoese]

This should now be fixed in develop. I texted it with the above seviri case and added a unit test for the behavior. Let me know if you see any issues with this.