meteoswiss-mdr/pyrad

Python Radar Data Processing

pyrad's Issues

Information on number of pulses per ray npulses not stored properly

Superpose data in plot output

It would be convinient to have the possibility to superpose other non-radar data to the output plots.
Two clear examples:
Position of lightning strikes
Position of the melting layer

refine filter based on position respect to the iso-0

in Py-ART, functions temp_based_gate_filter and iso0_based_gate_filter could be refined to account better for the beam broadening and the size of the pulse volume.

Implement ZDR columns

Flexible selection for ODIM file extensions

Right now only file extension .h5 is accepted as valid.

However hdf files can have the following extension:

.hdf, .h4, .hdf4, .he2, .h5, .hdf5, .he5

A solution has to be found so that the file extension can be either selected by the user or transparent.

volume data interpolation unstable

The interpolation used to adapt the temperature and iso0 profile from the volume reference to the current volume it is unstable and sometimes produces strange values

PPI map not saved properly

Function plot_ppi_map raises an exception when saving the figure:
Traceback (most recent call last):
File "/home/lom/users/kuz/.local/lib/python3.5/site-packages/mch_pyrad-0.1.1-py3.5.egg/pyrad/flow/flow_control.py", line 1242, in _generate_prod
result = prdfunc(dataset, prdcfg)
File "/home/lom/users/kuz/.local/lib/python3.5/site-packages/mch_pyrad-0.1.1-py3.5.egg/pyrad/prod/process_product.py", line 697, in generate_vol_products
plot_ppi_map(dataset, field_name, ind_el, prdcfg, fname)
File "/home/lom/users/kuz/.local/lib/python3.5/site-packages/mch_pyrad-0.1.1-py3.5.egg/pyrad/graph/plots.py", line 460, in plot_ppi_map
fig.savefig(fname_list[i], dpi=dpi)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/figure.py", line 1563, in savefig
self.canvas.print_figure(*args, **kwargs)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/backend_bases.py", line 2232, in print_figure
**kwargs)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 527, in print_png
FigureCanvasAgg.draw(self)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 474, in draw
self.figure.draw(self.renderer)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/artist.py", line 62, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/figure.py", line 1159, in draw
func(*args)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/artist.py", line 62, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/opt/anaconda3/lib/python3.5/site-packages/cartopy/mpl/geoaxes.py", line 360, in draw
inframe=inframe)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/artist.py", line 62, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 2319, in draw
a.draw(renderer)
File "/opt/anaconda3/lib/python3.5/site-packages/matplotlib/artist.py", line 62, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/opt/anaconda3/lib/python3.5/site-packages/cartopy/mpl/feature_artist.py", line 139, in draw
for geom in geoms:
File "/opt/anaconda3/lib/python3.5/site-packages/cartopy/feature.py", line 117, in
extent_geom.intersects(geom))
File "/opt/anaconda3/lib/python3.5/site-packages/shapely/geometry/base.py", line 652, in intersects
return bool(self.impl['intersects'](self, other))
File "/opt/anaconda3/lib/python3.5/site-packages/shapely/predicates.py", line 13, in call
self._validate(other, stop_prepared=True)
File "/opt/anaconda3/lib/python3.5/site-packages/shapely/topology.py", line 18, in _validate
raise ValueError("Null geometry supports no operations")
ValueError: Null geometry supports no operations

Release stable versions

Would be very useful to make pyrad more user friendly

Error reading OZC files

Likely it is only in the Python reader version

• master file: OZC172131700VL.860
  Traceback (most recent call last):
  File "main_process_data.py", line 175, in
  main()
  File "main_process_data.py", line 141, in main
  PROFILE_MULTIPROCESSING=args.PROFILE_MULTIPROCESSING)
  File "/users/jfigui/.local/lib/python3.7/site-packages/pyrad_mch-0.4.4-py3.7.egg/pyrad/flow/flow_control.py", line 229, in main
  num_radars=datacfg['NumRadars'])
  File "/users/jfigui/.local/lib/python3.7/site-packages/pyrad_mch-0.4.4-py3.7.egg/pyrad/flow/flow_aux.py", line 133, in wrapper
  return func(*args, **kwargs)
  File "/users/jfigui/.local/lib/python3.7/site-packages/pyrad_mch-0.4.4-py3.7.egg/pyrad/flow/flow_aux.py", line 714, in _get_radars_data
  get_data(master_voltime, datatypesdescr_list[0], datacfg))
  File "/users/jfigui/.local/lib/python3.7/site-packages/pyrad_mch-0.4.4-py3.7.egg/pyrad/io/read_data_radar.py", line 390, in get_data
  voltime, datatype_rad4alpgrid, cfg, ind_rad=ind_rad)
  File "/users/jfigui/.local/lib/python3.7/site-packages/pyrad_mch-0.4.4-py3.7.egg/pyrad/io/read_data_radar.py", line 2461, in merge_fields_rad4alp_grid
  filename_prod, reader=cfg['metranet_read_lib'])
  File "/users/jfigui/.local/lib/python3.7/site-packages/pyart/aux_io/metranet_cartesian_reader.py", line 111, in read_cartesian_metranet
  filename, physic_value=True, masked_array=True)
  File "/users/jfigui/.local/lib/python3.7/site-packages/pyart/aux_io/metranet_python.py", line 603, in read_product
  prd_data, (int(prd_header['row']), int(prd_header['column'])))
  File "<array_function internals>", line 6, in reshape
  File "/store/msrad/utils/anaconda3-pyrad/envs/pyrad/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 301, in reshape
  return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/store/msrad/utils/anaconda3-pyrad/envs/pyrad/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 61, in _wrapfunc
  return bound(*args, **kwds)
  ValueError: cannot reshape array of size 267282 into shape (640,710)

Zh bias retrieval using self-consistency algorithm gives unreasonable results

Vertical profile of TRT cell at its centre

Instead of using the median of the entire cell at each height level directly show the centre of the cell.

Clutter variance maps

There is a request for clutter variance maps. Should be easily implemented using the framework for ground clutter monitoring.
The maps would be computed on clear air days

Spectral processing

It requires a new radar object and its format should be coordinated with the ARM-DOE Py-ART group

Issue extracting only selected elevation of rad4alp data

It started with the STA data not being extracted, it turned out that the get_rad4alp_data_CSCS_2.sh file did not come round to this because it stopped at line 283 with the following error:

inflating: /store/msrad/radar/rad4alp/rawdata/18150/MLD18150/MLD1815023550U.020
Archive: /store/msrad/radar/swiss/data/2018/18150/YMD18150.zip
inflating: /store/msrad/radar/rad4alp/rawdata/18150/YMD18150/YMD1815022557L.801
Archive: /store/msrad/radar/swiss/data/2018/18150/YMD18150.zip
inflating: /store/msrad/radar/rad4alp/rawdata/18150/YMD18150/YMD1815022557L.802
Archive: /store/msrad/radar/swiss/data/2018/18150/YMD18150.zip
inflating: /store/msrad/radar/rad4alp/rawdata/18150/YMD18150/YMD1815022557L.803
Archive: /store/msrad/radar/swiss/data/2018/18150/YMD18150.zip
inflating: /store/msrad/radar/rad4alp/rawdata/18150/YMD18150/YMD1815022557L.804
Archive: /store/msrad/radar/swiss/data/2018/18150/YMD18150.zip
inflating: /store/msrad/radar/rad4alp/rawdata/18150/YMD18150/YMD1815022557L.805
Archive: /store/msrad/radar/swiss/data/2018/18150/YMD18150.zip
inflating: /store/msrad/radar/rad4alp/rawdata/18150/YMD18150/YMD1815022557L.806
Archive: /store/msrad/radar/swiss/data/2018/18150/YMD18150.zip
inflating: /store/msrad/radar/rad4alp/rawdata/18150/YMD18150/YMD1815022557L.807
/users/fvanden/pyrad/tools/copyData/get_rad4alp_data_CSCS_2.sh: line 283: 008: value too great for base (error token is "008")

extraction using no -- ele input worked fine .. so it really seems to be related to the part:
for ((iele=0; iele<${nele}; iele++)); do
ele="$((${ele_vec[${iele}]} + 800))"
unzip -o ${data_origpath}${filebase}.zip ${filebase}${time_rad}*.${ele} -d ${data_destpath}

Ingest NEXRAD data in Pyrad

Create the necessary steps to ingest NEXRAD data in Pyrad

Implement column vertical profiles

tools: get cosmo data and rad4alp cscs

Check that all the Tools involving fetching data from the cscs have been properly migrated from the old repository

Add entropy as output of the hydrometeor classification

Issue reading METRANET data from pre-polarimetry era

Not all the moments are present in the file. An exception have to be added when the moment is not in the returned data.

Saving data in ODIM file format

We are able to read ODIM files but we lack a proper writer

Check needed on MeteoSwiss C-band radars sun monitoring

The passage from reflectivity to received power at reference point is not clear because the radar constant of each radar is not clear. A check should be performed.

Software parallelization

We have attempted to parallelize the production of datasets and products but so far it seems that the code is unstable resulting in freezes of the data processing.

It is definitely a nice to have. Specially when processing high resolution data with super-computers.

Problems generating automatic pdf documentation with sphinx

Apparently the newest version of sphinx use the command latexmk to make the pdf and this is not installed by default in our servers.

Use more standard names for the data and make sure the plots are nicely produced

In several cases the letters are too big for the plots and there is ugly overlapping.

not able to insall all module

I have install the pyrad model using conda but when i run the script that time i given the error
1- can you give me proper setup file
2- readme file - https://github.com/meteoswiss-mdr/pyrad i am following this link but i am not able install packages
3 - https://github.com/meteoswiss-mdr/pyrad/blob/master/doc/pyrad_user_manual.pdf
above link i am following not able to install all packages
Please can you give me one proper setup document

the name of Weissfluh and Plaine Morte radars cannot be read from scanid in metranet file

The way the radars are identified should be changed

Output flight trajectory map in KML format

A nice addition allowing to display the data in Google Earth and other such platforms

Inconsistency in naming corrected dealised velocity

Internally there exists two names for the same field:

corrected_dealised_velocity
dealised_corrected_velocity

Compute radar cross section from reflectivity

Automatic correction of the ZDR bias according to values of monitoring

Ingest opera file format in Pyrad

A nice to have that would expand greatly the usability of Pyrad

Reading of ODIM files metadata

Currently no metadata from ODIM files is included in attributes instrument_parameters and radar_calibration of the radar object

gridmap using cartopy not working

No map is drawn when using cartopy gridmap

Add keywords REMOVE_FIELDS and REMOVE_PREVIOUS_OBJECT

In order to reduce the memory consumption and be able to sequentially process data even when the object changes these two keywords would come very handy

IQ processing

Implement standard I-Q processing techniques:

• time domain polarimetric and Doppler moments estimation (also lag1 estimators)
• Fourier transform of the I-Q data (with windowing)

Vulpiani KDP retrieval may have a bug

Apparently the KDP data is noiser than it should be. We have Vulpiani source code to compare with.

Move QVP/CVP functions to Py-ART

Functions currently implemented within Pyrad should be moved to Py-ART so that they can be transferred to ARM-DOE Py-ART

Wrong Doppler width in METRANET files

The conversion from binary to physical data of the Doppler width in METRANET files is wrong. The factor used should be doubled

Improve R(A) algorithm

More freedom in selecting the map projection in radarmapdisplay and gridmapdisplay. Better factoring

There is a lot of common code between plot_ppi in radarmap display and plot_surface and plot_surface_contour in gridmapdisplay which could be placed in auxiliary functions.

Moreover we misunderstood the fact that add_image already takes care of transforming the image from its native projection to the projection of the axes where it has to be added and at the moment when a background map is used the axes projection are changed to that of the background map which would not be necessary.

DSD retrieval from vertically poinint spectra