mpvginde/pysteps

Python framework for short-term ensemble prediction systems.

pysteps - Python framework for short-term ensemble prediction systems

docs
status
package
community

What is pysteps?

Pysteps is an open-source and community-driven Python library for probabilistic precipitation nowcasting, i.e. short-term ensemble prediction systems.

The aim of pysteps is to serve two different needs. The first is to provide a modular and well-documented framework for researchers interested in developing new methods for nowcasting and stochastic space-time simulation of precipitation. The second aim is to offer a highly configurable and easily accessible platform for practitioners ranging from weather forecasters to hydrologists.

The pysteps library supports standard input/output file formats and implements several optical flow methods as well as advanced stochastic generators to produce ensemble nowcasts. In addition, it includes tools for visualizing and post-processing the nowcasts and methods for deterministic, probabilistic, and neighbourhood forecast verification.

Quick start

Use pysteps to compute and plot a radar extrapolation nowcast in Google Colab with this interactive notebook.

Installation

The recommended way to install pysteps is with conda from the conda-forge channel:

$ conda install -c conda-forge pysteps

More details can be found in the installation guide.

Usage

Have a look at the gallery of examples to get a good overview of what pysteps can do.

For a more detailed description of all the available methods, check the API reference page.

Example data

A set of example radar data is available in a separate repository: pysteps-data. More information on how to download and install them is available here.

Contributions

We welcome contributions!

For feedback, suggestions for developments, and bug reports please use the dedicated issues page.

For more information, please read our contributors guidelines.

Reference publications

The overall library is described in

Pulkkinen, S., D. Nerini, A. Perez Hortal, C. Velasco-Forero, U. Germann, A. Seed, and L. Foresti, 2019: Pysteps: an open-source Python library for probabilistic precipitation nowcasting (v1.0). Geosci. Model Dev., 12 (10), 4185–4219, doi:10.5194/gmd-12-4185-2019.

While the more recent blending module is described in

Imhoff, R.O., L. De Cruz, W. Dewettinck, C.C. Brauer, R. Uijlenhoet, K-J. van Heeringen, C. Velasco-Forero, D. Nerini, M. Van Ginderachter, and A.H. Weerts, 2023: Scale-dependent blending of ensemble rainfall nowcasts and NWP in the open-source pysteps library. Q J R Meteorol Soc., 1-30, doi: 10.1002/qj.4461.

Contributors

pysteps's Issues

Correct noise cascade initialisation

Task related to this pySTEPS issue: 332

Now, the all noise cascade levels are initialized to zero.
It is probably better to initialize the noise with the appropriate noise field as provided by the EPS term.

The noise cascade has holds 2 time levels of noise. Both time levels should be initialized with the same noise field. If not the overall power of the noise cascade level will not converge to the correct power.

Implement no-rain check on radar fields/set default autocorrelation parameters

Task related to this pySTEPS issue 309

From the A. Seed comments as quoted by @ladc:

• Set threshold at around 3% raining fraction.
• Default auto-correlations [code from STEPS1] below this fraction.
  ▶ Default (lag2) auto-correlations use a power law for the correlation length as a function of scale + black magic

Parallellized ensemble run seems to lose track of ensemble number and order

If I run an ensemble using multiple cores (so, dask will parallellize the ensemble members over the cores), it seems that the ensemble order is lost, resulting in weird transitions (due to noise from a different member that ends up in that member). See for instance (these are three 15-min instance in the forecast):

Or maybe even clearer:

Has any of you come across this and/or know how to fix it?

If I run on 1 core, this problem does not occur, so it gives the impression that this has to do with parallellizing using dask.

Check impact of number of cascade levels /do we need 8 cascade levels for Belgian domain?

Identify the interpolation step in the SL-advection

Find and implement fix for the interpolation of smallest level cascades

Prepare flooding cases suggested by Ricardo

The 10 flooding events that occurred in Antwerp as proposed by @ReinosoRadar:

• 10/08/2014 ⬅ No appropriate NWP data
• 13/08/2015 ⬅ No appropriate NWP data
• 30/05/2016
• 05/09/2018 ⬅ No appropriate NWP data
• 09/02/2020
• 17/06/2020
• 16/08/2020
• 08/02/2021 ⬅ No appropriate NWP data
• 04/07/2021
• 24/07/2021

We could check for sensitivity to:

• number of members
• number of NPW-models
• number of lagged NWP-members
• Window size of rolling-average for climatological skill
• ?

Renormalization of the precipitation cascade levels after an AR iteration

Task related to this pySTEPS issue: 332

To prevent a loss of power in the precipitation cascade levels, each level should be renormalized after an AR iteration.
However if you use a global parameters for the AR iteration, the renormalization results in a constant (in time) field.

How should we proceed?

Experiment with localized AR processes

The currently implemented estimate_ar_params_yw_localized function is rather slow.
Let's look into ANVIL as suggested by S. Pulkkinen (slack).

Prepare no-radar-rain + NWP-rain case

Implement no-rain check on NWP fields and implement the no-rain-radar + no-rain-nwp case

Task related to this pySTEPS issue 309

From the A. Seed comments as quoted by @ladc:

• If QPE zero and NWP zero ➡ resulting forecast = all zeros.
• Checking for this case saves a lot of computer time since it is likely to be the case around 80-90% of the time.
• Maybe less likely in case of NWP EPS

Celebrate release of new pysteps.blending version

Once all steps are done, we have one final step before full operational implementation: celebrate!

Variable threshold for the "no rain" criterion

Prepare no-radar-rain + no-NWP-rain case

Implement no-rain-radar + rain-nwp case

Task related to this pySTEPS issue 309

From the A. Seed comments as quoted by @ladc:

• If QPE zero and NWP nonzero ➡ NWP provides the advection.
• Set radar mean and variance and advection to zero
• Use default power spectrum slopes (?) and the default auto-correlations set the rate of blending from zero rain to NWP forecasts through the calculations of nowcasting skill as a function of lead time and scale.

Add tests for no-rain combinations

Add tests for the different combinations of no rain in NWP, radar, and both.
@RubenImhoff could you pull the version in my hackaton branch and check if your case still works there? Thanks a lot!
I currently get errors for my tests (more specifically the decomposed precip still contains nans)