earthobservations / gribmagic Goto Github PK
View Code? Open in Web Editor NEWGeneric weather forecast downloader
Home Page: https://community.panodata.org/t/developing-gribmagic/233
License: MIT License
Generic weather forecast downloader
Home Page: https://community.panodata.org/t/developing-gribmagic/233
License: MIT License
Hi again,
we just found that data of DWD COSMO-D2 within the "temperature 2m" folder [1] is published with multiple grid variants like regular-lat-lon
vs. rotated-lat-lon
.
Parameter folders for "temperature" [2] also bring in the levels-axis like model-level
vs. pressure-level
as outlined within #3, so this folder and friends contains four (4) different sets of data in total.
As @mhaberler pointed out at #3 (comment), the docker-dwd-open-data-downloader by @EduardRosert also accounts for that by discriminating between "grids": ["regular-lat-lon", "rotated-lat-lon"]
, see models.json.
With kind regards,
Andreas.
[1] https://opendata.dwd.de/weather/nwp/cosmo-d2/grib/00/t_2m/
[2] https://opendata.dwd.de/weather/nwp/cosmo-d2/grib/00/t/
Dear Michael,
based on your improvements to the dwd-open-data-downloader
by @EduardRosert and @bjoern-reetz you published at EduardRosert/docker-dwd-open-data-downloader#2 (thanks a stack to all of you!), I've just added a demo of a humble pipeline machinery [1] which might resonate with your proposal at earthobservations/wetterdienst#271 (comment).
The goal is to have as much details available as Python code as possible. So, one detail that was important to me was to have the "download/processing recipe" available as Python code in order to lay the groundwork of evolving this into different directions, see [2].
It is really only a demo, completely unpolished yet and it might not even work on Windows. It has to be invoked from the toplevel directory. A short synopsis about installation and operation would be:
# Acquire sources
git clone https://github.com/earthobservations/gribmagic
cd gribmagic
# Install ECMWF ecCodes, Climate Data Operators (CDO) and ECMWF Magics. YMMV. See also `docs/magics.rst`.
# See also https://confluence.ecmwf.int/display/ECC and https://code.mpimet.mpg.de/projects/cdo.
# Linux
apt-get install --yes libeccodes0 cdo libmagplus3v5
# macOS
brew install eccodes cdo
make magics-install
make magics-info
# Python virtualenv FTW
python3 -m venv .venv
source .venv/bin/activate
# Install Python package
pip install --editable=.[plotting]
# Install improved DWD Open Data Downloader
gribmagic install dwd-grib-downloader
# Invoke the machinery
./demo/acquire_and_bbox.sh
While still in its infancy, I believed it would be a good idea to share this with you and will be happy about any feedback.
With kind regards,
Andreas.
[1] https://github.com/earthobservations/gribmagic/blob/8d5ffdc/demo/pipeline.sh
[2] https://github.com/earthobservations/gribmagic/blob/8d5ffdc/demo/pipeline/recipe.py
cc @gutzbenj, @wetterfrosch, @mhaberler
-- .gribmagic-data/subgrid/bbox_46.0_47.5_14.5_16.8/png/icon-d2_germany_regular-lat-lon_pressure-level_2020122212_001_950_v.grib2.png
@mhaberler suggested at #4 (comment):
Note Cosmo-D2 is on the way out and is currently being replaced by Icon-D2 which is sort-of preproduction status (but stable as far as I can judge).
Thanks. So, we should also add acquisition for ICON-D2 to the mix.
Hi. When i run the file i keep getting invalid syntax when i run python setup.py install
.
Hi there,
within model_variables_levels_mapping.yml
, temperature
is designated as a model-level
field. However, the model level is missing from the filename, so GribMagic croaks like
WARNING: Access failed: 404 Client Error: Not Found for url: [...]
The computed URL looks like [1] but it should be [2] instead.
With kind regards,
Andreas.
[1] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/t/icon-eu_europe_regular-lat-lon_model-level_2020121900_000_T.grib2.bz2
[2] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/t/icon-eu_europe_regular-lat-lon_model-level_2020121900_000_10_T.grib2.bz2
Hi there,
within model_variables_mapping.yml
, diffuse_horizontal_irradiance
is mapping to DWD ICON's aswdiff_s
parameter field. However, there [1-4] are only folders called aswdifd_s
and aswdifu_s
.
With kind regards,
Andreas.
[1] https://opendata.dwd.de/weather/nwp/icon/grib/00/
[2] https://opendata.dwd.de/weather/nwp/icon-d2/grib/00/
[3] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/
[4] https://opendata.dwd.de/weather/nwp/icon-eu-eps/grib/00/
Hi again,
while looking at #2, we found that the folders hosting parameters for temperature
[1,2] and wind
[3,4] not only contain model-level
data, but also pressure-level
data. So, we will probably have to improve model_variables_levels_mapping.yml
and the respective downstream infrastructure to accept a list of labels on the right hand side of the mapping.
With kind regards,
Andreas.
[1] https://opendata.dwd.de/weather/nwp/icon/grib/00/t/
[2] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/t/
[3] https://opendata.dwd.de/weather/nwp/icon/grib/00/u/
[4] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/u/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.