ecmwf / eccodes-python Goto Github PK

View Code? Open in Web Editor NEW

105.0 18.0 31.0 2.7 MB

Python interface to the ecCodes GRIB/BUFR decoder/encoder

License: Apache License 2.0

Dockerfile 0.29% Makefile 1.36% Python 90.02% C 8.33%

bufr grib meteorology

eccodes-python's People

Contributors

Stargazers

Watchers

eccodes-python's Issues

eccodes Table issues

I have used ecCodes for a while, and recently I tried to use it to decode some GNSS RO data from NOAA GDAS. I got some error message when I try to use
the command line
bufr_filter keys.filter gfs.t06z.gpsro.tm00.bufr_d.365
to grab the wanted subset, I got error like

1 of 1 total messages in 1 files
ECCODES ERROR : unable to get descriptor 063000 from table
ECCODES ERROR : unable to get descriptor 063000 from table
ECCODES ERROR : unable to get descriptor 361160 from table
ECCODES ERROR : unable to get descriptor 361160 from table
ECCODES ERROR : unable to get descriptor 102000 from table
ECCODES ERROR : unable to get descriptor 102000 from table
ECCODES ERROR : unable to get descriptor 063255 from table
ECCODES ERROR : unable to get descriptor 063255 from table
ECCODES ERROR : hash_array: no match for sequences=361160
ECCODES ERROR : Error while setting key unpack (Hash array no match)
ERROR: Hash array no match

The keys.filter looks like
set unpack=1;
set doExtractSubsets=1;
write "[bufrHeaderCentre:i]_[dataCategory].bufr[editionNumber]";

I know it’s related to some tables, but I don’t know how to find those table, and how to put them in the right place. Would you please help me to resolve the issue. Thank you very much in advance.

failing tests due to missing sample file.

After downloading eccodes-1.5.0.tar.gz from pypi it builds and installs just fine. But when I try to execute the tests using pytest-3 I get the following 5 test failures:

================================ short test summary info =================================
FAILED tests/test_eccodes.py::test_count_in_file - FileNotFoundError: [Errno 2] No such...
FAILED tests/test_highlevel.py::test_message_iter - FileNotFoundError: [Errno 2] No suc...
FAILED tests/test_highlevel.py::test_message_iter_missingvalues - FileNotFoundError: [E...
FAILED tests/test_highlevel.py::test_message_copy - FileNotFoundError: [Errno 2] No suc...
FAILED tests/test_highlevel.py::test_write_message - FileNotFoundError: [Errno 2] No su...
======================== 5 failed, 65 passed, 4 warnings in 2.30s ========================

After inspecting this, it becomes clear that this is caused by a missing sample file. The tests expect the file era5-levels-members.grib to be present in the tests/sample-data folder but that file is missing.
This in turn is due to a small bug in the MANIFEST.in file, which includes *.grib2 but not *.grib from the tests folder.
This is confirmed by manually downloading the file era5-levels-members.grib and placing it in the tests/sample-data folder. After doing this all tests run just fine.

ImportError: libeccodes.so: cannot open shared object file: No such file or directory

Using conda with the conda forge channel to install python-eccodes on linux (RHEL and ubuntu), we noticed today that the importing eccodes crashes with an import error.

Minimal example:

conda install python-eccodes

That downloads and install

    package                    |            build
    ---------------------------|-----------------
    eccodes-2.19.0             |       hf05d9b7_1         4.0 MB  conda-forge
    python-eccodes-2020.06.0   |   py38hab2c0dc_1          94 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         4.1 MB

And then:

❯ python
Python 3.8.6 | packaged by conda-forge | (default, Oct  7 2020, 19:08:05) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import eccodes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/a001673/miniconda3/envs/satpy-dev/lib/python3.8/site-packages/eccodes/__init__.py", line 15, in <module>
    from .eccodes import *
  File "/home/a001673/miniconda3/envs/satpy-dev/lib/python3.8/site-packages/eccodes/eccodes.py", line 12, in <module>
    from gribapi import __version__
  File "/home/a001673/miniconda3/envs/satpy-dev/lib/python3.8/site-packages/gribapi/__init__.py", line 13, in <module>
    from .gribapi import *  # noqa
  File "/home/a001673/miniconda3/envs/satpy-dev/lib/python3.8/site-packages/gribapi/gribapi.py", line 32, in <module>
    from .bindings import ENC, ffi, lib
  File "/home/a001673/miniconda3/envs/satpy-dev/lib/python3.8/site-packages/gribapi/bindings.py", line 29, in <module>
    from ._bindings import ffi, lib
ImportError: libeccodes.so: cannot open shared object file: No such file or directory
>>>

We see here in github that there is a 2020.10 release, could it be that it didn't make it to conda-forge for some reason?

Can eccodes decode buffr radar image?

Hi,
Can Eccodes decode bufr rain radar images?
Regards

Unexpected behavior if not all GRIB messages in a file are read

Hi,

I use the eccodes and python-eccodes libraries to read grib encoded satellite data from EUMETSAT. While looping over a set of files I noticed an issue where information from the previously opened grib file was sometimes retained and used, leading to conflicts in e.g. data dimensions. The problem occur if I do not read all grib messages, but break out from the while loop as soon as I find the message with the correct parameter number. This can be solved by looking at all grib messages, i.e. not breaking out from the loop until the message id is None. Is this an expected behavior? I would expect that it should possible to only read parts of the file and not have any left over information carried over to a subsequent eccodes instance.

Below is an example where I loop over four satellite data files, for which I want to read a given message/parameter number. To track the observed issue I print the number of messages in the file using codes_count_in_file()

import eccodes as ec

# dictionary with the following information:  <grib filename>: <parameter number to read>    # number of messages in grib file
files = {
    'CTHEncProd_20211102161500Z_00_OMPEFS01_MET08_FES_E0415': 2,     # 2 grib messages
    'OCAEncProd_20211102161500Z_00_OMPEFS01_MET08_FES_E0415': 30,    # 12 grib messages
    'CTHEncProd_20211102120000Z_00_OMPEFS04_MET11_FES_E0000': 2,     # 2 grib messages
    'OCAEncProd_20211102150000Z_00_OMPEFS04_MET11_FES_E0000': 30,    # 12 grib messages
}

i = 0
while i < 4:  # loop over the files four times in order to identify any changes in behavior
    for fname, pnum in files.items():
        ec.codes_grib_multi_support_on()

        with open(fname, 'rb') as fh:
            print('Number of messages:', ec.codes_count_in_file(fh))

            while True:
                gid = ec.codes_grib_new_from_file(fh)

                if gid is None:
                    # Reached end of file, break out of loop
                    break

                parameter_number = ec.codes_get(gid, 'parameterNumber')
                if parameter_number == pnum:
                    # Fond correct parameter number, load data
                    data = ec.codes_get_values(gid)
                    ec.codes_release(gid)
                    break                                                      #  !!!  If this break command is deleted, the code works ok  !!!
                else:
                    # The parameter number is not the correct one, continue to next message
                    ec.codes_release(gid)
    i += 1
    print('')

The code snippet above gives the following output

Number of messages: 2
Number of messages: 13
Number of messages: 2
Number of messages: 22

Number of messages: 3
Number of messages: 22
Number of messages: 3
Number of messages: 22

Number of messages: 3
Number of messages: 22
Number of messages: 3
Number of messages: 22

Number of messages: 3
Number of messages: 22
Number of messages: 3
Number of messages: 22

It's clear that the number of messages obtained using codes_count_in_file() doesn't correspond to the actual grib file content. If I remove the break command after loading the data and releasing the message id gid (line 30), i.e. if I stay in the while loop until I reach the end of the grib file, the output looks like this:

Number of messages: 2
Number of messages: 12
Number of messages: 2
Number of messages: 12

Number of messages: 2
Number of messages: 12
Number of messages: 2
Number of messages: 12

Number of messages: 2
Number of messages: 12
Number of messages: 2
Number of messages: 12

Number of messages: 2
Number of messages: 12
Number of messages: 2
Number of messages: 12

which is in line with the number of messages in the files.

Test setup:

Linux
python               3.8.12
eccodes               2.23.0
python-eccodes      2021.05.1

What is the difference between the eccodes and python-eccodes conda-forge packages ?

Apologies for a possibly stupid question, but I'm a python+netcdf user, and not a grib user

One of our users needs a/the python wrapper for eccodes and I see on conda-forge that there is

What is the difference between the packages? Does eccodes provides the binaries, and python-eccodes the python wrapper around the binaries. python-eccodes would be a conda-forge replacement of the pip python bindings mentioned in How to install ecCodes with Python bindings in conda - ecCodes FAQ

Needless to say I'd rather use conda than pip for installing the bindings, if possible :)

This kind of reminds me of a problem I had with cdo: Try2Code/cdo-bindings#33

RuntimeError: Could not load the ecCodes library!

Hi all,

I have installed the ecCodes with

pip3 install eccodes-python

but the python -m eccodes selfcheck fails, giving back :

RuntimeError: Could not load the ecCodes library!

I have installed it on a virtualenv and I am on Ubuntu 18.04

File handle leak when using GribIndex

Steps to reproduce:

import eccodes
import os

print(os.getpid())
with eccodes.GribIndex('gribtest.grib2', ['level']) as index:
    print(index.size())

After running this code, the gribtest file handle is still open:

$ lsof -p 5262 | grep gribtest
ipython3 5262 user   12r   REG  202,1 341741882  427972 gribtest.grb2

This file handle leak isn't normally a problem, but when processing lots of files that get downloaded then deleted after processing, it consumes disk space quickly because the OS doesn't free the space until the file is actually closed by the process.

I'm using eccodes version 2.17.0. I'm not sure whether the issue is in the high-level Python API code (which looks correct), or in the underlying eccodes library.

Fails at selfcheck

Hi, I tried to install eccodes for python3 in my virtualenv today.
I had libeccodes0 installed before
i A libeccodes0 - GRIB and BUFR enecoding/encoding software

>> pip install git+https://github.com/ecmwf/eccodes-python
Collecting git+https://github.com/ecmwf/eccodes-python
  Cloning https://github.com/ecmwf/eccodes-python to /tmp/pip-req-build-lpa590qq
Collecting attrs (from eccodes-python==0.8.0)
  Using cached https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
Collecting cffi (from eccodes-python==0.8.0)
  Downloading https://files.pythonhosted.org/packages/a7/3f/2667a1516d935938245bd256b56a2f96d739ac3683e6778380f06c6e4c79/cffi-1.12.2-cp37-cp37m-manylinux1_x86_64.whl (430kB)
    100% |████████████████████████████████| 440kB 8.1MB/s 
Collecting numpy (from eccodes-python==0.8.0)
  Using cached https://files.pythonhosted.org/packages/91/e7/6c780e612d245cca62bc3ba8e263038f7c144a96a54f877f3714a0e8427e/numpy-1.16.2-cp37-cp37m-manylinux1_x86_64.whl
Collecting pycparser (from cffi->eccodes-python==0.8.0)
  Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
    100% |████████████████████████████████| 163kB 47.4MB/s 
Building wheels for collected packages: eccodes-python, pycparser
  Building wheel for eccodes-python (setup.py) ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-uzfzt90s/wheels/c4/cf/53/3e513b6214aeacc46af40d3d9258aec5ab7f37107e982ee910
  Building wheel for pycparser (setup.py) ... done
  Stored in directory: /home/philipa8/.cache/pip/wheels/f2/9a/90/de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511
Successfully built eccodes-python pycparser
Installing collected packages: attrs, pycparser, cffi, numpy, eccodes-python
Successfully installed attrs-19.1.0 cffi-1.12.2 eccodes-python-0.8.0 numpy-1.16.2 pycparser-2.19

It says that its been installed:

pip list
Package        Version
-------------- -------
attrs          19.1.0 
cdsapi         0.1.3  
cffi           1.12.2 
eccodes-python 0.8.0  
numpy          1.16.2 
pip            19.0.3 
pycparser      2.19   
setuptools     40.8.0 
wheel          0.33.1

Unfortunately this didn't work:

python -m eccodes selfcheck
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.7/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/philipa8/Pyenvs/fe3/lib/python3.7/site-packages/eccodes/__init__.py", line 4, in <module>
    from .eccodes import *
  File "/home/philipa8/Pyenvs/fe3/lib/python3.7/site-packages/eccodes/eccodes.py", line 1, in <module>
    from gribapi import __version__
  File "/home/philipa8/Pyenvs/fe3/lib/python3.7/site-packages/gribapi/__init__.py", line 1, in <module>
    from .gribapi import *                # noqa
  File "/home/philipa8/Pyenvs/fe3/lib/python3.7/site-packages/gribapi/gribapi.py", line 26, in <module>
    from .bindings import ENC, ffi, lib
  File "/home/philipa8/Pyenvs/fe3/lib/python3.7/site-packages/gribapi/bindings.py", line 33, in <module>
    pkgutil.get_data(__name__, 'grib_api_prototypes.h').decode('utf-8') +
  File "/usr/lib/python3.7/pkgutil.py", line 637, in get_data
    return loader.get_data(resource_name)
  File "<frozen importlib._bootstrap_external>", line 916, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/philipa8/Pyenvs/fe3/lib/python3.7/site-packages/gribapi/grib_api.h'

Loading eccodes on MacOS with M1 chip fails with OSError

Hi,

I'm trying to install eccodes-python on Mac with M1 chip with MacOs Big Sur. I'm using Python 3.9.5.

I have first build and installed successfully eccodes-c library in /usr/local. Then when install eccodes-python with pip:

$ pip install eccodes

Then I try to load eccodes, I get the following error:

$ python -m eccodes selfcheck
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 147, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/Users/gle/dev/3e/dataservices/pipelines/gfs-datasync/eccodes-python-1.3.4/eccodes/__init__.py", line 13, in <module>
    from .eccodes import *  # noqa
  File "/Users/gle/dev/3e/dataservices/pipelines/gfs-datasync/eccodes-python-1.3.4/eccodes/eccodes.py", line 12, in <module>
    from gribapi import (
  File "/Users/gle/dev/3e/dataservices/pipelines/gfs-datasync/eccodes-python-1.3.4/gribapi/__init__.py", line 13, in <module>
    from .gribapi import *  # noqa
  File "/Users/gle/dev/3e/dataservices/pipelines/gfs-datasync/eccodes-python-1.3.4/gribapi/gribapi.py", line 34, in <module>
    from . import errors
  File "/Users/gle/dev/3e/dataservices/pipelines/gfs-datasync/eccodes-python-1.3.4/gribapi/errors.py", line 16, in <module>
    from .bindings import ENC, ffi, lib
  File "/Users/gle/dev/3e/dataservices/pipelines/gfs-datasync/eccodes-python-1.3.4/gribapi/bindings.py", line 45, in <module>
    lib = ffi.dlopen(library_path)
  File "/Users/gle/Envs/gfs-datasync/lib/python3.9/site-packages/cffi/api.py", line 150, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
  File "/Users/gle/Envs/gfs-datasync/lib/python3.9/site-packages/cffi/api.py", line 832, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
  File "/Users/gle/Envs/gfs-datasync/lib/python3.9/site-packages/cffi/api.py", line 828, in _load_backend_lib
    return backend.load_library(path, flags)
OSError: cannot load library '/usr/local/lib/libeccodes.dylib': dlopen(/usr/local/lib/libeccodes.dylib, 2): no suitable image found.  Did find:
        /usr/local/lib/libeccodes.dylib: mach-o, but wrong architecture
        /usr/local/lib/libeccodes.dylib: mach-o, but wrong architecture

I tried several things, also installing eccodes-python from source but it didn't work.

Could anyone help me with this?

Thanks,
Greg

Reappearance of segfault in saving non-double arrays, as in #30

On removing the workaround we introduced in iris-grib#208, the original signature problem as reported in #30, seems to have reappeared.

Context:

running under Linux, via Travis
The error is seen here in the Travis testing log : https://travis-ci.org/github/SciTools/iris-grib/jobs/731255839#L1237
The testcode that triggers this is in this test
The versions of all packages installed can be seen from the "conda list" command in the Travis log
- ( N.B. open the collapsed section to view installed packages )

"python3 setup.py build_sphinx" gives an error

After switching to the renamed version eccodes (in stead of eccodes-python) for release 0.9.9 then
if I execute "python3 setup.py build_sphinx" after building the module I get the following error:

Running Sphinx v2.2.2

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 361, in eval_config_file
    execfile_(filename, namespace)
  File "/usr/lib/python3.8/site-packages/sphinx/util/pycompat.py", line 81, in execfile_
    exec(code, _globals)
  File "/home/user_to_make_rpms/manual_install_test/eccodes-0.9.9/docs/conf.py", line 43, in <module>
    release = pkg_resources.get_distribution("eccodes-python").version
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 481, in get_distribution
    dist = get_provider(dist)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 357, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'eccodes-python' distribution was not found and is required by the application

The problem seems to be in file docs/conf.py which still refers to the project as eccodes-python in stead of plain eccodes on lines 35 and 43.
If I manually edit this file, and replace the 2 occurrences of eccodes-python with eccodes, then the documentation generation runs as expected.

Typo in gribapi.py causes AttributeError when calling codes_index_get_double

I suppose it should read grib_index_get_double instead of grib_index_get_doule.

eccodes-python/gribapi/gribapi.py

Line 1451 in 8b997a5

err = lib.grib_index_get_doule(ih, key.encode(ENC), values_p, size_p)

one failing testcase when upgrading eccodes from 2.18.0 to 2.19.0

If I execute the tests after building and installing the eccodes python module version 0.9.9 all 54 tests pass when I have eccodes v2.18.0 installed.
However, when I upgrade to eccodes v2.19.0 I see one failing testcase:
tests/test_eccodes.py::test_bufr_keys_iterator FAILED [ 87%]

It gives these details:

________________________________ test_bufr_keys_iterator _________________________________

    def test_bufr_keys_iterator():
        bid = codes_bufr_new_from_samples("BUFR3_local_satellite")
        # Header keys only
        iterid = codes_bufr_keys_iterator_new(bid)
        count = 0
        while codes_bufr_keys_iterator_next(iterid):
            keyname = codes_bufr_keys_iterator_get_name(iterid)
            assert "#" not in keyname
            count += 1
>       assert count == 53
E       assert 54 == 53
E         -54
E         +53

tests/test_eccodes.py:540: AssertionError
========================== 1 failed, 53 passed in 1.13 seconds ===========================

If I manually adjust this count 53 and modify it to be 54, I get another test failure:

________________________________ test_bufr_keys_iterator _________________________________

    def test_bufr_keys_iterator():
        bid = codes_bufr_new_from_samples("BUFR3_local_satellite")
        # Header keys only
        iterid = codes_bufr_keys_iterator_new(bid)
        count = 0
        while codes_bufr_keys_iterator_next(iterid):
            keyname = codes_bufr_keys_iterator_get_name(iterid)
            assert "#" not in keyname
            count += 1
        assert count == 54
    
        codes_set(bid, "unpack", 1)
        codes_bufr_keys_iterator_rewind(iterid)
        count = 0
        while codes_bufr_keys_iterator_next(iterid):
            keyname = codes_bufr_keys_iterator_get_name(iterid)
            count += 1
>       assert count == 156
E       assert 157 == 156
E         -157
E         +156

tests/test_eccodes.py:548: AssertionError
========================== 1 failed, 53 passed in 1.25 seconds ===========================

Changing this 156 to 157 finally lets me run the test sequence without any failures.
So it seems with the new eccodes 2.19.0 the number of BUFR keys has increased by one?
Please investigate and fix this if needed in the test script.

Handling segmentation faults

When using eccodes through Python to explore a grib file, I can often get "Segmentation Fault (core dumped)" errors. This kicks me out of the interpreter, and I lose any variables I had been working with. The most replicable instances I have found are trying to open an already-open grib file, or trying to close/release/delete an already closed grib file, index file, or iterator. A more subtle segmentation fault comes from trying to delete an iterator associated with a closed grib file. I also run into cases where looping over codes_keys_iterator, I will hit a number of known errors (ArrayTooSmallError, wrong size for pl, Function not yet implemented). Attempting to delete the iterator will segfault in some cases but not others.

Is there any functionality in SWIG to handle these kinds of seg faults? What about adding sanity checks to objects within Python? I have had some success in wrapping objects using contextlib, but a) that's python3 only and b) it ensures safety by opening and closing the grib files every time, which can hamper exploring them and will sometimes mess up loops. I'd be happy to try to help from the Python side, but I have no experience in C/C++/SWIG.

Cannot find the ecCodes library when pip installing eccodes 1.3.X

Reproducer

conda create -y  --name test_eccodes python=3.8 pip eccodes
conda activate test_eccodes
python -mvenv venv
source venv/bin/activate
pip install eccodes==1.3.0 # or 1.3.1 or 1.3.2
python -m eccodes selfcheck

results in

Traceback (most recent call last):
  File "/home/panos/.conda/envs/test_eccodes/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/panos/.conda/envs/test_eccodes/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/panos/.conda/envs/test_eccodes/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/eccodes/__init__.py", line 15, in <module>
    from .eccodes import *
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/eccodes/eccodes.py", line 12, in <module>
    from gribapi import __version__
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/gribapi/__init__.py", line 13, in <module>
    from .gribapi import *  # noqa
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/gribapi/gribapi.py", line 32, in <module>
    from .bindings import ENC, ffi, lib
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/gribapi/bindings.py", line 33, in <module>
    raise RuntimeError("Cannot find the ecCodes library")
RuntimeError: Cannot find the ecCodes library

Downgrading to 1.2.0 works just fine

$ pip install eccodes=1.2.0
$ python -m eccodes selfcheck

Found: ecCodes v2.21.0.
Your system is ready.

The eccodes package from conda is eccodes conda-forge/linux-64::eccodes-2.21.0-he2bb022_1

Cleaning up:

deactivate
conda deactivate
rm -rf venv
conda env remove --name test_eccodes

Problem when retrieving an attribute from BUFR file and then reopen the BUFR file

hi
i have a script that will retrieve an attribute from a BUFR file eg "TypicalDate" but i then want to reopen the BUFR file to retrieve data arrays from the BUFR messages.

Problem is I have to read in all of the BUFR file to retrieve the attribute - even though the first 1 i read is enough for my purposes. If I break from reading the file after the first read , when i reopen to read in the data array it only reads in a couple of messages before closing.

So is there a reason you need to read in the whole file

ie i would like to be able to do this , instead of a while True and break when bufr is None


bufr = ec.codes_bufr_new_from_file(fh)

ec.codes_set(bufr, 'unpack', 1)

attr = ec.codes_get(bufr, key)

ec.codes_release(bufr)

fh.close()

Issue with codes_grib_find_nearest not working

Hi,

My system:
Ubuntu 20.04
Eccodes 1.1.0
Eccodes-python 0.9.9

I'm trying to read a grib file and use the following. It seems to fail with the following:

Code:
from future import print_function
import traceback
import sys
from eccodes import *

INPUT = '/home/james/Desktop/File'
VERBOSE = 1 # verbose error reporting

lat=55.1
lon=-3.1
f = open(INPUT, 'rb')
gid = codes_grib_new_from_file(f)

nearest = codes_grib_find_nearest(gid, lat, lon)

version coherence with eccodes

Trying to create bindings for ecCodes 2.13.1 using eccodes-python 0.9.3 breaks due to

gribapi/_bindings.c:5679:12: error: 'GRIB_FUNCTIONALITY_NOT_ENABLED' undeclared

How does one know which version of ecCodes the eccodes-python packages needs?

Building the fast CFFI API?

Hi,

sorry, are the instructions for building the faster API still valid? Because on the ECMWF website https://confluence.ecmwf.int/display/ECC/ecCodes+installation it says "The Python 3 bindings are now built with CFFI and are packaged separately"?

If the instructions are still true and could lead to a dramatic speedup, I would definitely like to have them. But I don't understand what you mean by "clone the repo in the same folder as your ecCodes source tree"? I tried cloning to

eccodes-2.13.1-Source/eccodes-python and
eccodes-2.13.1-Source/src/eccodes-python

but every time I get

gcc -pthread -B /home/martin/.conda/envs/py3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/martin/.conda/envs/py3/include/python3.6m -c gribapi/_bindings.c -o ./gribapi/_bindings.o
gribapi/_bindings.c:492:10: fatal error: eccodes.h: No such file or directory
 #include <eccodes.h>
          ^~~~~~~~~~~

Could you give me a hint, please?
Thanks a lot,
Martin

Reading NCEP BUFR with Local Tables

Hello,

I am attempting to use eccodes-python to decode NCEP BUFR files that use local tables. I've been following the instructions located at https://confluence.ecmwf.int/display/UDOC/Local+configuration+-+ecCodes+BUFR+FAQ in order to create my own local table directory structure that reflects the NCEP Table B and Table D entries.

I have attached my formatted element.table and sequence.def files that are located in definitions/bufr/tables/0/local/1/7/3/, along with an example of the kind of BUFR file I am attempting to decode: eccodes-dump.zip

When I attempt to run the bufr_dump binary on the attached file using the attached configurations, I do successfully remove errors about definitions that have now been added. However, I cannot seem to decode message number 3 in the file due to a sequence that it cannot match:

ECCODES ERROR   :  hash_array: no match for sequences=360243
ERROR: unable to unpack data section: Hash array no match (message=3)

As best as I can tell, this sequence is a reference to an NCEP Table A entry about the data inside the BUFR file, which in this case it is a GFS BUFR sounding. There are no local subtypes for data type 243, but clearly eccodes isn't sure what it is or what to do with it.

Could I receive input/help on 1) whether or not I am approaching this correctly, and 2) how to fill out the rest of my local definitions to be able to decode this file? I'd be more than willing to share the final table configuration to help others who may wish to read NCEP BUFR data that uses local tables. I can see that there are plenty of other configuration files that can be touched, but I'm not entirely sure about what they are, what they do, and which ones are relevant.

Thanks in advance for the assistance!

When the package ecmwflibs is available, ECCODES_DEFINITION_PATH is removed after `from gribapi import *`

Hello

I'm using eccodes-python 1.4.1 with eccodes library 2.25.0
I have paths to local grib definitions in environment variable ECCODES_DEFINITION_PATH
In my python script, after from gribapi import * or from eccodes import *, the content of the environment variable ECCODES_DEFINITION_PATH is lost, as if an unset ECCODES_DEFINITION_PATH had been done. Same thing with GRIB_DEFINITION_PATH.
This was not the case with my previous usage of eccodes-python 1.3.3 with eccodes library 2.23.0

My python script :

import os
from gribapi import *
print (os.environ['ECCODES_DEFINITION_PATH'])

Best regards

Building eccodes

Hello all,

I'm trying to install eccodes on my windows 10 system by following the instructions provided in the GitHub page(https://github.com/ecmwf/eccodes). While running the -DCMAKE_INSTALL_PREFIX line of the code (in command prompt), it starts to build but soon exits with the following error:

"CMake Error at cmake/ecbuild_log.cmake:196 (message):
CRITICAL - Failed to replace windows symlinks. output=[] error=[/bin/bash:
E:/Akash/eccodes-2.18.0-Source/cmake/ecbuild_windows_replace_symlinks.sh:
No such file or directory

]
Call Stack (most recent call first):
cmake/ecbuild_check_os.cmake:450 (ecbuild_critical)
cmake/ecbuild_system.cmake:265 (include)
CMakeLists.txt:25 (include)

-- Configuring incomplete, errors occurred!
See also "E:/Akash/build/CMakeFiles/CMakeOutput.log".

I'm using cmake-3.18.3-win64-x64, eccodes-2.18.0-Source (downloaded from https://confluence.ecmwf.int/display/ECC/Releases and extracted as instructed), I have installed Ubuntu from windows store and I'm building in a clean directory. 'CMakeOutput.log' and 'ecbuild.log' files are also attached here for your reference.
CMakeOutput.log
.
ecbuild.log

If anybody has any solutions/suggestions on this, please help.

Thanks,
Akash

Wrong version or selfcheck error ?

I installed ecCodes 2.16.0 as suggested here, and bindings with PyPi.
The selfcheck command gives me
Warning: ecCodes 2.16.0 or higher is recommended. You are running version 2.6.0 Found: ecCodes v2.6.0. Your system is ready.

And yet in every ecCodes related file I can find, the version seems to be 2.16.0.

I'm not sure what the problem is, wrong version or error in selfcheck.

Does anyone know where exactly the selfcheck command looks to check the version ?

open up discussion tab on github?

I have some usage questions and wonder if would be suited for the discussion tab here on github so it's easily accessible to others? e.g. see https://github.com/pydata/xarray/discussions

there's info how to do it here: https://docs.github.com/en/discussions/quickstart

Windows Error: exit code -1073741819

Hi,

I found some threads according to this error but I could not really get an answer out of them:
So I am using the following on my Windows 10 machine with Anaconda as standard Python Interpreter:

-eccodes v2.16.0
-python-eccodes v2020.01.0

I solved the issue with ECCODES_DEFINITION_PATH in my PyCharm Environment but now it just exits like this:

Process finished with exit code -1073741819 (0xC0000005)

Any ideas how you can get it working? Thanks in advance!

read data from bytes object

Our application uses the ecCodes Python bindings to process streams of BUFR data as part of real-time processing and publication of same.

Are there any examples of reading data from a bytes object? Currently we are having to manage a temporary file in order to process.

cc @david-i-berry @webb-ben

Another (potential) user encountering version Warning on "selfcheck"

I performed the following steps exactly - thinking I was following the README correctly.

git clone https://github.com/ecmwf/eccodes-python.git
cd eccodes-python
git checkout master (based on the developer's reply to an issue from a year prior)

pip install -e .
python -m eccodes selfcheck

Warning: ecCodes 2.20.0 or higher is recommended. You are running version 2.12.3

Can you please advise a solution?

Please support Python 3.10 (and 3.11)

Some Linux distributions are using Python 3.10 as their default Python now, and it becomes inconvenient to install and use older versions. Would it be possible to support more recent versions of Python, such as 3.10 and 3.11?

f2py gets installed in $HOME/.local/bin

For some reason, if I run "python3 setup.py install --user" for the eccodes-python module, it installs a copy of the /usr/bin/f2py script (which is part of the numpy installation) in $HOME/.local/bin/.
I think this is wrong and should be prevented.

Segfault when closing GribFile if messages are closed manually

This code works fine:

with eccodes.GribFile('test.grib2') as gf:
    while True:
        try:
            message = gf.next()
        except StopIteration:
            break

On the other hand, this code, which manually closes each message to save memory, segfaults :

with eccodes.GribFile('test.grib2') as gf:
    while True:
        try:
            message = gf.next()
        except StopIteration:
            break
        message.close()

Here's the backtrace:

Current thread 0x00007efd4ed45740 (most recent call first):
  File "/usr/local/lib/python3.6/site-packages/gribapi/gribapi.py", line 474 in grib_release
  File "/usr/local/lib/python3.6/site-packages/eccodes/high_level/codesmessage.py", line 165 in __exit__
  File "/usr/local/lib/python3.6/site-packages/eccodes/high_level/gribmessage.py", line 61 in __exit__
  File "/usr/local/lib/python3.6/site-packages/eccodes/high_level/codesmessage.py", line 172 in close
  File "/usr/local/lib/python3.6/site-packages/eccodes/high_level/codesfile.py", line 52 in __exit__
  File "<ipython-input-5-abd830c3b0f5>", line 7 in <module>
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3296 in run_code
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3214 in run_ast_nodes
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3049 in run_cell_async
  File "/usr/local/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 67 in _pseudo_sync_runner
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2874 in _run_cell
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2848 in run_cell
  File "/usr/local/lib/python3.6/site-packages/IPython/terminal/interactiveshell.py", line 489 in interact
  File "/usr/local/lib/python3.6/site-packages/IPython/terminal/interactiveshell.py", line 498 in mainloop
  File "/usr/local/lib/python3.6/site-packages/IPython/terminal/ipapp.py", line 356 in start
  File "/usr/local/lib/python3.6/site-packages/traitlets/config/application.py", line 658 in launch_instance
  File "/usr/local/lib/python3.6/site-packages/IPython/__init__.py", line 125 in start_ipython
  File "/usr/local/bin/ipython3", line 10 in <module>
Segmentation fault

According to the class-level doc for CodesMessage, # If desired, messages can be closed manually or used in with, so this should work.

eccodes 2.17.0

Description of the grid types

I have a grib file with this grid specified:

gridDefinitionTemplateNumber = 50;
J = 639;
K = 639;
M = 639;
gridType = sh;
NV = 276;

From searching, I see that sh is a recognized grid that comes up in examples of allowed gridTypes, but I can't find any description of what that grid is. Despite being a known gridType, I can't get any of the grib tools to display lat/lon values. Can someone point me to what this grid is and how it should be used?

I am posting this here 1) because I plan to primarily use the Python tools, and 2) eccodes has no issues page.

Windows 10 eccodes-python not returning values from installed Eccodes library

On Windows 10 I have installed eccodes using conda and eccodes-python using pip. I have set the library paths in environment variables.

I obtained some validated bufr files for a Synop observation but when using the Python code example from the ECMWF website the code runs without throwing any exceptions and returns no data from the bufr file.

ECCODES is verified as installed from the selfcheck command.

> C:\eccodes\eccodes\eccodes-master\examples\python>python -m eccodes selfcheck Found: ecCodes v2.13.1. Your system is ready.
Running a sample Python bufr script: -

`C:\ProgramData\Miniconda3\python.exe C:/eccodes/eccodes/eccodes-master/examples/python/bufr_read_synop.py
message: 0

Process finished with exit code -1073741819 (0xC0000005)`

No errors are returned from the execution.

`headers_only` not used in `grib_new_from_file`

In https://github.com/ecmwf/eccodes-python/blob/develop/gribapi/gribapi.py#L402 and following the headers_only keyword argument is not passed to the C-bindings.

This appears deliberate, so maybe I'm missing something, but the functionality is not available any more.

I suggest to either add the functionality back or update the function documentation, and probably add a warning.

Build a wheel binary distribution that includes ecCodes

At the moment ecCodes must be installed on the system for eccodes-python to work.

Following thew strategy of https://github.com/mapbox/rasterio it would e possible to package the whole ecCodes library and data files in the multilinux wheel binary package.

Pro:

no need to have two install steps, only pip install eccodes-python
no need to install ecCodes as root

Con:

huge binary wheel package
possible conflicts with other Python packages that link to ecCodes (quite remote)

The infrastructure used by rasterio is here: https://github.com/sgillies/frs-wheel-builds

Was: ecmwf/cfgrib#22

Running into Runtime error while trying to import eccodes

Hi I am trying to import eccodes in my python code and I run into error :

    import eccodes

  File "C:\Users\admin1\anaconda3\lib\site-packages\eccodes\__init__.py", line 15, in <module>
    from .eccodes import *

  File "C:\Users\admin1\anaconda3\lib\site-packages\eccodes\eccodes.py", line 12, in <module>
    from gribapi import __version__

  File "C:\Users\admin1\anaconda3\lib\site-packages\gribapi\__init__.py", line 13, in <module>
    from .gribapi import *  # noqa

  File "C:\Users\admin1\anaconda3\lib\site-packages\gribapi\gribapi.py", line 2217, in <module>
    __version__ = grib_get_api_version()

  File "C:\Users\admin1\anaconda3\lib\site-packages\gribapi\gribapi.py", line 2207, in grib_get_api_version
    raise RuntimeError("Could not load the ecCodes library!")

RuntimeError: Could not load the ecCodes library!

Command
python -m cfgrib selfcheck
returns

Found: ecCodes v2.17.0.
Your system is ready.

I am using Windows 10 and latest relase of conda and python

Add binding to grib_multi_support_reset_file

In order to use multi_support play well with random access of MULTI-FIELD messages in cfgrib I need to reset the multi_support state. At the moment I'm using grib_multi_support_reset_file that appear to work well, but it is not available in eccodes-python.

Documentation lacking

Hi,
I am completely new to ecCodes. Thought it should be possible doing the equivalent to
os.system("grib_copy -w gridType=regular_ll " + nhsp_file + " " + tmp_filename) from Python instead of via a system call.

But can't really find any useful documentation on the Python interfaces. It must be there somewhere!?

codes_get_double_array returns a list, should return a np.ndarray

Please use the warnings library for selfcheck

On Ubuntu 21.04 (a very common docker environment), the easily available version of libeccodes is version 2.20.0. Any pip-installed version of eccodes-python newer than 1.1.0 gives the following warning at import time:

Warning: ecCodes 2.21.0 or higher is recommended. You are running version 2.20.0

It would be really nice if this warning could be emitted by the warnings library, so it would be possible to catch it if desired, with code like this:

import warnings
with warnings.catch_warnings():
    import eccodes

In the meantime, if I don't want this (practically unaddressable) warning emitted by my script, I see no choice but to revert to pip install eccodes==1.1.0.

Segfault from overwriting numpy array in grib_set_double_array().

I have been chasing down an extremely weird bug in my system where a whole bunch of data gets mysteriously zeroed out when converting from NetCDF4 to GRIB1. Specifically, out of 476700 float datapoints, the first 119174 datapoints end up as zeroes in the final GRIB1 file.

I eventually narrowed this down to a bad interaction between eccodes-python and all versions of numpy from 1.17.4 onwards. numpy 1.17.3 is unaffected.

The issue seems to come from grib_set_double_array() storing a numpy array in a variable, and then overwriting that variable with a FFI ctype object created from the same variable.

Test case:

#!/usr/bin/env python3
import numpy as np
from gribapi.bindings import ffi


inarray = np.ones((100000,), np.float32)
print(inarray)

# Copy numpy array to another local variable.
a = inarray

# Simulate type munging from grib_set_double_array().
a = a.astype(float)
print(a)

# Simulate FFI cast from grib_set_double_array() and overwrite variable.
a = ffi.cast('double*', a.ctypes.data)
print(a)

print(a[0])  # Sad trombone

On my amazon linux machine with eccodes 2.18.0, python 3.7.6, and any numpy >= 1.17.4, this code segfaults. Aren't pointers fun!? [1]

However, if we just store the FFI ctype object in a totally separate variable, and avoid clobbering the numpy array, it all works perfectly. Replace the final three lines of the test script with these:

b = ffi.cast('double*', a.ctypes.data)
print(b)

print(b[0])  # Cool sax

Speculating wildly, I think the issue might be that we're passing a.ctypes.data in to ffi.cast, which is the location in memory where the numpy array is stored. After we overwrite a, the numpy array is destroyed, so that memory location is no longer valid.

[1] They are not.

Module/attribute problem within Eccodes module

I'm trying to use Eccodes in opening a file and I get this error message:

AttributeError Traceback (most recent call last)
in ()
----> 1 ds = xr.open_dataset(os.path.join(pathfiles,datafile), engine='cfgrib')

~\Anaconda3\lib\site-packages\xarray\backends\api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime)
515 elif engine == "cfgrib":
516 store = backends.CfGribDataStore(
--> 517 filename_or_obj, lock=lock, **backend_kwargs
518 )
519

~\Anaconda3\lib\site-packages\xarray\backends\cfgrib_.py in init(self, filename, lock, **backend_kwargs)
36
37 def init(self, filename, lock=None, **backend_kwargs):
---> 38 import cfgrib
39
40 if lock is None:

~\Anaconda3\lib\site-packages\cfgrib_init_.py in ()
17
18 # cfgrib core API depends on the ECMWF ecCodes C-library only
---> 19 from .cfmessage import CfMessage
20 from .dataset import Dataset, DatasetBuildError, open_file, open_fileindex
21 from .messages import Message, FileStream

~\Anaconda3\lib\site-packages\cfgrib\cfmessage.py in ()
27 import numpy as np # noqa
28
---> 29 from . import messages
30
31 LOG = logging.getLogger(name)

~\Anaconda3\lib\site-packages\cfgrib\messages.py in ()
47 # MULTI-FIELD support is very tricky. Random access via the index needs multi support to be off.
48 #
---> 49 eccodes.codes_grib_multi_support_off()
50
51

AttributeError: module 'pyeccodes.compat' has no attribute 'codes_grib_multi_support_off'

Here is my line of code:
ds = xr.open_dataset(os.path.join(pathfiles,datafile), engine='cfgrib')

Not sure if it's a bug or just using pip instead of conda to install cfgrib as well as Eccodes. I tried to use conda but it wouldn't install in the base environment.

Using memory-based file-like object (BytesIO)

Is there a way to load a file-like object from memory? Looking for a way to work around local storage limitations and improve performance by using memory for input/output operations. The following example using a BytesIO object returns an error from the library.

This is with Python 3.6 and ecCodes 2.17.0.

Thanks!

f= open('file.grib','rb')
buffer = io.BytesIO(f.read())
codes_grib_new_from_file(buffer)
---------------------------------------------------------------------------
UnsupportedOperation                      Traceback (most recent call last)
<ipython-input-25-b3e3d2be12aa> in <module>
----> 1 codes_grib_new_from_file(stream)~/.local/lib/python3.6/site-packages/gribapi/gribapi.py in grib_new_from_file(fileobj, headers_only)
    403     # err, h = err_last(lib.grib_new_from_file)(ffi.NULL, fileobj, headers_only)
    404     err, h = err_last(lib.codes_handle_new_from_file)(
--> 405         ffi.NULL, fileobj, CODES_PRODUCT_GRIB
    406     )
    407     if err:~/.local/lib/python3.6/site-packages/gribapi/gribapi.py in wrapper(*args)
    150         err = ffi.new("int *")
    151         args += (err,)
--> 152         retval = func(*args)
    153         return err[0], retval
    154UnsupportedOperation: fileno

Versioning of eccodes-python releases

I'm trying to understand the versioning system of eccodes-python releases.

From a look at the releases page (https://github.com/ecmwf/eccodes-python/releases), there seem to be two types of releases:2020.??.?? and 0.9.? releases. The former seem to align with what is available on conda-forge, and the latter align with what is available on PyPI.

What is a bit confusing is that the two different types of releases don't seem to be synced up. For example I'm particularly interested in bug fixes that have gone into version 0.9.9 however, I'm a conda user and the fixes don't appear to have gone into a release to conda-forge yet.

Could you explain why the versions are diffferent, and give an idea of when to expect a next release to conda-forge?

import "from .bindings import ENC, ffi, lib" error

Hi,

I did a "conda install -c conda-forge eccodes". The his installed successfully the command line tools "grib_ls, grib_dump...". However, "python -c eccodes" failed with

ImportError: No module named eccodes

Then installed eccodes via pip:
pip install eccodes-python

and now I'm getting
(eccodes) niwa-1019500~/grib$ python -c "import eccodes"
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/site-packages/eccodes/init.py", line 4, in
from .eccodes import *
File "/usr/local/lib/python2.7/site-packages/eccodes/eccodes.py", line 1, in
from gribapi import version
File "/usr/local/lib/python2.7/site-packages/gribapi/init.py", line 1, in
from .gribapi import * # noqa
File "/usr/local/lib/python2.7/site-packages/gribapi/gribapi.py", line 21, in
from .bindings import ENC, ffi, lib
File "/usr/local/lib/python2.7/site-packages/gribapi/bindings.py", line 34, in
except ModuleNotFoundError:
NameError: name 'ModuleNotFoundError' is not defined

(eccodes) niwa-1019500~/grib$ conda list

packages in environment at /Users/pletzera/miniconda3/envs/eccodes:

Thanks for any help.

Iterating through eccodes.GribFile throws "ValueError: I/O operation on closed file"

I tried running an example similar to https://confluence.ecmwf.int/display/ECC/High-level+Pythonic+Interface+in+ecCodes

#!/usr/bin/env python3

import os, eccodes

input_path = os.path.join(eccodes.codes_samples_path(), 'GRIB2.tmpl')
with eccodes.GribFile(input_path) as grib:
    for msg in grib:
        print(msg['date'])

it throws an error:

Traceback (most recent call last):
  File "./x", line 7, in <module>
    for msg in grib:
ValueError: I/O operation on closed file

This can be fixed by adding ._next_() method in CodesFile class:

    def __next__(self):
        return self.next()

Otherwise ._next_() inherited from the FileIO base class is called and throws "ValueError: I/O operation on closed file".

Maybe fundamentally the problem is CodesFile.__init__() not calling the constructor of its base class (io.FileIO) and as a result CodesFile objects not being properly initialized. The fix above enables iteration, but other stuff still doesn't work properly:

>>> with eccodes.GribFile('sample.grb') as grib:
...     print(grib.closed)
...     print(grib.fileno())
True
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
ValueError: I/O operation on closed file

Selfcheck warning recommends ecCodes 2.17.0 or higher, but latest available release is 2.16.0

Using Anaconda3, I installed eccodes-2.15.0 (latest avail from conda-forge), then followed instructions to install eccodes-python with Fast CFFI bindings:
$ git clone https://github.com/ecmwf/eccodes-python
$ cd eccodes-python
$ pip install -e .

I then had to modify builder.py so that python builder.py would find my eccodes.h file and build properly -
ffibuilder.set_source(
"gribapi._bindings", "#include </home/user/anaconda3/include/eccodes.h>", libraries=["eccodes"],
)

Now when I run a selfcheck or >>>import eccodes, I get the warning:

(base) [user]:~]$ python -m eccodes selfcheck
Warning: ecCodes 2.17.0 or higher is recommended. You are running version 2.15.0
Found: ecCodes v2.15.0.
Your system is ready.

I see at https://confluence.ecmwf.int//display/ECC/Releases that the latest release is v2.16.0, which was released yesterday, 13 Jan 2020.

I then located the following in gribapi/init.py:min_reqd_version_str = "2.17.0". Should this be changed to read "2.15.0"?

Thanks!

data file tiggelam_cnmc_sfc.grib2 not included in released tar file eccodes-1.4.0.tar.gz

MANIFEST.in contains the line "recursive-include tests *.grib" and therefore only includes files with extension .grib.
However, currently the file below test/sample-data has extension .grib2:
"tests/sample-data/tiggelam_cnmc_sfc.grib2"

Therefore this file is not included in the release tar file eccodes-1.4.0.tar.gz
and this causes the 2 failures in the provided test suite when running them using the pytest-3 command:
================================ short test summary info =================================
FAILED tests/test_eccodes.py::test_count_in_file - FileNotFoundError: [Errno 2] No such...
FAILED tests/test_eccodes.py::test_extract_offsets - gribapi.errors.IOProblemError: Inp...
======================== 2 failed, 58 passed, 5 warnings in 2.18s =============================

So please update your manifest file to solve this issue.

Writing index file twice causes `ecCodes assertion failed: `field->file'`

Ubuntu 18.04.3 LTS
eccodes 2.16.0
conda install -c conda-forge eccodes and pip install eccodes-python

If I try to write an index file off the same grib file twice:

import eccodes

grib_name = 'era5-levels-members.grib'

for i in range(2):
    index_name = grib_name + f'.idx{i}'
    with eccodes.GribIndex(grib_name, ['level']) as index:
        index.write(index_name)

    with eccodes.GribIndex(file_index=index_name) as index:
        msgs = index.select({'level': 500})  # Fails here on second run of loop.
        print(f'Using {index_name}: {len(msgs)} messages')

I get an ecCodes assertion error:

Using era5-levels-members.grib.idx0: 192 messages
ecCodes assertion failed: `field->file' in /home/conda/feedstock_root/build_artifacts/eccodes_1579074340882/work/src/grib_index.c:1470
Aborted (core dumped)

The two index files written are not identical:

cmp era5-levels-members.grib.idx0 era5-levels-members.grib.idx1
era5-levels-members.grib.idx0 era5-levels-members.grib.idx1 differ: byte 36, line 1

However, if I replace index.write with a command line call to grib_index_build, it works:

import subprocess
import eccodes

grib_name = 'era5-levels-members.grib'

for i in range(2):
    index_name = grib_name + f'.idx{i}'
    subprocess.call(f'grib_index_build -o {index_name} -k level {grib_name}'.split(' '))

    with eccodes.GribIndex(file_index=index_name) as index:
        msgs = index.select({'level': 500})  # Fails here on second run of loop.
        print(f'Using {index_name}: {len(msgs)} messages')

Output:

--- grib_index_build: processing era5-levels-members.grib
--- grib_index_build: keys included in the index file era5-levels-members.grib.idx0:
--- level
--- level = { 500, 850 }
--- 160 messages indexed
Using era5-levels-members.grib.idx0: 192 messages
--- grib_index_build: processing era5-levels-members.grib
--- grib_index_build: keys included in the index file era5-levels-members.grib.idx1:
--- level
--- level = { 500, 850 }
--- 160 messages indexed
Using era5-levels-members.grib.idx1: 192 messages

Seems like some state is being retained incorrectly after the first call?

Downloading python-eccodes via Anaconda Cloud downgrades eccodes to 2.16.0

Hello,

If I want to install python-eccodes via Anaconda Cloud (conda install -c conda-forge python-eccodes) it downgrades my eccodes 2.17.0 to 2.16.0 even though eccodes got updated on Anaconda Cloud to 2.17.0 and I get this Selfcheck runtime warning and cannot work in my docker container. Would be nice if this gets fixed or shall I do a workaround?

Thanks for help!

Kind Regards,
Marko

ecmwf / eccodes-python Goto Github PK

eccodes-python's People

Contributors

Stargazers

Watchers

Forkers

eccodes-python's Issues

packages in environment at /Users/pletzera/miniconda3/envs/eccodes:

Name Version Build Channel

Recommend Projects

Recommend Topics

Recommend Org