jdkloe/pybufr-ecmwf

a python module that allows reading and writing BUFR formatted files, where BUFR stands for Binary Universal Form for the Representation of meteorological data.

pybufr-ecmwf's Issues

Big count of used memory

Library use many RAM. Sometimes server crashed, when library use more 6 Gb, it's server ram limit. Is it normal for a library to consume so many resources or can it be somehow fix?

backend_1          | 2019-04-16T10:23:13.113753044Z     from pybufr_ecmwf.bufr import BUFRReader
backend_1          | 2019-04-16T10:23:13.113760127Z   File "/usr/local/lib/python3.7/site-packages/pybufr_ecmwf/bufr.py", line 40, in <module>
backend_1          | 2019-04-16T10:23:13.113765844Z     from .bufr_interface_ecmwf import BUFRInterfaceECMWF
backend_1          | 2019-04-16T10:23:13.113778707Z   File "/usr/local/lib/python3.7/site-packages/pybufr_ecmwf/bufr_interface_ecmwf.py", line 52, in <module>
backend_1          | 2019-04-16T10:23:13.113787664Z     raise e
backend_1          | 2019-04-16T10:23:13.113795684Z   File "/usr/local/lib/python3.7/site-packages/pybufr_ecmwf/bufr_interface_ecmwf.py", line 48, in <module>
backend_1          | 2019-04-16T10:23:13.113804747Z     from . import ecmwfbufr
backend_1          | 2019-04-16T10:23:13.113813227Z ImportError: Error loading shared library /usr/local/lib/python3.7/site-packages/pybufr_ecmwf/ecmwfbufr.cpython-37m-x86_64-linux-gnu.so: Out of memory

Add even more convenient read only interface

We've got a BUFRReader class based on pybufr-ecmwf that makes it fairly straightforward to read BUFR files.

example usage

latitude = []
longitude = []

with bufr_reader.BUFRReader(self.filename) as bufr:
    for message in bufr.messages():

        latitude.append(message[:, 12])
        longitude.append(message[:, 13])

You can find the source code here. It uses the guessing of the array size as you have explained it in one of the examples. At the moment it is only tested on BUFR files we are actively using like the ASCAT Level-2 BUFR files from EUMETSAT.

Would you be interested to adding this to the library? I'm trying to focus our packages on their core functionality and move non-core functionality into appropriate packages.

If so then I can open a PR and we can talk about possible improvements.

BUFRReaderBUFRDC.get_value() method - incorrect values

Hello,

I have a Bufr file from which I want to pick out the values of a specific descriptor. However, it appears the values look wrong. In order for this problem to be reproduced I have uploaded a sample data file in the following anonymous FTP server,

Name of FTP server: ftp.bom.gov.au
Login name: anonymous
Password: (not required)
Location of Bufr file: /anon/home/cawcr/10day/jinlee/HIMCSR_1.bufr

Following is the set of commands I used to pick the values of the descriptor number, 96 ('BRIGHTNESS TEMPERATURE') from the message=2 and subsets=1, 2, etc.,

bfr=BUFRReader('HIMCSR_1.bufr')
bfr.get_next_msg()
bfr.get_next_msg()
bfr.get_value(96,1)

• bfr.get_value(96,1) returns 1.7e+38 and this is correct.
• bfr.get_value(96,2) returns 291.30000000000001 but the value should be missing
• bfr.get_value(96,3) returns 292.19999999999999 but the value should be missing
• etc.

I used ECMWF's Metview as an independent check on the values. I checked subsets of other messages and there there doesn't appear to be any discrepancy between the values printed out by get_value() and Metview. So the problem seems to be for message=2 and some of its subsets.

If you can have a look I would appreciate it.

Regards,

Jin

Pip install does not use symlinks for bufrtables if wheels are enabled

Running

pip install pybufr-ecmwf does not use symlinks for the BUFR tables.

This leads to 1.5 GB of used space for each environment in which it is installed in this way.

RADAR RAINFALL INTENSITY

Hi. I have a question, how calculated RADAR RAINFALL INTENSITY and always it have m/s as units? May be it's mm/h?

Entering installation sequence?

when i installed pybufr-ecmwf using python setup.py install command for tar.gz

Entering installation sequence：

what is the installation sequence?

cannot import name 'BUFRWriter'

I was reading the USAGE.md file for examples how to use this excellent software. However, I can not import BUFRWriter, but I am able to import BUFRReader. Is the documentation outdated or is this a bug?

`>>> from pybufr_ecmwf.bufr import BUFRWriter
Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name 'BUFRWriter'

from pybufr_ecmwf.bufr import BUFRReader
`

bfr.bufr_obj.print_sections_012() - return values

Hello,

I am not sure whether this is the right place to ask a question. Please point me to the right place if this is not an appropriate use of 'Issues'.

I have started using pybufr-ecmwf. I would like to print out sections 0, 1 and 2 of a Bufr message using "bfr.bufr_obj.print_sections_012()". However the method doesn't seem to return anything. I've tried other similar methods but without any success. Can you please advise me how I can assign the printed return values from "bfr.bufr_obj.print_sections_012()" to a variable?

Thank you.

Regards,

Jin

Symbol not found: _pbbufr_ (Mac OS Mojave)

First of all, Thank you for your valuable work, I always use your library to decode BUFR.

My Problem happened after i upgraded my OS to Mojave, at first a noticed that the library stopped importing correctly, producing this error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-a9afcbee3214> in <module>()
      9 import skimage.measure
     10 from pybufr_ecmwf.raw_bufr_file import RawBUFRFile
---> 11 from pybufr_ecmwf.bufr_interface_ecmwf import BUFRInterfaceECMWF
     12 import scipy
     13 import sklearn

~/.local/lib/python3.6/site-packages/pybufr_ecmwf/bufr_interface_ecmwf.py in <module>()
     50     print('import of ecmwfbufr failed.')
     51     print('sys.path = ', sys.path)
---> 52     raise e
     53 
     54 from . import ecmwfbufr_parameters

~/.local/lib/python3.6/site-packages/pybufr_ecmwf/bufr_interface_ecmwf.py in <module>()
     46 # import the raw wrapper interface to the ECMWF BUFR library
     47 try:
---> 48     from . import ecmwfbufr
     49 except ImportError as e:
     50     print('import of ecmwfbufr failed.')

ImportError: dlopen(/Users/mohamedakramzaytar/.local/lib/python3.6/site-packages/pybufr_ecmwf/ecmwfbufr.cpython-36m-darwin.so, 2): Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgfortran.4.dylib
  Referenced from: /Users/mohamedakramzaytar/.local/lib/python3.6/site-packages/pybufr_ecmwf/ecmwfbufr.cpython-36m-darwin.so
  Reason: image not found

I then proceeded to reinstall it following the usual steps described in here, it gave back this error:

a python wrapper to the ECMWF BUFR library has been generated
compilation of library wrapper finished
extracting library constants
Traceback (most recent call last):
  File "setup.py", line 414, in <module>
    provides=["pybufr_ecmwf"])
  File "/anaconda3/envs/indaba/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/anaconda3/envs/indaba/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/anaconda3/envs/indaba/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "setup.py", line 123, in run
    _build.run(self)
  File "/anaconda3/envs/indaba/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/anaconda3/envs/indaba/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/anaconda3/envs/indaba/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/anaconda3/envs/indaba/lib/python3.6/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/anaconda3/envs/indaba/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
    self._build_extensions_serial()
  File "/anaconda3/envs/indaba/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 322, in build_extension
    ibi.build()
  File "/Users/mohamedakramzaytar/libs/Downloads/pybufr-ecmwf/build_interface.py", line 893, in build
    self.extract_constants()
  File "/Users/mohamedakramzaytar/libs/Downloads/pybufr-ecmwf/build_interface.py", line 1985, in extract_constants
    import ecmwfbufr
ImportError: dlopen(/Users/mohamedakramzaytar/libs/Downloads/pybufr-ecmwf/build/lib.macosx-10.7-x86_64-3.6/pybufr_ecmwf/ecmwfbufr.cpython-36m-darwin.so, 2): Symbol not found: _pbbufr_
  Referenced from: /Users/mohamedakramzaytar/libs/Downloads/pybufr-ecmwf/build/lib.macosx-10.7-x86_64-3.6/pybufr_ecmwf/ecmwfbufr.cpython-36m-darwin.so
  Expected in: flat namespace
 in /Users/mohamedakramzaytar/libs/Downloads/pybufr-ecmwf/build/lib.macosx-10.7-x86_64-3.6/pybufr_ecmwf/ecmwfbufr.cpython-36m-darwin.so

After running tests:

Ran 65 tests in 6.096s

FAILED (failures=32, errors=31)
removing symlink pybufr_ecmwf
renaming pybufr_ecmwf.renamed to pybufr_ecmwf
just before exit: success =  False

Is there a way to fix this ?

Python 3.8 support

Are there any plans to support Python versions > 3.6?

At the moment installations via conda fails with e.g. Python 3.8

problem with code generating bad files

Hi-

We have been using your library to generate wind profiler BUFR products for a couple of years on one of our production Linux boxes (CentOS) and this has been working well. Thank you for a great product.

The original developer of this code is no longer available, and I'm trying to do some new development and have installed the library on a new machine (also CentOS). However, when I run the same script on the new machine the values for every level after the first are bad (very large values). It's like there is a memory or pointer issue when the data are either packed into cvalues. I have tried the pip install of version 0.82, the anaconda install and have built/installed it from scratch - all with the same result. Have you run into this issue before?

I'm attaching the script, the raw data used to create the BURF file, the "bad" bufr file and a good version produced on the production machine. Unzip them into a directory, cd there and you can run the script as:

./EncodeWindProfilerBUFR.py oth 74994 oth18067.16w . 520461292

I would appreciate any insight you can provide.

Thanks again for a great product.

badbufr.zip

Don Murray

get_next_msg() failure

Hello,

I am trying to read Bufr messages in a Bufr file which contains IASI data. I am doing repeated "bfr.get_next_msg()" call but for some reason at 3000th Bufr message "bfr.get_next_msg()" fails with the following error message,

/home/548/jtl548/da/ops/bufr/bufrexplore/tst.py in ()
4
5 for i in range(bfr.num_msgs-1):
----> 6 bfr.get_next_msg()
7 print bfr.msg_loaded

/projects/access/da/utilities/bufr/pybufr-ecmwf/install/lib/python2.7/site-packages/pybufr_ecmwf/bufr.pyc in get_next_msg(self)
183 self.bufr_obj.setup_tables(self.table_b_to_use, self.table_c_to_use,
184 self.table_d_to_use, self.tables_dir)
--> 185 self.bufr_obj.decode_data()
186
187 #nsub = int(self.bufr_obj.get_num_subsets())

/projects/access/da/utilities/bufr/pybufr-ecmwf/install/lib/python2.7/site-packages/pybufr_ecmwf/bufr_interface_ecmwf.pyc in decode_data(self)
1023 lines = self.get_fortran_stdout()
1024 self.display_fortran_stdout(lines)
-> 1025 raise e
1026 else:
1027 if self.verbose:

EcmwfBufrLibError: Sorry, call to bufrex failed, reported fortran error(s)"

If you would like to have a look I have uploaded a copy of the Bufr file on our anonymous ftp server,

ftp server: ftp.bom.gov.au
user: anonymous
directory: /anon/home/cawcr/10day/jinlee/IASIG_1.bufr

Any help would be greatly appreciated.

Regards,

Jin

rework high level interface

The current message iteration for reading works like this:

from pybufr_ecmwf.bufr import BUFRReader
with BUFRReader(input_bufr_file) as bufr:
    for data, names, units in bufr.messages():
        <do something>

(thanks to @cpaulik).

However, I also wish to support a higher level interface for writing. This includes exposing the message headers as well, and make them writable. Therefore I propose to change the interface to something like this:

from pybufr_ecmwf.bufr import BUFRReader
with BUFRReader(input_bufr_file) as bufr:
    for msg in bufr:
        data, names, units = msg.data, msg.names, msg.units
        <do something>

For writing I could then implement something like:

from pybufr_ecmwf.bufr import BUFRWriter
with BUFRWriter(output_bufr_file) as bufr:
    for data in array2d:
       msg = bufr.new()
       msg.headers = <some settings>
       msg.template = <choose template>
       msg.data = data

and they could be combined like this:

from pybufr_ecmwf.bufr import BUFRReader, BUFRWriter
with BUFRReader(input_bufr_file) as bufr_in:
    with BUFRWriter(output_bufr_file) as bufr_out:
       for msg_in in bufr_in:
           msg_out = bufr_out.newmsg()
           msg_out.template = msg_in.template
           headers = msg_in.headers
           <do something to headers>
           msg_out.headers = headers
           data = msg_in.data
           <do something to data>
           msg_out.data = data

So this question is especially directed to @cpaulik but also other users may respond:
would this be acceptable to you? Or do you think there is another, more convenient way to do these tasks?

documentation is outdated

user documentation needs to be rewritten (see comments in issue#6).

2 error when install pybufr-ecmwf on ubuntu

Executing command: python3 ./run_f2py_tool.py --build-dir f2py_build --f90flags='-fno-second-underscore -O -Dlinux -fPIC' --f77flags='-fno-second-underscore -O -Dlinux -fPIC' --fcompiler=gfortran ./f2py_build/signatures.pyf -L./ -lbufr -c 2>1 >> f2py_build.log
ERROR: build of python wrapper failed
the compilation or linking stage failed
..................................................................
..................................................................
..........................................................
Executing command: python3 ./run_f2py_tool.py --build-dir f2py_build --f90flags='-fno-second-underscore -O -Dlinux -fPIC' --f77flags='-fno-second-underscore -O -Dlinux -fPIC' --fcompiler=gfortran ./f2py_build/signatures.pyf -L./ -lbufr -c 2>1 >> f2py_build.log
ERROR: build of python wrapper failed
the compilation or linking stage failed
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/setup.py", line 410, in
provides=["pybufr_ecmwf"])
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/zhangqw/anaconda3/lib/python3.5/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/install.py", line 539, in run
self.run_command('build')
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/setup.py", line 119, in run
_build.run(self)
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/setup.py", line 318, in build_extension
ibi.build()
File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/build_interface.py", line 878, in build
self.generate_python_wrapper(source_dir)
File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/build_interface.py", line 1856, in generate_python_wrapper
raise InterfaceBuildError
pybufr_ecmwf.custom_exceptions.InterfaceBuildError

----------------------------------------

Command "/home/zhangqw/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-3dy3x23o/pybufr-ecmwf/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-ucj91k6_-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-3dy3x23o/pybufr-ecmwf/

Fortran error

Hi, I tryed read the bufr and writing data in arrays. Early this method work, but now that retur error: Sorry, call to bufrex failed, reported fortran error(s)". And when I call get_next_msg() method I get EOFError.
Why I do wrong, or how this fix?

    bufr = BUFRReader(path_to_file)

    for msg in bufr:
        for msg_or_subset_data in msg:
            all_data = np.dstack((msg_or_subset_data.names,
                                  msg_or_subset_data.data,
                                  msg_or_subset_data.units))[0]

ModuleNotFoundError: No module named 'ecmwfbufr'

In our internal continuous integration tests we experience occasionally the error ModuleNotFoundError: No module named 'ecmwfbufr'. We install pybufr-ecmwf via pip on a linux VM. However, sometimes it runs through without any exception - with the same settings though.

Full stack trace:

     Complete output (129 lines):
     running install
     running build
     build: self.user_options =  [('build-base=', 'b', 'base directory for build library'), ('build-purelib=', None, 'build directory for platform-neutral distributions'), ('build-platlib=', None, 'build directory for platform-specific distributions'), ('build-lib=', None, 'build directory for all distribution (defaults to either build-purelib or build-platlib'), ('build-scripts=', None, 'build directory for scripts'), ('build-temp=', 't', 'temporary build directory'), ('plat-name=', 'p', 'platform name to build for, if supported (default: linux-x86_64)'), ('compiler=', 'c', 'specify the compiler type'), ('parallel=', 'j', 'number of parallel build jobs'), ('debug', 'g', 'compile extensions and libraries with debugging information'), ('force', 'f', 'forcibly build everything (ignore file timestamps)'), ('executable=', 'e', 'specify final destination interpreter path (build.py)'), ('preferred-fortran-compiler=', None, 'name of preferred fortran compiler to be used'), ('preferred-c-compiler=', None, 'name of preferred c compiler to be used'), ('fortran-compiler=', None, 'name and full path of fortran compiler to be used'), ('fortran-ld-library-path=', None, 'path in which shared objects can be found that are needed by the choosen fortran compiler'), ('fortran-flags=', None, 'flags to be passed to the fortran compiler'), ('c-compiler=', None, 'name and full path of c compiler to be used'), ('c-ld-library-path=', None, 'path in which shared objects can be found that are needed by the choosen c compiler'), ('c-flags=', None, 'flags to be passed to the c compiler')]
     build: self.preferred_fortran_compiler =  gfortran
     build: self.preferred_c_compiler =  gcc
     running build_py
     creating build
     creating build/lib.linux-x86_64-3.6
     creating build/lib.linux-x86_64-3.6/pybufr_ecmwf
     copying pybufr_ecmwf/__init__.py -> build/lib.linux-x86_64-3.6/pybufr_ecmwf
     copying pybufr_ecmwf/bufr.py -> build/lib.linux-x86_64-3.6/pybufr_ecmwf
     copying pybufr_ecmwf/raw_bufr_file.py -> build/lib.linux-x86_64-3.6/pybufr_ecmwf
     copying pybufr_ecmwf/bufr_interface_ecmwf.py -> build/lib.linux-x86_64-3.6/pybufr_ecmwf
     copying pybufr_ecmwf/run_f2py_tool.py -> build/lib.linux-x86_64-3.6/pybufr_ecmwf
     copying pybufr_ecmwf/custom_exceptions.py -> build/lib.linux-x86_64-3.6/pybufr_ecmwf
     copying pybufr_ecmwf/bufr_table.py -> build/lib.linux-x86_64-3.6/pybufr_ecmwf
     copying pybufr_ecmwf/bufr_template.py -> build/lib.linux-x86_64-3.6/pybufr_ecmwf
     copying pybufr_ecmwf/helpers.py -> build/lib.linux-x86_64-3.6/pybufr_ecmwf
     running egg_info
     writing pybufr_ecmwf.egg-info/PKG-INFO
     writing dependency_links to pybufr_ecmwf.egg-info/dependency_links.txt
     writing top-level names to pybufr_ecmwf.egg-info/top_level.txt
     reading manifest file 'pybufr_ecmwf.egg-info/SOURCES.txt'
     writing manifest file 'pybufr_ecmwf.egg-info/SOURCES.txt'
     creating build/lib.linux-x86_64-3.6/pybufr_ecmwf/alt_bufr_tables
     copying pybufr_ecmwf/alt_bufr_tables/C0000000000098015001.TXT -> build/lib.linux-x86_64-3.6/pybufr_ecmwf/alt_bufr_tables
     copying pybufr_ecmwf/alt_bufr_tables/D0000000000098015001.TXT -> build/lib.linux-x86_64-3.6/pybufr_ecmwf/alt_bufr_tables
     copying pybufr_ecmwf/alt_bufr_tables/B0000000000098015001.TXT -> build/lib.linux-x86_64-3.6/pybufr_ecmwf/alt_bufr_tables
     copying pybufr_ecmwf/alt_bufr_tables/GENERIC_SCAT_BUFR_TABLE_C.TXT -> build/lib.linux-x86_64-3.6/pybufr_ecmwf/alt_bufr_tables
     copying pybufr_ecmwf/alt_bufr_tables/GENERIC_SCAT_BUFR_TABLE_D.TXT -> build/lib.linux-x86_64-3.6/pybufr_ecmwf/alt_bufr_tables
     copying pybufr_ecmwf/alt_bufr_tables/GENERIC_SCAT_BUFR_TABLE_B.TXT -> build/lib.linux-x86_64-3.6/pybufr_ecmwf/alt_bufr_tables
     running build_ext
     building 'pybufr_ecmwf.ecmwfbufr' extension
     initiating build in dir:  /tmp/pip-install-uy1hgito/pybufr-ecmwf/build/lib.linux-x86_64-3.6/pybufr_ecmwf
     base_build_dir =  /tmp/pip-install-uy1hgito/pybufr-ecmwf
     Entering installation sequence:
     Using bundled library copy...
     Executing command:  cp /tmp/pip-install-uy1hgito/pybufr-ecmwf/ecmwf_bufr_lib_sources/bufrdc_000409.tar.gz ./ecmwf_bufr_lib
     Executing command:  cd ./ecmwf_bufr_lib;tar zxf bufrdc_000409.tar.gz
     copied file: handle_stdout.F
     copied file: retrieve_settings.F
     copied file: set_nokey.F
     added file handle_stdout.F to the sources list
     added file retrieve_settings.F to the sources list
     added file set_nokey.F to the sources list
     selection fortran compiler
     ==>input: self.fortran_compiler =  None
     ==>input: self.preferred_fortran_compiler =  gfortran
     Executing command:  which gfortran
     selection fortran compiler
     ==>result: self.fortran_compiler_to_use =  gfortran
     selection c compiler
     ==>input: self.c_compiler =  None
     ==>input: self.preferred_c_compiler =  gcc
     Executing command:  which gcc
     selection c compiler
     ==>result: self.c_compiler_to_use =  gcc
     Using: gfortran as fortran compiler
     Using: gcc as c compiler
     Creating ECMWF-BUFR config file:  ./ecmwf_bufr_lib/bufrdc_000409/config/config.linux_compiler
     creating:  ./ecmwf_bufr_lib/bufrdc_000409/./Makefile
     creating:  ./ecmwf_bufr_lib/bufrdc_000409/bufrdc/Makefile
     creating:  ./ecmwf_bufr_lib/bufrdc_000409/bufrtables/Makefile
     creating:  ./ecmwf_bufr_lib/bufrdc_000409/pbio/Makefile
     creating:  ./ecmwf_bufr_lib/bufrdc_000409/fortranC/Makefile
     creating:  ./ecmwf_bufr_lib/bufrdc_000409/examples/Makefile
     Executing command:  gfortran -fno-second-underscore -O -Dlinux -fPIC -o pybufr_fortran_test_program pybufr_fortran_test_program.F90
     Executing command:  ./pybufr_fortran_test_program
     Fortran compilation test successfull...
     Executing command:  gcc -O -fPIC -o pybufr_c_test_program pybufr_c_test_program.c
     Executing command:  ./pybufr_c_test_program
     c compilation test successfull...
     Executing command:  gcc -O -fPIC -o GetByteSizeInt GetByteSizeInt.c
     Executing command:  ./GetByteSizeInt
     Executing command:  gcc -O -fPIC -o GetByteSizeLong GetByteSizeLong.c
     Executing command:  ./GetByteSizeLong
     Executing command:  gfortran -fno-second-underscore -O -Dlinux -fPIC -o GetByteSizeDefaultInteger GetByteSizeDefaultInteger.F90
     Executing command:  ./GetByteSizeDefaultInteger
     Executing command:  cd ./ecmwf_bufr_lib/bufrdc_000409;make ARCH=linux CNAME=_compiler R64= A64= 2>1 > bufrdc_build.log
     Build seems successfull
     compilation of BUFR library finished
     Entering wrapper generation sequence:
     Executing command:  python3 ./run_f2py_tool.py --overwrite-signature  --build-dir f2py_build -m ecmwfbufr -h signatures.pyf ./ecmwf_bufr_lib/bufrdc_000409/bufrdc/*.F ./ecmwf_bufr_lib/bufrdc_000409/pbio/pbbufr.F 2>1 > f2py_build.log
     Fixing array size definitions in signatures definition ...
     Using intlen =  8  to build the pbio interface
     Inserting hardcoded interface to pbio routines in signatures file ...
     Executing command:  python3 ./run_f2py_tool.py --build-dir f2py_build --f90flags='-fno-second-underscore -O -Dlinux -fPIC' --f77flags='-fno-second-underscore -O -Dlinux -fPIC' --fcompiler=gfortran ./f2py_build/signatures.pyf -L./ -lbufr -c 2>1 >> f2py_build.log
     a python wrapper to the ECMWF BUFR library has been generated
     compilation of library wrapper finished
     extracting library constants
     Traceback (most recent call last):
       File "<string>", line 1, in <module>
       File "/tmp/pip-install-uy1hgito/pybufr-ecmwf/setup.py", line 410, in <module>
         provides=["pybufr_ecmwf"])
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/core.py", line 148, in setup
         dist.run_commands()
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/dist.py", line 955, in run_commands
         self.run_command(cmd)
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/dist.py", line 974, in run_command
         cmd_obj.run()
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
         return orig.install.run(self)
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/command/install.py", line 545, in run
         self.run_command('build')
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/cmd.py", line 313, in run_command
         self.distribution.run_command(command)
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/dist.py", line 974, in run_command
         cmd_obj.run()
       File "/tmp/pip-install-uy1hgito/pybufr-ecmwf/setup.py", line 119, in run
         _build.run(self)
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/command/build.py", line 135, in run
         self.run_command(cmd_name)
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/cmd.py", line 313, in run_command
         self.distribution.run_command(command)
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/dist.py", line 974, in run_command
         cmd_obj.run()
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/command/build_ext.py", line 339, in run
         self.build_extensions()
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
         self._build_extensions_serial()
       File "/root/miniconda/envs/warp_h_nrt_env/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
         self.build_extension(ext)
       File "/tmp/pip-install-uy1hgito/pybufr-ecmwf/setup.py", line 318, in build_extension
         ibi.build()
       File "/tmp/pip-install-uy1hgito/pybufr-ecmwf/build_interface.py", line 888, in build
         self.extract_constants()
       File "/tmp/pip-install-uy1hgito/pybufr-ecmwf/build_interface.py", line 1874, in extract_constants
         import ecmwfbufr
     ModuleNotFoundError: No module named 'ecmwfbufr'
     ----------------------------------------
 ERROR: Command errored out with exit status 1: /root/miniconda/envs/warp_h_nrt_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-uy1hgito/pybufr-ecmwf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-uy1hgito/pybufr-ecmwf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-937_0s73/install-record.txt --single-version-externally-managed --compile --install-headers /root/miniconda/envs/warp_h_nrt_env/include/python3.6m/pybufr-ecmwf Check the logs for full command output.

Error in descr = self.table_b[int_descr] to ECCODES ERROR : unable to get descriptor Y from table

Hello,

I wanted to dump the content of a BUFR file using the file provided in the library "bufr_to_ascii.py", and i got the following error :

Traceback (most recent call last):
 File "org_bufr_to_csv.py", line 571, in <module>
   main()
 File "org_bufr_to_csv.py", line 540, in main
   separator, max_msg_nr, expand_flags)
 File "org_bufr_to_csv.py", line 58, in print_bufr_content1
   for msg_nr, msg in enumerate(bufr):
 File "/usr/local/lib/python2.7/dist-packages/pybufr_ecmwf/bufr.py", line 399, in messages
   self.get_next_msg()
 File "/usr/local/lib/python2.7/dist-packages/pybufr_ecmwf/bufr.py", line 378, in get_next_msg
   self.table_d_to_use, self.tables_dir)
 File "/usr/local/lib/python2.7/dist-packages/pybufr_ecmwf/bufr.py", line 91, in __init__
   self._bufr_obj.decode_data()
 File "/usr/local/lib/python2.7/dist-packages/pybufr_ecmwf/bufr_interface_ecmwf.py", line 964, in decode_data
   self.expand_raw_descriptor_list()
 File "/usr/local/lib/python2.7/dist-packages/pybufr_ecmwf/bufr_interface_ecmwf.py", line 1651, in expand_raw_descriptor_list
   self.bt.expand_descriptor_list(self.py_unexp_descr_list)
 File "/usr/local/lib/python2.7/dist-packages/pybufr_ecmwf/bufr_table.py", line 851, in expand_descriptor_list
   normalised_descriptor_list = self.normalise_descriptor_list(descr_list)
 File "/usr/local/lib/python2.7/dist-packages/pybufr_ecmwf/bufr_table.py", line 821, in normalise_descriptor_list
   descr = self.table_b[int_descr]
KeyError: 40239

After getting this error i thought i should try installing and dumping The BUFR using the ECCODES Library directly using the bufr_dump command, but then again i got a similar error :

{ "messages" : [ 
ECCODES ERROR   :  unable to get descriptor 040239 from table
ECCODES ERROR   :  unable to get descriptor 040232 from table
ECCODES ERROR   :  unable to get descriptor 040237 from table
ECCODES ERROR   :  unable to get descriptor 040231 from table
ECCODES ERROR   :  unable to get descriptor 040233 from table
ECCODES ERROR   :  unable to get descriptor 040235 from table
ECCODES ERROR   :  unable to get descriptor 040236 from table
ECCODES ERROR   :  unable to get descriptor 040197 from table
ECCODES ERROR   :  unable to get descriptor 040240 from table
ECCODES ERROR   :  unable to get descriptor 040220 from table
ECCODES ERROR   :  unable to get descriptor 040221 from table
ECCODES ERROR   :  unable to get descriptor 040223 from table
ECCODES ERROR   :  unable to get descriptor 010220 from table
ERROR: unable to unpack data section: Function not yet implemented (message=1)

I guess the error is related to my file but i know how, i am new to BUFR and i don't know how to read or debug related errors, any hint or word of advice would be much appreciated.

Thank you.