Giter Site home page Giter Site logo

ncar / pynio Goto Github PK

View Code? Open in Web Editor NEW
111.0 27.0 37.0 13.14 MB

PyNIO is a multi-format data I/O package with a NetCDF-style interface

Home Page: http://www.pyngl.ucar.edu/Nio.shtml

License: Apache License 2.0

Python 5.39% C 91.47% Fortran 3.10% Shell 0.05%

pynio's Introduction

PyNIO

Warning

PyNIO was placed in maintenance mode as of November of 2020. You can find more details in the full announcement here.

PyNIO ("pie-nee-oh") is a Python module that allows read and/or write access to a variety of scientific data formats popular in climate and weather:

  • NetCDF3 / NetCDF4
  • GRIB1 / GRIB2
  • HDF4
  • HDF-EOS2
  • HDF-EOS5
  • Shapefile

PyNIO has a data model interface that is modeled on the NetCDF data model.

PyNIO is developed by the Computational and Information Systems Lab at the National Center for Atmospheric Research (NCAR).

NCAR is sponsored by the National Science Foundation. Any opinions, findings and conclusions or recommendations expressed in this material do not necessarily reflect the views of the National Science Foundation.

Installation

PyNIO can be installed on Linux and MacOS systems via conda:

conda create --name pyn_env --channel conda-forge pynio
source activate pyn_env

Documentation and support

PyNIO is currently documented as part of the PyNGL website.

Bug reports and feature requests

Use the GitHub issue tracker to submit a bug or request.

pynio's People

Contributors

david-ian-brown avatar kafitzgerald avatar khallock avatar marylhaley avatar shoyer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pynio's Issues

How to disable debug ?

Hello !

When I'm using pynio to read a CDF file, I have debug like this :
nc_inq_attname: ncid 0x10000 varid -1 attnum 2
each time I use a data from the numpy arrays (created from the file)

How can I disable theses debugs ?

Thank you

does pynio & pyngl support Python3.x on win10

As title, I used conda create --name pyn_env --channel conda-forge pynio pyngl to install them.
But I just saw

Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  • pyngl
  • pynio

Not a valid GRIB2 code table

Hello

When I use PyNIO to process some gfs files, it gave such an warning:

2016-08-29 11:02:21,618 - INFO    PID: 31442 - Processing file gfs_3_20131226_1200_036.grb2
warning: Entry (224) not found in code table file /users/is/jackson/pyenvs/hello/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.2.table
warning: NclGRIB2: codetable file "/users/is/jackson/pyenvs/hello/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.2.4.table" not a valid GRIB2 code table

It seems the process cannot find a valid GRIB2 code table.

Can I know how to solve this problem?

Can't open a .nc file

f = Nio.open_file(ent_dir+'/merged/'+j+'.nc', 'c')
File "/public/home/soft/anaconda3/envs/meic2wrf/lib/python3.8/site-packages/PyNIO/Nio.py", line751, in open_file
file = _nio.open_file(filename,mode,options,history,format)
Nio.NIOError: Unable to open file

The .nc file is exist.

Towards a Python 3 compatible release

I'm wondering what the timetable is to making a full PyNio release with Python 3 support? At the end of #10, @bladwig1 suggested:

an official release on conda-forge won't take place until a large amount of bug fixes related to compound data types, HDF5, etc are included.

I don't see any open issues related to HDF5 and compound data type bugs that are open right now so its hard to gauge how that progress is coming.

For context, Xarray, which supports PyNio as a backend is dropping Python 2 support at the end of 2018. Ideally, we would have a Python 3 release well before then.

xref: pydata/xarray#1830, conda-forge/pynio-feedstock#23

libgdal undefined symbol

Just installed pynio/pyngl via conda, then activated (used conda activate instead of source activate, but should be the same)

Ubuntu Linux: Linux gose 4.18.0-13-generic #14-Ubuntu SMP Wed Dec 5 09:04:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

conda create --name pyn_env --channel conda-forge pynio pyngl
source activate pyn_env

Then:
(pyn_env) havenerk@gose:~$ python
Python 3.6.7 | packaged by conda-forge | (default, Nov 21 2018, 03:09:43)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import Nio
Traceback (most recent call last):
File "", line 1, in
File "/home/havenerk/miniconda3/envs/pyn_env/lib/python3.6/site-packages/PyNIO/Nio.py", line 83, in
from _nio import *
ImportError: /home/havenerk/miniconda3/envs/pyn_env/lib/python3.6/site-packages/PyNIO/../../../libgdal.so.20: undefined symbol: _ZN6libdap3DDSC1EPNS_15BaseTypeFactoryERKSs

What do I do now? Seems to be a conda/gdal issue, and the internet is all over the place about how to work around it.

Conda list reveals:

pyn_env) havenerk@gose:~$ conda list

packages in environment at /home/havenerk/miniconda3/envs/pyn_env:

Name Version Build Channel

blas 1.1 openblas conda-forge
boost-cpp 1.68.0 h11c811c_1000 conda-forge
bzip2 1.0.6 h14c3975_1002 conda-forge
ca-certificates 2018.11.29 ha4d7672_0 conda-forge
cairo 1.14.12 h80bd089_1005 conda-forge
certifi 2018.11.29 py36_1000 conda-forge
curl 7.63.0 h646f8bb_1000 conda-forge
esmf 7.1.0r ha678704_2 conda-forge
expat 2.2.5 hf484d3e_1002 conda-forge
fontconfig 2.13.1 h2176d3f_1000 conda-forge
freetype 2.9.1 h94bbf69_1005 conda-forge
freexl 1.0.5 h14c3975_1002 conda-forge
g2clib 1.6.0 3 conda-forge
geos 3.6.2 hfc679d8_4 conda-forge
geotiff 1.4.2 hfe6da40_1005 conda-forge
gettext 0.19.8.1 h9745a5d_1001 conda-forge
giflib 5.1.4 h14c3975_1001 conda-forge
glib 2.56.2 had28632_1001 conda-forge
gsl 2.2.1 blas_openblashddceaf2_6 [blas_openblas] conda-forge
hdf4 4.2.13 h9a582f1_1002 conda-forge
hdf5 1.10.2 hc401514_3 conda-forge
hdfeos2 2.20 h7a90ae3_0 conda-forge
hdfeos5 5.1.16 h647bee3_2 conda-forge
icu 58.2 hf484d3e_1000 conda-forge
jasper 1.900.1 h07fcdf6_1005 conda-forge
jpeg 9c h14c3975_1001 conda-forge
json-c 0.12.1 h470a237_1 conda-forge
kealib 1.4.9 h0bee7d0_2 conda-forge
krb5 1.16.3 hc83ff2d_1000 conda-forge
libcurl 7.63.0 h01ee5af_1000 conda-forge
libdap4 3.19.1 hd48c02d_1000 conda-forge
libedit 3.1.20170329 hf8c457e_1001 conda-forge
libffi 3.2.1 hf484d3e_1005 conda-forge
libgcc-ng 7.3.0 hdf63c60_0 conda-forge
libgdal 2.2.4 hbd6f514_9 conda-forge
libgfortran 3.0.0 1 conda-forge
libgfortran-ng 7.2.0 hdf63c60_3 conda-forge
libiconv 1.15 h14c3975_1004 conda-forge
libkml 1.3.0 h328b03d_1009 conda-forge
libnetcdf 4.6.1 h628ed10_200 conda-forge
libpng 1.6.36 h84994c4_1000 conda-forge
libpq 10.6 h13b8bad_1000 conda-forge
libspatialite 4.3.0a h9968ff2_1023 conda-forge
libssh2 1.8.0 h1ad7b7a_1003 conda-forge
libstdcxx-ng 7.3.0 hdf63c60_0 conda-forge
libtiff 4.0.10 h648cc4a_1001 conda-forge
libuuid 2.32.1 h14c3975_1000 conda-forge
libxcb 1.13 h14c3975_1002 conda-forge
libxml2 2.9.8 h143f9aa_1005 conda-forge
mpi 1.0 mpich conda-forge
mpich 3.2.1 h1c2f66e_1007 conda-forge
ncl 6.5.0 blas_openblashd40de8d_1 [blas_openblas] conda-forge
ncurses 6.1 hf484d3e_1002 conda-forge
netcdf-fortran 4.4.4 h4363f12_9 conda-forge
numpy 1.16.0 py36_blas_openblash1522bff_1000 [blas_openblas] conda-forge
openblas 0.3.3 h9ac9557_1001 conda-forge
openjpeg 2.3.0 hf38bd82_1003 conda-forge
openssl 1.0.2p h14c3975_1002 conda-forge
pcre 8.41 hf484d3e_1003 conda-forge
pip 18.1 py36_1000 conda-forge
pixman 0.34.0 h14c3975_1003 conda-forge
poppler 0.67.0 h2fc8fa2_1002 conda-forge
poppler-data 0.4.9 1 conda-forge
postgresql 10.6 h66cca7a_1000 conda-forge
proj4 4.9.3 h470a237_8 conda-forge
pthread-stubs 0.4 h14c3975_1001 conda-forge
pyngl 1.6.1 py36h4217dbb_0 conda-forge
pynio 1.5.4 py36h36f4ed2_0 conda-forge
python 3.6.7 hd21baee_1001 conda-forge
readline 7.0 hf8c457e_1001 conda-forge
setuptools 40.6.3 py36_0 conda-forge
sqlite 3.26.0 h67949de_1000 conda-forge
tk 8.6.9 h84994c4_1000 conda-forge
tzcode 2018g h14c3975_1001 conda-forge
udunits2 2.2.27.6 h4e0c4b3_1001 conda-forge
wheel 0.32.3 py36_0 conda-forge
xerces-c 3.2.0 h5d6a6da_2 conda-forge
xorg-imake 1.0.7 0 conda-forge
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.9 h14c3975_1004 conda-forge
xorg-libsm 1.2.3 h4937e3b_1000 conda-forge
xorg-libx11 1.6.6 h14c3975_1000 conda-forge
xorg-libxau 1.0.8 h14c3975_1006 conda-forge
xorg-libxaw 1.0.13 h14c3975_1002 conda-forge
xorg-libxdmcp 1.1.2 h14c3975_1007 conda-forge
xorg-libxext 1.3.3 h14c3975_1004 conda-forge
xorg-libxmu 1.1.2 h14c3975_1002 conda-forge
xorg-libxpm 3.5.12 h14c3975_1002 conda-forge
xorg-libxrender 0.9.10 h14c3975_1002 conda-forge
xorg-libxt 1.1.5 h14c3975_1002 conda-forge
xorg-makedepend 1.0.5 0 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h14c3975_1002 conda-forge
xorg-xproto 7.0.31 h14c3975_1007 conda-forge
xz 5.2.4 h14c3975_1001 conda-forge
zlib 1.2.11 h14c3975_1004 conda-forge

Importing Pynio image not found error

Hello,

I have installed pynio on MacOS using:

conda install -n pyn_env -c conda-forge pynio

But when I attempt to import Nio using:

import Nio, os, sys

I am getting the following error:

ImportError: dlopen(/Users/msgc/opt/miniconda3/envs/pyn_env/lib/python3.8/site-packages/PyNIO/_nio.cpython-38-darwin.so, 2): Library not loaded: @rpath/libtbb.dylib
  Referenced from: /Users/msgc/opt/miniconda3/envs/pyn_env/lib/libtiledb.dylib
  Reason: image not found

Here is the output from 'conda list':

packages in environment at /Users/msgc/opt/miniconda3/envs/pyn_env:
Name                    Version                   Build  Channel
alabaster                 0.7.12             pyhd3eb1b0_0  
appdirs                   1.4.4                      py_0  
applaunchservices         0.2.1                      py_0  
appnope                   0.1.2           py38hecd8cb5_1001  
argh                      0.26.2                   py38_0  
arrow                     0.13.1                   py38_0  
astroid                   2.5              py38hecd8cb5_1  
async_generator           1.10               pyhd3eb1b0_0  
atomicwrites              1.4.0                      py_0  
attrs                     20.3.0             pyhd3eb1b0_0  
autopep8                  1.5.6              pyhd3eb1b0_0  
babel                     2.9.0              pyhd3eb1b0_0  
backcall                  0.2.0              pyhd3eb1b0_0  
binaryornot               0.4.4              pyhd3eb1b0_1  
black                     19.10b0                    py_0  
bleach                    3.3.0              pyhd3eb1b0_0  
boost-cpp                 1.72.0               hdf9ef73_0    conda-forge
brotlipy                  0.7.0           py38h9ed2024_1003  
bzip2                     1.0.8                h0d85af4_4    conda-forge
c-ares                    1.17.1               h0d85af4_1    conda-forge
ca-certificates           2020.12.5            h033912b_0    conda-forge
cairo                     1.16.0            hec6a9b0_1003    conda-forge
certifi                   2020.12.5        py38h50d1736_1    conda-forge
cffi                      1.14.5           py38h2125817_0  
cfitsio                   3.470                h422484a_7    conda-forge
chardet                   4.0.0           py38hecd8cb5_1003  
click                     7.1.2              pyhd3eb1b0_0  
cloudpickle               1.6.0                      py_0  
colorama                  0.4.4              pyhd3eb1b0_0  
cookiecutter              1.7.2              pyhd3eb1b0_0  
cryptography              3.4.7            py38h2fd3fbb_0  
curl                      7.76.1               h06286d4_1    conda-forge
dbus                      1.13.18              h18a8e69_0  
decorator                 5.0.6              pyhd3eb1b0_0  
defusedxml                0.7.1              pyhd3eb1b0_0  
diff-match-patch          20200713                   py_0  
docutils                  0.17             py38hecd8cb5_1  
entrypoints               0.3                      py38_0  
expat                     2.3.0                h23ab428_2  
flake8                    3.9.0              pyhd3eb1b0_0  
fontconfig                2.13.1            h10f422b_1005    conda-forge
freetype                  2.10.4               h4cff582_1    conda-forge
freexl                    1.0.6                h0d85af4_0    conda-forge
future                    0.18.2                   py38_1  
g2clib                    1.6.0                h4e57d6e_9    conda-forge
geos                      3.8.1                h4a8c4bd_0    conda-forge
geotiff                   1.5.1               hd8796ba_10    conda-forge
gettext                   0.21.0               h7535e17_0  
giflib                    5.2.1                hbcb3906_2    conda-forge
glib                      2.68.1               hdf23fa2_0  
hdf4                      4.2.13            hefd3b78_1005    conda-forge
hdf5                      1.10.5          nompi_h34ad4e8_1111    conda-forge
hdfeos2                   2.20              h207910d_1000    conda-forge
hdfeos5                   5.1.16               hc5a5cdf_6    conda-forge
icu                       64.2                 h6de7cb9_1    conda-forge
idna                      2.10               pyhd3eb1b0_0  
imagesize                 1.2.0              pyhd3eb1b0_0  
importlib-metadata        3.10.0           py38hecd8cb5_0  
importlib_metadata        3.10.0               hd3eb1b0_0  
inflection                0.5.1            py38hecd8cb5_0  
intervaltree              3.1.0                      py_0  
ipykernel                 5.3.4            py38h5ca1d4c_0  
ipython                   7.22.0           py38h01d92e1_0  
ipython_genutils          0.2.0              pyhd3eb1b0_1  
isort                     5.8.0              pyhd3eb1b0_0  
jasper                    1.900.1           h636a363_1006    conda-forge
jedi                      0.17.2           py38hecd8cb5_1  
jinja2                    2.11.3             pyhd3eb1b0_0  
jinja2-time               0.2.0              pyhd3eb1b0_2  
jpeg                      9d                   hbcb3906_0    conda-forge
json-c                    0.13.1            h575e443_1002    conda-forge
jsonschema                3.2.0                      py_2  
jupyter_client            6.1.12             pyhd3eb1b0_0  
jupyter_core              4.7.1            py38hecd8cb5_0  
jupyterlab_pygments       0.1.2                      py_0  
kealib                    1.4.13               h2324030_0    conda-forge
keyring                   22.3.0           py38hecd8cb5_0  
krb5                      1.17.2               h60d9502_0    conda-forge
lazy-object-proxy         1.6.0            py38h9ed2024_0  
libblas                   3.8.0               17_openblas    conda-forge
libcblas                  3.8.0               17_openblas    conda-forge
libcurl                   7.76.1               h8ef9fac_1    conda-forge
libcxx                    11.1.0               habf9029_0    conda-forge
libdap4                   3.20.6               h3e144a0_2    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 haf1e3a3_1    conda-forge
libffi                    3.3                  hb1e8313_2  
libgdal                   3.0.4                ha92c1e4_6    conda-forge
libgfortran               4.0.0           7_5_0_h1a10cd1_22    conda-forge
libgfortran4              7.5.0               h1a10cd1_22    conda-forge
libiconv                  1.16                 h1de35cc_0  
libkml                    1.3.0             h88bc94a_1012    conda-forge
liblapack                 3.8.0               17_openblas    conda-forge
libnetcdf                 4.7.4           nompi_he461dc0_101    conda-forge
libnghttp2                1.43.0               h07e645a_0    conda-forge
libopenblas               0.3.10          openmp_h63d9170_3    conda-forge
libpng                    1.6.37               ha441bb4_0  
libpq                     12.3                 h7049927_3    conda-forge
libsodium                 1.0.18               h1de35cc_0  
libspatialindex           1.9.3                h23ab428_0  
libspatialite             4.3.0a            h658e6c1_1038    conda-forge
libssh2                   1.9.0                h52ee1ee_6    conda-forge
libtiff                   4.2.0                h355d032_0    conda-forge
libuuid                   2.32.1            h35c211d_1000    conda-forge
libwebp-base              1.2.0                h0d85af4_2    conda-forge
libxml2                   2.9.10               h53d96d6_0    conda-forge
llvm-openmp               10.0.0               h28b9765_0  
lz4-c                     1.9.2                hb1e8313_3    conda-forge
markupsafe                1.1.1            py38h1de35cc_1  
mccabe                    0.6.1                    py38_1  
mistune                   0.8.4           py38h1de35cc_1001  
mypy_extensions           0.4.3                    py38_0  
nbclient                  0.5.3              pyhd3eb1b0_0  
nbconvert                 6.0.7                    py38_0  
nbformat                  5.1.3              pyhd3eb1b0_0  
ncurses                   6.2                  h0a44026_1  
nest-asyncio              1.5.1              pyhd3eb1b0_0  
numpy                     1.20.2           py38had91d27_0    conda-forge
numpydoc                  1.1.0              pyhd3eb1b0_1  
openjpeg                  2.3.1                h6cbf5cd_3    conda-forge
openssl                   1.1.1k               h0d85af4_0    conda-forge
packaging                 20.9               pyhd3eb1b0_0  
pandoc                    2.12                 hecd8cb5_0  
pandocfilters             1.4.3            py38hecd8cb5_1  
parso                     0.7.0                      py_0  
pathspec                  0.7.0                      py_0  
pcre                      8.44                 hb1e8313_0  
pexpect                   4.8.0              pyhd3eb1b0_3  
pickleshare               0.7.5           pyhd3eb1b0_1003  
pip                       21.0.1           py38hecd8cb5_0  
pixman                    0.38.0            h01d97ff_1003    conda-forge
pluggy                    0.13.1           py38hecd8cb5_0  
poppler                   0.67.0               h16886b5_8    conda-forge
poppler-data              0.4.10                        0    conda-forge
postgresql                12.3                 he852019_3    conda-forge
poyo                      0.5.0              pyhd3eb1b0_0  
proj                      7.0.0                h45baca5_5    conda-forge
prompt-toolkit            3.0.17             pyh06a4308_0  
psutil                    5.8.0            py38h9ed2024_1  
ptyprocess                0.7.0              pyhd3eb1b0_2  
pycodestyle               2.6.0              pyhd3eb1b0_0  
pycparser                 2.20                       py_2  
pydocstyle                6.0.0              pyhd3eb1b0_0  
pyflakes                  2.2.0              pyhd3eb1b0_0  
pygments                  2.8.1              pyhd3eb1b0_0  
pylint                    2.7.4            py38hecd8cb5_1  
pyls-black                0.4.6                hd3eb1b0_0  
pyls-spyder               0.3.2              pyhd3eb1b0_0  
pynio                     1.5.5           py38h97f92f9_12    conda-forge
pyopenssl                 20.0.1             pyhd3eb1b0_1  
pyparsing                 2.4.7              pyhd3eb1b0_0  
pyqt                      5.9.2            py38h655552a_2  
pyrsistent                0.17.3           py38haf1e3a3_0  
pysocks                   1.7.1                    py38_1  
python                    3.8.8           h4e93d89_0_cpython    conda-forge
python-dateutil           2.8.1              pyhd3eb1b0_0  
python-jsonrpc-server     0.4.0                      py_0  
python-language-server    0.36.2             pyhd3eb1b0_0  
python-slugify            5.0.0              pyhd3eb1b0_0  
python.app                1.3              py38h5406a74_4    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytz                      2021.1             pyhd3eb1b0_0  
pyyaml                    5.4.1            py38h9ed2024_1  
pyzmq                     20.0.0           py38h23ab428_1  
qdarkstyle                3.0.2              pyhd3eb1b0_0  
qstylizer                 0.1.10             pyhd3eb1b0_0  
qt                        5.9.7                h8cf7e54_3    conda-forge
qtawesome                 1.0.2              pyhd3eb1b0_0  
qtconsole                 5.0.3              pyhd3eb1b0_0  
qtpy                      1.9.0                      py_0  
readline                  8.1                  h9ed2024_0  
regex                     2021.4.4         py38h9ed2024_0  
requests                  2.25.1             pyhd3eb1b0_0  
rope                      0.18.0                     py_0  
rtree                     0.9.7            py38hecd8cb5_1  
setuptools                52.0.0           py38hecd8cb5_0  
sip                       4.19.8           py38h0a44026_0  
six                       1.15.0           py38hecd8cb5_0  
snowballstemmer           2.1.0              pyhd3eb1b0_0  
sortedcontainers          2.3.0              pyhd3eb1b0_0  
sphinx                    3.5.3              pyhd3eb1b0_0  
sphinxcontrib-applehelp   1.0.2              pyhd3eb1b0_0  
sphinxcontrib-devhelp     1.0.2              pyhd3eb1b0_0  
sphinxcontrib-htmlhelp    1.0.3              pyhd3eb1b0_0  
sphinxcontrib-jsmath      1.0.1              pyhd3eb1b0_0  
sphinxcontrib-qthelp      1.0.3              pyhd3eb1b0_0  
sphinxcontrib-serializinghtml 1.1.4              pyhd3eb1b0_0  
spyder                    5.0.1            py38h50d1736_0    conda-forge
spyder-kernels            2.0.1            py38hecd8cb5_0  
sqlite                    3.35.4               hce871da_0  
testpath                  0.4.4              pyhd3eb1b0_0  
text-unidecode            1.3                        py_0  
textdistance              4.2.1              pyhd3eb1b0_0  
three-merge               0.1.1              pyhd3eb1b0_0  
tiledb                    1.7.7                h84aa2a7_2    conda-forge
tinycss                   0.4             pyhd3eb1b0_1002  
tk                        8.6.10               hb0a8c7a_0  
toml                      0.10.2             pyhd3eb1b0_0  
tornado                   6.1              py38h9ed2024_0  
traitlets                 5.0.5              pyhd3eb1b0_0  
typed-ast                 1.4.2            py38h9ed2024_1  
typing_extensions         3.7.4.3            pyha847dfd_0  
tzcode                    2021a                h0d85af4_1    conda-forge
ujson                     4.0.2            py38h23ab428_0  
unidecode                 1.2.0              pyhd3eb1b0_0  
urllib3                   1.26.4             pyhd3eb1b0_0  
watchdog                  1.0.2            py38h9ed2024_1  
wcwidth                   0.2.5                      py_0  
webencodings              0.5.1                    py38_1  
wheel                     0.36.2             pyhd3eb1b0_0  
whichcraft                0.6.1              pyhd3eb1b0_0  
wrapt                     1.12.1           py38haf1e3a3_1  
wurlitzer                 2.1.0            py38hecd8cb5_0  
xerces-c                  3.2.2             h8f8adb3_1004    conda-forge
xz                        5.2.5                h1de35cc_0  
yaml                      0.2.5                haf1e3a3_0  
yapf                      0.31.0             pyhd3eb1b0_0  
zeromq                    4.3.4                h23ab428_0  
zipp                      3.4.1              pyhd3eb1b0_0  
zlib                      1.2.11               h1de35cc_3  
zstd                      1.4.8                h62961b2_0    conda-forge

Any guidance that may be provided for getting Nio to properly import would be greatly appreciated!
Thank you!

No way to get location of opened file?

If I open a file as

f = Nio.open_file('xxx.nc', 'r')

there seems to be no way to recover the file location 'xxx.nc' from the object f (unless it happens to be an attribute of xxx.nc). Is this intended? It would be very convenient for scopes into which f is passed to be able to recover this information.

forecast time not always read correctly from grib2 files

The grib files belong to SubX models, in particular, I refer to one of these models:

EMC/GEFS hindcast - pr_sfc_GEFS_06oct1999_00z_d01_d35_m00.grb

if I explore the data with wgrib2

wgrib2 pr_sfc_GEFS_06oct1999_00z_d01_d35_m00.grb2
1:0:d=1999100600:APCP:surface:1 day fcst:ENS=low-res ctl
2:71544:d=1999100600:APCP:surface:2 day fcst:ENS=low-res ctl
3:142213:d=1999100600:APCP:surface:3 day fcst:ENS=low-res ctl
4:211380:d=1999100600:APCP:surface:4 day fcst:ENS=low-res ctl
5:280114:d=1999100600:APCP:surface:5 day fcst:ENS=low-res ctl
6:348933:d=1999100600:APCP:surface:6 day fcst:ENS=low-res ctl

and so on. Note, it is a daily forecast.
using wgrib2 with option -V


wgrib2 pr_sfc_GEFS_06oct1999_00z_d01_d35_m00.grb2 -V
1:0:vt=**1999100700**:surface:1 day fcst:APCP Total Precipitation [kg/m^2]:ENS=low-res ctl
    ndata=65160:undef=0:mean=2.85094:min=0:max=305.2
    grid_template=0:winds(N/S):
	lat-lon grid:(360 x 181) units 1e-06 input WE:NS output WE:SN res 48
	lat 90.000000 to -90.000000 by 1.000000
	lon 0.000000 to 359.000000 by 1.000000 #points=65160

2:71544:vt=**1999100800**:surface:2 day fcst:APCP Total Precipitation [kg/m^2]:ENS=low-res ctl
    ndata=65160:undef=0:mean=2.57202:min=0:max=320
    grid_template=0:winds(N/S):
	lat-lon grid:(360 x 181) units 1e-06 input WE:NS output WE:SN res 48
	lat 90.000000 to -90.000000 by 1.000000
	lon 0.000000 to 359.000000 by 1.000000 #points=65160

3:142213:vt=**1999100900**:surface:3 day fcst:APCP Total Precipitation [kg/m^2]:ENS=low-res ctl
    ndata=65160:undef=0:mean=2.51068:min=0:max=295.1
    grid_template=0:winds(N/S):
	lat-lon grid:(360 x 181) units 1e-06 input WE:NS output WE:SN res 48
	lat 90.000000 to -90.000000 by 1.000000
	lon 0.000000 to 359.000000 by 1.000000 #points=65160

and so on. Note the "vt" key, which is the validation time (? maybe) has the full calendar date of the lead time date.

When I open the same file both using Nio or xarray and the pynio engine I get:

Nio file:	pr_sfc_GEFS_06oct1999_00z_d01_d35_m00.grb2
   global attributes:
   dimensions:
      forecast_time0 = 35
      lat_0 = 181
      lon_0 = 360
   variables:
      float APCP_P1_L1_GLL0 [ forecast_time0, lat_0, lon_0 ]
         center :	US National Weather Service - NCEP (WMC)
         production_status :	Operational products
         long_name :	Total precipitation
         units :	kg m-2
         _FillValue :	1e+20
         grid_type :	Latitude/longitude
         parameter_discipline_and_category :	Meteorological products, Moisture
         parameter_template_discipline_category_number :	[1, 0, 1, 8]
         level_type :	Ground or water surface
         level :	0
         initial_time :	10/06/1999 (00:00)
      float lat_0 [ lat_0 ]
         long_name :	latitude
         grid_type :	Latitude/Longitude
         units :	degrees_north
         Dj :	1
         Di :	1
         Lo2 :	359
         La2 :	-90
         Lo1 :	0
         La1 :	90
      float lon_0 [ lon_0 ]
         long_name :	longitude
         grid_type :	Latitude/Longitude
         units :	degrees_east
         Dj :	1
         Di :	1
         Lo2 :	359
         La2 :	-90
         Lo1 :	0
         La1 :	90
      **integer forecast_time0 [ forecast_time0 ]
         long_name :	Forecast offset from initial time
         units :	hours**

and

<xarray.Dataset>
Dimensions:          (forecast_time0: 35, lat_0: 181, lon_0: 360)
Coordinates:
  * lat_0            (lat_0) float32 90.0 89.0 88.0 87.0 ... -88.0 -89.0 -90.0
  * lon_0            (lon_0) float32 0.0 1.0 2.0 3.0 ... 356.0 357.0 358.0 359.0
  * forecast_time0   (forecast_time0) timedelta64[ns] 01:00:00 ... 1 days 11:00:00
Data variables:
    APCP_P1_L1_GLL0  (forecast_time0, lat_0, lon_0) float32 ...

more specifically, it reads the forecast time as hours

temp.forecast_time0/np.timedelta64(1,'h')
<xarray.DataArray 'forecast_time0' (forecast_time0: 35)>
array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13., 14.,
       15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.,
       29., 30., 31., 32., 33., 34., 35.])
Coordinates:
  * forecast_time0  (forecast_time0) timedelta64[ns] 01:00:00 ... 1 days 11:00:00

the same happens for the EMC/GEFS forecast data - so the same issue is in both the hindcast and realtime forecast data.

Any hint on how to fix this?

Variable names inferred from grib2 files through Nio.open_file - or xr.open_dataset with engine pynio

I posted this on SO, but I think it belongs here as well.

Below an edited version:

I am using the data found here (Note, these are rotating files of forecast data, so the actual date will change as time goes by, you might need to update the date in my example in a few days)

!wget ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20180524/00/6hrly_grib_01/pgbf2018052400.01.2018052400.grb2


tmp2 = xr.open_dataset('pgbf2018052400.01.2018052400.grb2', 
                   engine='pynio')

which is read as a dataset with some coordinates and variables

<xarray.Dataset>
Dimensions:                   (lat_0: 181, lon_0: 360, lv_AMSL1: 4, lv_HTGL6: 2, lv_ISBL0: 37, lv_ISBL4: 32, lv_ISBL5: 2, lv_PVL2: 2, lv_SIGL3: 4)
Coordinates:
  * lon_0                     (lon_0) float32 0.0 1.0 2.0 3.0 4.0 5.0 6.0 ...
  * lv_ISBL0                  (lv_ISBL0) float32 100.0 200.0 300.0 500.0 ...
  * lv_ISBL5                  (lv_ISBL5) float32 50000.0 100000.0
  * lv_ISBL4                  (lv_ISBL4) float32 1000.0 2000.0 3000.0 5000.0 ...
  * lat_0                     (lat_0) float32 90.0 89.0 88.0 87.0 86.0 85.0 ...
  * lv_PVL2                   (lv_PVL2) float32 -2e-06 2e-06
  * lv_AMSL1                  (lv_AMSL1) float32 1829.0 2743.0 3658.0 4572.0
Dimensions without coordinates: lv_HTGL6, lv_SIGL3
Data variables:
    TOZNE_P0_L200_GLL0        (lat_0, lon_0) float32 ...
    HGT_P0_L100_GLL0          (lv_ISBL0, lat_0, lon_0) float32 ...
    VVEL_P0_L100_GLL0         (lv_ISBL0, lat_0, lon_0) float32 ...
    UGRD_P0_L6_GLL0           (lat_0, lon_0) float32 ...
    HGT_P0_L7_GLL0            (lat_0, lon_0) float32 ...
    CIN_P0_2L108_GLL0         (lat_0, lon_0) float32 ...
    PRES_P0_L109_GLL0         (lv_PVL2, lat_0, lon_0) float32 ...
    UGRD_P0_L7_GLL0           (lat_0, lon_0) float32 ...
    UGRD_P0_L109_GLL0         (lv_PVL2, lat_0, lon_0) float32 ...
    PRES_P0_L7_GLL0           (lat_0, lon_0) float32 ...
    PLI_P0_2L108_GLL0         (lat_0, lon_0) float32 ...

and many other variables... i am not copying here...

my struggle right now is to understand how these variable names (i.e. PLI_P0_2L108_GLL0) are generated.

If I run directly Nio:

f = Nio.open_file('pgbf2018052400.01.2018052400.grb2')
print(f)

I get:

Nio file:	pgbf2018052400.01.2018052400.grb2
   global attributes:
   dimensions:
      lat_0 = 181
      lon_0 = 360
      lv_ISBL0 = 37
      lv_AMSL1 = 4
      lv_PVL2 = 2
      lv_SIGL3 = 4
      lv_ISBL4 = 32
      lv_ISBL5 = 2
      lv_HTGL6 = 2
   variables:
      float TMP_P0_L6_GLL0 [ lat_0, lon_0 ]
         center :	US National Weather Service - NCEP (WMC)
         production_status :	Operational products
         long_name :	Temperature
         units :	K
         _FillValue :	1e+20
         grid_type :	Latitude/longitude
         parameter_discipline_and_category :	Meteorological products, Temperature
         parameter_template_discipline_category_number :	[0, 0, 0, 0]
         level_type :	Maximum wind level
         level :	0
         forecast_time :	0
         forecast_time_units :	hours
         initial_time :	05/24/2018 (00:00)
      float TMP_P0_L7_GLL0 [ lat_0, lon_0 ]

etc...

but the names of the variables are already there. So it seems like these names are defined at Nio level (I think).

However, when I inspect the data with wgrib2 I have the following:

wgrib2 pgbf2018052400.01.2018052400.grb2

1:0:d=2018052400:PRES:mean sea level:anl:
2:66570:d=2018052400:HGT:1 mb:anl:
3:102113:d=2018052400:TMP:1 mb:anl:
4:118939:d=2018052400:RH:1 mb:anl:
5:122609:d=2018052400:SPFH:1 mb:anl:
6:141106:d=2018052400:VVEL:1 mb:anl:
7.1:219137:d=2018052400:UGRD:1 mb:anl:
7.2:219137:d=2018052400:VGRD:1 mb:anl:
8:283014:d=2018052400:ABSV:1 mb:anl:
9:319471:d=2018052400:O3MR:1 mb:anl:
10:372630:d=2018052400:HGT:2 mb:anl:
11:408159:d=2018052400:TMP:2 mb:anl:
12:425397:d=2018052400:RH:2 mb:anl:
13:427490:d=2018052400:SPFH:2 mb:anl:
14:459259:d=2018052400:VVEL:2 mb:anl:
15.1:517010:d=2018052400:UGRD:2 mb:anl:
15.2:517010:d=2018052400:VGRD:2 mb:anl:
16:580043:d=2018052400:ABSV:2 mb:anl:
17:617506:d=2018052400:O3MR:2 mb:anl:
18:672805:d=2018052400:HGT:3 mb:anl:
19:707799:d=2018052400:TMP:3 mb:anl:
20:725593:d=2018052400:RH:3 mb:anl:
21:729682:d=2018052400:SPFH:3 mb:anl:
22:765030:d=2018052400:VVEL:3 mb:anl:
23.1:828383:d=2018052400:UGRD:3 mb:anl:
23.2:828383:d=2018052400:VGRD:3 mb:anl:
24:892759:d=2018052400:ABSV:3 mb:anl:
25:931422:d=2018052400:O3MR:3 mb:anl:
26:979930:d=2018052400:HGT:5 mb:anl:

The variable names are PRES, HGT, ABSV, VGRD, ACPCP, and so on.

The problem I have is that I am trying to run an open_mfdataset in xarray to concatenate along multiple dimensions (in my case forecast start time and forecast lead time) thousands of these little files, unfortunately the variable names (of what should be the same quantity, i.e. convective precipitation, which in the grib file is simply named ACPC) generated by xarray/PyNIO change throughout the dataset going from ACPCP_P8_L1_GLL0_acc, to ACPCP_P8_L1_GLL0_acc6h, to ACPCP_P0_L1_GLL0, making things complicated.

I tried to overwrite the variable names with some preprocessing function, but I wanted to understand the rationale behind those names, to be sure that I am doing it correctly.

So, where does the part of the variable name, like "_P0_L6_GLL0", attached to the grib variable name come from? is there a way to control that?

Can not import pynio on yellowstone// libmfhdf.so.0: cannot open shared object file

Hi,

I installed miniconda on yellowstone and tried tp install conda on both dbrown and conda-forge channel. Both can not import pynio :-(

the error message:

yslogin1:~> python -c "import nio"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: libmfhdf.so.0: cannot open shared object file: No such file or directory

below is my conda list

backports                 1.0                      py27_0
backports_abc             0.4                      py27_0
bokeh                     0.11.1                   py27_0
boto3                     1.3.1                    py27_0
cairo                     1.12.18                       6
chest                     0.2.3                    py27_0
cloog                     0.18.0                        0
cloudpickle               0.2.1                    py27_0
conda                     4.1.11                   py27_0
botocore                  1.4.34                   py27_0    conda-forge
bottleneck                1.0.0               np110py27_0    conda-forge
click                     6.6                      py27_0    conda-forge
conda-env                 2.5.2                    py27_0    conda-forge
cyordereddict             1.0.0                    py27_1    conda-forge
dask                      0.10.1                   py27_1    conda-forge
distributed               1.11.2                   py27_0    conda-forge
freexl                    1.0.2                         1    conda-forge
gdal                      1.11.4             np110py27_10    conda-forge
giflib                    5.1.2                         1    conda-forge
h5netcdf                  0.2.2                    py27_0    conda-forge
h5py                      2.6.0               np110py27_4    conda-forge
hdf5                      1.8.17                        2    conda-forge
jmespath                  0.9.0                    py27_0    conda-forge
jpeg                      9b                            0    conda-forge
json-c                    0.12                          0    conda-forge
krb5                      1.14.2                        0    conda-forge
libnetcdf                 4.4.0                         2    conda-forge
libspatialite             4.3.0a                        8    conda-forge
libtiff                   4.0.6                         5    conda-forge
netcdf4                   1.2.4               np110py27_2    conda-forge
openjpeg                  2.1.0                         5    conda-forge
pillow                    3.2.0                    py27_1    conda-forge
postgresql                9.5.3                    py27_0    conda-forge
psutil                    4.3.0                    py27_0    conda-forge
pynio                     1.5.0               np110py27_7    conda-forge
s3fs                      0.0.6                    py27_0    conda-forge
xarray                    0.7.2                    py27_1    conda-forge
configparser              3.5.0                    py27_0
curl                      7.45.0                        0
cycler                    0.10.0                   py27_0
g2clib                    1.5.0                         1    dbrown
hdf4                      4.2.11                        1    dbrown
hdfeos2                   2.19.1                        1    dbrown
hdfeos5                   5.1.15                        1    dbrown
jasper                    1.900.1                       1    dbrown
libgdal                   2.0.2                 geos342_3    dbrown
libiconv                  1.14                          1    dbrown
szip                      2.1                           1    dbrown
decorator                 4.0.9                    py27_0
docutils                  0.12                     py27_2
entrypoints               0.2.2                    py27_0
expat                     2.1.0                         0
fontconfig                2.11.1                        5
freetype                  2.5.5                         0
funcsigs                  1.0.0                    py27_0
functools32               3.2.3.2                  py27_0
futures                   3.0.5                    py27_0
gcc                       4.8.5                         6
get_terminal_size         1.0.0                    py27_0
gmp                       5.1.2                         2
heapdict                  1.0.0                    py27_0
ipykernel                 4.3.1                    py27_0
ipython                   4.2.0                    py27_0
ipython_genutils          0.1.0                    py27_0
ipywidgets                4.1.1                    py27_0
isl                       0.12.2                        0
jbig                      2.1                           0
jinja2                    2.8                      py27_0
jsonschema                2.5.1                    py27_0
jupyter                   1.0.0                    py27_3
jupyter_client            4.3.0                    py27_0
jupyter_console           5.0.0                    py27_0
jupyter_core              4.1.0                    py27_0
kealib                    1.4.5                         0
libgcc                    5.2.0                         0
libgfortran               3.0.0                         1
libpng                    1.6.17                        0
libsodium                 1.0.10                        0
libxml2                   2.9.2                         0
libxslt                   1.1.28                        0
locket                    0.2.0                    py27_0
lxml                      3.6.0                    py27_0
markupsafe                0.23                     py27_0
matplotlib                1.5.1               np110py27_0
mistune                   0.7.2                    py27_0
mkl                       11.3.1                        0
mock                      2.0.0                    py27_0
mpc                       1.0.1                         0
mpfr                      3.1.2                         0
msgpack-python            0.4.7                    py27_0
nbconvert                 4.2.0                    py27_0
nbformat                  4.0.1                    py27_0
nose                      1.3.7                    py27_0
notebook                  4.2.1                    py27_0
numpy                     1.10.4                   py27_1
openssl                   1.0.2g                        0
pandas                    0.18.1              np110py27_0
partd                     0.3.3                    py27_0
path.py                   8.2.1                    py27_0
pbr                       1.8.0                    py27_0
pexpect                   4.0.1                    py27_0
pickleshare               0.5                      py27_0
pip                       8.1.1                    py27_1
pixman                    0.32.6                        0
proj4                     4.9.2                         0
prompt_toolkit            1.0.3                    py27_0
ptyprocess                0.5                      py27_0
pycairo                   1.10.0                   py27_0
pycosat                   0.6.1                    py27_0
pycrypto                  2.6.1                    py27_0
pygments                  2.1.3                    py27_0
pyparsing                 2.1.1                    py27_0
pyproj                    1.9.4                    py27_1
pyqt                      4.11.4                   py27_1
python                    2.7.12                        1
python-dateutil           2.5.3                    py27_0
pytz                      2016.4                   py27_0
pyyaml                    3.11                     py27_1
pyzmq                     15.3.0                   py27_0
qt                        4.8.7                         1
qtconsole                 4.2.1                    py27_0
quickImpt                 1.0                       <pip>
readline                  6.2                           2
requests                  2.9.1                    py27_0
ruamel_yaml               0.11.7                   py27_0
scipy                     0.17.0              np110py27_3
cartopy                   0.13.1              np110py27_0    scitools
geos                      3.4.2                         4    scitools
owslib                    0.10.3                   py27_0    scitools
proj.4                    4.9.1                         1    scitools
pyepsg                    0.2.0                    py27_0    scitools
pyshp                     1.2.3                    py27_0    scitools
shapely                   1.5.13              np110py27_1    scitools
setuptools                20.3                     py27_0
simplegeneric             0.8.1                    py27_0
singledispatch            3.4.0.3                  py27_0
sip                       4.16.9                   py27_0
six                       1.10.0                   py27_0
sqlite                    3.13.0                        0
ssl_match_hostname        3.4.0.2                  py27_1
tblib                     1.3.0                    py27_0
terminado                 0.6                      py27_0
tk                        8.5.18                        0
toolz                     0.7.4                    py27_0
tornado                   4.3                      py27_0
traitlets                 4.2.1                    py27_0
wcwidth                   0.1.7                    py27_0
wheel                     0.29.0                   py27_0
xerces-c                  3.1.4                         0
xz                        5.0.5                         1
yaml                      0.1.6                         0
zeromq                    4.1.4                         0
zlib                      1.2.8                         0

Any suggestions? Many thanks!

when shape is unexpected, pynio return "check dtype"

The error is:

Traceback (most recent call last):
File "interp2d.py", line 60, in
outf.variables['lat'].assign_value(ersst_lat)
File "/etc/anaconda2/envs/ml/lib/python3.8/site-packages/PyNIO/Nio.py", line 444, in assign_value
self._obj.assign_value(value)
Nio.NIOError: failed to write array (check dtype)

but acctually this caused by wrong numpy.shape

installation linux (ubuntu ) in the window 10 sub system

I am trying to install pynio in windows 10 system ubuntu and getting an error
python3 -m pip install https://github.com/NCAR/pynio/archive/develop.zip
Collecting https://github.com/NCAR/pynio/archive/develop.zip
Downloading https://github.com/NCAR/pynio/archive/develop.zip
- 12.6 MB 499 kB/s
Building wheels for collected packages: PyNIO
Building wheel for PyNIO (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-3kmcfei0/setup.py'"'"'; file='"'"'/tmp/pip-req-build-3kmcfei0/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-1bse7e2k
cwd: /tmp/pip-req-build-3kmcfei0/
Complete output (353 lines):
====> Installing Nio to the "PyNIO" site packages directory.
non-existing path in '': ''
running bdist_wheel
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building library "nio" sources
building extension "PyNIO._nio" sources
building data_files sources
build_src: building npy-pkg config files
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/PyNIO
copying alt-setup.py -> build/lib.linux-x86_64-3.8/PyNIO
copying coordsel.py -> build/lib.linux-x86_64-3.8/PyNIO
copying Nio.py -> build/lib.linux-x86_64-3.8/PyNIO
copying niodict.py -> build/lib.linux-x86_64-3.8/PyNIO
copying pynio_version.py -> build/lib.linux-x86_64-3.8/PyNIO
copying _xarray.py -> build/lib.linux-x86_64-3.8/PyNIO
copying init.py -> build/lib.linux-x86_64-3.8/PyNIO
creating build/lib.linux-x86_64-3.8/PyNIO/ncarg
creating build/lib.linux-x86_64-3.8/PyNIO/ncarg/data
creating build/lib.linux-x86_64-3.8/PyNIO/ncarg/data/netcdf
copying ncarg/data/netcdf/pop.nc -> build/lib.linux-x86_64-3.8/PyNIO/ncarg/data/netcdf
running build_clib
customize UnixCCompiler
customize UnixCCompiler using build_clib
get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_clib
building 'nio' library
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/libsrc
compile options: '-DNeedFuncProto -DNIO_LIB_ONLY -D_POSIX_SOURCE -DByteSwapped -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -Ilibsrc -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -c'
extra options: '-O2 -w'
x86_64-linux-gnu-gcc: libsrc/AddFileFormats.c
x86_64-linux-gnu-gcc: libsrc/Formats.c
x86_64-linux-gnu-gcc: libsrc/Memory.cx86_64-linux-gnu-gcc: libsrc/ctoiee.c

x86_64-linux-gnu-gcc: libsrc/DataSupport.c
x86_64-linux-gnu-gcc: libsrc/AdvancedFileSupport.c
x86_64-linux-gnu-gcc: libsrc/NclAdvancedFile.c
x86_64-linux-gnu-gcc: libsrc/GetGrids.c
x86_64-linux-gnu-gcc: libsrc/AttSupport.c
x86_64-linux-gnu-gcc: libsrc/cmpf.c
x86_64-linux-gnu-gcc: libsrc/date.c
x86_64-linux-gnu-gcc: libsrc/NclCCM.c
x86_64-linux-gnu-gcc: libsrc/FileSupport.c
x86_64-linux-gnu-gcc: libsrc/NclCoordVar.c
x86_64-linux-gnu-gcc: libsrc/NclData.c
x86_64-linux-gnu-gcc: libsrc/NclAdvancedGroup.c
x86_64-linux-gnu-gcc: libsrc/NclFile.c
x86_64-linux-gnu-gcc: libsrc/NclFileVar.c
x86_64-linux-gnu-gcc: libsrc/NclAtt.c
x86_64-linux-gnu-gcc: libsrc/InitData.c
x86_64-linux-gnu-gcc: libsrc/NclGRIB.c
x86_64-linux-gnu-gcc: libsrc/ListSupport.c
x86_64-linux-gnu-gcc: libsrc/NclMultiDValData.c
x86_64-linux-gnu-gcc: libsrc/NclNetCDF4.c
x86_64-linux-gnu-gcc: libsrc/NclTypechar.c
x86_64-linux-gnu-gcc: libsrc/NclOneDValCoordData.c
x86_64-linux-gnu-gcc: libsrc/NclMultiDVallistData.c
x86_64-linux-gnu-gcc: libsrc/NclTypedouble.c
x86_64-linux-gnu-gcc: libsrc/NclType.c
x86_64-linux-gnu-gcc: libsrc/NclTypebyte.c
x86_64-linux-gnu-gcc: libsrc/NclMultiDValnclfileData.c
x86_64-linux-gnu-gcc: libsrc/NclTypefloat.c
x86_64-linux-gnu-gcc: libsrc/NclTypeint64.c
x86_64-linux-gnu-gcc: libsrc/NclNetCdf.c
x86_64-linux-gnu-gcc: libsrc/NclTypeint.c
x86_64-linux-gnu-gcc: libsrc/NclTypelist.c
x86_64-linux-gnu-gcc: libsrc/NclTypeobj.c
x86_64-linux-gnu-gcc: libsrc/NclGroup.c
x86_64-linux-gnu-gcc: libsrc/NclTypelogical.c
x86_64-linux-gnu-gcc: libsrc/NclTypeshort.c
x86_64-linux-gnu-gcc: libsrc/NclList.c
x86_64-linux-gnu-gcc: libsrc/NclTypeuint.c
x86_64-linux-gnu-gcc: libsrc/NclTypelong.c
x86_64-linux-gnu-gcc: libsrc/NclVar.c
x86_64-linux-gnu-gcc: libsrc/NclTypestring.c
x86_64-linux-gnu-gcc: libsrc/NclTypeuint64.c
x86_64-linux-gnu-gcc: libsrc/nioConvert.c
x86_64-linux-gnu-gcc: libsrc/NclTypeubyte.c
x86_64-linux-gnu-gcc: libsrc/nioError.c
x86_64-linux-gnu-gcc: libsrc/NclTypeulong.c
x86_64-linux-gnu-gcc: libsrc/niohlu.c
x86_64-linux-gnu-gcc: libsrc/nio.c
x86_64-linux-gnu-gcc: libsrc/nioNclHLUStub.c
x86_64-linux-gnu-gcc: libsrc/nioBase.c
x86_64-linux-gnu-gcc: libsrc/nioNresDB.c
x86_64-linux-gnu-gcc: libsrc/NclTypeushort.c
x86_64-linux-gnu-gcc: libsrc/nio_path.c
x86_64-linux-gnu-gcc: libsrc/nioCallbacks.c
x86_64-linux-gnu-gcc: libsrc/TypeSupport.c
x86_64-linux-gnu-gcc: libsrc/VarSupport.c
x86_64-linux-gnu-gcc: libsrc/nioQuarks.c
x86_64-linux-gnu-gcc: libsrc/nioUtils.c
x86_64-linux-gnu-gcc: libsrc/nio_io.c
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-DNeedFuncProto -DNIO_LIB_ONLY -D_POSIX_SOURCE -DByteSwapped -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -Ilibsrc -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -c'
gfortran:f77: libsrc/llmisc.fgfortran:f77: libsrc/ncepcode.f

gfortran:f77: libsrc/qu2reg3.f
gfortran:f77: libsrc/llmap.f
libsrc/qu2reg3.f:274:5:

274 |   200 CONTINUE
    |     1

Warning: Label 200 at (1) defined but not used [-Wunused-label]
libsrc/qu2reg3.f:208:5:

208 |   100 CONTINUE
    |     1

Warning: Label 100 at (1) defined but not used [-Wunused-label]
libsrc/ncepcode.f:2015:72:

2015 | 100 E(I-1) = E(I)
| 1
Warning: Fortran 2018 deleted feature: DO termination statement which is not END DO or CONTINUE with label 100 at (1)
libsrc/ncepcode.f:1878:18:

1878 | SLAT(J)=SLATD(J)
| 1
Warning: Possible change of value in conversion from REAL(8) to REAL(4) at (1) [-Wconversion]
libsrc/ncepcode.f:1879:18:

1879 | WLAT(J)=(2.(1.-SLATD(J)**2))/(JMAXPKM1(J))**2
| 1
Warning: Possible change of value in conversion from REAL(8) to REAL(4) at (1) [-Wconversion]
libsrc/ncepcode.f:1990:5:

1990 | 20 DO 40 I = 1, N
| 1
Warning: Label 20 at (1) defined but not used [-Wunused-label]
libsrc/ncepcode.f:1950:10:

1949 | DO 100 J=2,N
| 2
1950 | WRK(J)= (J-1.D0)/DSQRT((2.D0J-1.D0)(2.D0*J-3.D0))
| 1
Warning: Array reference at (1) out of bounds (2 > 1) in loop beginning at (2)
libsrc/ncepcode.f:1952:17:

1949 | DO 100 J=2,N
| 2
......
1952 | E(J) = WRK(J)
| 1
Warning: Array reference at (1) out of bounds (2 > 1) in loop beginning at (2)
libsrc/ncepcode.f:1980:38:

1980 | SUBROUTINE DRSTNIO(NM,N,W,E,MATZ,Z,IERR)
| 1
Warning: Unused dummy argument ‘matz’ at (1) [-Wunused-dummy-argument]
libsrc/ncepcode.f:1837:21:

1837 | INTEGER JHE,JHO,J0,NEXP
| 1
Warning: Unused variable ‘jho’ declared at (1) [-Wunused-variable]
libsrc/llmap.f:2251:72:

2251 | IF (IPRJ-11) 100,197,113
| 1
Warning: Fortran 2018 deleted feature: Arithmetic IF statement at (1)
libsrc/llmap.f:2256:72:

2256 | 100 IF (IPRJ-1) 101,102,103
| 1
Warning: Fortran 2018 deleted feature: Arithmetic IF statement at (1)
libsrc/llmap.f:2426:72:

2426 | 113 IF (IPRJ-13) 114,115,116
| 1
Warning: Fortran 2018 deleted feature: Arithmetic IF statement at (1)
libsrc/llmap.f:1466:5:

1466 | 641 RLAT=CLAT
| 1
Warning: Label 641 at (1) defined but not used [-Wunused-label]
libsrc/llmap.f:2460:5:

2460 | 199 U=U-UOFF
| 1
Warning: Label 199 at (1) defined but not used [-Wunused-label]
libsrc/llmap.f:2222:31:

2222 | DOUBLE PRECISION MDGDDP,RBGDFE,RBGLEN
| 1
Warning: Unused variable ‘mdgddp’ declared at (1) [-Wunused-variable]
libsrc/llmap.f:2218:29:

2218 | REAL USNG,VSNG
| 1
Warning: Unused variable ‘usng’ declared at (1) [-Wunused-variable]
libsrc/llmap.f:2218:34:

2218 | REAL USNG,VSNG
| 1
Warning: Unused variable ‘vsng’ declared at (1) [-Wunused-variable]
libsrc/llmap.f:1832:29:

1832 | REAL SLAT,SLON
| 1
Warning: Unused variable ‘slat’ declared at (1) [-Wunused-variable]
libsrc/llmap.f:1832:34:

1832 | REAL SLAT,SLON
| 1
Warning: Unused variable ‘slon’ declared at (1) [-Wunused-variable]
gfortran:f77: libsrc/rowina3.f
libsrc/rowina3.f:308:5:

308 |   300   CONTINUE
    |     1

Warning: Label 300 at (1) defined but not used [-Wunused-label]
libsrc/rowina3.f:248:5:

248 |   200 CONTINUE
    |     1

Warning: Label 200 at (1) defined but not used [-Wunused-label]
libsrc/rowina3.f:130:5:

130 |   100 CONTINUE
    |     1

Warning: Label 100 at (1) defined but not used [-Wunused-label]
libsrc/rowina3.f:172:19:

172 |             IP   = ZWT
    |                   1

Warning: Possible change of value in conversion from REAL(4) to INTEGER(4) at (1) [-Wconversion]
libsrc/rowina3.f:217:21:

217 |               IP   = ZWT
    |                     1

Warning: Possible change of value in conversion from REAL(4) to INTEGER(4) at (1) [-Wconversion]
libsrc/rowina3.f:295:17:

295 |             IP = ZWT+1
    |                 1

Warning: Possible change of value in conversion from REAL(4) to INTEGER(4) at (1) [-Wconversion]
libsrc/rowina3.f:209:0:

209 |                 P(JL) = PW(IP+1,1)
    |

Warning: ‘ip’ may be used uninitialized in this function [-Wmaybe-uninitialized]
gfortran:f77: libsrc/scm0.f
libsrc/scm0.f:41:5:

 41 |   100 CONTINUE
    |     1

Warning: Label 100 at (1) defined but not used [-Wunused-label]
x86_64-linux-gnu-gcc-ar: adding 50 object files to build/temp.linux-x86_64-3.8/libnio.a
x86_64-linux-gnu-gcc-ar: adding 19 object files to build/temp.linux-x86_64-3.8/libnio.a
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
extending extension 'PyNIO._nio' defined_macros with [('NeedFuncProto', None), ('NIO_LIB_ONLY', None), ('_POSIX_SOURCE', None), ('ByteSwapped', None)]
customize UnixCCompiler
customize UnixCCompiler using build_ext
get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
customize Gnu95FCompiler
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'PyNIO._nio' extension
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-DNeedFuncProto -DNIO_LIB_ONLY -DNeedFuncProto -DNIO_LIB_ONLY -D_POSIX_SOURCE -DByteSwapped -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -Ilibsrc -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -I/usr/include/python3.8 -c'
x86_64-linux-gnu-gcc: niomodule.c
niomodule.c: In function ‘data_type’:
niomodule.c:688:3: warning: ‘NPY_CHAR’ is deprecated: Use NPY_STRING [-Wdeprecated-declarations]
688 | return NPY_CHAR;
| ^~~~~~
In file included from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from niomodule.c:24:
/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:21: note: declared here
84 | NPY_CHAR NPY_ATTR_DEPRECATE("Use NPY_STRING"),
| ^~~~~~~~
niomodule.c: In function ‘typecode’:
niomodule.c:790:2: warning: ‘NPY_CHAR’ is deprecated: Use NPY_STRING [-Wdeprecated-declarations]
790 | case NPY_CHAR:
| ^~~~
In file included from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from niomodule.c:24:
/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:21: note: declared here
84 | NPY_CHAR NPY_ATTR_DEPRECATE("Use NPY_STRING"),
| ^~~~~~~~
niomodule.c: In function ‘_convertCOMPOUND2Obj’:
niomodule.c:5273:17: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
5273 | for (n = 0; n < comprec->n_comps; ++n) {
| ^
niomodule.c:5290:5: warning: ‘NPY_CHAR’ is deprecated: Use NPY_STRING [-Wdeprecated-declarations]
5290 | tmpobj = PyArray_SimpleNewFromData(1, &length, NPY_CHAR,
| ^~~~~~
In file included from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from niomodule.c:24:
/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:21: note: declared here
84 | NPY_CHAR NPY_ATTR_DEPRECATE("Use NPY_STRING"),
| ^~~~~~~~
niomodule.c:5299:5: warning: ‘NPY_CHAR’ is deprecated: Use NPY_STRING [-Wdeprecated-declarations]
5299 | tmpobj = PyArray_SimpleNewFromData(1, &length, NPY_CHAR,
| ^~~~~~
In file included from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from niomodule.c:24:
/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:21: note: declared here
84 | NPY_CHAR NPY_ATTR_DEPRECATE("Use NPY_STRING"),
| ^~~~~~~~
niomodule.c: In function ‘_convertObj2COMPOUND’:
niomodule.c:5701:13: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
5701 | if (len2 != comprec->n_comps) {
| ^~
niomodule.c:5723:42: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘ng_size_t’ {aka ‘long int’} [-Wformat=]
5723 | "\tFailed to allocate memory of %d char.\n",
| ~^
| |
| int
| %ld
5724 | compnode->nvals);
| ~~~~~~~~~~~~~~~
| |
| ng_size_t {aka long int}
niomodule.c: In function ‘NioVariable_WriteArray’:
niomodule.c:5776:20: warning: variable ‘dimrec’ set but not used [-Wunused-but-set-variable]
5776 | NclFileDimRecord* dimrec = NULL;
| ^~~~~~
niomodule.c: In function ‘NioVariableObject_ass_slice’:
niomodule.c:6556:40: warning: integer overflow in expression of type ‘long int’ results in ‘0’ [-Woverflow]
6556 | if (indices->start < PY_SSIZE_T_MIN * 100)
| ^
niomodule.c: In function ‘NioVariableObject_ass_subscript’:
niomodule.c:6626:41: warning: integer overflow in expression of type ‘long int’ results in ‘0’ [-Woverflow]
6626 | if (indices->start < PY_SSIZE_T_MIN * 100)
| ^
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/niomodule.o -Llibsrc -Lbuild/temp.linux-x86_64-3.8 -lnio -ljpeg -lpng -lz -lnetcdf -o build/lib.linux-x86_64-3.8/PyNIO/_nio.cpython-38-x86_64-linux-gnu.so
x86_64-linux-gnu-g++: error: : No such file or directory
error: Command "x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/niomodule.o -Llibsrc -Lbuild/temp.linux-x86_64-3.8 -lnio -ljpeg -lpng -lz -lnetcdf -o build/lib.linux-x86_64-3.8/PyNIO/_nio.cpython-38-x86_64-linux-gnu.so" failed with exit status 1

ERROR: Failed building wheel for PyNIO
Running setup.py clean for PyNIO
Failed to build PyNIO
Installing collected packages: PyNIO
Running setup.py install for PyNIO ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-3kmcfei0/setup.py'"'"'; file='"'"'/tmp/pip-req-build-3kmcfei0/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-2u8mgvoj/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/hafez/.local/include/python3.8/PyNIO
cwd: /tmp/pip-req-build-3kmcfei0/
Complete output (353 lines):
====> Installing Nio to the "PyNIO" site packages directory.
non-existing path in '': ''
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building library "nio" sources
building extension "PyNIO._nio" sources
building data_files sources
build_src: building npy-pkg config files
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/PyNIO
copying alt-setup.py -> build/lib.linux-x86_64-3.8/PyNIO
copying coordsel.py -> build/lib.linux-x86_64-3.8/PyNIO
copying Nio.py -> build/lib.linux-x86_64-3.8/PyNIO
copying niodict.py -> build/lib.linux-x86_64-3.8/PyNIO
copying pynio_version.py -> build/lib.linux-x86_64-3.8/PyNIO
copying _xarray.py -> build/lib.linux-x86_64-3.8/PyNIO
copying init.py -> build/lib.linux-x86_64-3.8/PyNIO
creating build/lib.linux-x86_64-3.8/PyNIO/ncarg
creating build/lib.linux-x86_64-3.8/PyNIO/ncarg/data
creating build/lib.linux-x86_64-3.8/PyNIO/ncarg/data/netcdf
copying ncarg/data/netcdf/pop.nc -> build/lib.linux-x86_64-3.8/PyNIO/ncarg/data/netcdf
running build_clib
customize UnixCCompiler
customize UnixCCompiler using build_clib
get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_clib
building 'nio' library
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/libsrc
compile options: '-DNeedFuncProto -DNIO_LIB_ONLY -D_POSIX_SOURCE -DByteSwapped -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -Ilibsrc -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -c'
extra options: '-O2 -w'
x86_64-linux-gnu-gcc: libsrc/ctoiee.cx86_64-linux-gnu-gcc: libsrc/Formats.cx86_64-linux-gnu-gcc: libsrc/Memory.cx86_64-linux-gnu-gcc: libsrc/AddFileFormats.c



x86_64-linux-gnu-gcc: libsrc/NclAdvancedFile.c
x86_64-linux-gnu-gcc: libsrc/DataSupport.c
x86_64-linux-gnu-gcc: libsrc/AdvancedFileSupport.c
x86_64-linux-gnu-gcc: libsrc/GetGrids.c
x86_64-linux-gnu-gcc: libsrc/AttSupport.c
x86_64-linux-gnu-gcc: libsrc/cmpf.c
x86_64-linux-gnu-gcc: libsrc/date.c
x86_64-linux-gnu-gcc: libsrc/NclCCM.c
x86_64-linux-gnu-gcc: libsrc/FileSupport.c
x86_64-linux-gnu-gcc: libsrc/NclCoordVar.c
x86_64-linux-gnu-gcc: libsrc/NclData.c
x86_64-linux-gnu-gcc: libsrc/NclFile.c
x86_64-linux-gnu-gcc: libsrc/NclFileVar.c
x86_64-linux-gnu-gcc: libsrc/NclAdvancedGroup.c
x86_64-linux-gnu-gcc: libsrc/InitData.c
x86_64-linux-gnu-gcc: libsrc/NclGRIB.c
x86_64-linux-gnu-gcc: libsrc/ListSupport.c
x86_64-linux-gnu-gcc: libsrc/NclAtt.c
x86_64-linux-gnu-gcc: libsrc/NclMultiDValData.c
x86_64-linux-gnu-gcc: libsrc/NclNetCDF4.c
x86_64-linux-gnu-gcc: libsrc/NclTypechar.c
x86_64-linux-gnu-gcc: libsrc/NclOneDValCoordData.c
x86_64-linux-gnu-gcc: libsrc/NclMultiDVallistData.c
x86_64-linux-gnu-gcc: libsrc/NclTypedouble.c
x86_64-linux-gnu-gcc: libsrc/NclType.c
x86_64-linux-gnu-gcc: libsrc/NclTypebyte.c
x86_64-linux-gnu-gcc: libsrc/NclMultiDValnclfileData.c
x86_64-linux-gnu-gcc: libsrc/NclTypefloat.c
x86_64-linux-gnu-gcc: libsrc/NclTypeint64.c
x86_64-linux-gnu-gcc: libsrc/NclGroup.c
x86_64-linux-gnu-gcc: libsrc/NclNetCdf.c
x86_64-linux-gnu-gcc: libsrc/NclList.c
x86_64-linux-gnu-gcc: libsrc/NclTypeint.c
x86_64-linux-gnu-gcc: libsrc/NclTypelist.c
x86_64-linux-gnu-gcc: libsrc/NclTypeobj.c
x86_64-linux-gnu-gcc: libsrc/NclTypelogical.c
x86_64-linux-gnu-gcc: libsrc/NclTypeuint.c
x86_64-linux-gnu-gcc: libsrc/NclTypeshort.c
x86_64-linux-gnu-gcc: libsrc/NclTypelong.c
x86_64-linux-gnu-gcc: libsrc/NclVar.c
x86_64-linux-gnu-gcc: libsrc/NclTypeuint64.c
x86_64-linux-gnu-gcc: libsrc/NclTypestring.c
x86_64-linux-gnu-gcc: libsrc/nioConvert.c
x86_64-linux-gnu-gcc: libsrc/nio.c
x86_64-linux-gnu-gcc: libsrc/NclTypeulong.c
x86_64-linux-gnu-gcc: libsrc/nioError.c
x86_64-linux-gnu-gcc: libsrc/NclTypeubyte.c
x86_64-linux-gnu-gcc: libsrc/nioBase.c
x86_64-linux-gnu-gcc: libsrc/niohlu.c
x86_64-linux-gnu-gcc: libsrc/nioCallbacks.c
x86_64-linux-gnu-gcc: libsrc/nioNclHLUStub.c
x86_64-linux-gnu-gcc: libsrc/nioNresDB.c
x86_64-linux-gnu-gcc: libsrc/nio_path.c
x86_64-linux-gnu-gcc: libsrc/TypeSupport.c
x86_64-linux-gnu-gcc: libsrc/NclTypeushort.c
x86_64-linux-gnu-gcc: libsrc/VarSupport.c
x86_64-linux-gnu-gcc: libsrc/nioQuarks.c
x86_64-linux-gnu-gcc: libsrc/nioUtils.c
x86_64-linux-gnu-gcc: libsrc/nio_io.c
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-DNeedFuncProto -DNIO_LIB_ONLY -D_POSIX_SOURCE -DByteSwapped -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -Ilibsrc -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -c'
gfortran:f77: libsrc/llmisc.f
gfortran:f77: libsrc/qu2reg3.f
gfortran:f77: libsrc/ncepcode.f
gfortran:f77: libsrc/llmap.f
libsrc/qu2reg3.f:274:5:

  274 |   200 CONTINUE
      |     1
Warning: Label 200 at (1) defined but not used [-Wunused-label]
libsrc/qu2reg3.f:208:5:

  208 |   100 CONTINUE
      |     1
Warning: Label 100 at (1) defined but not used [-Wunused-label]
libsrc/ncepcode.f:2015:72:

 2015 |   100 E(I-1) = E(I)
      |                                                                        1
Warning: Fortran 2018 deleted feature: DO termination statement which is not END DO or CONTINUE with label 100 at (1)
libsrc/ncepcode.f:1878:18:

 1878 |           SLAT(J)=SLATD(J)
      |                  1
Warning: Possible change of value in conversion from REAL(8) to REAL(4) at (1) [-Wconversion]
libsrc/ncepcode.f:1879:18:

 1879 |           WLAT(J)=(2.*(1.-SLATD(J)**2))/(JMAX*PKM1(J))**2
      |                  1
Warning: Possible change of value in conversion from REAL(8) to REAL(4) at (1) [-Wconversion]
libsrc/ncepcode.f:1990:5:

 1990 |    20 DO 40 I = 1, N
      |     1
Warning: Label 20 at (1) defined but not used [-Wunused-label]
libsrc/ncepcode.f:1950:10:

 1949 |       DO 100 J=2,N
      |                                                                        2
 1950 |       WRK(J)= (J-1.D0)/DSQRT((2.D0*J-1.D0)*(2.D0*J-3.D0))
      |          1
Warning: Array reference at (1) out of bounds (2 > 1) in loop beginning at (2)
libsrc/ncepcode.f:1952:17:

 1949 |       DO 100 J=2,N
      |                                                                        2
......
 1952 |       E(J) = WRK(J)
      |                 1
Warning: Array reference at (1) out of bounds (2 > 1) in loop beginning at (2)
libsrc/ncepcode.f:1980:38:

 1980 |       SUBROUTINE DRSTNIO(NM,N,W,E,MATZ,Z,IERR)
      |                                      1
Warning: Unused dummy argument ‘matz’ at (1) [-Wunused-dummy-argument]
libsrc/ncepcode.f:1837:21:

 1837 |       INTEGER JHE,JHO,J0,NEXP
      |                     1
Warning: Unused variable ‘jho’ declared at (1) [-Wunused-variable]
gfortran:f77: libsrc/rowina3.f
libsrc/rowina3.f:308:5:

  308 |   300   CONTINUE
      |     1
Warning: Label 300 at (1) defined but not used [-Wunused-label]
libsrc/rowina3.f:248:5:

  248 |   200 CONTINUE
      |     1
Warning: Label 200 at (1) defined but not used [-Wunused-label]
libsrc/rowina3.f:130:5:

  130 |   100 CONTINUE
      |     1
Warning: Label 100 at (1) defined but not used [-Wunused-label]
libsrc/rowina3.f:172:19:

  172 |             IP   = ZWT
      |                   1
Warning: Possible change of value in conversion from REAL(4) to INTEGER(4) at (1) [-Wconversion]
libsrc/rowina3.f:217:21:

  217 |               IP   = ZWT
      |                     1
Warning: Possible change of value in conversion from REAL(4) to INTEGER(4) at (1) [-Wconversion]
libsrc/rowina3.f:295:17:

  295 |             IP = ZWT+1
      |                 1
Warning: Possible change of value in conversion from REAL(4) to INTEGER(4) at (1) [-Wconversion]
gfortran:f77: libsrc/scm0.f
libsrc/llmap.f:2251:72:

 2251 |         IF (IPRJ-11) 100,197,113
      |                                                                        1
Warning: Fortran 2018 deleted feature: Arithmetic IF statement at (1)
libsrc/llmap.f:2256:72:

 2256 |   100   IF (IPRJ-1) 101,102,103
      |                                                                        1
Warning: Fortran 2018 deleted feature: Arithmetic IF statement at (1)
libsrc/llmap.f:2426:72:

 2426 |   113   IF (IPRJ-13) 114,115,116
      |                                                                        1
Warning: Fortran 2018 deleted feature: Arithmetic IF statement at (1)
libsrc/rowina3.f:209:0:

  209 |                 P(JL) = PW(IP+1,1)
      |
Warning: ‘ip’ may be used uninitialized in this function [-Wmaybe-uninitialized]
libsrc/llmap.f:1466:5:

 1466 |   641   RLAT=CLAT
      |     1
Warning: Label 641 at (1) defined but not used [-Wunused-label]
libsrc/llmap.f:2460:5:

 2460 |   199   U=U-UOFF
      |     1
Warning: Label 199 at (1) defined but not used [-Wunused-label]
libsrc/llmap.f:2222:31:

 2222 |         DOUBLE PRECISION MDGDDP,RBGDFE,RBGLEN
      |                               1
Warning: Unused variable ‘mdgddp’ declared at (1) [-Wunused-variable]
libsrc/llmap.f:2218:29:

 2218 |         REAL             USNG,VSNG
      |                             1
Warning: Unused variable ‘usng’ declared at (1) [-Wunused-variable]
libsrc/llmap.f:2218:34:

 2218 |         REAL             USNG,VSNG
      |                                  1
Warning: Unused variable ‘vsng’ declared at (1) [-Wunused-variable]
libsrc/llmap.f:1832:29:

 1832 |         REAL             SLAT,SLON
      |                             1
Warning: Unused variable ‘slat’ declared at (1) [-Wunused-variable]
libsrc/llmap.f:1832:34:

 1832 |         REAL             SLAT,SLON
      |                                  1
Warning: Unused variable ‘slon’ declared at (1) [-Wunused-variable]
libsrc/scm0.f:41:5:

   41 |   100 CONTINUE
      |     1
Warning: Label 100 at (1) defined but not used [-Wunused-label]
x86_64-linux-gnu-gcc-ar: adding 50 object files to build/temp.linux-x86_64-3.8/libnio.a
x86_64-linux-gnu-gcc-ar: adding 19 object files to build/temp.linux-x86_64-3.8/libnio.a
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
extending extension 'PyNIO._nio' defined_macros with [('NeedFuncProto', None), ('NIO_LIB_ONLY', None), ('_POSIX_SOURCE', None), ('ByteSwapped', None)]
customize UnixCCompiler
customize UnixCCompiler using build_ext
get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
customize Gnu95FCompiler
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'PyNIO._nio' extension
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-DNeedFuncProto -DNIO_LIB_ONLY -DNeedFuncProto -DNIO_LIB_ONLY -D_POSIX_SOURCE -DByteSwapped -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -Ilibsrc -I/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include -I/usr/include/python3.8 -c'
x86_64-linux-gnu-gcc: niomodule.c
niomodule.c: In function ‘data_type’:
niomodule.c:688:3: warning: ‘NPY_CHAR’ is deprecated: Use NPY_STRING [-Wdeprecated-declarations]
  688 |   return NPY_CHAR;
      |   ^~~~~~
In file included from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from niomodule.c:24:
/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:21: note: declared here
   84 |                     NPY_CHAR NPY_ATTR_DEPRECATE("Use NPY_STRING"),
      |                     ^~~~~~~~
niomodule.c: In function ‘typecode’:
niomodule.c:790:2: warning: ‘NPY_CHAR’ is deprecated: Use NPY_STRING [-Wdeprecated-declarations]
  790 |  case NPY_CHAR:
      |  ^~~~
In file included from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from niomodule.c:24:
/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:21: note: declared here
   84 |                     NPY_CHAR NPY_ATTR_DEPRECATE("Use NPY_STRING"),
      |                     ^~~~~~~~
niomodule.c: In function ‘_convertCOMPOUND2Obj’:
niomodule.c:5273:17: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
 5273 |   for (n = 0; n < comprec->n_comps; ++n) {
      |                 ^
niomodule.c:5290:5: warning: ‘NPY_CHAR’ is deprecated: Use NPY_STRING [-Wdeprecated-declarations]
 5290 |     tmpobj = PyArray_SimpleNewFromData(1, &length, NPY_CHAR,
      |     ^~~~~~
In file included from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from niomodule.c:24:
/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:21: note: declared here
   84 |                     NPY_CHAR NPY_ATTR_DEPRECATE("Use NPY_STRING"),
      |                     ^~~~~~~~
niomodule.c:5299:5: warning: ‘NPY_CHAR’ is deprecated: Use NPY_STRING [-Wdeprecated-declarations]
 5299 |     tmpobj = PyArray_SimpleNewFromData(1, &length, NPY_CHAR,
      |     ^~~~~~
In file included from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from niomodule.c:24:
/home/hafez/.local/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:21: note: declared here
   84 |                     NPY_CHAR NPY_ATTR_DEPRECATE("Use NPY_STRING"),
      |                     ^~~~~~~~
niomodule.c: In function ‘_convertObj2COMPOUND’:
niomodule.c:5701:13: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]
 5701 |    if (len2 != comprec->n_comps) {
      |             ^~
niomodule.c:5723:42: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘ng_size_t’ {aka ‘long int’} [-Wformat=]
 5723 |         "\tFailed to allocate memory of %d char.\n",
      |                                         ~^
      |                                          |
      |                                          int
      |                                         %ld
 5724 |         compnode->nvals);
      |         ~~~~~~~~~~~~~~~
      |                 |
      |                 ng_size_t {aka long int}
niomodule.c: In function ‘NioVariable_WriteArray’:
niomodule.c:5776:20: warning: variable ‘dimrec’ set but not used [-Wunused-but-set-variable]
 5776 |  NclFileDimRecord* dimrec = NULL;
      |                    ^~~~~~
niomodule.c: In function ‘NioVariableObject_ass_slice’:
niomodule.c:6556:40: warning: integer overflow in expression of type ‘long int’ results in ‘0’ [-Woverflow]
 6556 |    if (indices->start < PY_SSIZE_T_MIN * 100)
      |                                        ^
niomodule.c: In function ‘NioVariableObject_ass_subscript’:
niomodule.c:6626:41: warning: integer overflow in expression of type ‘long int’ results in ‘0’ [-Woverflow]
 6626 |     if (indices->start < PY_SSIZE_T_MIN * 100)
      |                                         ^
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/niomodule.o  -Llibsrc -Lbuild/temp.linux-x86_64-3.8 -lnio -ljpeg -lpng -lz -lnetcdf -o build/lib.linux-x86_64-3.8/PyNIO/_nio.cpython-38-x86_64-linux-gnu.so
x86_64-linux-gnu-g++: error: : No such file or directory
error: Command "x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/niomodule.o  -Llibsrc -Lbuild/temp.linux-x86_64-3.8 -lnio -ljpeg -lpng -lz -lnetcdf -o build/lib.linux-x86_64-3.8/PyNIO/_nio.cpython-38-x86_64-linux-gnu.so" failed with exit status 1
----------------------------------------

ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-3kmcfei0/setup.py'"'"'; file='"'"'/tmp/pip-req-build-3kmcfei0/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-2u8mgvoj/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/hafez/.local/include/python3.8/PyNIO Check the logs for full command output.
hafez@LAPTOP-G6UK4AFP:~$

pynio & netCDF4 memory leak

When using pynio and netCDF4 in the same script, there appears to be a memory leak.

Currently, I only use netCDF4 to write out .nc files due to the ability to include significant digit and zlib compression. E.g.,

nc.createVariable('test_var','d',('time','lat','lon',),zlib=True, least_significant_digit=2)

when running the script with only pynio for netCDF output, there is no memory leak. Are there any plans to include zlib and least_significant_digit arguments (available in netCDF4) in pynio so users can take advantage of this compression?

Unable to install PyNIO and PyNGL in Debian Linux

Hi, I try to install pynio and pyngl in Debian Linux (running in a virtual machine) by using "conda create --name pyn_env --channel conda-forge pynio pyngl", but it failed.

These are details:

  • Virtual machine version: VMware® Workstation 14 Pro 14.1.2 build-8497320
  • Operating system: Windows 10, 64-bit (Build 17134) 10.0.17134
  • Debian version: Linux debian 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u1 (2018-08-03) x86_64 GNU/Linux
  • Enter commands at the console:
shigan@debian:~$ conda create --name pyn_env --channel conda-forge pynio pyngl
Fetching package metadata .............
Solving package specifications: .


UnsatisfiableError: The following specifications were found to be in conflict:
  - pynio
Use "conda info <package>" to see the dependencies for each package.


shigan@debian:~$ conda create --name pyn_env --channel conda-forge pyngl
Fetching package metadata .............
Solving package specifications: .


UnsatisfiableError: The following specifications were found to be in conflict:
  - pyngl
Use "conda info <package>" to see the dependencies for each package.

  • conda list:
shigan@debian:~$ conda list
# packages in environment at /home/shigan/miniconda2:
#
cffi                      1.9.1                    py27_0    defaults
conda                     4.3.14                   py27_0    defaults
conda-env                 2.6.0                         0    defaults
cryptography              1.7.1                    py27_0    defaults
enum34                    1.1.6                    py27_0    defaults
idna                      2.2                      py27_0    defaults
ipaddress                 1.0.18                   py27_0    defaults
libffi                    3.2.1                         1    defaults
openssl                   1.0.2k                        1    defaults
pip                       9.0.1                    py27_1    defaults
pyasn1                    0.1.9                    py27_0    defaults
pycosat                   0.6.1                    py27_1    defaults
pycparser                 2.17                     py27_0    defaults
pyopenssl                 16.2.0                   py27_0    defaults
python                    2.7.13                        0    defaults
readline                  6.2                           2    defaults
requests                  2.12.4                   py27_0    defaults
ruamel_yaml               0.11.14                  py27_1    defaults
setuptools                27.2.0                   py27_0    defaults
six                       1.10.0                   py27_0    defaults
sqlite                    3.13.0                        0    defaults
tk                        8.5.18                        0    defaults
wheel                     0.29.0                   py27_0    defaults
yaml                      0.1.6                         0    defaults
zlib                      1.2.8                         3    defaults

  • conda info:
shigan@debian:~$ conda info
Current conda install:

               platform : linux-64
          conda version : 4.3.14
       conda is private : False
      conda-env version : 4.3.14
    conda-build version : not installed
         python version : 2.7.13.final.0
       requests version : 2.12.4
       root environment : /home/shigan/miniconda2  (writable)
    default environment : /home/shigan/miniconda2
       envs directories : /home/shigan/miniconda2/envs
                          /home/shigan/.conda/envs
          package cache : /home/shigan/miniconda2/pkgs
                          /home/shigan/.conda/pkgs
           channel URLs : https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/linux-64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : /home/shigan/.condarc
           offline mode : False
             user-agent : conda/4.3.14 requests/2.12.4 CPython/2.7.13 Linux/4.9.0-7-amd64 debian/9.5 glibc/2.24
                UID:GID : 1000:1000

Library not loaded: @rpath/libnetcdf.11.dylib

To whom it may concern,

I have tried installing pynio on my Mac osX system by running conda install -c dbrown pynio. The results give:

Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
Solving package specifications: .........

Package plan for installation in environment /Users/UchidaTakaya/anaconda/envs/PyNIO:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pynio-1.5.0beta            |      np110py27_8         3.1 MB

The following NEW packages will be INSTALLED:

    szip:  2.1-0                

The following packages will be UPDATED:

    pynio: 1.4.3-np110py27_3 --> 1.5.0beta-np110py27_8

Proceed ([y]/n)? y

Fetching packages ...
pynio-1.5.0bet 100% |#############################################################################| Time: 0:00:01   2.05 MB/s
Extracting packages ...
[      COMPLETE      ]|################################################################################################| 100%
Unlinking packages ...
[      COMPLETE      ]|################################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|################################################################################################| 100%

so it seems to be installed. However, when I try to import Nio, an error shows up as:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "Nio.py", line 83, in <module>
    from nio import *
ImportError: dlopen(./nio.so, 2): Library not loaded: @rpath/libnetcdf.11.dylib
  Referenced from: /Users/UchidaTakaya/anaconda/envs/PyNIO/lib/python2.7/site-packages/PyNIO/nio.so
  Reason: image not found

I looked for the libnetcdf.11.dylib library by running otool -L /(package_path)/PyNIO/nio.so, which shows that it is downloaded:

/Users/UchidaTakaya/anaconda/envs/PyNIO/lib/python2.7/site-packages/PyNIO/nio.so:
    @rpath/libjpeg.8.dylib (compatibility version 13.0.0, current version 13.0.0)
    @rpath/libpng16.16.dylib (compatibility version 34.0.0, current version 34.0.0)
    @rpath/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
    @rpath/libnetcdf.11.dylib (compatibility version 12.0.0, current version 12.0.0)
    @rpath/libhdf5_hl.10.dylib (compatibility version 11.0.0, current version 11.2.0)
    @rpath/libhdf5.10.dylib (compatibility version 12.0.0, current version 12.0.0)
    @rpath/libcurl.4.dylib (compatibility version 9.0.0, current version 9.0.0)
    @rpath/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    @rpath/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    @rpath/libgdal.20.dylib (compatibility version 21.0.0, current version 21.2.0)
    @rpath/./libproj.9.dylib (compatibility version 10.0.0, current version 10.0.0)
    @rpath/./libiconv.2.dylib (compatibility version 8.0.0, current version 8.1.0)
    @rpath/./libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    @rpath/./libstdc++.6.dylib (compatibility version 7.0.0, current version 7.19.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    @rpath/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Could you please tell me how to resolve this issue? Thank you very much for your attention and support.

python: /lib64/libc.so.6: version `GLIBC_2.10' not found (required by python)

After I installed the PyNgl and PyNio with this command ,conda create -n ncl_to_python -c conda-forge xarray netcdf4 scipy pyngl pynio ncl, I entered the ncl_to_python, but there was a mistake happen when I use python, and import nio was also unsuccessful.

Then I tried the method showed in #30, after that the same mistake happened.

What should I do?

Putting variables in files, NCL style

Hi everyone,

One of the most useful functionalities of NCL was that metadata was passed along with the variables, which made writing NC files from scratch (among many other things) much easier. For example, if you wanted to isolate just variable T from a big file containing several others in a file with just it, it was as simple as:

fbig = addfile("thebigfile.nc","r")
var = fbig->T
fnew = addfile("thenewfile.nc","c")
fnew->T = var

However, in PyNIO you apparently have to create the variable beforehand in the file, and have the dimensions created beforehand. Also, subsetting generally breaks the metadata and there is no copy_VarCoods-like function. Is there a plan to implement those behaviours, or maybe something already implemented that I'm not aware of?

If there's a plan, I'd be glad to contribute if someone points me in the right direction, at least with some testing.

Thanks,
Gabriel

Nio.pth not installed with absolute pkgs_pth

I've been working on installing PyNIO from source on Ubuntu 18.04. I'm able to install the package, with the exception of the Nio.pth file. The result is that, when I try to import the package, I get

ImportError: No module named Nio

After some experimentation I found that distutils doesn't respect absolute file paths for data_files in the setup function. I was able to fix this by replacing the absolute path with a relative path:

setup (version      = pynio_version,
       description  = 'Multi-format data I/O package',
       author       = 'David I. Brown',
       author_email = '[email protected]',
       url          = 'http://www.pyngl.ucar.edu/Nio.shtml',
       long_description = '''
       Enables NetCDF-like access for NetCDF (rw), HDF (rw), HDF-EOS2 (r), HDF-EOS5, GRIB (r), and CCM (r) data files
       ''',
       package_data = { pynio_pkg_name : data_files },
       data_files   = [('.', pynio_pth_file)],
       **configuration().todict())

I tested this out on Ubuntu 18.04 with both python 2 and 3, and both pip and python setup.py install. In each case, only the relative path copied Nio.pth to the correct location.

Problems Loading Nio

I tried to install pynio through anaconda. I'm having some issues.

After a fresh install

conda create -c conda-forge pynio

[barryb@aitken ~]$ python
Python 2.7.12 |Anaconda custom (64-bit)| (default, Jul 2 2016, 17:42:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org

import Nio
Traceback (most recent call last):
File "", line 1, in
File "/data/aqf/barryb/anaconda2/lib/python2.7/site-packages/PyNIO/Nio.py", line 63, in
from nio import *
ImportError: /data/aqf/barryb/anaconda2/lib/python2.7/site-packages/PyNIO/../../.././libgeos_c.so.1: ELF file OS ABI invalid

Clean environment with latest PyNio on conda-forge appears to be broken

Hi all,

Today I was unable to import Nio after updating my default anaconda environment, so I reproduced the problem with a clean environment:

[ddawson@dawsonlaptop notebooks (master)]$ conda create --name pyniotest pynio
Solving environment: done

## Package Plan ##

environment location: /Users/ddawson/anaconda/envs/pyniotest

added / updated specs:
- pynio

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
certifi-2018.4.16          |           py27_0         142 KB  conda-forge
libpq-9.6.3                |                0          89 KB  conda-forge
------------------------------------------------------------
                                       Total:         232 KB

The following NEW packages will be INSTALLED:

boost:           1.66.0-py27_1         conda-forge
boost-cpp:       1.66.0-1              conda-forge
bzip2:           1.0.6-1               conda-forge
ca-certificates: 2018.4.16-0           conda-forge
cairo:           1.14.10-0             conda-forge
certifi:         2018.4.16-py27_0      conda-forge
curl:            7.59.0-1              conda-forge
expat:           2.2.5-0               conda-forge
fontconfig:      2.12.6-0              conda-forge
freetype:        2.8.1-0               conda-forge
freexl:          1.0.5-0               conda-forge
g2clib:          1.6.0-5               conda-forge
geos:            3.6.2-1               conda-forge
geotiff:         1.4.2-1               conda-forge
gettext:         0.19.8.1-0            conda-forge
giflib:          5.1.4-0               conda-forge
glib:            2.55.0-0              conda-forge
hdf4:            4.2.13-0              conda-forge
hdf5:            1.10.1-2              conda-forge
hdfeos2:         2.19.1-8              conda-forge
hdfeos5:         5.1.16-0              conda-forge
icu:             58.2-0                conda-forge
intel-openmp:    2018.0.0-8
jasper:          1.900.1-4             conda-forge
jpeg:            9b-2                  conda-forge
json-c:          0.13.1-h470a237_0     conda-forge
kealib:          1.4.7-4               conda-forge
krb5:            1.14.6-0              conda-forge
libdap4:         3.18.3-2              conda-forge
libffi:          3.2.1-3               conda-forge
libgdal:         2.2.4-2               conda-forge
libgfortran:     3.0.1-h93005f0_2
libiconv:        1.15-0                conda-forge
libkml:          1.3.0-6               conda-forge
libnetcdf:       4.6.1-2               conda-forge
libpng:          1.6.34-0              conda-forge
libpq:           9.6.3-0               conda-forge
libspatialite:   4.3.0a-19             conda-forge
libssh2:         1.8.0-2               conda-forge
libtiff:         4.0.9-0               conda-forge
libuuid:         1.0.3-1               conda-forge
libxml2:         2.9.8-0               conda-forge
mkl:             2018.0.2-1
mkl_fft:         1.0.2-py27_0          conda-forge
mkl_random:      1.0.1-py27_0          conda-forge
ncurses:         5.9-10                conda-forge
numpy:           1.14.2-py27ha9ae307_1
openjpeg:        2.3.0-2               conda-forge
openssl:         1.0.2o-0              conda-forge
pcre:            8.41-1                conda-forge
pip:             9.0.3-py27_0          conda-forge
pixman:          0.34.0-2              conda-forge
poppler:         0.61.1-3              conda-forge
poppler-data:    0.4.9-0               conda-forge
proj4:           4.9.3-5               conda-forge
pynio:           1.5.0-py27_4          conda-forge
python:          2.7.15-0              conda-forge
readline:        7.0-0                 conda-forge
setuptools:      39.1.0-py27_0         conda-forge
sqlite:          3.20.1-2              conda-forge
tk:              8.6.7-0               conda-forge
wheel:           0.31.0-py27_0         conda-forge
xerces-c:        3.2.0-0               conda-forge
xz:              5.2.3-0               conda-forge
zlib:            1.2.11-0              conda-forge

My condarc is as follows:

channels:
  - conda-forge
  - ncar
  - defaults

And here is the error message I get upon attempting to import Nio:

[ddawson@dawsonlaptop notebooks (master)]$ source activate pyniotest
(pyniotest) [ddawson@dawsonlaptop notebooks (master)]$ python
Python 2.7.15 | packaged by conda-forge | (default, May  8 2018, 14:53:14)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import Nio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ddawson/anaconda/envs/pyniotest/lib/python2.7/site-packages/PyNIO/Nio.py", line 83, in <module>
    from nio import *
ImportError: dlopen(/Users/ddawson/anaconda/envs/pyniotest/lib/python2.7/site-packages/PyNIO/nio.so, 2): Library not loaded: @rpath/libjson-c.2.dylib
  Referenced from: /Users/ddawson/anaconda/envs/pyniotest/lib/libgdal.20.dylib
  Reason: image not found

Any ideas on how to fix this? Many thanks!

Errors opening GRIB file with xarray, engine="pynio" (using python 3 pynio developer branch)

Hi folks,

Thanks for updating pynio for python3 (#10). I created a virtual environment for the developer branch (as advised) with conda create -n pynio -c ncar -c conda-forge pynio=dev python=3, activated it with . activate pynio, and installed xarray to the virtual environment with conda install xarray. Unfortunately when trying to open an ERA-Interim GRIB file I get a ModuleNotFound error (the module Nio can't be found); included is the full traceback.

In [1]: import xarray as xr

In [2]: f = xr.open_dataset('ei.moda.an.sfc.regn128sc.2012120100.grb', engine='pynio')
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-9da81691cb02> in <module>()
----> 1 f = xr.open_dataset('ei.moda.an.sfc.regn128sc.2012120100.grb', engine='pynio')

/home/user/anaconda3/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables)
    291         elif engine == 'pynio':
    292             store = backends.NioDataStore(filename_or_obj,
--> 293                                           autoclose=autoclose)
    294         else:
    295             raise ValueError('unrecognized engine for open_dataset: %r'

/home/user/anaconda3/lib/python3.6/site-packages/xarray/backends/pynio_.py in __init__(self, filename, mode, autoclose)
     40     """
     41     def __init__(self, filename, mode='r', autoclose=False):
---> 42         import Nio
     43         opener = functools.partial(Nio.open_file, filename, mode=mode)
     44         self.ds = opener()

ModuleNotFoundError: No module named 'Nio'

Also, for what it's worth, I also tried installing pynio without creating a virtual environment into my existing anaconda environment; in this case, I get an entirely different error having to do with a missing object file from the library "libkea":

In [1]: import xarray as xr
In [2]: f = xr.open_dataset('ei.moda.an.sfc.regn128sc.2012120100.grb', engine='pynio')
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-4-9da81691cb02> in <module>()
----> 1 f = xr.open_dataset('ei.moda.an.sfc.regn128sc.2012120100.grb', engine='pynio')

/home/user/anaconda3/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables)
    291         elif engine == 'pynio':
    292             store = backends.NioDataStore(filename_or_obj,
--> 293                                           autoclose=autoclose)
    294         else:
    295             raise ValueError('unrecognized engine for open_dataset: %r'

/home/user/anaconda3/lib/python3.6/site-packages/xarray/backends/pynio_.py in __init__(self, filename, mode, autoclose)
     40     """
     41     def __init__(self, filename, mode='r', autoclose=False):
---> 42         import Nio
     43         opener = functools.partial(Nio.open_file, filename, mode=mode)
     44         self.ds = opener()

/home/user/anaconda3/lib/python3.6/site-packages/PyNIO/Nio.py in <module>()
     81 '''
     82 from __future__ import print_function
---> 83 from _nio import *
     84 
     85 #

ImportError: libkea.so.1.4: cannot open shared object file: No such file or directory

Any idea what these issues could be?

pynio installation downgrades esmf/esmpy to no mpi versions

I have an environment where esmf and esmpy are installed with MPI support.
When I attempt to install pynio (conda install --channel conda-forge pynio), I get notified that many of the libraries I have already installed will be updated or downgraded to versions without MPI support:

libnetcdf 4.7.4-mpi_mpich_hfd9c5b6_5 --> 4.7.4-nompi_h9f9fd6a_101
netcdf-fortran 4.5.3-mpi_mpich_h3923e1a_0 --> 4.5.2-nompi_h45d7149_104
netcdf4 1.5.4-nompi_py38hec8b9af_102 --> 1.5.3-nompi_py38heb6102f_103
hdf5 1.10.6-mpi_mpich_ha7d0aea_1 --> 1.10.5-nompi_h3c11f04_1104
esmf 8.0.1-mpi_mpich_h213fab7_100 --> 8.0.0-nompi_hb0fcdcb_6
esmpy 8.0.1-mpi_mpich_py38h08b7006_100 --> 8.0.0-nompi_py38hf0e99fa_1

However, disabling MPI support disables some functionality I use that depends on MPI.
How can I install PYNIO without these packages loosing MPI support?

Thank you,
Adam

Unable to install pynio under linux-aarch64

Attempting to install pynio using the recommended command, 'conda install -c conda-forge pynio' produces the following result:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  • pynio

Current channels:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

cannot open hdf or netcdf files

Hi Pynio team!
I cannot open cdf or hdf files using nio.

I have a windows 10, so I've installed PyNio and PyNgl using WSL. I run python 3.6.6 using WSL, but I cannot open hdf or .nc file containing lightning data from NASA. I suspect, that something is wrong with the filepath - but have a hard time confirming it. Do you have any good suggestions?

I have no trouble importing NIO og Ngl.

Thanks !

build of h5reader.c fails when compiling against hdf5-openmpi 1.12.0

I have been trying to set up a package of pynio on archlinux with the following package versions:

pynio 1.5.5
python 3.9.1-2
ncl 6.6.2-1
hdf-eos2 2.20-1
hdf-eos5 1.16-2
hdf4 4.2.15-1
hdf5-openmpi 1.12.0-2

Trying to build pynio I encountered the following errors:

libsrc/h5reader.c:42:14: error: too few arguments to function ‘H5Oget_info_by_name3’
   42 |     status = H5Oget_info_by_name(fid, root_name, &oi, H5P_DEFAULT);

libsrc/h5reader.c:2491:83: error: ‘H5O_info2_t’ has no member named ‘addr’
 2491 |             if(NULL == (already_visited = _NclHDF5_addr_visited(udata->seen, oinfo.addr)))

libsrc/h5reader.c:2693:12: error: too few arguments to function ‘H5Oget_info3’
 2693 |         if(H5Oget_info(type, &oi) >= 0)

libsrc/h5reader.c:4486:8: error: too few arguments to function ‘H5Oget_info_by_name3’
 4486 |     if(H5Oget_info_by_name(fid, grp_name, &oinfo, H5P_DEFAULT) < 0)

and similar repetitions. This looks to me like a version incompatibility or a bug in pynio's h5reader.c. All of the errors point to a discrepancy to the header files associated to my hdf5 installation (with openmpi support). Has anyone else encountered this problem?

Any information on a fix would be very much appreciated.

Alma Linux ModuleNotFoundError: No module named '_nio'

I have successfully built and installed pynio from the github repository for centos-7. I am migrating all my servers to AlmaLinux because of the change in how Centos is configured. When I try to install pynio from github I get several failures: 1) NCLnoPrintElem is multiply defined and 2) ModuleNotFoundError: No module named '_nio'

I have been able to fix the NCLnoPrintElem definitions in niomodule.c and NclMultiDValData.c which then allows all the library member to compile. Importing Nio from python3.9 gives ModuleNotFoundError: No module named '_nio'

I tried the suggestions in the mailing list but those were from 2012 and python2.7 and nothing changed

curl error details... xarray dataset variable only returns array full of 0 values

Disclaimer: I'm a relative python newbie...

I have installed pynio in a python 2.7 environment by doing the following:

>> ./Miniconda3-latest-MacOSX-x86_64.sh
>> conda create -n py27 python=2.7 anaconda
>> source activate py27
>> conda install numpy pandas jupyter basemap xarray netcdf4 dask
>> conda install -c ncar pynio

Then in ipython (IPython 5.7.0) using Python 2.7.15 |Anaconda custom (64-bit) I do:
In [1]: import xarray as xr
In [2]: url = "http://nomads.ncep.noaa.gov:9090/dods/gfs_0p25_1hr/gfs20180725/gfs_0p25_1hr_00z"
In [3]: ds = xr.open_dataset(url, engine = 'pynio')
In [5]: ds.ugrd10m
Out[5]:
<xarray.DataArray 'ugrd10m' (time: 121, lat: 721, lon: 1440)>
[125627040 values with dtype=float32]
Coordinates:
* lon (lon) float64 0.0 0.25 0.5 0.75 1.0 1.25 1.5 1.75 2.0 2.25 2.5 ...
* lat (lat) float64 -90.0 -89.75 -89.5 -89.25 -89.0 -88.75 -88.5 ...
* time (time) datetime64[ns] 2018-07-25 2018-07-25T01:00:00 ...
Attributes:
long_name: ** 10 m above ground u-component of wind [m/s]

When I try to get the values I it returns an array filled with 0's:
In [6]: ds.ugrd10m.values
**curl error details:**
Out[6]:
array([[[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
...,
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.]],

[[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
...,
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.]]], dtype=float32)

Interestingly when select only one timestep of the dataset it seems to work:
In [7]: ds = xr.open_dataset(url, engine = 'pynio').isel(time = 0)
In [8]: ds.ugrd10m
Out[8]:
<xarray.DataArray 'ugrd10m' (lat: 721, lon: 1440)>
[1038240 values with dtype=float32]
Coordinates:
* lon (lon) float64 0.0 0.25 0.5 0.75 1.0 1.25 1.5 1.75 2.0 2.25 2.5 ...
* lat (lat) float64 -90.0 -89.75 -89.5 -89.25 -89.0 -88.75 -88.5 ...
time datetime64[ns] 2018-07-25
Attributes:
long_name: ** 10 m above ground u-component of wind [m/s]

In [9]: ds.ugrd10m.values
Out[9]:
array([[-3.88510251, -3.85510254, -3.83510256, ..., -3.96510243,
-3.93510246, -3.91510248],
[-4.59510231, -4.56510258, -4.52510262, ..., -4.68510246,
-4.65510225, -4.62510252],
[-5.53510237, -5.51510239, -5.47510242, ..., -5.62510252,
-5.59510231, -5.56510258],
...,
[-3.43510246, -3.43510246, -3.43510246, ..., -3.43510246,
-3.43510246, -3.43510246],
[-3.22510242, -3.22510242, -3.22510242, ..., -3.22510242,
-3.22510242, -3.22510242],
[-3.0051024 , -3.0051024 , -3.0051024 , ..., -3.0051024 ,
-3.0051024 , -3.0051024 ]], dtype=float32)

I'm not sure what the cause of the error is, but it seems to be related to this curl error details: message. And I was thinking it might have to do with a memory issue?

I want to write the xarray Dataset to a netcdf file so I can work with it in py36.

GRIB2 memory leak

There appears to be a memory leak when opening GRIB2 files.

Example

while True:
    ds = Nio.open_file('ruc2_130_20110624_0100_000.grb2')
    ds.close()

In this example the amount of memory used on my computer increases by about 9MB each iteration. (Download the example file here.)

I found a discussion on the NCL mailing list about a memory leak with GRIB2 files, and it was apparently fixed last year. If this is related to the PyNIO memory leak, maybe it can be fixed just by updating the NCL files?

conda installation not working on Mac OS

I have tried installing PyNIO through anaconda. Installation goes fine but when I import the library in my script I get the following error (I have tried both version 1.4.3 and 1.5.0):

File "myscript.py", line 1, in
import Nio
File "/Users/Romonet/anaconda/lib/python2.7/site-packages/PyNIO/Nio.py", line 63, in
from nio import *
ImportError: dlopen(/Users/Romonet/anaconda/lib/python2.7/site-packages/PyNIO/nio.so, 2): Library not loaded: @rpath/./libhdf5_cpp.10.dylib
Referenced from: /Users/Romonet/anaconda/lib//libkea.1.4.5.dylib
Reason: image not found

I then tried creating a symbolic link to libhdf5_cpp.10.dylib in /usr/lib but when I do that I get a different error.

pynio does not work for conda create under centos 8

'conda create -n pynio_env -c conda-forge pynio`

Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.

Unexpected behavior when opening Canadian global GEM 25km grib files using pynio

Hi all,

I am experiencing interesting behavior when opening a GEM global 25km resolution grib files using pynio.

ds=xr.open_dataset('CMC_glb_TMP_ISBL_700_latlon.24x.24_2020050600_P198.grb',engine='pynio')
print(ds)
<xarray.Dataset>
Dimensions: (lat_0: 751, lon_0: 1500)
Coordinates:

lat_0 (lat_0) float32 -90.0 -89.76 -89.52 ... 89.52 89.76 90.0
lon_0 (lon_0) float32 180.0 180.24 180.48 ... 539.52 539.76
Data variables:
TMP_P0_L100_GLL0 (lat_0, lon_0) float32 ...
print(ds['lon_0'])
<xarray.DataArray 'lon_0' (lon_0: 1500)>
array([180. , 180.24, 180.48, ..., 539.28, 539.52, 539.76], dtype=float32)
Coordinates:
lon_0 (lon_0) float32 180.0 180.24 180.48 180.72 ... 539.28 539.52 539.76
Attributes:
long_name: longitude
grid_type: Latitude/Longitude
units: degrees_east
Dj: [0.24]
Di: [0.24000001]
Lo2: [179.76]
La2: [90.]
Lo1: [180.]
La1: [-90.]

Notice that the range for lon_0 is 180->539.76. When I open using cfgrib as the xarray engine:

ds=xr.open_dataset('CMC_glb_TMP_ISBL_700_latlon.24x.24_2020050600_P198.grb',engine='cfgrib')
print(ds['longitude'])
<xarray.DataArray 'longitude' (longitude: 1500)>
array([-180. , -179.76, -179.52, ..., 179.28, 179.52, 179.76])
Coordinates:
time datetime64[ns] ...
step timedelta64[ns] ...
isobaricInhPa int64 ...

longitude (longitude) float64 -180.0 -179.8 -179.5 ... 179.5 179.8
valid_time datetime64[ns] ...
Attributes:
units: degrees_east
standard_name: longitude
long_name: longitude

Therefore, it appears 'cgfrib' shows the longitude correctly while 'pynio' does not.

This was originally posted as an xarray issue:
pydata/xarray#4040

However, after investigating further, I found that this occurs when using pynio directly also:
ds=Nio.open_file('CMC_glb_TMP_ISBL_700_latlon.24x.24_2020050600_P198.grb') ds.variables['lon_0'].get_value()

gives a result of:
array([180. , 180.24, 180.48, ..., 539.28, 539.52, 539.76], dtype=float32)

Installing pynio with anaconda

I'm trying to install pynio through anaconda.

I have a fresh environments named "pynio_env" created using conda create --name pynio_env --channel conda-forge pynio pyngl .

When importing with import pynio I get the following error message : ImportError: libicui18n.so.58: cannot open shared object file: No such file or directory.

Can I know how to solve this problem?

Using pynio on windows

Is there any workaround to help install and run pynio on windows system. Or is there any future plan to support this on windows?

Reading grib2 file causes an exception TypeError: illegal subscript type in Python3

When i reading a grib2 file via:
dataset = xarray.open_dataset('example.grib2', engine='pynio')
and then trying get a variable with specific index:
dataset.EXAMPLE_VARIABLE[1,2]
i getting the following error:
TypeError: illegal subscript type

Stacktrace:

Traceback (most recent call last):
File "", line 1, in
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/common.py", line 100, in repr
return formatting.array_repr(self)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/formatting.py", line 393, in array_repr
summary.append(short_array_repr(arr.values))
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/dataarray.py", line 412, in values
return self.variable.values
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/variable.py", line 396, in values
return _as_array_or_item(self._data)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/variable.py", line 217, in _as_array_or_item
data = np.asarray(data)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/numpy/core/numeric.py", line 531, in asarray
return array(a, dtype, copy=False, order=order)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 418, in array
self._ensure_cached()
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 415, in _ensure_cached
self.array = NumpyIndexingAdapter(np.asarray(self.array))
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/numpy/core/numeric.py", line 531, in asarray
return array(a, dtype, copy=False, order=order)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 399, in array
return np.asarray(self.array, dtype=dtype)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/numpy/core/numeric.py", line 531, in asarray
return array(a, dtype, copy=False, order=order)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 366, in array
return np.asarray(array[self.key], dtype=None)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/conventions.py", line 389, in getitem
self.scale_factor, self.add_offset, self.dtype)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/conventions.py", line 74, in mask_and_scale
values = np.array(array, dtype=dtype, copy=True)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 366, in array
return np.asarray(array[self.key], dtype=None)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/backends/pynio
.py", line 44, in getitem
return array[key]
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/PyNIO/Nio.py", line 347, in getitem
ret = get_variable(self.file, self.varname, xsel)
File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/PyNIO/coordsel.py", line 68, in get_variable
ret = file.file.variables[varname][xsel]

As i found problem in that items in xsel variable are instances of numpy.int64
So if i add a dirty hack:

if isinstance(xsel, tuple) and len(xsel) > 0 and isinstance(xsel[0], N.int64):
xsel = tuple(map(lambda x: int(x) if x else x, xsel))

all works properly

Can you help me this problem?

Non-conda build instructions

Are build instructions for non-conda users documented anywhere? Not all users want to use conda to install software on their systems. It would be nice if this could be packaged within Debian/Ubuntu at some point. Having some build instructions would be a step in that direction.

ncl_filedump for GRIB2 files

Hi,

An RDA user noticed that NCL treats GRIB2 files differently. Specifically, for a GRIB2 file that contains u- and v-components of the wind at multiple vertical levels (and where both the u- and v-components are in the same GRIB2 message), if the grid is global, ncl_filedump show variables like this:
109 cheyenne5:DATTORE341712 -> ncl_filedump global.grb2 |grep GRD
float UGRD_P0_L100_GLL0 ( lv_ISBL0, lat_0, lon_0 )
float VGRD_P0_L100_GLL0 ( lv_ISBL0, lat_0, lon_0 )

but if the grid is a spatial subset of the global grid, the u-component is still okay, but the v-component is broken up into separate variables by vertical level:
111 cheyenne5:DATTORE341713 -> ncl_filedump spatial_subset.grb2 | grep GRD
float UGRD_P0_L100_GLL0 ( lv_ISBL0, lat_0, lon_0 )
float VGRD_P0_L100_GLL0 ( lat_0, lon_0 )
float VGRD_P0_L100_GLL1 ( lat_1, lon_1 )
float VGRD_P0_L100_GLL2 ( lat_2, lon_2 )
float VGRD_P0_L100_GLL3 ( lat_3, lon_3 )
float VGRD_P0_L100_GLL4 ( lat_4, lon_4 )
float VGRD_P0_L100_GLL5 ( lat_5, lon_5 )
float VGRD_P0_L100_GLL6 ( lat_6, lon_6 )
float VGRD_P0_L100_GLL7 ( lat_7, lon_7 )
float VGRD_P0_L100_GLL8 ( lat_8, lon_8 )
float VGRD_P0_L100_GLL9 ( lat_9, lon_9 )
float VGRD_P0_L100_GLL10 ( lat_10, lon_10 )
float VGRD_P0_L100_GLL11 ( lat_11, lon_11 )
float VGRD_P0_L100_GLL12 ( lat_12, lon_12 )
float VGRD_P0_L100_GLL13 ( lat_13, lon_13 )
float VGRD_P0_L100_GLL14 ( lat_14, lon_14 )
float VGRD_P0_L100_GLL15 ( lat_15, lon_15 )
float VGRD_P0_L100_GLL16 ( lat_16, lon_16 )
float VGRD_P0_L100_GLL17 ( lat_17, lon_17 )
float VGRD_P0_L100_GLL18 ( lat_18, lon_18 )
float VGRD_P0_L100_GLL19 ( lat_19, lon_19 )
float VGRD_P0_L100_GLL20 ( lat_20, lon_20 )
float VGRD_P0_L100_GLL21 ( lat_21, lon_21 )
float VGRD_P0_L100_GLL22 ( lat_22, lon_22 )
float VGRD_P0_L100_GLL23 ( lat_23, lon_23 )
float VGRD_P0_L100_GLL24 ( lat_24, lon_24 )
float VGRD_P0_L100_GLL25 ( lat_25, lon_25 )
float VGRD_P0_L100_GLL26 ( lat_26, lon_26 )
float VGRD_P0_L100_GLL27 ( lat_27, lon_27 )
float VGRD_P0_L100_GLL28 ( lat_28, lon_28 )
float VGRD_P0_L100_GLL29 ( lat_29, lon_29 )
float VGRD_P0_L100_GLL30 ( lat_30, lon_30 )
float VGRD_P0_L100_GLL31 ( lat_31, lon_31 )
float VGRD_P0_L100_GLL32 ( lat_32, lon_32 )
float VGRD_P0_L100_GLL33 ( lat_33, lon_33 )
float VGRD_P0_L100_GLL34 ( lat_34, lon_34 )
float VGRD_P0_L100_GLL35 ( lat_35, lon_35 )
float VGRD_P0_L100_GLL36 ( lat_36, lon_36 )

Can someone please check this out when you get a chance? If you want me to supply sample files, let me know.

Thanks,
Bob

open .ncf files

I'm using the test data from the MEGANv3 model. These files are in .ncf format, but have a NetCDF structure. How can I open it using pynio?
Thanks :)

not finding libpoppler (Py3)

Using conda package (env file coming soon).
I get

>>> import Nio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/scollis/anaconda/envs/pchange/lib/python3.6/site-packages/PyNIO/Nio.py", line 83, in <module>
    from _nio import *
ImportError: dlopen(/Users/scollis/anaconda/envs/pchange/lib/python3.6/site-packages/PyNIO/_nio.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.66.dylib
  Referenced from: /Users/scollis/anaconda/envs/pchange/lib/libgdal.20.dylib
  Reason: image not found
>>> 

I have Poppler 0.61 installed.. There is no conda package on conda-forge for poppler-0.66

Duplicate symbols when compiling pynio from source using Python 3.7.4

Hello All:

I’m jumping into the migration from NCL into the python realm and am having difficulty getting started with pyNIO and pyNGL.

My primary program manager has been homebrew and my gcc compiler is 9.2.0. I am working with python 3.7.4 as my main default distro from homebrew and to date, it’s worked well and pip3 doesn’t create any major conflicts. But pip’s version of pyNIO are pyNGL are out of date (2006).

(Because of other projects I would prefer not to work through the conda manager due to previous [recent] extreme difficulties with conda with some packages that I need for other projects. Also I’d like a consistent development platform between my Mac and our larger department UNIX fleet since I have to manage both but do not have the r€$ouR¢e$ or time to do so).

I had luck getting pyNIO and pyNGL working with python 3.6.8

However.

In trying to bring pyNIO and pyNGL up and running with python 3.7.4, I’ve been getting “duplicate symbol” errors crashing the builds (_NCLnoPrintElem w/ pyNIO, and pcmp04 w/ pyNGL).

The output from the command ...

python3 ./setup.py install

... is in the gist link below.

https://gist.github.com/wjcapehart/d44f5618b5baa613cd5be4d328cdfa01

Please advise.
Bill Capehart
SD Mines

grib2_codetables not closing -> too many open files exception

Hi all,

We've been doing some work on grib2 files with xarray, and I think I've found a strange error that traces back to PyNIO. While I can't really go into great detail on the specifics regarding the work, I think I can be general enough about the error.

Basically, we're opening up several days worth of proprietary grib2 files, but some of the variables within the grib file aren't listed in the code table, resulting in this warning:

warning: Entry (250) not found in code table file /.../PyNIO/ncarg/grib2_codetables/centers.table
warning: Entry (250) not found in code table file /.../PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table

After 2430 iterations (every time!), PyNIO is convinced the previously accessed (or any) codetables are neither valid nor exist:

warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.16.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.16.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.5.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/0/0.0.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.2.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/3/3.1.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/1/1.3.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.5.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/0/0.0.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.2.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/3/3.1.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/1/1.3.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.5.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/0/0.0.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.2.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/3/3.1.table" not a valid GRIB2 code table.
warning: NclGRIB2: codetable file "/.../PyNIO/ncarg/grib2_codetables/ncep/1/1.3.table" not a valid GRIB2 code table.

Unfortunately, we do not create these grib2 files in house, so it isn't quite as easy as simply fixing the processing of the grib2 file. Editing the specific code table used would also fix the problem, but this seems like an underlying issue with how NclGRIB2.c handles fields that aren't an entry within the referenced grib table. As it turns out, the grib2 code tables are never closed:

In [6]: for i in xrange(10000):
...: f = Nio.open_file(fname)
...: f.close()
...: del(f)

fatal: NclGRIB2: codetable directory "/.../PyNIO/ncarg/grib2_codetables" invalid fatal:Could not open ()

In [9]: p = psutil.Process(os.getpid())

In [10]: open_files = [f for f in p.open_files() if 'PyNIO' in f.path]

In [11]: len(open_files)
Out[11]: 4856

In [12]: open_files[0]
Out[12]: popenfile(path='/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table', fd=5)

In [13]: open_files[1]
Out[13]: popenfile(path='/.../PyNIO/ncarg/grib2_codetables/centers.table', fd=6)

In [14]: open_files[-1]
Out[14]: popenfile(path='/.../PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table', fd=4863)

This is confirmed by trying to import/open/do much of anything else in the same iPython session:

In [15]: import pandas as pd
ERROR: Internal Python error in the inspect module.
Below is the traceback from this internal error.

Traceback (most recent call last):
File "/.../lib/python2.7/site-packages/IPython/core/ultratb.py", line 1118, in get_records
File "/.../lib/python2.7/site-packages/IPython/core/ultratb.py", line 300, in wrapped
File "/.../lib/python2.7/site-packages/IPython/core/ultratb.py", line 345, in _fixed_getinnerframes
File "/.../lib/python2.7/inspect.py", line 1049, in getinnerframes
File "/.../lib/python2.7/inspect.py", line 1009, in getframeinfo
File "/.../lib/python2.7/inspect.py", line 454, in getsourcefile
File "/.../lib/python2.7/inspect.py", line 483, in getmodule
File "/.../lib/python2.7/inspect.py", line 467, in getabsfile
File "/.../lib/python2.7/posixpath.py", line 364, in abspath
OSError: [Errno 24] Too many open files

This is a shot in the dark, but I suspect this is a result of a missing (void) fclose(fp) here for when an entry isn't found in the table: https://github.com/yyr/pynio/blob/master/libsrc/NclGRIB2.c#L7979, though I'm not the best at reading C and I haven't looked too much into it tbqh.

If this, in fact, is the issue, I could throw together a PR to resolve this issue.

For reference: OS X 10.11.3, installed via Conda, tried both 1.4.3 on the ncar channel and the 1.5 beta.

Thanks!

Edit: For further reference, I tried closing the files like so, but the error persisted:

p = psutil.Process(os.getpid())
fds = [f.fd for f in p.open_files() if 'PyNIO' in f.path]
for f in fds:
    print f
    os.close(f)

Sunset PyNIO

PyNIO hasn't had a commit since 2019. Is it safe to assume the project is finished? Perhaps it would be worth updating the readme to articulate the status of the project.

All numpy data arrays read from PyNIO variables are being marked as read only

In the current (1.5.1) release of PyNIO, accessed through the conda package (see below), we can no longer update data arrays retrieved from PyNIO variables in place with code like the following:

x = nc.variables["skintemp"][:]
x *= 2.0

This raises an exception: ValueError: output array is read-only

Somehow the arrays have their writeable flag set to False.

You can set x to be writeable though and the operation will succeed:
x = nc.variables["skintemp"][:]
x.setflags(write=True)
x *= 2.0

This behavior is consistent across any variables created for all files we've thrown at it (GRIB-2, NetCDF versions 3 and 4).

A sample program to illustrate the issue where we print the flags for the data arrays retrieved from all variables in a file:
#!/usr/bin/env python
from future import print_function
import sys
import Nio

nc = Nio.open_file(sys.argv[1])
for v in nc.variables.keys():
arr = nc.variables[v][:]
print(v)
print(arr.flags)
nc.close()

Below are some tests using one of NCAR's sample NetCDF files:
https://www.unidata.ucar.edu/software/netcdf/examples/test_echam_spectral.nc

The operation fails in a conda environment created with the current release:

conda create -n pynio-test -c conda-forge pynio
conda list | grep pynio

packages in environment at /home/mkay/.conda/envs/pynio-test:

pynio 1.5.1 py27h36f4ed2_2 conda-forge
(pynio-test) > python ~/tmp/wrtest.py /tmp/test_echam_spectral-deflated.nc | grep WRITEABLE | sort | uniq -c
136 WRITEABLE : False

The above works if we fall back to the 1.5.0:

conda create -n pynio-test150 -c conda-forge pynio=1.5.0
source activate pynio-test150
conda list | grep pynio

packages in environment at /home/mkay/.conda/envs/pynio-test150:

pynio 1.5.0 py27h10838b4_5 conda-forge
(pynio-test150) > python ~/tmp/wrtest.py /tmp/test_echam_spectral-deflated.nc | grep WRITEABLE | sort | uniq -c
136 WRITEABLE : True

Reference issue for functions returning variables

The following code fails:

>>> def test():
...     x = Nio.open_file("wrfout_d02_2005-08-28_12:00:00.nc", "r")
...     y = x.variables["P"]
...     return y
... 
>>> q = test()
>>> q[:]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ladwig/miniconda2/envs/pynio2_build/lib/python2.7/site-packages/PyNIO/Nio.py", line 347, in __getitem__
    ret = get_variable(self.file, self.varname, xsel)
  File "/Users/ladwig/miniconda2/envs/pynio2_build/lib/python2.7/site-packages/PyNIO/coordsel.py", line 39, in get_variable
    dims = file.variables[varname].cf_dimensions
ReferenceError: weakly-referenced object no longer exists

It looks like in the newer 1.5.x series of PyNIO, variables are being stored as weak references. This is going to be difficult to fix. Ideally, the weak reference stuff needs to be removed in favor of cyclic garbage collection (e.g. (traverseproc)nio_traverse).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.