Comments (26)
Any progress on Py3.x support. Xarray is using Pynio as a ingest for grib... I am nearly 100% 3.x now and don't want to go back.
from pynio.
from pynio.
PyNIO for Python 3.x is now merged in to the 'develop' branch on GitHub. However, an official release on conda-forge won't take place until a large amount of bug fixes related to compound data types, HDF5, etc are included. That being said, you should be able to pull down PyNIO from GitHub and start trying it out with Python 3.x.
from pynio.
Unfortunately, the PyNIO port is taking a little longer than I expected, so I wouldn't say that release is imminent. It's currently in the debug stage, soon approaching testing.
from pynio.
I'm working on an ML project for NASA (related to @darothen 's mention of earthio
above). We're glad to hear Python 3 support is coming for pynio
. Thanks!
from pynio.
The issue here is a mismatch between the ncar
channel and the conda-forge
channel for the dependencies used ni both pynio
and xarray
. I recommend to release the dev version of pynio
in a sub-label in conda-forge
, that way people could do:
$ conda install --channel conda-forge/label/dev pynio
this would be in sync with the other dependencies we have in conda-forge
and would avoid this issue.
from pynio.
Thanks for reporting this @darothen and @scollis, I'm looking into this now.
@ocefpaf, I will create an issue to further discuss your idea on the pynio-feedstock repo.
from pynio.
My .condarc
was completely empty. However, I copied/pasted yours and it miraculously worked...
from pynio.
@david-ian-brown any late summer updates on this? I have an application using HRRR GRIB2 output which leans heavily on xarray; if I could skip the step of converting GRIB2 to NetCDF (which would be possible with PyNIO support for Python 3), it would greatly help.
from pynio.
I'm also super excited to see pynio
with python 3 support.
But for anyone reading this thread: in the meantime, if you use anaconda, you can easily create a python 2.7 virtual environment. This link has some great info. Create the environment with conda create -n python2 python=2.7 anaconda
(where python2
is the name I chose for the virtualenv), install the relevant packages with conda install -n python2 xarray
and conda install -n python2 -c conda-forge pynio
, activate the environment from the command-line with source activate python2
or . activate python2
, and launch an ipython terminal, jupyter notebook, etc while the environment is "activated."
This is useful for any other packages that haven't yet been ported to python 3.
from pynio.
@lukelbd that's true - but the scientific Python community is quickly moving away from Python 2, and there is little reason to maintain two fully functioning development environments for daily work. Virtual environments aren't a solution here in many cases. For instance, I'm building a workflow to ingest/parse GRIB2 output via xarray for work, and our entire stack is Python 3; it's hugely disruptive to maintain a separate Python stack on all our cloud services just to read those files into our system.
from pynio.
@darothen you're absolutely right; just wanted to share a band-aid solution (viable for not-too-intensive workflows) until the python 3 update arrives. Though it looks like that will be imminent, so maybe it was not necessary.
from pynio.
Thanks for hte update @bladwig1 . Is there anything the community to do at this stage to help out?
from pynio.
There is now a "dev" snapshot build on conda (ncar channel), which you can get by doing:
conda create -n (your_env_name) -c ncar -c conda-forge pynio=dev python=3
If you notice any issues, please create a new GitHub issue.
Thanks!
from pynio.
Hey all - I know this thread is technically closed, but there's an issue with this old protocol.
Somewhere along the line, a dependency issue was introduced. If you install a clean Python 3 environment with just xarray, a la:
$ conda create -n test_nio -c conda-forge python=3.6 xarray
then activate it and attempt to install pynio as referenced above:
$ conda install -c ncar pynio=dev
you'll run into an issue with libnetcdf versions, where it requests a downgrade:
The following packages will be DOWNGRADED:
libnetcdf: 4.5.0-3 conda-forge --> 4.4.1.1-10 conda-forge
netcdf4: 1.3.1-py36_2 conda-forge --> 1.3.1-py36_1 conda-forge
Somewhere along the line, hdf5
gets messed up and it renders pynio un-importable. On my MacBook I'll get import errors such as:
ImportError: dlopen(/Users/daniel/anaconda/envs/test_nio/lib/python3.6/site-packages/PyNIO/_nio.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libhdf5_hl.10.dylib
Referenced from: /Users/daniel/anaconda/envs/test_nio/lib/python3.6/site-packages/PyNIO/_nio.cpython-36m-darwin.so
Reason: image not found
Any thoughts and/or solutions? This is new behavior; last time I helped someone get set-up with pynio/Python 3 back in January we did not run into this problem.
from pynio.
Yeah add me to the list of people this no longer works for..
from pynio.
I can confirm that @ocefpaf's solution has worked just fine, and fixes the dependency issues.
from pynio.
Thanks to @khallock who promptly created the dev version on conda-forge
.
PS: he also created a ncl
dev package too, install it with
conda install --channel conda-forge/label/dev ncl
🎉
from pynio.
from pynio.
FWIW, there may be a new issue - the resolved method isn't working, and is forcing a downgrade to Python 2.7. I think I've traced this down to an older version of cyordereddict which is pinned to Python 2.7, and is a dependency of xarray.
from pynio.
@darothen what happens if you specify the python version too? Like
conda create --name TEST --channel conda-forge/label/dev python=3.6 xarray pynio
The conda solver tries to wiggle its way out of complex dependencies by downgrading package version and build number, sometimes it even breaks channel preference! So the more you specify in the install command the better.
from pynio.
@ocefpaf No dice, unfortunately:
from pynio.
Works for me, do you have other channel in your .condarc
?
> conda create --name TEST --channel conda-forge/label/dev python=3.6 xarray pynio
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.4.10
latest version: 4.5.0
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: /home/filipe/miniconda3/envs/TEST
added / updated specs:
- ipykernel
- pip
- pynio
- python=3.6
- xarray
The following packages will be downloaded:
package | build
---------------------------|-----------------
krb5-1.14.6 | 0 4.0 MB conda-forge
backcall-0.1.0 | py_0 13 KB conda-forge
distributed-1.21.5 | py36_0 772 KB conda-forge
bokeh-0.12.15 | py36_0 4.2 MB conda-forge
pynio-1.5.1.dev0 | py36_1 4.3 MB conda-forge/label/dev
wheel-0.31.0 | py36_0 62 KB conda-forge
msgpack-python-0.5.6 | py36_0 300 KB conda-forge
ipython-6.3.0 | py36_0 1.0 MB conda-forge
------------------------------------------------------------
Total: 14.7 MB
The following NEW packages will be INSTALLED:
backcall: 0.1.0-py_0 conda-forge
bokeh: 0.12.15-py36_0 conda-forge
boost: 1.66.0-py36_1 conda-forge
boost-cpp: 1.66.0-1 conda-forge
bottleneck: 1.2.1-py36_1 conda-forge
bzip2: 1.0.6-1 conda-forge
ca-certificates: 2018.1.18-0 conda-forge
cairo: 1.14.10-0 conda-forge
certifi: 2018.1.18-py36_0 conda-forge
click: 6.7-py_1 conda-forge
cloudpickle: 0.5.2-py_0 conda-forge
curl: 7.59.0-0 conda-forge
cytoolz: 0.9.0.1-py36_0 conda-forge
dask: 0.17.2-py_0 conda-forge
dask-core: 0.17.2-py_0 conda-forge
decorator: 4.2.1-py36_0 conda-forge
distributed: 1.21.5-py36_0 conda-forge
expat: 2.2.5-0 conda-forge
fontconfig: 2.12.6-0 conda-forge
freetype: 2.8.1-0 conda-forge
freexl: 1.0.5-0 conda-forge
g2clib: 1.6.0-5 conda-forge
geos: 3.6.2-1 conda-forge
geotiff: 1.4.2-1 conda-forge
gettext: 0.19.8.1-0 conda-forge
giflib: 5.1.4-0 conda-forge
glib: 2.55.0-0 conda-forge
h5netcdf: 0.5.0-py_0 conda-forge
h5py: 2.7.1-py36_2 conda-forge
hdf4: 4.2.13-0 conda-forge
hdf5: 1.10.1-2 conda-forge
hdfeos2: 2.19.1-8 conda-forge
hdfeos5: 5.1.16-0 conda-forge
heapdict: 1.0.0-py36_0 conda-forge
icu: 58.2-0 conda-forge
intel-openmp: 2018.0.0-8 defaults
ipykernel: 4.8.2-py36_0 conda-forge
ipython: 6.3.0-py36_0 conda-forge
ipython_genutils: 0.2.0-py36_0 conda-forge
jasper: 1.900.1-4 conda-forge
jedi: 0.11.1-py36_0 conda-forge
jinja2: 2.10-py36_0 conda-forge
jpeg: 9b-2 conda-forge
json-c: 0.12.1-0 conda-forge
jupyter_client: 5.2.3-py36_0 conda-forge
jupyter_core: 4.4.0-py_0 conda-forge
kealib: 1.4.7-4 conda-forge
krb5: 1.14.6-0 conda-forge
libdap4: 3.18.3-2 conda-forge
libffi: 3.2.1-3 conda-forge
libgcc-ng: 7.2.0-hdf63c60_3 defaults
libgdal: 2.2.4-0 conda-forge
libgfortran: 3.0.0-1 defaults
libgfortran-ng: 7.2.0-hdf63c60_3 defaults
libiconv: 1.15-0 conda-forge
libkml: 1.3.0-6 conda-forge
libnetcdf: 4.4.1.1-10 conda-forge
libpng: 1.6.34-0 conda-forge
libpq: 9.6.3-0 conda-forge
libsodium: 1.0.16-0 conda-forge
libspatialite: 4.3.0a-19 conda-forge
libssh2: 1.8.0-2 conda-forge
libstdcxx-ng: 7.2.0-hdf63c60_3 defaults
libtiff: 4.0.9-0 conda-forge
libxml2: 2.9.8-0 conda-forge
locket: 0.2.0-py36_1 conda-forge
markupsafe: 1.0-py36_0 conda-forge
mkl: 2018.0.2-1 defaults
mkl_fft: 1.0.1-py36_1 conda-forge
mkl_random: 1.0.1-py36_0 conda-forge
msgpack-python: 0.5.6-py36_0 conda-forge
ncurses: 5.9-10 conda-forge
netcdf4: 1.3.1-py36_1 conda-forge
numpy: 1.14.2-py36hdbf6ddf_1 defaults
openjpeg: 2.3.0-2 conda-forge
openssl: 1.0.2n-0 conda-forge
packaging: 17.1-py_0 conda-forge
pandas: 0.22.0-py36_0 conda-forge
parso: 0.1.1-py_0 conda-forge
partd: 0.3.8-py36_0 conda-forge
pcre: 8.41-1 conda-forge
pexpect: 4.4.0-py36_0 conda-forge
pickleshare: 0.7.4-py36_0 conda-forge
pip: 9.0.3-py36_0 conda-forge
pixman: 0.34.0-1 conda-forge
poppler: 0.61.1-3 conda-forge
poppler-data: 0.4.8-0 conda-forge
proj4: 4.9.3-5 conda-forge
prompt_toolkit: 1.0.15-py36_0 conda-forge
psutil: 5.4.3-py36_0 conda-forge
ptyprocess: 0.5.2-py36_0 conda-forge
pygments: 2.2.0-py36_0 conda-forge
pynio: 1.5.1.dev0-py36_1 conda-forge/label/dev
pyparsing: 2.2.0-py36_0 conda-forge
python: 3.6.5-0 conda-forge
python-dateutil: 2.7.2-py_0 conda-forge
pytz: 2018.3-py_0 conda-forge
pyyaml: 3.12-py36_1 conda-forge
pyzmq: 17.0.0-py36_4 conda-forge
readline: 7.0-0 conda-forge
scipy: 1.0.1-py36hfc37229_0 defaults
setuptools: 39.0.1-py36_0 conda-forge
simplegeneric: 0.8.1-py36_0 conda-forge
six: 1.11.0-py36_1 conda-forge
sortedcontainers: 1.5.9-py36_0 conda-forge
sqlite: 3.20.1-2 conda-forge
tblib: 1.3.2-py36_0 conda-forge
tk: 8.6.7-0 conda-forge
toolz: 0.9.0-py_0 conda-forge
tornado: 5.0.1-py36_1 conda-forge
traitlets: 4.3.2-py36_0 conda-forge
util-linux: 2.21-0 defaults
wcwidth: 0.1.7-py36_0 conda-forge
wheel: 0.31.0-py36_0 conda-forge
xarray: 0.10.2-py36_0 conda-forge
xerces-c: 3.2.0-0 conda-forge
xz: 5.2.3-0 conda-forge
yaml: 0.1.7-0 conda-forge
zeromq: 4.2.5-1 conda-forge
zict: 0.1.3-py_0 conda-forge
zlib: 1.2.11-0 conda-forge
Proceed ([y]/n)?
from pynio.
Oddly, no. I spun up a fresh VM, grabbed the latest Miniconda3-latest-Linux-x86_64.sh
, and executed exactly the same command. The vanilla environment just contends that pynio and python=3.6 are in conflict.
from pynio.
Can you post the contents of your .condarc
? Here is mine BTW:
> cat ~/.condarc
show_channel_urls: true
add_pip_as_python_dependency: true
channels:
- conda-forge
- defaults
create_default_packages:
- pip
- ipykernel
from pynio.
Probably b/c the defaults added pynio
to the main channel and conda was trying to get that for you. The the .condarc
conda will look on conda-forge first.
Note that, according to the docs, the --channel
should act like that .condarc
configuration and put conda-forge on top of the search, but that never worked properly for me and I have no idea if that works for sub-channels (labels).
from pynio.
Related Issues (20)
- Duplicate symbols when compiling pynio from source using Python 3.7.4 HOT 2
- Does this work with HDF5? Also does it work with netdf4 versions 1.10?
- when shape is unexpected, pynio return "check dtype" HOT 1
- Unexpected behavior when opening Canadian global GEM 25km grib files using pynio
- Failed to read a large grib2 file correctly
- pynio installation downgrades esmf/esmpy to no mpi versions HOT 3
- installation linux (ubuntu ) in the window 10 sub system
- build of h5reader.c fails when compiling against hdf5-openmpi 1.12.0 HOT 2
- Unable to install pynio under linux-aarch64 HOT 1
- Importing Pynio image not found error HOT 1
- No way to get location of opened file?
- Can't open a .nc file
- Non-conda build instructions
- pynio does not work for conda create under centos 8
- open .ncf files
- Sunset PyNIO HOT 2
- Alma Linux ModuleNotFoundError: No module named '_nio'
- failing to read grib2 files from ftp.bsh.de
- Can i use scipy instead of pynio engine in xarray HOT 3
- CI failing on environment creation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pynio.