melizalab / arf Goto Github PK
View Code? Open in Web Editor NEWadvanced recording format for physiology and behavior
License: BSD 3-Clause "New" or "Revised" License
arf's Issues
transition to setuptools
uuid attribute fails for byte string
This error comes up if you try to copy attributes from one entry/dataset to another, because the uuid is stored as an ascii-encoded byte array, but UUID(hex=) won't accept byte arrays.
add python 3 support
Module should work under both Python 2 and 3. Target versions 2.6 to 3.3
Warnings attempting to (double) close invalid groups and datasets (c++)
In some cases closing an HDF5 file invalidates open group and dataset handles (in spite of what the spec says). Need to check whether the handles are valid in the c++ interface before calling the close methods to avoid generating a warning.
Eliminate nose
I am a maintainer of Python packages in openSUSE, and I am on the crusade of eliminating nose1 from our distribution. When I look at its repository on https://github.com/nose-devs/nose, the last release 1.3.7 was on 2 Jun 2015, and even the last commit on the master branch was on 4 Mar 2016.
This patch eliminates dependency on nose. Resulting tests are pure unittest-based ones requiring nothing than the standard library.
Unfortunately, I had to skip several tests:
- test01_create_existing_entry
- test03_set_attributes
- test04_create_bad_dataset
- test04_create_bad_dataset
- test10_select_from_timeseries
- test11_copy_entry_attrs
otherwise these tests fail:
[ 28s] + pytest-3.8 --ignore=_build.python2 --ignore=_build.python3 --ignore=_build.pypy3 -v
[ 28s] ============================= test session starts ==============================
[ 28s] platform linux -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
[ 28s] cachedir: .pytest_cache
[ 28s] rootdir: /home/abuild/rpmbuild/BUILD/arf-2.5.1
[ 28s] collecting ... collected 13 items
[ 28s]
[ 28s] tests/test_arf.py::TestArf::test00_create_entries PASSED [ 7%]
[ 28s] tests/test_arf.py::TestArf::test01_create_existing_entry FAILED [ 15%]
[ 28s] tests/test_arf.py::TestArf::test02_create_datasets PASSED [ 23%]
[ 28s] tests/test_arf.py::TestArf::test03_set_attributes FAILED [ 30%]
[ 28s] tests/test_arf.py::TestArf::test04_create_bad_dataset FAILED [ 38%]
[ 28s] tests/test_arf.py::TestArf::test05_null_uuid FAILED [ 46%]
[ 28s] tests/test_arf.py::TestArf::test06_creation_iter PASSED [ 53%]
[ 28s] tests/test_arf.py::TestArf::test07_append_to_table PASSED [ 61%]
[ 29s] tests/test_arf.py::TestArf::test08_check_file_version PASSED [ 69%]
[ 29s] tests/test_arf.py::TestArf::test09_timestamp_conversion PASSED [ 76%]
[ 29s] tests/test_arf.py::TestArf::test10_select_from_timeseries FAILED [ 84%]
[ 29s] tests/test_arf.py::TestArf::test11_copy_entry_attrs PASSED [ 92%]
[ 29s] tests/test_arf.py::TestArf::test99_various PASSED [100%]
[ 29s]
[ 29s] =================================== FAILURES ===================================
[ 29s] _____________________ TestArf.test01_create_existing_entry _____________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test01_create_existing_entry>
[ 29s]
[ 29s] def test01_create_existing_entry(self):
[ 29s] with self.assertRaises(ValueError):
[ 29s] > arf.create_entry(fp, entry_base % 0, tstamp, **entry_attributes)
[ 29s] E AssertionError: ValueError not raised
[ 29s]
[ 29s] tests/test_arf.py:109: AssertionError
[ 29s] ________________________ TestArf.test03_set_attributes _________________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test03_set_attributes>
[ 29s]
[ 29s] def test03_set_attributes(self):
[ 29s] # tests the set_attributes convenience function
[ 29s] > arf.set_attributes(fp["entry_001/spikes"], mystr="myvalue", myint=5000)
[ 29s]
[ 29s] tests/test_arf.py:123:
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[ 29s] oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s]
[ 29s] > ???
[ 29s] E KeyError: 'Unable to open object (component not found)'
[ 29s]
[ 29s] h5py/h5o.pyx:190: KeyError
[ 29s] ______________________ TestArf.test04_create_bad_dataset _______________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test04_create_bad_dataset>
[ 29s]
[ 29s] def test04_create_bad_dataset(self):
[ 29s] # f = raises(ValueError)(self.create_dataset)
[ 29s] > e = fp['entry_001']
[ 29s]
[ 29s] tests/test_arf.py:131:
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[ 29s] oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s]
[ 29s] > ???
[ 29s] E KeyError: "Unable to open object (object 'entry_001' doesn't exist)"
[ 29s]
[ 29s] h5py/h5o.pyx:190: KeyError
[ 29s] ___________________________ TestArf.test05_null_uuid ___________________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test05_null_uuid>
[ 29s]
[ 29s] def test05_null_uuid(self):
[ 29s] # nulls in a uuid can make various things barf
[ 29s] from uuid import UUID
[ 29s] uuid = UUID(bytes=b''.rjust(16, b'\0'))
[ 29s] > e = fp['entry_001']
[ 29s]
[ 29s] tests/test_arf.py:141:
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[ 29s] oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s]
[ 29s] > ???
[ 29s] E KeyError: "Unable to open object (object 'entry_001' doesn't exist)"
[ 29s]
[ 29s] h5py/h5o.pyx:190: KeyError
[ 29s] ____________________ TestArf.test10_select_from_timeseries _____________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test10_select_from_timeseries>
[ 29s]
[ 29s] def test10_select_from_timeseries(self):
[ 29s] entry = fp[entry_base % 0]
[ 29s] for data in datasets:
[ 29s] > dset = entry[data["name"]]
[ 29s]
[ 29s] tests/test_arf.py:186:
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[ 29s] oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s]
[ 29s] > ???
[ 29s] E KeyError: "Unable to open object (object 'acoustic' doesn't exist)"
[ 29s]
[ 29s] h5py/h5o.pyx:190: KeyError
[ 29s] =========================== short test summary info ============================
[ 29s] FAILED tests/test_arf.py::TestArf::test01_create_existing_entry - AssertionEr...
[ 29s] FAILED tests/test_arf.py::TestArf::test03_set_attributes - KeyError: 'Unable ...
[ 29s] FAILED tests/test_arf.py::TestArf::test04_create_bad_dataset - KeyError: "Una...
[ 29s] FAILED tests/test_arf.py::TestArf::test05_null_uuid - KeyError: "Unable to op...
[ 29s] FAILED tests/test_arf.py::TestArf::test10_select_from_timeseries - KeyError: ...
[ 29s] ========================= 5 failed, 8 passed in 0.34s ==========================
[ 29s] error: Bad exit status from /var/tmp/rpm-tmp.pSIDnG (%check)
(using pytest only as the test runner, nothing in my patch depends on pytest).
c++ library segfaults on debian 8
Appears to result from a stack overflow in the error handler. Unclear from backtrace what the initial problem is:
58201 0x00007ffff79b9703 in H5E_push_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58202 0x00007ffff79b98f1 in H5E_printf_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58203 0x00007ffff7a292ea in H5I_inc_ref () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58204 0x00007ffff79b9703 in H5E_push_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58205 0x00007ffff79b98f1 in H5E_printf_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58206 0x00007ffff7a292ea in H5I_inc_ref () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58207 0x00007ffff79b9703 in H5E_push_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58208 0x00007ffff79b98f1 in H5E_printf_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58209 0x00007ffff79dc55b in H5FL_fac_term () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58210 0x00007ffff7aaa3cc in H5SL_term_interface ()
from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58211 0x00007ffff7955f86 in H5_term_library () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58212 0x00007ffff6b57b29 in __run_exit_handlers (status=0, listp=0x7ffff6ec55a8 <__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#58213 0x00007ffff6b57b75 in __GI_exit (status=<optimized out>) at exit.c:104
#58214 0x00007ffff6b41b4c in __libc_start_main (main=0x404b3a <main(int, char**)>, argc=1,
argv=0x7fffffffe288, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7fffffffe278) at libc-start.c:321
more h5py internal API changes
Similar to #8, many tests now start to fail because of private API changes in h5py:
[ 42s] _______________ TestArfNavigation.test10_select_from_timeseries ________________
[ 42s]
[ 42s] self = <tests.test_arf.TestArfNavigation testMethod=test10_select_from_timeseries>
[ 42s]
[ 42s] def setUp(self):
[ 42s] > self.fp = arf.open_file("test", 'w', driver="core", backing_store=False)
[ 42s]
[ 42s] tests/test_arf.py:156:
[ 42s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 42s]
[ 42s] name = b'test', mode = 'w', driver = 'core', libver = None
[ 42s] userblock_size = None
[ 42s] kwargs = {'backing_store': False, 'rdcc_nbytes': None, 'rdcc_nslots': None, 'rdcc_w0': None}
[ 42s] sys = <module 'sys' (built-in)>
[ 42s] os = <module 'os' from '/usr/lib64/python3.9/os.py'>, h5py_version = '3.6.0'
[ 42s] StrictVersion = <class 'distutils.version.StrictVersion'>
[ 42s] h5p = <module 'h5py.h5p' from '/usr/lib64/python3.9/site-packages/h5py/h5p.cpython-39-x86_64-linux-gnu.so'>
[ 42s] files = <module 'h5py._hl.files' from '/usr/lib64/python3.9/site-packages/h5py/_hl/files.py'>
[ 42s] exists = False
[ 42s]
[ 42s] def open_file(name, mode=None, driver=None, libver=None, userblock_size=None, **kwargs):
[ 42s] """Open an ARF file, creating as necessary.
[ 42s]
[ 42s] Use this instead of h5py.File to ensure that root-level attributes and group
[ 42s] creation property lists are set correctly.
[ 42s]
[ 42s] """
[ 42s] import sys
[ 42s] import os
[ 42s] from h5py.version import version as h5py_version
[ 42s] from distutils.version import StrictVersion
[ 42s] from h5py import h5p
[ 42s] from h5py._hl import files
[ 42s]
[ 42s] try:
[ 42s] # If the byte string doesn't match the default
[ 42s] # encoding, just pass it on as-is. Note Unicode
[ 42s] # objects can always be encoded.
[ 42s] name = name.encode(sys.getfilesystemencoding())
[ 42s] except (UnicodeError, LookupError):
[ 42s] pass
[ 42s] exists = os.path.exists(name)
[ 42s] try:
[ 42s] fcpl = h5p.create(h5p.FILE_CREATE)
[ 42s] fcpl.set_link_creation_order(
[ 42s] h5p.CRT_ORDER_TRACKED | h5p.CRT_ORDER_INDEXED)
[ 42s] except AttributeError:
[ 42s] # older version of h5py
[ 42s] fp = files.File(name, mode=mode, driver=driver,
[ 42s] libver=libver, **kwargs)
[ 42s] else:
[ 42s] if StrictVersion(h5py_version) >= StrictVersion('2.9'):
[ 42s] kwargs.update(rdcc_nslots=None, rdcc_nbytes=None, rdcc_w0=None)
[ 42s] > fapl = files.make_fapl(driver, libver, **kwargs)
[ 42s] E TypeError: make_fapl() missing 4 required positional arguments: 'locking', 'page_buf_size', 'min_meta_keep', and 'min_raw_keep'
[ 42s]
[ 42s] arf.py:77: TypeError
arf.open_file(fn, mode="a") fails with h5py 3.3
The internals of h5py have changed, and arf's usage of private _hl fails now:
...
[ 16s] python39-h5py-3.3.0-1.1 ########################################
...
[ 19s] + PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-arf-2.6.1-1.4.x86_64/usr/lib/python3.9/site-packages
[ 19s] + PYTHONDONTWRITEBYTECODE=1
[ 19s] + pytest-3.9 --ignore=_build. --ignore=_build.python39 --ignore=_build.python38 -v
...
[ 20s] =================================== FAILURES ===================================
[ 20s] ____________________ TestArfNavigation.test01_creation_iter ____________________
[ 20s]
[ 20s] self = <tests.test_arf.TestArfNavigation testMethod=test01_creation_iter>
[ 20s]
[ 20s] def test01_creation_iter(self):
[ 20s] > self.fp = arf.open_file("test06", mode="a", driver="core", backing_store=False)
[ 20s]
[ 20s] tests/test_arf.py:162:
[ 20s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 20s] arf.py:78: in open_file
[ 20s] fp = files.File(files.make_fid(name, mode, userblock_size, fapl, fcpl))
[ 20s] /usr/lib64/python3.9/site-packages/h5py/_hl/files.py:211: in make_fid
[ 20s] fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
[ 20s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 20s] ???
[ 20s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 20s] ???
[ 20s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 20s]
[ 20s] > ???
[ 20s] E OSError: Unable to open file (unable to open file)
[ 20s]
[ 20s] h5py/h5f.pyx:100: OSError
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.