melizalab / arf Goto Github PK
View Code? Open in Web Editor NEWadvanced recording format for physiology and behavior
License: BSD 3-Clause "New" or "Revised" License
advanced recording format for physiology and behavior
License: BSD 3-Clause "New" or "Revised" License
This error comes up if you try to copy attributes from one entry/dataset to another, because the uuid is stored as an ascii-encoded byte array, but UUID(hex=) won't accept byte arrays.
Module should work under both Python 2 and 3. Target versions 2.6 to 3.3
In some cases closing an HDF5 file invalidates open group and dataset handles (in spite of what the spec says). Need to check whether the handles are valid in the c++ interface before calling the close methods to avoid generating a warning.
I am a maintainer of Python packages in openSUSE, and I am on the crusade of eliminating nose1 from our distribution. When I look at its repository on https://github.com/nose-devs/nose, the last release 1.3.7 was on 2 Jun 2015, and even the last commit on the master branch was on 4 Mar 2016.
This patch eliminates dependency on nose. Resulting tests are pure unittest-based ones requiring nothing than the standard library.
Unfortunately, I had to skip several tests:
[ 28s] + pytest-3.8 --ignore=_build.python2 --ignore=_build.python3 --ignore=_build.pypy3 -v
[ 28s] ============================= test session starts ==============================
[ 28s] platform linux -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
[ 28s] cachedir: .pytest_cache
[ 28s] rootdir: /home/abuild/rpmbuild/BUILD/arf-2.5.1
[ 28s] collecting ... collected 13 items
[ 28s]
[ 28s] tests/test_arf.py::TestArf::test00_create_entries PASSED [ 7%]
[ 28s] tests/test_arf.py::TestArf::test01_create_existing_entry FAILED [ 15%]
[ 28s] tests/test_arf.py::TestArf::test02_create_datasets PASSED [ 23%]
[ 28s] tests/test_arf.py::TestArf::test03_set_attributes FAILED [ 30%]
[ 28s] tests/test_arf.py::TestArf::test04_create_bad_dataset FAILED [ 38%]
[ 28s] tests/test_arf.py::TestArf::test05_null_uuid FAILED [ 46%]
[ 28s] tests/test_arf.py::TestArf::test06_creation_iter PASSED [ 53%]
[ 28s] tests/test_arf.py::TestArf::test07_append_to_table PASSED [ 61%]
[ 29s] tests/test_arf.py::TestArf::test08_check_file_version PASSED [ 69%]
[ 29s] tests/test_arf.py::TestArf::test09_timestamp_conversion PASSED [ 76%]
[ 29s] tests/test_arf.py::TestArf::test10_select_from_timeseries FAILED [ 84%]
[ 29s] tests/test_arf.py::TestArf::test11_copy_entry_attrs PASSED [ 92%]
[ 29s] tests/test_arf.py::TestArf::test99_various PASSED [100%]
[ 29s]
[ 29s] =================================== FAILURES ===================================
[ 29s] _____________________ TestArf.test01_create_existing_entry _____________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test01_create_existing_entry>
[ 29s]
[ 29s] def test01_create_existing_entry(self):
[ 29s] with self.assertRaises(ValueError):
[ 29s] > arf.create_entry(fp, entry_base % 0, tstamp, **entry_attributes)
[ 29s] E AssertionError: ValueError not raised
[ 29s]
[ 29s] tests/test_arf.py:109: AssertionError
[ 29s] ________________________ TestArf.test03_set_attributes _________________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test03_set_attributes>
[ 29s]
[ 29s] def test03_set_attributes(self):
[ 29s] # tests the set_attributes convenience function
[ 29s] > arf.set_attributes(fp["entry_001/spikes"], mystr="myvalue", myint=5000)
[ 29s]
[ 29s] tests/test_arf.py:123:
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[ 29s] oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s]
[ 29s] > ???
[ 29s] E KeyError: 'Unable to open object (component not found)'
[ 29s]
[ 29s] h5py/h5o.pyx:190: KeyError
[ 29s] ______________________ TestArf.test04_create_bad_dataset _______________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test04_create_bad_dataset>
[ 29s]
[ 29s] def test04_create_bad_dataset(self):
[ 29s] # f = raises(ValueError)(self.create_dataset)
[ 29s] > e = fp['entry_001']
[ 29s]
[ 29s] tests/test_arf.py:131:
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[ 29s] oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s]
[ 29s] > ???
[ 29s] E KeyError: "Unable to open object (object 'entry_001' doesn't exist)"
[ 29s]
[ 29s] h5py/h5o.pyx:190: KeyError
[ 29s] ___________________________ TestArf.test05_null_uuid ___________________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test05_null_uuid>
[ 29s]
[ 29s] def test05_null_uuid(self):
[ 29s] # nulls in a uuid can make various things barf
[ 29s] from uuid import UUID
[ 29s] uuid = UUID(bytes=b''.rjust(16, b'\0'))
[ 29s] > e = fp['entry_001']
[ 29s]
[ 29s] tests/test_arf.py:141:
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[ 29s] oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s]
[ 29s] > ???
[ 29s] E KeyError: "Unable to open object (object 'entry_001' doesn't exist)"
[ 29s]
[ 29s] h5py/h5o.pyx:190: KeyError
[ 29s] ____________________ TestArf.test10_select_from_timeseries _____________________
[ 29s]
[ 29s] self = <tests.test_arf.TestArf testMethod=test10_select_from_timeseries>
[ 29s]
[ 29s] def test10_select_from_timeseries(self):
[ 29s] entry = fp[entry_base % 0]
[ 29s] for data in datasets:
[ 29s] > dset = entry[data["name"]]
[ 29s]
[ 29s] tests/test_arf.py:186:
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[ 29s] oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[ 29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 29s] ???
[ 29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 29s]
[ 29s] > ???
[ 29s] E KeyError: "Unable to open object (object 'acoustic' doesn't exist)"
[ 29s]
[ 29s] h5py/h5o.pyx:190: KeyError
[ 29s] =========================== short test summary info ============================
[ 29s] FAILED tests/test_arf.py::TestArf::test01_create_existing_entry - AssertionEr...
[ 29s] FAILED tests/test_arf.py::TestArf::test03_set_attributes - KeyError: 'Unable ...
[ 29s] FAILED tests/test_arf.py::TestArf::test04_create_bad_dataset - KeyError: "Una...
[ 29s] FAILED tests/test_arf.py::TestArf::test05_null_uuid - KeyError: "Unable to op...
[ 29s] FAILED tests/test_arf.py::TestArf::test10_select_from_timeseries - KeyError: ...
[ 29s] ========================= 5 failed, 8 passed in 0.34s ==========================
[ 29s] error: Bad exit status from /var/tmp/rpm-tmp.pSIDnG (%check)
(using pytest only as the test runner, nothing in my patch depends on pytest).
Appears to result from a stack overflow in the error handler. Unclear from backtrace what the initial problem is:
58201 0x00007ffff79b9703 in H5E_push_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58202 0x00007ffff79b98f1 in H5E_printf_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58203 0x00007ffff7a292ea in H5I_inc_ref () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58204 0x00007ffff79b9703 in H5E_push_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58205 0x00007ffff79b98f1 in H5E_printf_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58206 0x00007ffff7a292ea in H5I_inc_ref () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58207 0x00007ffff79b9703 in H5E_push_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58208 0x00007ffff79b98f1 in H5E_printf_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58209 0x00007ffff79dc55b in H5FL_fac_term () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58210 0x00007ffff7aaa3cc in H5SL_term_interface ()
from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58211 0x00007ffff7955f86 in H5_term_library () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58212 0x00007ffff6b57b29 in __run_exit_handlers (status=0, listp=0x7ffff6ec55a8 <__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#58213 0x00007ffff6b57b75 in __GI_exit (status=<optimized out>) at exit.c:104
#58214 0x00007ffff6b41b4c in __libc_start_main (main=0x404b3a <main(int, char**)>, argc=1,
argv=0x7fffffffe288, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7fffffffe278) at libc-start.c:321
Similar to #8, many tests now start to fail because of private API changes in h5py:
[ 42s] _______________ TestArfNavigation.test10_select_from_timeseries ________________
[ 42s]
[ 42s] self = <tests.test_arf.TestArfNavigation testMethod=test10_select_from_timeseries>
[ 42s]
[ 42s] def setUp(self):
[ 42s] > self.fp = arf.open_file("test", 'w', driver="core", backing_store=False)
[ 42s]
[ 42s] tests/test_arf.py:156:
[ 42s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 42s]
[ 42s] name = b'test', mode = 'w', driver = 'core', libver = None
[ 42s] userblock_size = None
[ 42s] kwargs = {'backing_store': False, 'rdcc_nbytes': None, 'rdcc_nslots': None, 'rdcc_w0': None}
[ 42s] sys = <module 'sys' (built-in)>
[ 42s] os = <module 'os' from '/usr/lib64/python3.9/os.py'>, h5py_version = '3.6.0'
[ 42s] StrictVersion = <class 'distutils.version.StrictVersion'>
[ 42s] h5p = <module 'h5py.h5p' from '/usr/lib64/python3.9/site-packages/h5py/h5p.cpython-39-x86_64-linux-gnu.so'>
[ 42s] files = <module 'h5py._hl.files' from '/usr/lib64/python3.9/site-packages/h5py/_hl/files.py'>
[ 42s] exists = False
[ 42s]
[ 42s] def open_file(name, mode=None, driver=None, libver=None, userblock_size=None, **kwargs):
[ 42s] """Open an ARF file, creating as necessary.
[ 42s]
[ 42s] Use this instead of h5py.File to ensure that root-level attributes and group
[ 42s] creation property lists are set correctly.
[ 42s]
[ 42s] """
[ 42s] import sys
[ 42s] import os
[ 42s] from h5py.version import version as h5py_version
[ 42s] from distutils.version import StrictVersion
[ 42s] from h5py import h5p
[ 42s] from h5py._hl import files
[ 42s]
[ 42s] try:
[ 42s] # If the byte string doesn't match the default
[ 42s] # encoding, just pass it on as-is. Note Unicode
[ 42s] # objects can always be encoded.
[ 42s] name = name.encode(sys.getfilesystemencoding())
[ 42s] except (UnicodeError, LookupError):
[ 42s] pass
[ 42s] exists = os.path.exists(name)
[ 42s] try:
[ 42s] fcpl = h5p.create(h5p.FILE_CREATE)
[ 42s] fcpl.set_link_creation_order(
[ 42s] h5p.CRT_ORDER_TRACKED | h5p.CRT_ORDER_INDEXED)
[ 42s] except AttributeError:
[ 42s] # older version of h5py
[ 42s] fp = files.File(name, mode=mode, driver=driver,
[ 42s] libver=libver, **kwargs)
[ 42s] else:
[ 42s] if StrictVersion(h5py_version) >= StrictVersion('2.9'):
[ 42s] kwargs.update(rdcc_nslots=None, rdcc_nbytes=None, rdcc_w0=None)
[ 42s] > fapl = files.make_fapl(driver, libver, **kwargs)
[ 42s] E TypeError: make_fapl() missing 4 required positional arguments: 'locking', 'page_buf_size', 'min_meta_keep', and 'min_raw_keep'
[ 42s]
[ 42s] arf.py:77: TypeError
The internals of h5py have changed, and arf's usage of private _hl
fails now:
...
[ 16s] python39-h5py-3.3.0-1.1 ########################################
...
[ 19s] + PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-arf-2.6.1-1.4.x86_64/usr/lib/python3.9/site-packages
[ 19s] + PYTHONDONTWRITEBYTECODE=1
[ 19s] + pytest-3.9 --ignore=_build. --ignore=_build.python39 --ignore=_build.python38 -v
...
[ 20s] =================================== FAILURES ===================================
[ 20s] ____________________ TestArfNavigation.test01_creation_iter ____________________
[ 20s]
[ 20s] self = <tests.test_arf.TestArfNavigation testMethod=test01_creation_iter>
[ 20s]
[ 20s] def test01_creation_iter(self):
[ 20s] > self.fp = arf.open_file("test06", mode="a", driver="core", backing_store=False)
[ 20s]
[ 20s] tests/test_arf.py:162:
[ 20s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 20s] arf.py:78: in open_file
[ 20s] fp = files.File(files.make_fid(name, mode, userblock_size, fapl, fcpl))
[ 20s] /usr/lib64/python3.9/site-packages/h5py/_hl/files.py:211: in make_fid
[ 20s] fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
[ 20s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[ 20s] ???
[ 20s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[ 20s] ???
[ 20s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 20s]
[ 20s] > ???
[ 20s] E OSError: Unable to open file (unable to open file)
[ 20s]
[ 20s] h5py/h5f.pyx:100: OSError
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.