melizalab/arf

advanced recording format for physiology and behavior

arf's Issues

add python 3 support

Module should work under both Python 2 and 3. Target versions 2.6 to 3.3

uuid attribute fails for byte string

This error comes up if you try to copy attributes from one entry/dataset to another, because the uuid is stored as an ascii-encoded byte array, but UUID(hex=) won't accept byte arrays.

c++ library segfaults on debian 8

Appears to result from a stack overflow in the error handler. Unclear from backtrace what the initial problem is:

58201 0x00007ffff79b9703 in H5E_push_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58202 0x00007ffff79b98f1 in H5E_printf_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58203 0x00007ffff7a292ea in H5I_inc_ref () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58204 0x00007ffff79b9703 in H5E_push_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58205 0x00007ffff79b98f1 in H5E_printf_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58206 0x00007ffff7a292ea in H5I_inc_ref () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58207 0x00007ffff79b9703 in H5E_push_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58208 0x00007ffff79b98f1 in H5E_printf_stack () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58209 0x00007ffff79dc55b in H5FL_fac_term () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58210 0x00007ffff7aaa3cc in H5SL_term_interface ()
   from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58211 0x00007ffff7955f86 in H5_term_library () from /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8
#58212 0x00007ffff6b57b29 in __run_exit_handlers (status=0, listp=0x7ffff6ec55a8 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#58213 0x00007ffff6b57b75 in __GI_exit (status=<optimized out>) at exit.c:104
#58214 0x00007ffff6b41b4c in __libc_start_main (main=0x404b3a <main(int, char**)>, argc=1,
    argv=0x7fffffffe288, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7fffffffe278) at libc-start.c:321

arf.open_file(fn, mode="a") fails with h5py 3.3

The internals of h5py have changed, and arf's usage of private _hl fails now:

...
[   16s] python39-h5py-3.3.0-1.1               ########################################
...
[   19s] + PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-arf-2.6.1-1.4.x86_64/usr/lib/python3.9/site-packages
[   19s] + PYTHONDONTWRITEBYTECODE=1
[   19s] + pytest-3.9 --ignore=_build. --ignore=_build.python39 --ignore=_build.python38 -v
...
[   20s] =================================== FAILURES ===================================
[   20s] ____________________ TestArfNavigation.test01_creation_iter ____________________
[   20s] 
[   20s] self = <tests.test_arf.TestArfNavigation testMethod=test01_creation_iter>
[   20s] 
[   20s]     def test01_creation_iter(self):
[   20s] >       self.fp = arf.open_file("test06", mode="a", driver="core", backing_store=False)
[   20s] 
[   20s] tests/test_arf.py:162: 
[   20s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   20s] arf.py:78: in open_file
[   20s]     fp = files.File(files.make_fid(name, mode, userblock_size, fapl, fcpl))
[   20s] /usr/lib64/python3.9/site-packages/h5py/_hl/files.py:211: in make_fid
[   20s]     fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
[   20s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   20s]     ???
[   20s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   20s]     ???
[   20s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   20s] 
[   20s] >   ???
[   20s] E   OSError: Unable to open file (unable to open file)
[   20s] 
[   20s] h5py/h5f.pyx:100: OSError

Eliminate nose

I am a maintainer of Python packages in openSUSE, and I am on the crusade of eliminating nose1 from our distribution. When I look at its repository on https://github.com/nose-devs/nose, the last release 1.3.7 was on 2 Jun 2015, and even the last commit on the master branch was on 4 Mar 2016.

This patch eliminates dependency on nose. Resulting tests are pure unittest-based ones requiring nothing than the standard library.

Unfortunately, I had to skip several tests:

• test01_create_existing_entry
• test03_set_attributes
• test04_create_bad_dataset
• test04_create_bad_dataset
• test10_select_from_timeseries
• test11_copy_entry_attrs
  otherwise these tests fail:

[   28s] + pytest-3.8 --ignore=_build.python2 --ignore=_build.python3 --ignore=_build.pypy3 -v
[   28s] ============================= test session starts ==============================
[   28s] platform linux -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
[   28s] cachedir: .pytest_cache
[   28s] rootdir: /home/abuild/rpmbuild/BUILD/arf-2.5.1
[   28s] collecting ... collected 13 items
[   28s] 
[   28s] tests/test_arf.py::TestArf::test00_create_entries PASSED                 [  7%]
[   28s] tests/test_arf.py::TestArf::test01_create_existing_entry FAILED          [ 15%]
[   28s] tests/test_arf.py::TestArf::test02_create_datasets PASSED                [ 23%]
[   28s] tests/test_arf.py::TestArf::test03_set_attributes FAILED                 [ 30%]
[   28s] tests/test_arf.py::TestArf::test04_create_bad_dataset FAILED             [ 38%]
[   28s] tests/test_arf.py::TestArf::test05_null_uuid FAILED                      [ 46%]
[   28s] tests/test_arf.py::TestArf::test06_creation_iter PASSED                  [ 53%]
[   28s] tests/test_arf.py::TestArf::test07_append_to_table PASSED                [ 61%]
[   29s] tests/test_arf.py::TestArf::test08_check_file_version PASSED             [ 69%]
[   29s] tests/test_arf.py::TestArf::test09_timestamp_conversion PASSED           [ 76%]
[   29s] tests/test_arf.py::TestArf::test10_select_from_timeseries FAILED         [ 84%]
[   29s] tests/test_arf.py::TestArf::test11_copy_entry_attrs PASSED               [ 92%]
[   29s] tests/test_arf.py::TestArf::test99_various PASSED                        [100%]
[   29s] 
[   29s] =================================== FAILURES ===================================
[   29s] _____________________ TestArf.test01_create_existing_entry _____________________
[   29s] 
[   29s] self = <tests.test_arf.TestArf testMethod=test01_create_existing_entry>
[   29s] 
[   29s]     def test01_create_existing_entry(self):
[   29s]         with self.assertRaises(ValueError):
[   29s] >           arf.create_entry(fp, entry_base % 0, tstamp, **entry_attributes)
[   29s] E           AssertionError: ValueError not raised
[   29s] 
[   29s] tests/test_arf.py:109: AssertionError
[   29s] ________________________ TestArf.test03_set_attributes _________________________
[   29s] 
[   29s] self = <tests.test_arf.TestArf testMethod=test03_set_attributes>
[   29s] 
[   29s]     def test03_set_attributes(self):
[   29s]         # tests the set_attributes convenience function
[   29s] >       arf.set_attributes(fp["entry_001/spikes"], mystr="myvalue", myint=5000)
[   29s] 
[   29s] tests/test_arf.py:123: 
[   29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[   29s]     oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[   29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   29s] 
[   29s] >   ???
[   29s] E   KeyError: 'Unable to open object (component not found)'
[   29s] 
[   29s] h5py/h5o.pyx:190: KeyError
[   29s] ______________________ TestArf.test04_create_bad_dataset _______________________
[   29s] 
[   29s] self = <tests.test_arf.TestArf testMethod=test04_create_bad_dataset>
[   29s] 
[   29s]     def test04_create_bad_dataset(self):
[   29s]         # f = raises(ValueError)(self.create_dataset)
[   29s] >       e = fp['entry_001']
[   29s] 
[   29s] tests/test_arf.py:131: 
[   29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[   29s]     oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[   29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   29s] 
[   29s] >   ???
[   29s] E   KeyError: "Unable to open object (object 'entry_001' doesn't exist)"
[   29s] 
[   29s] h5py/h5o.pyx:190: KeyError
[   29s] ___________________________ TestArf.test05_null_uuid ___________________________
[   29s] 
[   29s] self = <tests.test_arf.TestArf testMethod=test05_null_uuid>
[   29s] 
[   29s]     def test05_null_uuid(self):
[   29s]         # nulls in a uuid can make various things barf
[   29s]         from uuid import UUID
[   29s]         uuid = UUID(bytes=b''.rjust(16, b'\0'))
[   29s] >       e = fp['entry_001']
[   29s] 
[   29s] tests/test_arf.py:141: 
[   29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[   29s]     oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[   29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   29s] 
[   29s] >   ???
[   29s] E   KeyError: "Unable to open object (object 'entry_001' doesn't exist)"
[   29s] 
[   29s] h5py/h5o.pyx:190: KeyError
[   29s] ____________________ TestArf.test10_select_from_timeseries _____________________
[   29s] 
[   29s] self = <tests.test_arf.TestArf testMethod=test10_select_from_timeseries>
[   29s] 
[   29s]     def test10_select_from_timeseries(self):
[   29s]         entry = fp[entry_base % 0]
[   29s]         for data in datasets:
[   29s] >           dset = entry[data["name"]]
[   29s] 
[   29s] tests/test_arf.py:186: 
[   29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] /usr/lib64/python3.8/site-packages/h5py/_hl/group.py:264: in __getitem__
[   29s]     oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
[   29s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   29s]     ???
[   29s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   29s] 
[   29s] >   ???
[   29s] E   KeyError: "Unable to open object (object 'acoustic' doesn't exist)"
[   29s] 
[   29s] h5py/h5o.pyx:190: KeyError
[   29s] =========================== short test summary info ============================
[   29s] FAILED tests/test_arf.py::TestArf::test01_create_existing_entry - AssertionEr...
[   29s] FAILED tests/test_arf.py::TestArf::test03_set_attributes - KeyError: 'Unable ...
[   29s] FAILED tests/test_arf.py::TestArf::test04_create_bad_dataset - KeyError: "Una...
[   29s] FAILED tests/test_arf.py::TestArf::test05_null_uuid - KeyError: "Unable to op...
[   29s] FAILED tests/test_arf.py::TestArf::test10_select_from_timeseries - KeyError: ...
[   29s] ========================= 5 failed, 8 passed in 0.34s ==========================
[   29s] error: Bad exit status from /var/tmp/rpm-tmp.pSIDnG (%check)

(using pytest only as the test runner, nothing in my patch depends on pytest).

Full build log without skipping failing tests

transition to setuptools

more h5py internal API changes

Similar to #8, many tests now start to fail because of private API changes in h5py:

[   42s] _______________ TestArfNavigation.test10_select_from_timeseries ________________
[   42s] 
[   42s] self = <tests.test_arf.TestArfNavigation testMethod=test10_select_from_timeseries>
[   42s] 
[   42s]     def setUp(self):
[   42s] >       self.fp = arf.open_file("test", 'w', driver="core", backing_store=False)
[   42s] 
[   42s] tests/test_arf.py:156: 
[   42s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   42s] 
[   42s] name = b'test', mode = 'w', driver = 'core', libver = None
[   42s] userblock_size = None
[   42s] kwargs = {'backing_store': False, 'rdcc_nbytes': None, 'rdcc_nslots': None, 'rdcc_w0': None}
[   42s] sys = <module 'sys' (built-in)>
[   42s] os = <module 'os' from '/usr/lib64/python3.9/os.py'>, h5py_version = '3.6.0'
[   42s] StrictVersion = <class 'distutils.version.StrictVersion'>
[   42s] h5p = <module 'h5py.h5p' from '/usr/lib64/python3.9/site-packages/h5py/h5p.cpython-39-x86_64-linux-gnu.so'>
[   42s] files = <module 'h5py._hl.files' from '/usr/lib64/python3.9/site-packages/h5py/_hl/files.py'>
[   42s] exists = False
[   42s] 
[   42s]     def open_file(name, mode=None, driver=None, libver=None, userblock_size=None, **kwargs):
[   42s]         """Open an ARF file, creating as necessary.
[   42s]     
[   42s]         Use this instead of h5py.File to ensure that root-level attributes and group
[   42s]         creation property lists are set correctly.
[   42s]     
[   42s]         """
[   42s]         import sys
[   42s]         import os
[   42s]         from h5py.version import version as h5py_version
[   42s]         from distutils.version import StrictVersion
[   42s]         from h5py import h5p
[   42s]         from h5py._hl import files
[   42s]     
[   42s]         try:
[   42s]             # If the byte string doesn't match the default
[   42s]             # encoding, just pass it on as-is.  Note Unicode
[   42s]             # objects can always be encoded.
[   42s]             name = name.encode(sys.getfilesystemencoding())
[   42s]         except (UnicodeError, LookupError):
[   42s]             pass
[   42s]         exists = os.path.exists(name)
[   42s]         try:
[   42s]             fcpl = h5p.create(h5p.FILE_CREATE)
[   42s]             fcpl.set_link_creation_order(
[   42s]                 h5p.CRT_ORDER_TRACKED | h5p.CRT_ORDER_INDEXED)
[   42s]         except AttributeError:
[   42s]             # older version of h5py
[   42s]             fp = files.File(name, mode=mode, driver=driver,
[   42s]                             libver=libver, **kwargs)
[   42s]         else:
[   42s]             if StrictVersion(h5py_version) >= StrictVersion('2.9'):
[   42s]                 kwargs.update(rdcc_nslots=None, rdcc_nbytes=None, rdcc_w0=None)
[   42s] >           fapl = files.make_fapl(driver, libver, **kwargs)
[   42s] E           TypeError: make_fapl() missing 4 required positional arguments: 'locking', 'page_buf_size', 'min_meta_keep', and 'min_raw_keep'
[   42s] 
[   42s] arf.py:77: TypeError

Warnings attempting to (double) close invalid groups and datasets (c++)

In some cases closing an HDF5 file invalidates open group and dataset handles (in spite of what the spec says). Need to check whether the handles are valid in the c++ interface before calling the close methods to avoid generating a warning.