bluesky/area-detector-handlers

Bluesky — An Experiment Specification & Orchestration Engine

Source	https://github.com/bluesky/bluesky
PyPI	pip install bluesky
Documentation	https://bluesky.github.io/bluesky
Releases	https://github.com/bluesky/bluesky/releases

Bluesky is a library for experiment control and collection of scientific data and metadata. It emphasizes the following virtues:

• Live, Streaming Data: Available for inline visualization and processing.
• Rich Metadata: Captured and organized to facilitate reproducibility and searchability.
• Experiment Generality: Seamlessly reuse a procedure on completely different hardware.
• Interruption Recovery: Experiments are "rewindable," recovering cleanly from interruptions.
• Automated Suspend/Resume: Experiments can be run unattended, automatically suspending and resuming if needed.
• Pluggable I/O: Export data (live) into any desired format or database.
• Customizability: Integrate custom experimental procedures and commands, and get the I/O and interruption features for free.
• Integration with Scientific Python: Interface naturally with numpy and Python scientific stack.

Bluesky Documentation.

The Bluesky Project enables experimental science at the lab-bench or facility scale. It is a collection of Python libraries that are co-developed but independently useful and may be adopted a la carte.

Bluesky Project Documentation.

See https://bluesky.github.io/bluesky for more detailed documentation.

area-detector-handlers's Issues

tifffile can no longer read (at least some) AD TIFF files

Example:

In [1]: import tifffile                                                                                                                                                                       

In [2]: fn = '/XF11ID/data/2020/08/25/f448221a-8f02-4a97-903a_000000.tiff'                                                                                                                    

In [3]: tifffile.TiffFile(fn)                                                                                                                                                                 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-3c6068d96329> in <module>
----> 1 tifffile.TiffFile(fn)

/opt/conda_envs/collection-2019-3.0-chx/lib/python3.7/site-packages/tifffile/tifffile.py in __init__(self, arg, name, offset, size, multifile, movie, **kwargs)
   1785 
   1786             # file handle is at offset to offset to first page
-> 1787             self.pages = TiffPages(self)
   1788 
   1789             if self.is_lsm and (self.filehandle.size >= 2**32 or

/opt/conda_envs/collection-2019-3.0-chx/lib/python3.7/site-packages/tifffile/tifffile.py in __init__(self, parent)
   2845         # always read and cache first page
   2846         fh.seek(offset)
-> 2847         page = TiffPage(parent, index=0)
   2848         self.pages.append(page)
   2849         self._keyframe = page

/opt/conda_envs/collection-2019-3.0-chx/lib/python3.7/site-packages/tifffile/tifffile.py in __init__(self, parent, index, keyframe)
   3171             self.andor_tags
   3172         elif self.is_epics:
-> 3173             self.epics_tags
   3174 
   3175         if self.is_lsm or (self.index and self.parent.is_lsm):

/opt/conda_envs/collection-2019-3.0-chx/lib/python3.7/site-packages/tifffile/tifffile.py in __get__(self, instance, owner)
    611             return self
    612         try:
--> 613             value = self.func(instance)
    614         except AttributeError as e:
    615             raise RuntimeError(e)

/opt/conda_envs/collection-2019-3.0-chx/lib/python3.7/site-packages/tifffile/tifffile.py in epics_tags(self)
   3795             if code == 65000:
   3796                 result['timeStamp'] = datetime.datetime.fromtimestamp(
-> 3797                     float(value))
   3798             elif code == 65001:
   3799                 result['uniqueID'] = int(value)

ValueError: year 357964404 is out of range

Write schemas for Resource, Datum expected by each handler.

Express3 handler error at QAS

/opt/conda_envs/collection-2020-2.0rc7-1/lib/python3.7/site-packages/dask/utils.py in apply(func=<bound method Xspress3HDF5Handler.__call__ of Su...m/2020/08/17/e7b09d10-7ad4-433c-84d3_000000.h5')>, args=[], kwargs={'frame': 0})
     29 def apply(func, args, kwargs=None):
     30     if kwargs:
---> 31         return func(*args, **kwargs)
        func = <bound method Xspress3HDF5Handler.__call__ of Subclass(filename='/nsls2/xf07bm/data/x3m/2020/08/17/e7b09d10-7ad4-433c-84d3_000000.h5')>
        args = []
        kwargs = {'frame': 0}
     32     else:
     33         return func(*args)

/opt/conda_envs/collection-2020-2.0rc7-1/lib/python3.7/site-packages/area_detector_handlers/_xspress3.py in __call__(self=Subclass(filename='/nsls2/xf07bm/data/x3m/2020/08/17/e7b09d10-7ad4-433c-84d3_000000.h5'), frame=0, channel=None)
     87         # Don't read out the dataset until it is requested for the first time.
     88         self._get_dataset()
---> 89         return self._dataset[frame, channel - 1, :].squeeze()
        self._dataset = array([[[ 1.0022392 ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [27.06106184, 32.07236958, 23.05201564, ...,  0.        ,
          0.        ,  6.0135693 ]],

       [[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [24.05472318, 38.08664504, 27.06156358, ...,  0.        ,
          0.        ,  4.00912053]],

       [[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [24.05443389, 23.05216581, 33.0748466 , ...,  0.        ,
          0.        ,  6.01360847]],

       ...,

       [[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [43.01456008, 46.0155759 , 37.01252844, ...,  0.        ,
          0.        ,  1.00033861]],

       [[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 1.00035137,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [44.01525092, 33.01143819, 50.01733059, ...,  0.        ,
          0.        ,  2.00069322]],

       [[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [39.01331298, 40.01365434, 36.0122889 , ...,  0.        ,
          0.        ,  1.00034136]]])
        frame = 0
        channel.squeeze = undefined
     90
     91     def get_file_list(self, datum_kwarg_gen):

TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

IMM detector and get_files_list

While trying to export data 8-ID-I, we found that the handler for the IMM detector was missing a get_files_list() method.

We played around in the time we had, but were unable to test our proposed implementation. @gwbischof or @prjemian can you please add the implementation you were testing?

Latest stable version is missing get_file_list method on AD_HDF5_TS

I have not confirmed whether one of the open PRs already addresses this.

Is AD_HDF5_SINGLE for general use?

Via conversations on Slack, it seems the AD_HDF5_SINGLE handler is specific to some HDF5 files generated by the SIX instrument at facility NSLS-II. If so, that's too bad since the name suggests a more general use.

Maybe acknowledge (by stating boldly in the documentation) that AD_HDF5_SINGLE is only for NSLS-II instrument SIX. Make a new handler (say AD_HDF5_GENERAL) which matches the files generated via FileStoreHDF5SingleIterativeWrite (which may need some modification as well). Then, if SIX still uses this mixin, they, alone, would need to customize for their specific use.

One issue noted by @tacaswell with the AD_HDF5_SINGLE handler:

that handler does not conform to what we expect handlers to do (return an ND sliceable thing)

Related to: BCDA-APS/apstools#691

Failing unit tests on Windows

The following error was observed when running the full test suite for a conda-forge recipe (see conda-forge/staged-recipes#15389 (comment)):

(%PREFIX%) %SRC_DIR%>pytest -v --pyargs area_detector_handlers 
============================= test session starts =============================
platform win32 -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- %PREFIX%\python.exe
cachedir: .pytest_cache
rootdir: %SRC_DIR%
collecting ... collected 13 items

tests/test_eiger.py::test_eiger[EigerHandler] PASSED                     [  7%]
tests/test_eiger.py::test_eiger2[EigerHandler] PASSED                    [ 15%]
tests/test_eiger.py::test_eiger2[EigerHandler] ERROR                     [ 15%]
tests/test_entrypoints.py::test_entrypoint_consisetncy PASSED            [ 23%]
tests/test_examples.py::test_context PASSED                              [ 30%]
tests/test_hdf5.py::test_hdf5[1-AreaDetectorHDF5Handler] PASSED          [ 38%]
tests/test_hdf5.py::test_hdf5_slicing_bound_exception[1-AreaDetectorHDF5Handler] PASSED [ 46%]
tests/test_hdf5.py::test_hdf5[5-AreaDetectorHDF5Handler] PASSED          [ 53%]
tests/test_hdf5.py::test_hdf5_slicing_bound_exception[5-AreaDetectorHDF5Handler] PASSED [ 61%]
tests/test_imm.py::test_imm[IMMHandler-B020_bluesky_demo_00001-00021.imm-21-frame_shape0] PASSED [ 69%]
tests/test_tiff.py::test_tiff[1-AreaDetectorTiffHandler] PASSED          [ 76%]
tests/test_tiff.py::test_tiff[5-AreaDetectorTiffHandler] PASSED          [ 84%]
tests/test_xs3.py::test_bulk[BulkXSPRESS] PASSED                         [ 92%]
tests/test_xs3.py::test_pre_pixel[Xspress3HDF5Handler] PASSED            [100%]

=================================== ERRORS ====================================
_______________ ERROR at teardown of test_eiger2[EigerHandler] ________________

    def finalize():
        os.remove(out_name)
>       os.remove("sample_data_000001.h5")
E       PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'sample_data_000001.h5'

..\_test_env\lib\site-packages\area_detector_handlers\tests\conftest.py:82: PermissionError
============================== warnings summary ===============================
..\_test_env\lib\site-packages\pyreadline\py3k_compat.py:8
  %PREFIX%\lib\site-packages\pyreadline\py3k_compat.py:8: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    return isinstance(x, collections.Callable)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ===========================
ERROR tests/test_eiger.py::test_eiger2[EigerHandler] - PermissionError: [WinE...
=================== 13 passed, 1 warning, 1 error in 2.88s ====================

Migrate from Travis-CI to GitHub Actions

See bluesky/bluesky-enhancement-proposals#21 for details.

HDF5DatasetSliceHandler should check that it is slicing within valid bounds

As a result of some configuration issues, I encountered a symptom where the xarrays extracted from databroker had 0 length on one dimensions. @danielballan and I traced this to

area-detector-handlers/area_detector_handlers/handlers.py

Line 131 in 86d406f

return self._dataset[start:stop]

As an effect of having wrong point_number values in my datum_pages, the handler was slicing beyond the size of the dataset, giving back an empty (0 len in one dim) array.

This handler should check that the slicing bounds are valid, and raise otherwise. This will help in tracing any potential future configuration issues.

get_file_list

A talk with @danielballan led us to these questions.

Can the handler methods get_file_list be part of the handler class or be moved to event_model?

It appears that none of the handlers in handlers.py need to open the file to get the list of files for the run. And the file list can be determined from the mongo documents.

If get_file_list is very specific to the handler, maybe we leave this method as part of the Handler class, but change it so that it can be called, without calling the init first.

Some of the handler inits call open on the file, and I would like to get the list of files with access to the file.

Thoughts?

Use a finalizer instead of relying on __del__

Consider refactoring

area-detector-handlers/area_detector_handlers/__init__.py

Lines 25 to 29 in 316e126

	def __del__(self):
	try:
	self.close()
	except Exception:
	pass

according to the advice in https://docs.python.org/3/library/weakref.html#comparing-finalizers-with-del-methods

Update conf.py for current Sphinx

The CI failure in #32 is because the conf.py file is out of date with current Sphinx. (need to comment the language= line now, the default is acceptable).

Originally posted by @prjemian in #32 (comment)