prjemian/punx

Python Utilities for NeXus HDF5 files

punx - Python Utilities for NeXus HDF5 files

GitHub	PyPI	conda-forge

Python version(s)	Unit Tests	Code Coverage	License

Documentation

The punx package provides these features:

• Validate NeXus HDF5 data files
• Choose the NeXus release to use for validation
• Validate NeXus NXDL files
• Display NeXus HDF5 data file structure
• Display NeXus class hierarchy (stretch goal, graphical output)

Package Details

• author: Pete R. Jemian
• email: [email protected]
• copyright: 2014-2022, Pete R. Jemian
• license: Creative Commons Attribution 4.0 International Public License (see LICENSE.txt)
• URL: https://prjemian.github.io/punx
• git: https://github.com/prjemian/punx
• issues: https://github.com/prjemian/punx/issues
• citation:

punx's Issues

make MyArgumentParser.parse_args() more robust

make more robust for variations in optional commands

current assumption is that sys.argv[1] has the subcommand

It is possible to add arguments to the main parser that add optional arguments which may precede a subcommand.

validate: comment not clear

/entry1/DMC/DMC-BF3-Detector    NXinstrument subgroup    ERROR    NXpsd: not known NeXus class NXinstrument      

python 3

support both python 2 and python 3

verify that shape of each dimension scale is 1-D

in validate.version1 (and possibly version2 & version3) dimension scales should be 1-D

test & report about each one

verify algorithm of h5structure._get_group_niac2014()

this code may benefit from the validate.py code to get this assessment

The original comment in the code reads:

Does this search ALL POSSIBLE {nxclass_name} groups for at least one compliance or just the first one it finds?

TypeError exception when validating non-NeXus HDF5 file

mintadmin@mintadmin-VirtualBox ~ $ punx validate /home/mintadmin/Apps/anaconda/pkgs/bokeh-0.8.1-np19py27_1/lib/python2.7/site-packages/bokeh/server/tests/data/AAPL.hdf5
Traceback (most recent call last):
  File "/home/mintadmin/Apps/anaconda/bin/punx", line 11, in <module>
    sys.exit(main())
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 326, in main
    args.func(args)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 157, in func_validate
    validator.validate()
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/validate.py", line 559, in validate
    self.validate_group(obj, 'NXentry')
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/validate.py", line 588, in validate_group
    self.validate_attributes(group, nx_class)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/validate.py", line 673, in validate_attributes
    msg = 'unknown: ' + nxdl_class
TypeError: cannot concatenate 'str' and 'NoneType' objects
mintadmin@mintadmin-VirtualBox ~ $ 

files from Eiger detector need new libraries

(Related to issue #39)
We have received a linked fileset from an Eiger 1M and want to read it.
Cannot seem to progress past knowing that there is data.
Here is an excerpt from h5dump (version 1.8.16):

   DATASET "data" {
      DATATYPE H5T_STD_U32LE
      DATASPACE SIMPLE { ( 1, 1065, 1030 ) / ( H5S_UNLIMITED, 1065, 1030 ) }
      DATA {h5dump error: unable to print data

      }
      ATTRIBUTE "image_nr_high" {
         DATATYPE H5T_STD_I32LE
         DATASPACE SCALAR
         DATA {
         (0): 1
         }
      }
      ATTRIBUTE "image_nr_low" {
         DATATYPE H5T_STD_I32LE
         DATASPACE SCALAR
         DATA {
         (0): 1
         }
      }
   }

It's the error: "h5dump error: unable to print data"

missing NXentry group not reported in validation

When validating a non-NeXus HDF5 data file, the report does not mention that there is no NXentry group. (This file was a component of a linked file set with master and two linked files.)

mintadmin@mintadmin-VirtualBox ~ $ punx va /home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/external_angles.hdf5
Validation findings
:file: external_angles.hdf5
:validation results shown:  COMMENT, ERROR, NOTE, OK, TODO, UNUSED, WARN
============= ============================ ======= ================================
address       validation                   status  comment(s)                      
============= ============================ ======= ================================
/             validate all names           COMMENT ----------                      
/angles       validItemName-strict         OK      strict re: [a-z_][a-z0-9_]*     
/angles@units validItemName-strict         OK      strict re: [a-z_][a-z0-9_]*     
/             validate all objects         COMMENT ----------                      
/angles       field data type              OK      data type: float64              
/angles@units field units attribute        OK      exists                          
/angles@units field units attribute        OK      value: degrees                  
/             review file root with NXroot COMMENT ----------                      
/             NXDL known                   OK      NXroot                          
/angles       NXroot defined field         NOTE    acceptable                      
/angles       NXroot field                 TODO    need to complete check with NXDL
/             check for default plot       COMMENT ----------                      
/@default     @default attribute           NOTE    does not exist                  
============= ============================ ======= ================================

summary statistics
======= ===== ===========================================================
status  count description                                                
======= ===== ===========================================================
OK      6     meets NeXus specification                                  
NOTE    2     does not meet NeXus specification, but acceptable          
WARN    0     does not meet NeXus specification, not generally acceptable
ERROR   0     violates NeXus specification                               
TODO    1     validation not implemented yet                             
UNUSED  0     optional NeXus item not used in data file                  
COMMENT 4     comment from the punx source code                          
--      --    --                                                         
TOTAL   13    --                                                         
======= ===== ===========================================================

mintadmin@mintadmin-VirtualBox ~ $ 

update user cache with named version of NXDL

in the update module, allow the user to specify a specific version of NeXus, named either by release, tag or SHA

show local NeXus base class hierarchy from a given base class

the full hierarchy looks very complicated, this might be easier to read

make GUI for any/all features

this is a stretch goal, beyond the initial release

punx update fails

punx update fails

I can download commits but punx update doesn't work:

ich@AirBook:~ punx update
INFO: get repo info: https://api.github.com/repos/nexusformat/definitions/commits
ERROR: IOError from https://api.github.com/repos/nexusformat/definitions/commits
INFO: GitHub not available

ich@AirBook:~ wget --no-check-certificate https://api.github.com/repos/nexusformat/definitions/commits
--2016-06-28 08:51:20--  https://api.github.com/repos/nexusformat/definitions/commits
Resolving api.github.com... 192.30.253.116
Connecting to api.github.com|192.30.253.116|:443... connected.
WARNING: cannot verify api.github.com's certificate, issued by ‘CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 107156 (105K) [application/json]
Saving to: ‘commits’

commits                  100%[==================================>] 104.64K   339KB/s   in 0.3s

2016-06-28 08:51:21 (339 KB/s) - ‘commits’ saved [107156/107156]

versioneer-supplied version number wrong when installed with pip

The version number, when installed with pip (using versioneer) is wrong. Can this be fixed?

use logging

various operations would benefit from the logging package being used

error rendering 4-D data from DLS

cannot render 4-D data from DLS, problem with this syntax

eval(obj[0, :::])

validating fails with some non-NeXus files

when attempting to validate against obviously non-NeXus HDF5 files (such as legacy_table.h5 from pandas), exceptions are raised when NoneType objects are added to str objects for various output

These should be trapped and suitable response made to report None was discovered.

need to learn *minOccurs* from NXDL

this is needed in various places of validate.py

h5structure: UnicodeDecodeError

mintadmin@mintadmin-VirtualBox ~/test_data $ punx st file_1.nxs 
Traceback (most recent call last):
  File "/home/mintadmin/Apps/anaconda/bin/punx", line 11, in <module>
    sys.exit(main())
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 326, in main
    args.func(args)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 129, in func_structure
    report = mc.report(show_attributes)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 67, in report
    structure = self._renderGroup(f, txt, indentation = "")
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 118, in _renderGroup
    s += self._renderGroup(value, itemname, indentation+"  ")
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 118, in _renderGroup
    s += self._renderGroup(value, itemname, indentation+"  ")
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 118, in _renderGroup
    s += self._renderGroup(value, itemname, indentation+"  ")
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 106, in _renderGroup
    s += self._renderDataset(value, itemname, indentation+"  ")
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 163, in _renderDataset
    s += self._renderAttributes(dset, indentation)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 127, in _renderAttributes
    s.append("%s  @%s = %s" % (indentation, name, str(value)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 0: ordinal not in range(128)

implement subcommand: show

show internal configuration, status of each cache, ...

improve exception trace report in h5structure

Hard to diagnose the problem with user's data file from this exception report:

> user@AirBook:~/Data punx structure !$
> punx structure file_master.h5
> Traceback (most recent call last):
>   File "/sw/bin/punx", line 9, in <module>
>     load_entry_point('punx==0+unknown', 'console_scripts', 'punx')()
>   File "/sw/lib/python2.7/site-packages/punx/main.py", line 210, in main
>     args.func(args)
>   File "/sw/lib/python2.7/site-packages/punx/main.py", line 106, in
> func_structure
>     print '\n'.join(mc.report(show_attributes) or '')
>   File "/sw/lib/python2.7/site-packages/punx/h5structure.py", line 58,
> in report
>     structure = self._renderGroup(f, txt, indentation = "")
>   File "/sw/lib/python2.7/site-packages/punx/h5structure.py", line 109,
> in _renderGroup
>     s += self._renderGroup(value, itemname, indentation+"  ")
>   File "/sw/lib/python2.7/site-packages/punx/h5structure.py", line 109,
> in _renderGroup
>     s += self._renderGroup(value, itemname, indentation+"  ")
>   File "/sw/lib/python2.7/site-packages/punx/h5structure.py", line 97,
> in _renderGroup
>     s += self._renderDataset(value, itemname, indentation+"  ")
>   File "/sw/lib/python2.7/site-packages/punx/h5structure.py", line 158,
> in _renderDataset
>     value = self._renderArray(dset, indentation + '  ')
>   File "/sw/lib/python2.7/site-packages/punx/h5structure.py", line 205,
> in _renderArray
>     r = self._renderNdArray(obj, indentation + '  ')
>   File "/sw/lib/python2.7/site-packages/punx/h5structure.py", line 234,
> in _renderNdArray
>     r.append( __render(obj, rank, i, indentation + '  ') )
>   File "/sw/lib/python2.7/site-packages/punx/h5structure.py", line 224,
> in __render
>     part = eval('obj[%s]' % indices)
>   File "<string>", line 1, in <module>
>   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
> (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2687)
>   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
> (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2645)
>   File "/sw/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 482,
> in __getitem__
>     self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
>   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
> (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2687)
>   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
> (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2645)
>   File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
> (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/h5d.c:3231)
>   File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
> (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_proxy.c:1860)
>   File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
> (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_proxy.c:1508)
> IOError: Can't read data (Can't stat file: no such file or directory)

adopt unit testing

this will be especially useful for the validation process

*update* command needs console output

the update command has not console output
users need to know what is happening

NeXus HDF5 not identified as such by h5structure

structure

/home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/prj_test.nexus.hdf5
  @default = entry
  @file_name = prj_test.nexus.hdf5
  @file_time = 2010-10-18T17:17:04-0500
  @instrument = APS USAXS at 32ID-B
  @creator = BasicWriter.py
  @NeXus_version = 4.3.0
  @HDF5_Version = 1.8.15
  @h5py_version = 2.5.0
  entry:NXentry
    @NX_class = NXentry
    @default = mr_scan
    title:CHAR = 1-D scan of I00 v. mr
    mr_scan:NXdata
      @NX_class = NXdata
      @signal = I00
      @axes = mr
      @mr_indices = [0]
      I00:int32[31] = [1037, 1318, 1704, '...', 1321]
        @units = counts
        @long_name = USAXS I00 (counts)
      mr:float64[31] = [17.926079999999999, 17.925909999999998, 17.925750000000001, '...', 17.92108]
        @units = degrees
        @long_name = USAXS mr (degrees)

validation

mintadmin@mintadmin-VirtualBox ~ $ punx va /home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/prj_test.nexus.hdf5
Validation findings
:file: prj_test.nexus.hdf5
:validation results shown:  COMMENT, ERROR, NOTE, OK, TODO, UNUSED, WARN
============================ =========================================== ======= ===============================================
address                      validation                                  status  comment(s)                                     
============================ =========================================== ======= ===============================================
/                            validate all names                          COMMENT ----------                                     
/@HDF5_Version               validItemName                               NOTE    relaxed re: [A-Za-z_][\w_]*                    
/@NeXus_version              validItemName                               NOTE    relaxed re: [A-Za-z_][\w_]*                    
/@creator                    validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/@default                    validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/@file_name                  validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/@file_time                  validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/@h5py_version               validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/@instrument                 validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry                       validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry@NX_class              validItemName                               NOTE    relaxed re: [A-Za-z_][\w_]*                    
/entry@default               validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/mr_scan               validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/mr_scan@NX_class      validItemName                               NOTE    relaxed re: [A-Za-z_][\w_]*                    
/entry/mr_scan@axes          validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/mr_scan@mr_indices    validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/mr_scan@signal        validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/mr_scan/I00           validItemName                               NOTE    relaxed re: [A-Za-z_][\w_]*                    
/entry/mr_scan/I00@long_name validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/mr_scan/I00@units     validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/mr_scan/mr            validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/mr_scan/mr@long_name  validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/mr_scan/mr@units      validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/entry/title                 validItemName-strict                        OK      strict re: [a-z_][a-z0-9_]*                    
/                            validate all objects                        COMMENT ----------                                     
/entry                       NX_class known                              OK      value: NXentry                                 
/entry                       NXDL known                                  OK      NXentry                                        
/entry@default               NXentry attribute                           OK      defined in NXentry                             
/entry@default               NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/entry/mr_scan               NXentry subgroup                            OK      NXdata                                         
/entry/title                 NXentry defined field                       NOTE    acceptable                                     
/entry/title                 NXentry field                               TODO    need to complete check with NXDL               
/entry@default               NXentry attribute                           OK      defined in NXentry                             
/entry@default               NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/entry/mr_scan               NX_class known                              OK      value: NXdata                                  
/entry/mr_scan               NXDL known                                  OK      NXdata                                         
/entry/mr_scan@axes          NXdata attribute                            OK      defined in NXdata                              
/entry/mr_scan@axes          NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/entry/mr_scan@signal        NXdata attribute                            OK      defined in NXdata                              
/entry/mr_scan@signal        NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/entry/mr_scan/I00           NXdata defined field                        OK      acceptable                                     
/entry/mr_scan/I00           NXdata field                                TODO    need to complete check with NXDL               
/entry/mr_scan/mr            NXdata defined field                        OK      acceptable                                     
/entry/mr_scan/mr            NXdata field                                TODO    need to complete check with NXDL               
/entry/mr_scan@axes          NXdata attribute                            OK      defined in NXdata                              
/entry/mr_scan@axes          NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/entry/mr_scan@signal        NXdata attribute                            OK      defined in NXdata                              
/entry/mr_scan@signal        NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/entry/mr_scan/I00           field data type                             OK      data type: int32                               
/entry/mr_scan/I00@units     field units attribute                       OK      exists                                         
/entry/mr_scan/I00@units     field units attribute                       OK      value: counts                                  
/entry/mr_scan/mr            field data type                             OK      data type: float64                             
/entry/mr_scan/mr@units      field units attribute                       OK      exists                                         
/entry/mr_scan/mr@units      field units attribute                       OK      value: degrees                                 
/                            review file root with NXroot                COMMENT ----------                                     
/                            NXDL known                                  OK      NXroot                                         
/@HDF5_Version               NXroot attribute                            OK      defined in NXroot                              
/@HDF5_Version               NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/@NeXus_version              NXroot attribute                            OK      defined in NXroot                              
/@NeXus_version              NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/@creator                    NXroot attribute                            OK      defined in NXroot                              
/@creator                    NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/@default                    NXroot attribute                            OK      defined in NXroot                              
/@default                    NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/@file_name                  NXroot attribute                            OK      defined in NXroot                              
/@file_name                  NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/@file_time                  NXroot attribute                            OK      defined in NXroot                              
/@file_time                  NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/@h5py_version               NXroot attribute                            OK      defined in NXroot                              
/@h5py_version               NXDL NX_type                                OK      <type 'str'> : NX_CHAR                         
/entry                       NXroot subgroup                             OK      NXentry                                        
/                            check for default plot                      COMMENT ----------                                     
/entry/mr_scan               default plot v3 @signal attribute           OK      exists                                         
/entry/mr_scan               default plot v3 @signal value               OK      value: I00                                     
/entry/mr_scan               default plot v3 @axes attribute             OK      exists                                         
/entry/mr_scan               default plot v3 @axes value                 OK      value: mr                                      
/entry/mr_scan@mr_indices    attribute data type                         NOTE    [variable length string]: [0]                  
/entry/mr_scan@mr_indices    default plot v3 @AXISNAME_indices attribute OK      value: 0                                       
/entry/mr_scan@mr_indices    default plot v3 @AXISNAME_indices value     OK      rank test: 0 <= 0 < 1                          
/entry/mr_scan               default plot v3 dimension scales            TODO    unchecked                                      
/entry/mr_scan               NXdata default plottable data               OK      version 3                                      
/entry                       /NXentry/NXdata/signal_data                 OK      basic NeXus requirement: default plot described
/@default                    @default attribute                          OK      exists                                         
/@default                    @default value                              OK      HDF5 item exists                               
/@default                    @default value test                         OK      HDF5 item is NXentry                           
/entry@default               @default attribute                          OK      exists                                         
/entry@default               @default value                              OK      HDF5 item exists                               
/entry@default               @default value test                         OK      HDF5 item is NXdata                            
/entry/mr_scan               /NXroot@default/NXentry@default/NXdata      OK      absolute path to default plot                  
============================ =========================================== ======= ===============================================

summary statistics
======= ===== ===========================================================
status  count description                                                
======= ===== ===========================================================
OK      74    meets NeXus specification                                  
NOTE    7     does not meet NeXus specification, but acceptable          
WARN    0     does not meet NeXus specification, not generally acceptable
ERROR   0     violates NeXus specification                               
TODO    4     validation not implemented yet                             
UNUSED  0     optional NeXus item not used in data file                  
COMMENT 4     comment from the punx source code                          
--      --    --                                                         
TOTAL   89    --                                                         
======= ===== ===========================================================

mintadmin@mintadmin-VirtualBox ~ $ 

validate has undefined "title" error

ich@AirBook:~ punx validate ~/Data/SUM2_180dps_0p18T_master.h5
Traceback (most recent call last):
  File "/sw/bin/punx", line 9, in <module>
    load_entry_point('punx==0+unknown', 'console_scripts', 'punx')()
  File "/sw/lib/python2.7/site-packages/punx/main.py", line 201, in main
    args.func(args)
  File "/sw/lib/python2.7/site-packages/punx/main.py", line 119, in func_validate
    validator.validate()
  File "/sw/lib/python2.7/site-packages/punx/validate.py", line 559, in validate
    self.validate_default_plot()
  File "/sw/lib/python2.7/site-packages/punx/validate.py", line 431, in validate_default_plot
    elif version1(nxdata_group):
  File "/sw/lib/python2.7/site-packages/punx/validate.py", line 291, in version1
    self.new_finding(title, field.name + '@axis', finding.TODO, 'not checked')
UnboundLocalError: local variable 'title' referenced before assignment

apply NXDL rules when parsing NXDL files

The NXDL rules parsed from nxdl.xsd and nxdlTypes.xsd establish defaults for the various elements of NeXus, as used in NXDL files. Subsequently, the parsed NXDL files are used to validate NeXus HDF5 data files.

• file
• attribute
• definition
• field
• group
• link
• symbol

dimension scales can define bin boundaries or bin centers

see nexpy/nexpy#95 (comment)

Basically, this test defines a valid length for a NeXus dimension scale

length_ok = len(scale) - len(signal_data_axis) in (0, 1)

Make sure this is the validation test that is applied

cache fallback algorithm needs improvement

Code was added today to workaround the current algorithm, which was failing.
The problem is that cache.py would need to import nxdlstructure.py and this would create an infinite loop of imports. Undesirable.

For a fresh install of the package, when no user cache directory contains a valid pickle file OR a valid directory of NXDL definitions AND GitHub is not available to update the user cache, the code must fall back to the source cache installed with the package.

@axes generates an ERROR

not obvious why this is a problem

suspect it is objecting to the ":" which is correct usage for v2 default data rules (http://download.nexusformat.org/doc/html/datarules.html?highlight=colon#version-2) but not for v3 (http://download.nexusformat.org/doc/html/datarules.html?highlight=colon#version-3) which specifies a string or string array

/entry/data    default plot v3 @axes value    ERROR    value: sample_times:vertical:horizontal        

the comment should be more clear

arg parsing fails in some cases

PS C:\Users\Pete\Documents\eclipse\punx> punx hi
Traceback (most recent call last):
  File "d:\apps\anaconda\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "d:\apps\anaconda\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "D:\Apps\Anaconda\Scripts\punx.exe\__main__.py", line 9, in <module>
  File "d:\apps\anaconda\lib\site-packages\punx\main.py", line 317, in main
    if args.logfile == '__console__':
AttributeError: 'Namespace' object has no attribute 'logfile'
PS C:\Users\Pete\Documents\eclipse\punx>

check field dimensions against "rank" attribute

nxdlstructure.field_dimensions()

IOError when rendering structure of non-NeXus HDF5 file

mintadmin@mintadmin-VirtualBox ~ $ punx st /home/mintadmin/Apps/anaconda/pkgs/bokeh-0.8.1-np19py27_1/lib/python2.7/site-packages/bokeh/server/tests/data/array.hdf5
Traceback (most recent call last):
  File "/home/mintadmin/Apps/anaconda/bin/punx", line 11, in <module>
    sys.exit(main())
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 326, in main
    args.func(args)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 129, in func_structure
    report = mc.report(show_attributes)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 67, in report
    structure = self._renderGroup(f, txt, indentation = "")
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 80, in _renderGroup
    s += self._renderAttributes(obj, indentation)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 126, in _renderAttributes
    for name, value in obj.attrs.iteritems():
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/h5py/_hl/base.py", line 381, in iteritems
    yield (x, self.get(x))
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/h5py/_hl/base.py", line 340, in get
    return self[name]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2579)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2538)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/h5py/_hl/attrs.py", line 73, in __getitem__
    attr.read(arr, mtype=htype)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2579)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2538)
  File "h5py/h5a.pyx", line 355, in h5py.h5a.AttrID.read (/home/ilan/minonda/conda-bld/work/h5py/h5a.c:5125)
  File "h5py/_proxy.pyx", line 36, in h5py._proxy.attr_rw (/home/ilan/minonda/conda-bld/work/h5py/_proxy.c:915)
IOError: Unable to read attribute (No appropriate function for conversion path)
mintadmin@mintadmin-VirtualBox ~ $ 

nxdl_rules.NxdlRules() as defaults in nxdlstructure.NXDL_mixin subclasses

apply nxdl_rules.NxdlRules() as defaults when creating NXDL_mixin subclasses

check if unknown names are allowed to be flexible

in validate.review_with_NXDL(), check if an otherwise unrecognized field or group name (not known to NXDL class) can fit some specification of the NXDL

validation comment not clear

/entry/link_rules    NXentry subgroup    ERROR    link_rules: not known NeXus class NXentry      

implement warnings as advised in NeXus manual

the search:
http://download.nexusformat.org/doc/html/search.html?q=warning&check_keywords=yes&area=default

Actually, flag them as NOTE unless WARN is compelling

version number wrong when installed by pip

mintadmin@mintadmin-VirtualBox ~ $ pip install punx
Collecting punx
  Downloading punx-0.0.5.zip (355kB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 358kB 2.0MB/s 
Requirement already satisfied (use --upgrade to upgrade): h5py in ./Apps/anaconda/lib/python2.7/site-packages (from punx)
Requirement already satisfied (use --upgrade to upgrade): lxml in ./Apps/anaconda/lib/python2.7/site-packages (from punx)
Requirement already satisfied (use --upgrade to upgrade): numpy in ./Apps/anaconda/lib/python2.7/site-packages (from punx)
Requirement already satisfied (use --upgrade to upgrade): pyRestTable in ./Apps/anaconda/lib/python2.7/site-packages (from punx)
Requirement already satisfied (use --upgrade to upgrade): six in ./Apps/anaconda/lib/python2.7/site-packages (from h5py->punx)
Building wheels for collected packages: punx
  Running setup.py bdist_wheel for punx ... done
  Stored in directory: /home/mintadmin/.cache/pip/wheels/be/d5/58/3e1b2ff3914e49aae887275fa501e84573f863f9974ca7dde8
Successfully built punx
Installing collected packages: punx
Successfully installed punx-0-unknown
mintadmin@mintadmin-VirtualBox ~ $ 

check item names for UnicodeDecodeError

related to #37, report ERROR if rendering the item name raises this exception

change ERROR to WARN

under the notion of being more tolerant for structures which are valid for HDF5 but not NeXus compliant, consider changing the ERROR assignment below to a WARN:

/image_1/axis/id@NX_class                                            validItemName                NOTE    relaxed re: [A-Za-z_][\w_]*                        
/image_1/axis/id/data                                                validItemName-strict         OK      strict re: [a-z_][a-z0-9_]*                        
/image_1/axis/id/data@items                                          validItemName-strict         OK      strict re: [a-z_][a-z0-9_]*                        
/image_1/axis/offset[1]                                              validItemName                ERROR   relaxed re: [A-Za-z_][\w_]*                        
/image_1/axis/offset[1]@NX_class                                     validItemName                NOTE    relaxed re: [A-Za-z_][\w_]*                        

That is, for names which do not pass the strict or the relaxed validItemName regular expression, change the assignment from ERROR to WARN

base class hierarchy drawing could be improved

discussed at today's NeXus telco (http://wiki.nexusformat.org/Telco_20160531)

"looks very complex, too many lines" - interpret this to mean too much overlap
"idea is good for the manual but image is not yet ready"
"in its current form: likely to create more questions than answers"
"very informative for those who know NeXus"
"very instructive to those learning NeXus"
"note the recursion involving NXgeometry"

• try to spread this out: make drawing sheet larger?
• straighten lines
• automate .png file production
• consider alternative renderer for directed graph

IOError reading HDF5 files

Received a sample file set from Diamond Light Source with data from the Eiger detector.

Neither structure nor validate can handle this yet. Here's the exception:

 jemian@gov ~/Downloads $ punx st PB103A_2_98_master.h5
 Traceback (most recent call last):
   File "/home/oxygen/JEMIAN/Apps/anaconda/bin/punx", line 9, in <module>
     load_entry_point('punx==0.0.6+6.g2614f69', 'console_scripts', 'punx')()
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 387, in main
     args.func(args)
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 159, in func_structure
     report = mc.report(show_attributes)
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 90, in report
     structure = self._renderGroup(f, txt, indentation = "")
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 141, in _renderGroup
     s += self._renderGroup(value, itemname, indentation+"  ")
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 141, in _renderGroup
     s += self._renderGroup(value, itemname, indentation+"  ")
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 129, in _renderGroup
     s += self._renderDataset(value, itemname, indentation+"  ")
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 200, in _renderDataset
     value = self._renderArray(dset, indentation + '  ')
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 247, in _renderArray
     r = self._renderNdArray(obj, indentation + '  ')
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 279, in _renderNdArray
     r.append( __render(obj, rank, i, indentation + '  ') )
   File "/local/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 269, in __render
     part = eval('obj[%s]' % indices)
   File "<string>", line 1, in <module>
   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2582)
   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2541)
   File "/local/Apps/anaconda/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 482, in __getitem__
     self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2582)
   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2541)
   File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read (-------src-dir-------/h5py/h5d.c:3128)
   File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw (-------src-dir-------/h5py/_proxy.c:1774)
   File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread (-------src-dir-------/h5py/_proxy.c:1416)
 IOError: Can't read data (Can't open directory)

validate: comment unclear for unknown NX_class

given this structure

facility:Facility
    @NX_class = Facility

this report from validation is not clear
(should be an ERROR: @NX_class = "Facility" is unknown
also report that it is unknown as a child group of ...)

/facility                                                  NXDL known                            OK      expected NX_class unknown: Facility            
<HDF5 group "/facility" (4 members)>                       NXDL NX_class                         ERROR   unknown: Facility                              
/facility/facility_beamline                                unknown NX_class                      ERROR   found: Facility                                
<HDF5 dataset "facility_beamline": shape (1,), type "|S6"> NXDL NX_class                         ERROR   unknown: Facility                              
/facility/facility_name                                    unknown NX_class                      ERROR   found: Facility                                
<HDF5 dataset "facility_name": shape (1,), type "|S3">     NXDL NX_class                         ERROR   unknown: Facility                              
/facility/facility_sector                                  unknown NX_class                      ERROR   found: Facility                                
<HDF5 dataset "facility_sector": shape (1,), type "|S7">   NXDL NX_class                         ERROR   unknown: Facility                              
/facility/facility_station                                 unknown NX_class                      ERROR   found: Facility                                
<HDF5 dataset "facility_station": shape (1,), type "|S1">  NXDL NX_class                         ERROR   unknown: Facility                              

NXdata is now optional

see: nexusformat/NIAC#16

Adjust validation accordingly

validate: @signal as integer type: assign with either NOTE or WARN

/entry1/data/data@signal    NXDL NX_type    ERROR    <type 'numpy.int32'> : NX_CHAR                 

KeyError exception when validating NeXus file with incorrect @default attribute

This file set is from the NeXus manual.

mintadmin@mintadmin-VirtualBox ~ $ punx va /home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/external_master.hdf5
Traceback (most recent call last):
  File "/home/mintadmin/Apps/anaconda/bin/punx", line 11, in <module>
    sys.exit(main())
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 326, in main
    args.func(args)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 157, in func_validate
    validator.validate()
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/validate.py", line 572, in validate
    self.validate_default_plot()
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/validate.py", line 470, in validate_default_plot
    self.new_finding(title, self.h5[nxentry][nxdata].name, finding.OK, m)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2579)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2538)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/h5py/_hl/group.py", line 164, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2579)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2538)
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open (/home/ilan/minonda/conda-bld/work/h5py/h5o.c:3551)
KeyError: "Unable to open object (Object 'entry' doesn't exist)"

determine Python types for use in validate.NXDL_DATA_TYPES

through some experience, it will be necessary to expand this dictionary.

NX_BINARY and NX_BOOLEAN are two keys that need defined values

validate: implement command line options to select reported *status* levels

allow user to change which status levels are reported, this is the default set:

:validation results shown:  COMMENT, ERROR, NOTE, OK, TODO, UNUSED, WARN
================================================== ===================================== ======= ===============================================
address                                            validation                            status  comment(s)                                     
================================================== ===================================== ======= ===============================================
/                                                  validate all names                    COMMENT ----------                                     
/@HDF5_Version                                     validItemName                         NOTE    relaxed re: [A-Za-z_][\w_]*                    
/@NeXus_version                                    validItemName                         NOTE    relaxed re: [A-Za-z_][\w_]*                    

implement method h5structure.isNeXusFile_ByAxisAttr()

listed as a TODO item

is `filename` a NeXus HDF5 file?

This is the oldest method in NeXus to define the default plot.

NOTE: **Not implemented yet!**

may benefit from code in validate.py

cannot validate HDF5 file with missing external file link

The missing link should be noted as an ERROR or WARNING and the remainder of validation should proceed.

IOError when rendering structure of non-NeXus HDF5 file

mintadmin@mintadmin-VirtualBox ~ $ punx st /home/mintadmin/Apps/anaconda/pkgs/pytables-3.2.2-np110py27_0/lib/python2.7/site-packages/tables/tests/Table2_1_lzo_nrv2e_shuffle.h5
Traceback (most recent call last):
  File "/home/mintadmin/Apps/anaconda/bin/punx", line 11, in <module>
    sys.exit(main())
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 326, in main
    args.func(args)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/main.py", line 129, in func_structure
    report = mc.report(show_attributes)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 67, in report
    structure = self._renderGroup(f, txt, indentation = "")
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 106, in _renderGroup
    s += self._renderDataset(value, itemname, indentation+"  ")
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 167, in _renderDataset
    value = self._renderArray(dset, indentation + '  ')
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 214, in _renderArray
    r = self._renderNdArray(obj, indentation + '  ')
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 243, in _renderNdArray
    r.append( __render(obj, rank, i, indentation + '  ') )
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/punx/h5structure.py", line 229, in __render
    item = obj[key]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2579)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2538)
  File "/home/mintadmin/Apps/anaconda/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 451, in __getitem__
    self.id.read(mspace, fspace, arr, mtype)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2579)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2538)
  File "h5py/h5d.pyx", line 177, in h5py.h5d.DatasetID.read (/home/ilan/minonda/conda-bld/work/h5py/h5d.c:3128)
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw (/home/ilan/minonda/conda-bld/work/h5py/_proxy.c:1774)
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread (/home/ilan/minonda/conda-bld/work/h5py/_proxy.c:1416)
IOError: Can't read data (Can't open directory)
mintadmin@mintadmin-VirtualBox ~ $ 

h5tree: report external group links in the right place

in h5structure.h5structure._renderGroup()
The problem is the link file and path need to be fed into the next call to _renderGroup(). No such design exists now for that.