nexusformat / cnxvalidate Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 5.0 1.49 MB

NeXus validation tool written in C (not via NAPI)

CMake 1.73% C 20.43% Makefile 3.41% C++ 70.61% Python 3.75% Tcl 0.07%

cnxvalidate's People

Contributors

Stargazers

Watchers

Forkers

yayahjb epanepucci hdrmx eothred bmaranville

cnxvalidate's Issues

test do not pass at build on debian11

$ make test
Running tests...
Test project /home/jerome/workspace/cnxvalidate/build
    Start 1: nxvtest
1/1 Test #1: nxvtest ..........................***Failed    0.22 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.22 sec

The following tests FAILED:
	  1 - nxvtest (Failed)
Errors while running CTest
make: *** [Makefile:149 : test] Erreur 8

make test fails

After compiling, nxvtest failed on the first test:

mintadmin@mintadmin-VirtualBox ~/.../cnxvalidate/build $ make test
Running tests...
Test project /home/mintadmin/Documents/eclipse/NeXus/cnxvalidate/build
    Start 1: nxvtest
1/1 Test #1: nxvtest ..........................***Failed    0.28 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.28 sec

The following tests FAILED:
      1 - nxvtest (Failed)
Errors while running CTest
make: *** [test] Error 8

Here is the output when compiling:

mintadmin@mintadmin-VirtualBox ~/.../NeXus/cnxvalidate $ cd build/
mintadmin@mintadmin-VirtualBox ~/.../cnxvalidate/build $ cmake ..
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- /home/mintadmin/Documents/eclipse/NeXus/cnxvalidate
-- /home/mintadmin/Documents/eclipse/NeXus/cnxvalidate/cmake_include
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.26")
-- Found HDF5: /usr/lib/x86_64-linux-gnu/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so
-- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.1")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/mintadmin/Documents/eclipse/NeXus/cnxvalidate/build
mintadmin@mintadmin-VirtualBox ~/.../cnxvalidate/build $ make
Scanning dependencies of target NXV_Static_Library
[  4%] Building C object src/CMakeFiles/NXV_Static_Library.dir/nxvmainloop.c.o
[  8%] Building C object src/CMakeFiles/NXV_Static_Library.dir/nxvcontext.c.o
[ 13%] Building C object src/CMakeFiles/NXV_Static_Library.dir/hash.c.o
[ 17%] Building C object src/CMakeFiles/NXV_Static_Library.dir/nxdlutil.c.o
[ 21%] Building C object src/CMakeFiles/NXV_Static_Library.dir/nxvgroup.c.o
[ 26%] Building C object src/CMakeFiles/NXV_Static_Library.dir/nxvfield.c.o
[ 30%] Building C object src/CMakeFiles/NXV_Static_Library.dir/nxvutil.c.o
Linking C static library libnxvalidate.a
[ 30%] Built target NXV_Static_Library
Scanning dependencies of target nxvalidate
[ 34%] Building C object CMakeFiles/nxvalidate.dir/nxvmain.c.o
Linking C executable nxvalidate
[ 34%] Built target nxvalidate
Scanning dependencies of target NXV_Shared_Library
[ 39%] Building C object src/CMakeFiles/NXV_Shared_Library.dir/nxvmainloop.c.o
[ 43%] Building C object src/CMakeFiles/NXV_Shared_Library.dir/nxvcontext.c.o
[ 47%] Building C object src/CMakeFiles/NXV_Shared_Library.dir/hash.c.o
[ 52%] Building C object src/CMakeFiles/NXV_Shared_Library.dir/nxdlutil.c.o
[ 56%] Building C object src/CMakeFiles/NXV_Shared_Library.dir/nxvgroup.c.o
[ 60%] Building C object src/CMakeFiles/NXV_Shared_Library.dir/nxvfield.c.o
[ 65%] Building C object src/CMakeFiles/NXV_Shared_Library.dir/nxvutil.c.o
Linking C shared library libnxvalidate.so
[ 65%] Built target NXV_Shared_Library
Scanning dependencies of target nxvtest
[ 69%] Building CXX object test/CMakeFiles/nxvtest.dir/testbasics.cpp.o
[ 73%] Building CXX object test/CMakeFiles/nxvtest.dir/testmain.cpp.o
[ 78%] Building CXX object test/CMakeFiles/nxvtest.dir/testutil.cpp.o
/home/mintadmin/Documents/eclipse/NeXus/cnxvalidate/test/testutil.cpp: In function \u2018void TestLogger(hash_table*, void*)\u2019:
/home/mintadmin/Documents/eclipse/NeXus/cnxvalidate/test/testutil.cpp:44:51: warning: deprecated conversion from string constant to \u2018char*\u2019 [-Wwrite-strings]
  dataPath = (char *)hash_lookup("dataPath",logData);
                                                   ^
[ 82%] Building CXX object test/CMakeFiles/nxvtest.dir/testgroup.cpp.o
[ 86%] Building CXX object test/CMakeFiles/nxvtest.dir/testlink.cpp.o
[ 91%] Building CXX object test/CMakeFiles/nxvtest.dir/testunidim.cpp.o
[ 95%] Building CXX object test/CMakeFiles/nxvtest.dir/testsimpleatt.cpp.o
[100%] Building CXX object test/CMakeFiles/nxvtest.dir/testdepends.cpp.o
Linking CXX executable nxvtest
[100%] Built target nxvtest

Wrong validation failure when rank is specified as a symbol

With example_nx_mx.h5 and the NXmx application definition the rank is defined as the symbol dataRank. This is allowed but validation fails because it is not properly handled.

Write tests for the integer attributes

Write tests for the primary, axis and signal attributes

Wrong type
OK

Prepare 1.0

Fix as many problems as possible with cmake
Reduce or eliminate warnings

Document the -p option

The -p option which allows to specify a path to validate in the file is not properly documented.

Write tests for links

Write tests for NeXus links:

Link OK
Link into nirvana
Link good in H5, but in wrong group
Target attribute broken

Test basics

Write tests for the basic functionality of nxvalidate:

finding application definitions in H5 files
Opening H5 files
Recursing through a file

Test for offset/stride dataset attributes

Write tests for the offset/stride pair of attributes:

either missing
wrong type
wrong dimensions

validate each NXDL file before using it

Qualify each NXDL file by validating it against the NXDL Schema (nxdl.xsd) before it is used. This will ensure the NXDL is proper (and has no errors introduced).

Here is some python code that does this:

def validate_xml(xml_file_name):
    '''
    validate an NXDL XML file against an XML Schema file
    :param str xml_file_name: name of XML file
    '''
    xml_tree = lxml.etree.parse(xml_file_name)
    xsd = cache.get_XML_Schema()
return xsd.assertValid(xml_tree)

Use the libxml2 library to perform this validation step.

Test group validation

Write tests for:

Group/ dataset found OK
group missing
group finding by NX_class
missing NX_class
optional group warning
optional dataset warning
additional group/dataset from base class warning
additional group/dataset not in base warning

Decide upon test framework and test it

Decide upon one of the my available C/C++ unit testing frameworks and test if it integrates well.

Automatic installation/local cache of definitions

When you install cnxvalidate a copy of the definitions should be installed locally. With an option to update it from github.

Pete has code to do this.

Write Dataset Test

Write tests for validating datasets:

type testing
dimension OK testing
dimension not matching NXDL
dimension not matching symbols

Extend cnxvalidate to validate any NeXus file

Currently cnxvalidate balks if there is no application definition. There is a desire to continue validating against the base classes then, with a hard coded empty application definition. Thus cnxvalidates scope is extended to a general NeXus file validation tool. There may be changes required to the recursion to support that.

NXcollection unvalidated

The NeXus manual states that NXcollection is 'An unvalidated set of terms...'.
Does this mean that base classes within NXcollection are not validated individually, or just that there is no validation of what NXcollection contains?

Validating attributes got stuck in an endless loop

cnxvalidate fails to build against HDF5 1.12

root attribute "HDF5_version" is not required

Using the cnxvalidate-web tool, it is shown that for the writer_1_3.hdf5 example file that attribute HDF5_version is missing and that this is an error. These lines in the source code are relevant:

cnxvalidate/src/nxvmainloop.c

Lines 76 to 82 in 4a21051

    
           attID = H5NXget_attribute_string(fileID,"/","HDF5_Version",data,sizeof(data)); 
        
           if(attID < 0){ 
        
           	NXVsetLog(self,"sev","error"); 
        
           	NXVsetLog(self,"message","Missing required global HDF5_Version attribute"); 
        
           	NXVlog(self); 
        
           	self->errCount++; 
        
                  }

This assignment is not correct. The HDF5_version attribute is described as optional in the NXroot base class. A search of the definitions repository shows this attribute in two NXDL files (NXroot and NXspecdata).

The presence of the HDF5_version attribute should only be checked based on the NXDL file. If it is marked optional, then the evaluation should be a warning (warnopt) and not an error.

Valgrind cnxvalidate

Run cnxvalidate with valgrind and make sure that it does not leak memory by the bucket

Errors with NX_BOOLEAN for h5py

I'm using h5py for storing a file in NeXus format and I'm having problems with values stored in NX_BOOLEAN type.
h5py stores boolean values as a H5T_ENUM according to their FAQ, which creates an error for me when validating a file:

definition=NXmpes.nxdl.xml message="Data type mismatch, expected NX_BOOLEAN, got H5T_ENUM {
      H5T_STD_I8LE;
      "FALSE"            0;
      "TRUE"             1;
   }" 
... nxdlPath=/NXentry/NXprocess/NXcalibration/applied 
... sev=error dataPath=/entry/process/energy_calibration/applied dataFile=MoTe.mpes.nxs

It would be really nice if nxvalidate would also accept this exact enum as a form of NX_BOOLEAN.

Write dataset attribute tests against NXDL

Write tests which check that NXDL specified attributes exist and are OK/Not OK

as field attributes
as children of field

Failed to load base class - errors

Hello,

I want to use cnxvalidate to validate nexus files. With my files and test files from https://github.com/nexusformat/exampledata I get the following error for the NXxas defintion:

I am not sure if there is an issue with the nexus files or with cnxvalidate.

Let me know what you need to reproduce this.

Best Regards
Michael Götte

`nxvalidate` does not run on MacOs with conda environment

In the cmake/make files, I think that I have been careful to overwrite all references to '/opt/local/' with references to the conda environment directory, but nevertheless I get the error message below, which indicates that nxvalidate is trying to link to system libraries, rather than the equivalent conda environment libraries.

Has anyone found a workaround?

NOTE: I have HDF5 1.12.1 installed in the conda environment, and believe that I have linked to these libraries and their corresponding headers during the build. I have a version installed at /opt/local/... from May 2017 that I suppose is 1.10.

$ nxvalidate -l nexus-definitions jupyter/SigScan19245.h5
Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
You can, at your own risk, disable this warning by setting the environment
variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.12.1, library is 1.10.1
	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
                   HDF5 Version: 1.10.1
                  Configured on: Mon May 22 14:39:30 UTC 2017
                  Configured by: [email protected]
                    Host system: x86_64-apple-darwin15.6.0
              Uname information: Darwin elcapitan.internal.macports.net 15.6.0 Darwin Kernel Version 15.6.0: Fri Feb 17 10:21:18 PST 2017; root:xnu-3248.60.11.4.1~1/RELEASE_X86_64 x86_64
                       Byte sex: little-endian
             Installation point: /opt/local

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables: 
                        LDFLAGS: -L/opt/local/lib -Wl,-headerpad_max_install_names 
                     H5_LDFLAGS: 
                     AM_LDFLAGS: 
                Extra libraries: -lz -ldl -lm 
                       Archiver: ar
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /usr/bin/clang
                       CPPFLAGS: -I/opt/local/include
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS: 
                        C Flags: -pipe -Os 
                     H5 C Flags:     
                     AM C Flags: 
               Shared C Library: yes
               Static C Library: yes


                        Fortran: no

                            C++: yes
                   C++ Compiler: /usr/bin/clang++
                      C++ Flags: -pipe -Os -stdlib=libc++ 
                   H5 C++ Flags:    
                   AM C++ Flags: 
             Shared C++ Library: yes
             Static C++ Library: yes

                           Java: no


Features:
---------
                  Parallel HDF5: no
             High-level library: yes
                   Threadsafety: no
            Default API mapping: v110
 With deprecated public symbols: yes
         I/O filters (external): deflate(zlib)
                            MPE: no
                     Direct VFD: no
                        dmalloc: no
 Packages w/ extra debug output: none
                    API tracing: no
           Using memory checker: no
Memory allocation sanity checks: no
            Metadata trace file: no
         Function stack tracing: no
      Strict file format checks: no
   Optimization instrumentation: no
Bye...
Abort trap: 6

EDIT: "cmake", not "cake"; "headers", not "haters" :)

file_name and file_time are required by nxvalidate but not by NXroot

nxvalidate requires file_name and file_time as attributes of NXroot:

https://github.com/nexusformat/cnxvalidate/blob/master/src/nxvmainloop.c#L53-L74

However, these attributes are marked as optional in NXroot:
http://download.nexusformat.org/doc/html/classes/base_classes/NXroot.html

Are they optional? If not, shouldn't the NXroot spec and the application definitions note this?

depends_on . throws an error

For NXmx, which has an NXdetector with a depends_on attribute, if that attribute is set to ., then nx validate throws this error:

Cannot even find the starting point of the depends_on chain

Instead it should accept . since in some cases the full depends_on chain is specified elsewhere.

Write tests for depends_on chain

Write tests for the depends_on chain:

An OK example
Missing attributes
Attributes with wrong data types or dimensions
Following depends_on in three different ways
Failing to find depends_on dataset
Non terminated chain

Failing to read variable length string data

example_nx_mx.nxs contains the definition as a variable length string. H5Tread_dataset_string() fails to
read this properly, the definition is wrong and all else goes pear shape after this. There may be other places in the code where variable length string data messes up things. There are 5 calls to this function sprinkled across the code base.

NXdetector_module/fast_axis_direction@depends_on is not validated

I added a printf to validateDependsOnField in nxvgroup.c and ran it on a NeXus file using NXdetector_module. I observed that the fast_axis_direction@depends_on attribute wasn't validated.

I don't anticipate this being fixed in time for the next milestone. I'll put more details and steps to reproduce here after I finish with nexusformat/example_data#7.

Getting "required group missing" for existing NXdata.

I have a NXdata entry in one of my NeXus files, which nxvalidate complains about:

definition=NXmpes.nxdl.xml message="Required group missing" 
... nxdlPath=/NXentry/NXdata 
... sev=error dataPath=/entry dataFile=MoTe.mpes.nxs

It is properly written to the file in the path /entry/data with the following attributes:

@NX_class: NXdata
@axes: energy
@signal: intensity
energy_indices: 0

there are also two arrays of the same length /entry/data/energy and /entry/data/intensity holding the data values. I'm also able to plot the data in h5web without any problems. So as far as I see it the NXdata requirements are fully met and I don't understand why nxvalidate is complaining about the missing group. Am I missing something?

failed to find application definition

Using the cnxvalidate-web tool, it is shown that for the writer_1_3.hdf5 example file that

Cannot validate Scan, failed to find application definition

which is reported as an error. I believe this message originates from this code:

cnxvalidate/src/nxvmainloop.c

Lines 248 to 256 in 4a21051

    
           if(self->subEntryFound == 0){ 
        
           	NXVsetLog(self,"sev","error"); 
        
           	NXVprintLog(self,"dataPath","/%s",name); 
        
           	NXVprintLog(self,"message", 
        
           		"Cannot validate %s, failed to find application definition", 
        
           		name); 
        
           	self->errCount++; 
        
           	NXVlog(self); 
        
           }

This is incorrect. An NXentry is not required to be governed by an application definition nor is it required to have a NXsubentry group.

mintadmin@mintadmin-VirtualBox ~/.../NeXus/cnxvalidate/build $ ./nxvalidate /home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/writer_2_1.hdf5
message="Missing required global file_name attribute" sev=error dataPath=/ dataFile=/home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/writer_2_1.hdf5 
message="Missing required global file_time attribute" sev=error dataPath=/ dataFile=/home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/writer_2_1.hdf5 
message="Wrong root group class ???�, expected NXentry" dataPath=/entry sev=error dataFile=/home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/writer_2_1.hdf5 
message="Cannot validate entry, failed to find application definition" dataPath=/entry sev=error dataFile=/home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/writer_2_1.hdf5 
3 errors and 0 warnings found when validating /home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/writer_2_1.hdf5

Not sure about the "Missing required global file_name/ file_time attribute" errors. Are there any required global attributes?
root group class is NXentry (see below)
test file uses no application definition, should be able to validate the file anyway

test data file structure:

/home/mintadmin/Documents/eclipse/NeXus/definitions/manual/source/examples/h5py/writer_2_1.hdf5 : NeXus data file
  entry:NXentry
    @NX_class = NXentry
    data:NXdata
      @NX_class = NXdata
      @signal = counts
      @axes = two_theta
      @two_theta_indices = [0]
      counts --> /entry/instrument/detector/counts
      two_theta --> /entry/instrument/detector/two_theta
    instrument:NXinstrument
      @NX_class = NXinstrument
      detector:NXdetector
        @NX_class = NXdetector
        counts:NX_INT32[31] = [1037, 1318, 1704, '...', 1321]
          @units = counts
          @target = /entry/instrument/detector/counts
        two_theta:NX_FLOAT64[31] = [17.926079999999999, 17.925909999999998, 17.925750000000001, '...', 17.92108]
          @units = degrees
          @target = /entry/instrument/detector/two_theta

	attID = H5NXget_attribute_string(fileID,"/","HDF5_Version",data,sizeof(data));
	if(attID < 0){
	NXVsetLog(self,"sev","error");
	NXVsetLog(self,"message","Missing required global HDF5_Version attribute");
	NXVlog(self);
	self->errCount++;
	}

	if(self->subEntryFound == 0){
	NXVsetLog(self,"sev","error");
	NXVprintLog(self,"dataPath","/%s",name);
	NXVprintLog(self,"message",
	"Cannot validate %s, failed to find application definition",
	name);
	self->errCount++;
	NXVlog(self);
	}