osoceanacoustics/echopype-checker

Compliance checker for the echopype adaptation of SONAR-netCDF v1

echopype-checker's Issues

Update the Platform and Sonar/Beam_group1 CDLs

The CDLs are lagging and don't include some of the last updates that went into echopype v0.8.0

Eliminate creation of netcdf files from CDLs

In cc._process_cdl(), the nc4.Dataset.fromcdl statement creates a netcdf file for the CDL file being read. But my reading of nc4.Dataset.fromcdl suggested that no netcdf would be created when the argument ncfilename is set to None.

https://github.com/OSOceanAcoustics/sonarnetcdf4-echopype-checker/blob/6ab5ea3e485d940a83a2e0bfd66b43de931f0aa1/sonarnetcdf4_echopype_checker/cc.py#L75

This is harmless but clutters the cdls folder with netcdf files. Look into it.

Resolve failure to recognize as identical `string` type and numpy `object` type containing only strings

cc._dtype_compare() is supposed to recognize as identical (when using dtype_strict=False) string type and numpy object type containing only strings, in this block:

https://github.com/OSOceanAcoustics/sonarnetcdf4-echopype-checker/blob/6ab5ea3e485d940a83a2e0bfd66b43de931f0aa1/sonarnetcdf4_echopype_checker/cc.py#L23-L32

I tested it when I developed the scheme, and it worked. But I'm seeing one case, Platform.sentence_type is string (<U3) type in the EchoData object being tested and object type in the CDL-derived source:

Look into this, to ensure they're treated as equivalent.

Create instrument-specific `Vendor` group CDLs for the compliance checker

Leverage this package to help monitor changes in the Vendor group as well. Unlike all other groups, SONAR-netCDF4 v1 convention does not specify anything for the content of the group. So, all the content is based on echopype decisions.

• @leewujung Create small netcdf files from the Vendor group from each major echopype instrument type: ek60, ek80 and azfp.
• @emiliom Use those netcdf files to create a CDL file per group; fine-tune and prune each CDL; then incorporate each CDL into the code base

Use case to ensure data conversion outcome

We are converting large volumes of echosounder data using echopype, and would like to have something in the workflow to ensure the integrity of data conversion outcome. This package would be a natural way to do it, if we can add an API to split out some outputs to indicate if the converted data is in good order (and if not what is missing).

In particular, @Sohambutala is configuring echoflow to handle automatic and parallel data conversion, and it'd be good to have a task/flow after the conversion step to ensure data integrity. Also adding @ctuguinay here since he brought this needs up yesterday.

Each test should optionally return the results in a dict, rather than just printing out the result

Currently the test only prints to screen. Returning a variable holding the information will provide more flexibility for acting on test results.