A Proposal to
The Open Navigation Surface Working Group
for updating the
Bathymetric Attributed Grid
specification with tables and layers for metadata (updated proposal)
2019 March 13
Introduction & Background
The Bathymetric Attributed Grid (BAG) was created to service the needs of the hydrographic community and target the concept of a Navigation Surface on a survey by survey basis. The design incorporates depth, uncertainty, and some metadata for the associated survey, but the ability to provide metadata with an associated geospatial context is limited. Some metadata, such as those attributes associated with S-101 (e.g., Quality of Bathymetric Data), may be better described with some geospatial context to enable different quality designations within an area. Importantly, these data may describe survey information that are not covered directly by the bathymetric layer. For example, it would be valuable to carry survey wide quality information, even when some of the survey coverage was contributed by side scan. Here we propose a key-raster and value-table pair that would enable the description of metadata on a node by node basis (Figure 1).
Figure 1 - georeference metadata as key-raster value-table pairs.
The previous pre-proposal also discussed a stand alone table for vector georeferenced data. This approach will not be discussed further in this proposal but may be redressed in the future.
Generically, a key-raster and value-table pair could serve the needs of many different types of metadata. The driving desire here is to enable the transport of a S-101 Quality of Bathymetric Data-like meta attribute within the BAG. In recognition of these two distinct parts of this proposal the generic data type to satisfy storing this data within the BAG is in Part A, while a specific implementation of this data type will be discussed for the Quality of Bathymetric Data in Part B. As a matter of convenience we have added a Part C, not previously discussed, such that georeferenced data licensing could be considered as part of this proposal and as a distinct implementation of Part A.
Part A - Georeferenced Metadata
Describing metadata on a node by node basis is important when full spatial resolution of the metadata is desirable and there is only one result for each node. Assuming many of the metadata values will be repeated as a set, a raster containing a key at each node can reference a table row of metadata values.
Requirements
-
Spatially keyed metadata included in the BAG.
-
Minimally impact the size of a BAG.
-
Leave the current XML metadata unchanged.
Project Description
As a generic metadata container the key-raster value-table pair may be reused to describe various metadata types. To enable reuse of this type of container without polluting the root node we propose storing all georeferenced metadata in a dedicated node under the BAG_root node named "Georef_metadata". Each implemented instance of the key-raster value-table pair would then be in an additional sub-node with the name of the represented layer as the name of the subnode. The subnode metadata header would contain a georeferenced metadata type, such as the types described in part B or part C of this proposal. These specific types should not need to be interpreted directly by a library using the BAG, but instead are meant to provide context for the specific meaning of the values stored in the value table for the downstream user. See part B of this proposal for an example.
Within each subnode the single resolution key-raster and value-table pair would be named as "keys" and “values” respectively. In the case that variable resolution metadata keys are available it would be stored as “varres_keys”, reference the same “value” metadata table, and correspond to the indices described in varres_metadata in the same manner as varres_refinements.
An overview level layout of the proposed update is given in Figure 2.
Figure 2 - The BAG structure with proposed structure additions. Each of the georeferenced metadata field have their own type (eg "Type A").
In Figure 2 there happen to be three georeferenced metadata layers. The metadata Alpha is of metadata type A, while the Cheese and Juice layers are of types B and C respectively.
If the assumption holds that there are a limited number of valid combinations of the metadata values in the raster, the number of entries in the table should be small and the raster should be highly compressible (raster in Figure 3, table in Table 1).
Figure 3 - Left, the elevation layer, right, the node metadata layer with table indice keys. The resolution of both rasters is the same despite their depiction here for clarity.
Table 1 - An example of a table corresponding to Figure 3.
The table information is matched to the relevant raster, and vice versa, through corresponding keys. This enables finding metadata information based on location using the georeferenced raster keys to look up metadata in the table, or finding the locations that have particular metadata by searching the raster for keys that correspond to particular metadata in the table.
Because the key-raster layer corresponds to the other raster layers found in the root node, the georeferencing information within the BAG should also govern this layer. The same number of rows and columns should exist in the key-raster as exist in the root level arrays for elevation and uncertainty.
Keys in the raster are only unsigned integers and should correspond to the row number in the metadata table. Because the HDF structure is self describing there is not a need to predefine the byte size of the key-raster data type. A single byte will likely satisfy the needs of many BAG files, but could be expanded if a large number of rows are required in the metadata table. The byte size required can be determined at the time of BAG formulation. Because the data type for the key-raster may differ from other raster arrays in the file, the no data value is zero ("0"), and the first row in the value-table shall be correspond to integer one (“1”).
Impact to the existing BAG library should be minimal as the reading and writing of the proposed additions should not change the existing code and could be handled entirely separately. The level of effort to implement a reader and writer for the proposed format has not been formally undertaken, however we expect this to be a modest effort.
Feedback on this proposal to consider:
Some preliminary feedback and suggestions on this proposal are included with additional thoughts.
-
There is concern that needing to write a raster or refinements layer when the values are the same for the surveyed area is a waste of space, even despite compression. If this is a cause for concern we propose that if there is only one row in the value-table metadata that the key raster or varres_key not be required with the assumption that the metadata applies to all locations where there is data in the elevation layer.
-
Making this layer type compatible with* CF conventions** (http://cfconventions.org/) would enable easy viewing without special parsers.* A thorough investigation into the CF conventions has not been undertaken. The value of enabling the CF convention needs to be further discussed and defined.
-
Making each metadata value (column in the value-table) its own layer would be another way to convey this information and also be potentially more flexible. While this is true, it could require many more raster layers which would contain largely the same values. While compressible, lots of rasters could also add significant clutter while increasing the ambiguity of the metadata available. Having specific metadata types means there can be an expectation of the information available, which is valuable when targeting specific needs, and reduce the number of datasets available in the file structure.
-
What would happen if one of the columns that we need to read are not there for some reason? This is the reason for the data type definitions in the following parts of this proposal. We think a check to ensure compliance with the data type definition can be put in place when the BAG is accessed for read. In this case the layer would be labeled as invalid and ignored if it doesn’t meet the define table parameters.
-
How would this type of data structure fit within the GDAL data structure? The raster attribute table (RAT) in GDAL is attached to a raster band and would be a natural structure to house proposed raster and table pair. The only difficulty anticipated by the GDAL team is the RAT is not compatible with all HDF data types, thus forcing some form of data type mapping if the GDAL library is not updated to be include these additional types in the RAT format. More information can be found here: https://www.gdal.org/classGDALRasterAttributeTable.html#details
Part B - Quality of Bathymetric Data Metalayer
As noted in the Cathedral and the Bazaar (E.S. Raymond 2001), "Every good work of Software starts by scratching a developer's personal itch." In this case the NOAA National Bathymetric Source (NBS) Project desires to use the BAG format to convey both bathymetry and a description of the quality of the bathymetry. While the current format contains a vertical uncertainty layer we wish to include additional information we often provide in the sub attributes of the S57 metaobject M_QUAL. Thinking forward, this should be formulated as S-101 Quality of Bathymetric Data meta type with the assumption that the information may be backported to S-57 where needed. We think adding this information aligns well with enabling the idea of a navigation surface. We also think this additional information will increase the value of the products we produce and enable seamless internal use of our data.
The specific use case envisioned for the NBS project adds value in two phases. First, processes downstream can utilize the bathymetry quality information directly in the BAG without having to carry around a supplemental file. The NBS supersession logic uses this quality information to sort which data should be used as part of the national bathymetry, so having quality directly integrated into the bag is valuable. Also, this is the same information that is currently provided to our Marine Chart Division as a supplemental file, so it would streamline that processes as well. Second, the NBS project currently creates a BAG which contains the amalgamation of many different sources with different qualities. Having these various qualities tracked within the BAG would simplify the products from the NBS and streamline our process.
It is worth noting that NOAA commissioned surveys rarely have more than one or two designations for the quality of data within the S-57 format. We expect the structure proposed in Part A to be able to hold many surveys with a roughly proportional number of entries in the value-table.
Requirements
-
Carry the Quality of Bathymetric Data attributes, or some reformulation thereof, forward such that the ideas of object detection capability, coverage, and uncertainty can be captured for use with the bathymetry.
-
Enable the representation of survey coverage as distinct information from bathymetry.
Project Description
As an implementation of the georeferenced metadata layer type discussed in Part A of this proposal, this section is meant to define a georeferenced metadata type corresponding to the S-101 Quality of Bathymetric Data. This definition is expected to be more of a registration of the data type as a library accessing this information should just pass it to a user for interpretation. A writer of this information may wish to implement some checks for consistency in the metadata type dependencies declared in S-101 and included the discussion here.
The data type declared in the node containing the key and value data should be labeled as "quality_of_bathy_data_1_0", where the “1_0” (one_zero) is meant to convey a version number for the data type. While changes in data type are not expected to impact reading the data, it may imply changes to how to use the data or the expected fields that may be present in the value table columns. The version “1_0” is meant to correspond to the S-101 standard version 1.0.
The proposed fields for the metadata table are depicted in Figure 4 and summarized in Table 2.
Figure 4 - The quality_of_bathy_data_1_0 type as an implementation of Figure 2.
Table 2 - A summary of the metadata fields corresponding to S-101 quality of bathymetric data to be contained in the "values" table. Values which directly reference the S-101 definition are noted. See later discussion on the other items.
| Column Name |
Column Type |
Note |
| temporal_variability |
Unsigned Integer |
Valid numbers are 1 to 6, corresponding to S-101 encoding |
| data_assessment |
Unsigned Integer |
Valid numbers are 1 to 3,
corresponding to S-101 encoding |
| feature_least_depth |
Boolean |
See S-101 least depth of detected feature measured. |
| significant_features |
Boolean |
See S-101 significant features detected. |
| feature_size |
Float |
See S-101 feature size. |
| feature_size_var |
Float |
See further discussion (2) |
| full_coverage |
Boolean |
See S-101 full seafloor coverage achieved |
| bathy_coverage |
Boolean |
See further discussion (3) |
| horizontal_uncert_fixed |
Float |
See S-101 horizontal position uncertainty fixed |
| horizontal_uncert_var |
Float |
See S-101 horizontal position uncertainty variable factor |
| survey_date_start |
String |
See S-101 Survey date start |
| survey_date_end |
String |
See S-101 Survey date end |
Most of the proposed fields for the value table correspond directly to the S-101 metaclass Quality of Bathymetric Data. There are, however, some exceptions.
-
Depth range maximum and minimum in S-101 are omitted. The assumption is that if this information is required than the corresponding nodes in the elevation layer can be queried for a minimum and maximum depth for each table row.
-
feature_size_var is meant to augment feature_size which corresponds to S-101 size of features detected. As noted in S-101, size of features detected is intended to be described as the smallest size in cubic metres the survey was capable of detecting. Depending on the type of survey this definition might force different depth ranges to have different values. For example, a survey vessel that works at a fixed height off the seafloor could maintain a fixed feature detection size capability over a wide range of depths. A surface vessel working over those same range of depths may have a feature detection capability that varies with depth causing the detection capability to be ambiguous and potentially misrepresented. For this reason feature_size_var is the percentage of depth that a feature of such size could be detected. When both feature_size and feature_size_var are present the greater of the two should be considered valid. The expectation is that feature_size_var will be set to zero if the feature size does not scale with depth. As with feature_size, feature_size_var should be ignored if significant_features is False.
-
NOAA surveys often use side scan and therefore have areas of coverage that do not contain direct depth measurements. In these cases the nodes with survey coverage but without bathymetry would be set to False. A condition with full_coverage = True and bathy_coverage = False is a useful indicator for how to work with these nodes within our workflow. If full_coverage is False, bathy_coverage must also be False.
-
Vertical uncertainty is excluded from this table as the vertical uncertainty is reported node by node within the BAG structure.
Feedback on this proposal to consider:
Some preliminary feedback and suggestions on this proposal are included with additional thoughts.
- Can the metadata column names follow the S-101 names exactly so as not to create a new set of names that need to be mapped? The answer is yes, but this would make the names very long. We are happy to support a clear convention suggested by the Open Navigation Surface Working Group.
Part C - Data Licensing Information
As NOAA Office of Coast Survey has begun to ingest more data not acquired under NOAA’s direction, proper tracking of data licensing is important for where data can be used internally or in public. Particularly in the case of the National Bathymetric Source Project, tracking the various contributing sources within a final product can be helpful. While BAG already maintains a use qualifier within the top level metadata, a georeference metadata layer would allow for proper tracking of the data license by node to ensure information regarding restrictions on redistribution of data is not lost.
The specific use case envisioned for the described licensing information is within the workflow of the National Bathymetric Source (NBS) Project. The NBS project intends to filter contributing data to a product to match the intended user and the licensing terms of the source data. For example, the NBS project may create bathymetry to support NOAA hydrodynamic models. In this case the bathymetry would only be used to support internal NOAA work and therefore enable the use of data that might not be licensed for general distribution.
Requirements
-
Carry georeferenced data license information for all contributing data.
-
The license must be available through a URL or DOI.
-
Carry information about the source institution and survey.
Project Description
As an implementation of the georeferenced metadata layer type discussed in Part A of this proposal, this section is meant to define a georeferenced metadata type corresponding to the license applied to source data. This definition is expected to be more of a registration of the data type as a library accessing this information should just pass it to a user for interpretation.
The license information referenced in second requirement above are to allow for the referencing of generic licenses without requiring the full text be carried within the BAG. This means the BAG is only able to reference generic licenses, and custom ones would need to be put online or obtain a DOI. While restrictive, this limits the size of data in the the BAG. Also, this encourages the use of generic (rather than custom) licenses. The following might be useful for some background on data licensing: https://data.research.cornell.edu/content/intellectual-property
The data type declared within the node metadata containing the corresponding key and value data should be labeled as "data_license_1_0", where the “1_0” (one_zero) is meant to convey a version number for the data type.
Four values are required for this data type as depicted in Figure 5 and described in Table 3.
Table 3 - A summary of the metadata fields to the contained in the values table for data_license_1_0.
| Column Name |
Column Type |
| Source Institution |
String |
| Source Survey ID |
String |
| License Name |
String |
| License URL or DOI |
String |
Figure 5 - The data_licence_1_0 type
Feedback on this proposal to consider:
Some preliminary feedback and suggestions on this proposal are included with additional thoughts.
- What if there is more than one contributing source for a node? In this case we suggest each contributing source be added within a single table entry, but separated by a semicolon within each value in the table.
Appreciation
We wish to thank Dr. Giuseppe Masetti at the University of New Hampshire Center for Coastal and Ocean Mapping for his substantial contributions to this proposal.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent