schwehr/generic-sensor-format

Sonar Generic Sensor Format (gsf) codec

Introduction

Library for reading and writing Generic Sensor Format (GSF) sonar files.

This sonar format and library was originally created by SAIC for the US Navy. Leidos was spun out of SAIC and they released version 03.06 under the LGPL 2.1 license.

https://www.leidos.com/maritime/gsf

Kurt Schwehr forked from this version to add unit testing and do a general code audit and cleanup with the work done being Copyright Google and under the same LGPL license in the GSF c code and Apache 2.0 for all new code.

License summary

Code in src/gsf/*.[ch] is LGPL 2.1 and mixed copyright Leidos/SAIC and Google.

All else is Apache 2.0 and copyright Google.

generic-sensor-format's Issues

gcc 5 used maybe uninitialized warnings

make clean; make OPTIMIZE=1 all
rm -f *.o
rm -f libgsf.a
gcc-5 -Wall -Wextra -Werror -O3 -funroll-loops -ffast-math -DNDEBUG   -c -o gsf.o gsf.c
gsf.c: In function ‘gsfOpen’:
gsf.c:463:9: error: ‘headerSize’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
     if (fseek(gsfFileTable[fileTableIndex].fp, headerSize, SEEK_SET))
         ^
gsf.c: In function ‘gsfOpenBuffered’:
gsf.c:800:13: error: ‘headerSize’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
         if (fseek(gsfFileTable[fileTableIndex].fp, headerSize, SEEK_SET))
             ^
cc1: all warnings being treated as errors
make: *** [gsf.o] Error 1

Why is the endian test not running on mac osx?

./tests/gsfxx/util_test 
Running main() from gtest_main.cc
[==========] Running 0 tests from 0 test cases.
[==========] 0 tests from 0 test cases ran. (0 ms total)
[  PASSED  ] 0 tests.

using sizeof(struct foo) is a bad idea for serialization

The character array needs to be 13 to allow for a trailing \0. However, that bumps the struct size up to 16 causing the gsf write code to write a 16 byte block instead of the original 12.

gsfHeader uses the size of the struct to determine the size of what to write.

Travis continuous integration

Setup Travis for gsf.

Simplify and max_size should be a gsfuLong or dataSize should be unsigned in gsfUnpackStream

    assert(max_size > 0);
    if ((buf) && (dataSize <= (gsfuLong)max_size))
    {
        memcpy(buf, dptr, dataSize);
    }
    else if ((buf) && (dataSize > (gsfuLong)max_size))
    {
        gsfError = GSF_INSUFFICIENT_SIZE;
        return (-1);
    }

Should be something more like

    if (buf) {
      if (dataSize > max_size)  {
        gsfError = GSF_INSUFFICIENT_SIZE;
        return (-1);
      } else {
        memcpy(buf, dptr, dataSize);
      }
    }     

Format all returns the same way

Almost all returns of values are in parents. e.g.

  return (-1);

Make sure that they are all the same style. These need changing:

grep return *.c | egrep -v '[(]| ?\* |return;'
gsf.c:        return -1;
gsf.c:    return rc;
gsf.c:    return gsfError;
gsf.c:    return checkSum;
gsf.c:            return data->mb_ping.sensor_data.gsfGeoSwathPlusSpecific.side;
gsf.c:            return data->mb_ping.sensor_data.gsfKlein5410BssSpecific.side;
gsf_dec.c:                        return -1;
gsf_enc.c:        return -1;
gsf_enc.c:            return -1;
gsf_enc.c:    return 0;
gsf_geo.c:    return gp_info[k].a0;
gsf_geo.c:    return gp_info[k].b0;
gsf_geo.c:    return &new_gp;
gsf_geo.c:    return &offsets;

egrep 'return \(' *.c | wc -l
     791

gsfHistory questions

• Why do host_name and operator_name have maximum lengths, while command_line and comment are char *?
• Why is does one have "NAME" and the other not with GSF_HOST_NAME_LENGTH and GSF_OPERATOR_LENGTH when the struct members are host_name and operator_name?

Both should have NAME or neither.

A clean output format for BigQuery

Can I write a clean schema and output writer for BigQuery? It would be good to have an S2 indexing column.

Use glog

https://github.com/google/glog

Compile without warnings

Using -Wall -Wextra, need to get the C code to compile warning free.

Python interface

Would be good to have pure python and/or C/C++ wrapped python support for GSF.

This would be especially good for helping to produce small test files cut down from larger original GSF files.

gsfNumberParams() can be written without a string copy

And the function does not do what its description says. It never looks for the equal sign.

Description : This function parses a KEYWORD=VALUE style parameter
 *   and returns the number of comma delimited values which follow the
 *   equal sign.

Compare the performance between fread and mmap

Back to the discussions of 2010 or 2011 that included Moe of IVS (now QPS). We thought back then that using mmap might be substantially faster when reading GSF from disk. I don't think that is worth doing in the old C gsf, but it would be worthwhile in a new C++ library.

S2 spatial indexing

It would be nice to be able to write a record that has spatial and record indexing to make search faster for reading.

Malicious input and fuzz testing

Make sure that handling nasty files does the correct thing.

Add old versions to the repo?

Is it possible to add old versions of gsf tacked onto the beginning of the git repo? e.g. from mbsystem, we have 1.07, 1.09, 1.10, 2.0, 2.0.5 (or 2.05?), 3.03, and 3.04. It would be really nice to have those in the history even if they aren't LGPL. As Shannon if it okay to put them in.

svn log | egrep '^r[0-9]|GSF' | grep -v MBF_ | egrep -B 1 '[1-3].+GSF|GSF.+[1-3]'
r1964 | caress | 2012-06-24 00:33:29 -0700 (Sun, 24 Jun 2012) | 10 lines
Updated GSF library to new release 3.04 from SAIC.
--
r1913 | caress | 2011-11-23 22:34:58 -0800 (Wed, 23 Nov 2011) | 1 line
GSF 3.03 update.
r1869 | caress | 2010-09-28 14:10:48 -0700 (Tue, 28 Sep 2010) | 1 line
Update to GSF version 3.02
--
r1596 | caress | 2007-07-03 20:10:09 -0700 (Tue, 03 Jul 2007) | 2 lines
Release 2.0.5 of GSF.
--
r1348 | caress | 2005-01-04 10:28:39 -0800 (Tue, 04 Jan 2005) | 2 lines
GSF Release 2.0.3
--
r984 | caress | 2002-04-04 15:41:22 -0800 (Thu, 04 Apr 2002) | 2 lines
GSF release 2.0
--
r766 | caress | 2000-10-25 14:41:35 -0700 (Wed, 25 Oct 2000) | 3 lines
Inserted GSF 1.10 distribution with minor modification.
--
r694 | caress | 1999-12-28 16:32:24 -0800 (Tue, 28 Dec 1999) | 2 lines
Updated to GSF version 1.09
r635 | caress | 1999-04-15 18:40:35 -0700 (Thu, 15 Apr 1999) | 2 lines
Updated GSF to version 1.07

Allow for building with hardened

https://wiki.debian.org/Hardening
http://security.stackexchange.com/questions/24444/what-is-the-most-hardened-set-of-options-for-gcc-compiling-c-c

CFLAGS += -Wformat -Wformat-security -Werror=format-security
CFLAGS += -D_FORTIFY_SOURCE=2
# CFLAGS += -fstack-protector-strong
CFLAGS += -fstack-protector-all

gsf_indx.c array access beyond bounds / buffer overflow

gcc-5 -Wall -Wextra -Werror -O3 -funroll-loops -ffast-math -DNDEBUG   -c -o gsf_indx.o gsf_indx.c
gsf_indx.c: In function ‘gsfOpenIndex’:
gsf_indx.c:2063:27: error: array subscript is above array bounds [-Werror=array-bounds]
             data.bytevalue[8-j] = data.bytevalue[j];
                           ^
gsf_indx.c:1232:31: error: ‘last_record_number’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
         data_id.record_number = last_record_number;
                               ^
gsf_indx.c:1047:22: note: ‘last_record_number’ was declared here
     int              last_record_number;
                      ^
gsf_indx.c:1229:8: error: ‘last_record_type’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
     if (last_record_type != GSF_RECORD_SWATH_BATHYMETRY_PING)
        ^
gsf_indx.c:1046:22: note: ‘last_record_type’ was declared here
     int              last_record_type;
                      ^
gsf_indx.c: In function ‘SwapLongLong’:
gsf_indx.c:2063:27: error: array subscript is above array bounds [-Werror=array-bounds]
             data.bytevalue[8-j] = data.bytevalue[j];
                           ^
cc1: all warnings being treated as errors
make: *** [gsf_indx.o] Error 1

Consider converting gsf to C++, but with C extern linkage.

This will make cleanup easier if I can use C++ internally.

Spelling in the docs

tr -s '[[:punct:][:space:]]' '\n' < GSF_spec_03-06.md | egrep -i '[a-z]' | egrep -v '[0-9]' | sort -u > words
cat words | aspell list > words2

Some key misspellings found:

ABSORP
ACHIVED
ATION
AVIGATION
Chebychhev
DECIMA
DECIMAT
DESIGNA
EXAMIN
GATION
ORRECTOR
OUTSTA
REJECTE
RRECTOR
Subecord
UNKN
UNKNWN
VECHICLE
degress
observerd
rheadin
subecords
supercedes

Convert #defines to enums

Make function args and variables const whenever possible

int OPTLK       gsfWrite(int handle, gsfDataID * id, gsfRecords * record);

to

int OPTLK       gsfWrite(int handle, const gsfDataID * id, const gsfRecords * record);

Remove the OS/2 Optlink

Migrate the UNUSED macro into gsf.h and make it GSF_UNUSED

First issue with asan

==9364== ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffccf47248c at pc 0x4b01ad bp 0x7ffccf471120 sp 0x7ffccf471118
READ of size 4 at 0x7ffccf47248c thread T0
    #0 0x4b01ac (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x4b01ac)
    #1 0x47e6e8 (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x47e6e8)
    #2 0x480339 (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x480339)
    #3 0x40632f (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x40632f)
    #4 0x47bee0 (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x47bee0)
    #5 0x463597 (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x463597)
    #6 0x463825 (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x463825)
    #7 0x463b90 (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x463b90)
    #8 0x46422a (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x46422a)
    #9 0x47c7c9 (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x47c7c9)
    #10 0x4648fb (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x4648fb)
    #11 0x47dcc9 (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x47dcc9)
    #12 0x2b6197ba6ec4 (/lib/x86_64-linux-gnu/libc-2.19.so+0x21ec4)
    #13 0x405568 (/usr/local/google/home/schwehr/src/generic-sensor-format/tests/read_gsf_2_9_test+0x405568)
Address 0x7ffccf47248c is located at offset 44 in frame <gsfDecodeSwathBathymetryPing> of T0's stack:
  This frame has 2 object(s):
    [32, 44) 'test_sizes'
    [96, 100) 'bytes_to_unpack'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
Shadow bytes around the buggy address:
  0x100019e86440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100019e86450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100019e86460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100019e86470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100019e86480: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
=>0x100019e86490: 00[04]f4 f4 f2 f2 f2 f2 04 f4 f4 f4 00 00 00 00
  0x100019e864a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100019e864b0: 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 f4
  0x100019e864c0: f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00
  0x100019e864d0: 00 00 f1 f1 f1 f1 00 00 f4 f4 f3 f3 f3 f3 00 00
  0x100019e864e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==9364== ABORTING

Are OD_PROFILE_REJECTED_MASK and OD_PROFILE_REJECTED_BY_HYDROG_MASK really supposed to have the same value?

Appendix C.3 Flag Mapping Between GSF and HDCS

Large file testing

How to manage testing of files > 4GB?

Perhaps xz compress a simple file that is mostly repetitive filler? Then the test unpacks the file and does a couple read checks?

test failure with comment.gsf index

https://travis-ci.org/schwehr/generic-sensor-format/builds/70490659

[ RUN      ] GsfIndex.Comment
index_test.cc:61: Failure
Value of: buf_new[i]
  Actual: '0' (48, 0x30)
Expected: buf_old[i]
Which is: '4' (52, 0x34)
[  FAILED  ] GsfIndex.Comment (1 ms)

This stems from #47 The original golden file index was written based on the 16 character header and is now broken with the 12 character index from change 0b0cd72 The old orig files need to be rewritten with the values for the correct new comment.gsf and history.gsf

cmake build setup

Would be good to have cmake in addition to autoconf build.

Use gflags

http://gflags.github.io/gflags/

Should "unknown" really be truncated?

egrep -n 'UNKN[^O]' *.[ch]
gsf.c:88:#define GSF_UNKNOWN_PARAM_TEXT "UNKNWN"   /* Flag value for unknown parameter value. */
gsf.c:7010:            sprintf(temp, "GEOID=UNKNWN");
gsf.h:1538:#define GSF_POS_TYPE_UNKN "UNKN"               /* Unknown positioning system type. */

Looks like "UNKNWN" is in use, but UNKN is not. Can it be removed or should it be changed to UNKNWN?

grep  -n GSF_UNKNOWN_PARAM_TEXT *.[ch] | wc -l 
     189
grep  -n GSF_POS_TYPE_UNKN *.[ch] | wc -l 
      1

Create a C++ GSF library

Start from scratch and create a pure C++ (gsfpp or gsfxx?) library that is less confusing than the original C interface. It would be nice to have a system that would make it easy to switch between stdio reading and mmap reading. If I start from scratch, I can make this library be under the Apache 2.0 license.

Test writing GSF

I have two initial read tests, but not started any round trip write and then read testing.

Fix travis build

$ make test
(cd src && make all)
make[1]: Entering directory `/home/travis/build/schwehr/generic-sensor-format/src'
gcc    -c -o gsf.o gsf.c
gsf.c: In function ‘gsfStat’:
gsf.c:258:24: error: storage size of ‘stbuf’ isn’t known
make[1]: *** [gsf.o] Error 1
make[1]: Leaving directory `/home/travis/build/schwehr/generic-sensor-format/src'
make: *** [all] Error 2
The command "make test" exited with 2.

Add basic gunit testing

I've started, but there is a lot left to cover and it still needs serious cleanup.

gsfEncodeHeader writes then copies when it could just directly write to the destination

Why not write GSF_VERSION directly into the on output memory?

Run code audit tools

I have yet to try using asan/msan/heaper checker/valgrind on the existing gsf code.

Also https://scan.coverity.com is free for open source projects

2038 seconds roll over issue

GSF uses seconds since the epoc unix time in a 4-byte integer in multiple places. No time like the present to fix this.

https://en.wikipedia.org/wiki/Year_2038_problem

GSF v1 testing

There is at least one comment about supporting GSF v1 being an issue. Need to test reading really old GSF files. Where can I get samples for testing?

gcc 4.8 does not have std::regex implemented

Breaks Travis-CI:

https://travis-ci.org/schwehr/generic-sensor-format/builds/70794915

gcc-4.9 is the first version with std::regex working:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53631

g++-4.8 -isystem ../../third_party/gtest/include -isystem ../../third_party/gmock/include -I../../src/gsfxx -I../../third_party/gtest -I../../third_party/gmock -std=c++11 -Wall -Wextra -pthread -g -O0 -c \
        ../../third_party/gtest/src/gtest_main.cc
ar rv libgtest_main.a gtest-all.o gtest_main.o
ar: creating libgtest_main.a
a - gtest-all.o
a - gtest_main.o
g++-4.8 -std=c++11 -Wall -Wextra -pthread -g -O0 -isystem ../../third_party/gtest/include -isystem ../../third_party/gmock/include -I../../src/gsfxx -I../../third_party/gtest -I../../third_party/gmock -L../../src/gsfxx -lgsfxx -L. -lgtest_main  records_test.cc libgtest_main.a   -o records_test
/tmp/ccJXsz9C.o: In function `gsfxx::test::(anonymous namespace)::GsfxxRecordHeader_Simple1_10_Test::TestBody()':
/home/travis/build/schwehr/generic-sensor-format/tests/gsfxx/records_test.cc:37: undefined reference to `gsfxx::Header::Decode(gsfxx::RecordBuffer const&)'
/tmp/ccJXsz9C.o: In function `gsfxx::test::(anonymous namespace)::GsfxxRecordHeader_Simple2_9_Test::TestBody()':

[SNIP]

/tmp/ccJXsz9C.o:/home/travis/build/schwehr/generic-sensor-format/tests/gsfxx/records_test.cc:85: more undefined references to `gsfxx::Header::Decode(gsfxx::RecordBuffer const&)' follow
collect2: error: ld returned 1 exit status
make[2]: *** [records_test] Error 1
make[2]: Leaving directory `/home/travis/build/schwehr/generic-sensor-format/tests/gsfxx'

gsfAttitude decode appears to have a bug with negative heading values

-2.3 heading became 653.06. The code for heading decode is slightly different than the over parameters.

Create an autoconf build setup

I noticed that S2 has pkg-config. It would be good to have pkg-config support.

Remove version comments

There are lots of comments that look like they were version control change comments. They make little sense out of context of a version control diff, so remove them. e.g.

 * jsb 01-16-02  Added support for Simrad EM120.

The comments are always still available in the original git commit: b7fed33

If those comments are important to have around, they should be put into a ChangeLog

Can we make desiredRecord be unsigned?

#define GSF_NEXT_RECORD 0

typedef struct t_gsfDataID
{
    int             checksumFlag;       /* boolean */
    int             reserved;           /* up to 9 bits */
    unsigned int    recordID;
   /* ... */
}

int gsfRead(int handle, int desiredRecord
static int gsfUnpackStream (int handle, int desiredRecord,

See this short term work around: 201041b

depth_calculation and vertical datum defaults changed in gsfInitializeMBParams()

There were very scary implicit conversions from DBL_MIN to zero happening before. I stuck with explicit values that are 0 for everything but depth_calculation and vertical datum. Will this break anything?

What version of C to allow for the library? C89, C95, C99, or C11?

https://en.wikipedia.org/wiki/ANSI_C

• https://en.wikipedia.org/wiki/C99
• https://en.wikipedia.org/wiki/C11_(C_standard_revision)

Or follow issue #28 and just go with C++11/14.

Code in the GSF_lib document

What is the point of the code in the GSF_lib document that is repeating what is in gsf.h?

In the image, the left is from the PDF and the right is viewing gsf.h.

Create markdown versions of the GSF docs

So that I can fix errors in the docs.

Starting with poppler's pdftotext.

How can max_size be less than 0 in gsfUnpackStream?

Basic command line tools

It would be great to have a gsfinfo tool like gdalinfo for GSF files. What else should be in a basic command line suite? Should these be in C, C++ or python?