hydroffice/hyo2_openbst

Open BackScatter Toolchain (OpenBST)

The Open BackScatter Toolchain Project

• Code Repository: github
• Project Page: url
• License: LGPLv3

General info

The Open BackScatter Toolchain (OpenBST) package provides a library and an app for processing acoustic backscatter.

Copyright Notice

Copyright (c) 2019, University of New Hampshire, Center for Coastal and Ocean Mapping; IFREMER. All rights reserved. Portions of this project were developed under a cooperative agreement with NOAA Coast Survey Development Laboratory, and contain NOAA-developed code in the public domain.

hyo2_openbst's Issues

Reduce Code Duplication

Issue: Codeacy indicates significant code duplication. There are large sections of the code base that have similar functionality that could be compressed. An example is below.

Example: When adding a raw file, three different 'add' methods are called, all with almost the exact same code.
project_info.add_raw:

raws.add_raw:

process.add_process:

Objective: Refactor code to reduce complexity and duplication

Codacy Invalid Syntax Issue

For each run of Codacy, I get an invalid syntax issue (see below)

This is not invalid syntax.

The impression from this link indicates that codacy uses either pylint or pyflakes, or other to run the static analysis. Not sure how that is set up.
@giumas it would be nice if I could be added to the codacy organization.

Either way, the closest matching issue I found was a Flake8 issue seen [here]
(https://gitlab.com/pycqa/flake8/issues/284) and I think the issue on our end is similar with codacy not recognizing typing maybe. Though this would be inconsistent as typing is used throughout the code base

This is a weird bug and I would like to not be failing codacy every time so it would be nice to get this fixed. Otherwise I will continue to ignore it.

Reson T50-P Notebook

Issue: OpenBST is designed to work from individual notebooks to present backscatter workflows and results

Aim: Complete first Notebook for public use.

Tasks

• Setup principal groups and headings:
  Title
  Parameter Modification
  Process Order
  Run Process
  Results Visualization
• Fill sub headings in 'Parameter Modification Headings based on processing method:
  Raw Decode, Static gain, TVG, Area correction etc.
• Set subheadings to display method types and parameter options.
• Setup Calculate widget:
  Options: run notebook according to order imposed by Notebook Structure, or pass list with method enum implying order.
• Setup results visualization section Related #10

WaterColumnCompressed implementation?

Are you working on implementing the 7042 record, waterColumnCompressed? Or are there any open source alternatives that handles this data?

Add processing methods

Issue: There are still backscatter processing methods/stages that need to be added to the code base.

Object: Add all processing methods on the task list below

Tasks:

• TVG
• Interpolation
• Gridding
• Area Correction
• Radiation Pattern Compensation
• Geo-Referencing
• Bathy Prep
• Transmission Loss
• Average
• Remove Angle dependence

DAG Visualization

Issue: Backscatter processing follows a directed acyclic graph pattern. Visualizing the different processing trees is useful to the user.

Task: Insert a graphical figure into the notebooks that tracks the processing chains within the project

Tasks:

• Create an empty DAG in the notebooks:
  Package is NetworkX
• Link processes in netCDF file to DAG for visualization.
• Add meta data on process (method type, process parameters, etc)

Start documentation

Issue: There is no documentation on how to use the project.

Objective: Begin working on the user-manual.

Tasks:

• Write basic how - to on how to use the project
• Write a tutorial notebook
• Add formulaic representation of processing algorithms for reference (user-man/glossary?)

Wrongly named examples are recognized as tests

The py.test library used in CI (AppVeyor and Travis CI) runs as test any file that starts with "test*".

The solution is to rename the examples under: examples/workground/reson_smith/test*.py

Process manager re-calculates prior steps and same parameter processes

Issue: Process manager does not catch a process already run in the chain. Also processes with same hashes not caught

Objective: N/A

Tasks: N/A

Using type() instead of isinstance() for a typecheck.

Codacy detected an issue:

Message: Using type() instead of isinstance() for a typecheck.

Currently on:

• Commit: 76ddfec
• File: hyo2/openbst/lib/processing/process.py
• LineNum: 115

Pythonize Code

Issue: @zxcvbnmike15 is not quite the Python Wizard he wishes he was. The code base is somewhat sloppy and not Pythonic

Objective :Refactor code base to make it neater, more explicit, etc.. NO MAJOR REFACTORS, major changes to code structure should be discussed first

Tasks: N/A

Covert example test files to unit test

Issue: A number of example scripts were written to test modules for the appropriate functionality. These need to be converted to unit tests within the project.

Objective: Convert example testing scripts to unit-tests.

Tasks:

• Write unit-tests based off of example scripts (see note for location of scripts)
• Expand unit-test files to cover entire test module
• Delete example scripts

Design widget frame

Issue: Results of processing steps must be available for viewing inthe Jupyter Notebook.

Task: Use ipywidgets to create a custom visualization widget.

Tasks:

• Design Hbox frame to contain widgets
• Tabs for the following plot vizualizations - Ping x beam, Geo-referenced, statistics
• Drop down menus for each tab, contents are linked to file list

Unused variable 'process_written'

Codacy detected an issue:

Message: Unused variable 'process_written'

Currently on:

• Commit: 76ddfec
• File: hyo2/openbst/lib/processing/process.py
• LineNum: 120

Plot a 2d array (pings x beams) with depth values

Run the prr's Pydro example in Python 3

Add new unit-tests for modules

Issue: Portions of the code base are not covered by either unit-tests nor example test scripts [#15 ]. To shift back towards a test driven development cycle, the coverage of the current code base must be increased

Objective: Write new unit-tests for modules not currently covered within the test directory or issue 15.

Tasks:

• Identify portions of the code base not covered
• Write unit-tests for uncovered modules

process_manager refactor

Issue: There is missing functionality from the process_manager. Some of this has been addressed individually in various processing modules; however, this is sloppy and buggy. Additionally, the process_manager currently lives in the process modules. Is this the best place for it? Refactors related to issues #16 and #45 could result in a better location for this module within the code base.

Finally, it could be more advantageous to use networkX than the current module.

Objective: Discuss if it makes more sense to use networkX packages to manage the in-project graph and how much work would be required to make that transition and interface with the netCDF files. If not using networkX, refactor the process_manager to reduce code complexity.

Tasks:

• Discuss transition to networkX
• Add missing functionality to process_manager
• Refactor

Add ability to apply a calibration file to the backscatter values

project_info utilization

Issue: The project_info class was originally written to manage the status of input/output files, when the project was modified and other general information about the project. In it's current form it is either poorly utilized, or containing redundant information.

Objective: Open discussion on future utilization of the class. Should its usage be expanded further? Could it's functionality be better refactored into other modules.

Notes: Currently the class manages the info.nc file. Information stored within the file:

1. added/removed raws
2. added/removed process
3. added/removed products (of which, the current project does not generate)
4. Storage of last process run

Plot the navigation of a single surveyline

Update Process Manager - Reset functionality

Issue: The current process manager stores parent process information, even after closing and re-opening. The notebooks are to be setup currently in a 'run' format.

Objective: Add a reset process manager method. This will set the parent process tracking to null to essentially rerun the processing chain for each calculate run in the notebook.

Tasks:

• Write method to reset process chain
• Check reset method does not break the standard process manager workflow
• Write a children parameter into the nc storage.
• Write a check children method

Plot an angular response curve for a single ping

Bug - Windows does not release control of netcdf files on rebuild of project

When re-creating a project (using force_new flag is true)

Windows does not tend to release control of one of the project .nc files (typically project_info.nc).

This causes a permission error when attempting to delete the file during project creation.