stumelius/cranio

Force measurements, data analysis and visualization in posterior calvarial vault osteodistraction (PCVO) using a torque sensor and Python

cranio's Issues

Logging configuration file

See http://docs.python-guide.org/en/latest/writing/logging/

Set up a basic CI system

For GitHub, Travis CI is the recommended

Set up a basic CI pipeline that:

• installs the application with pip install .

This is the "Hello World" verion of a CI system, but requires you to read up a bit on getting the basics running.

A minimal Travis setup looks like:

  language: python
  python:
    - 3.6
    - nightly
  script:
    - pip install -r requirements.txt
    - pip install .

SQLite foreign keys are not enforced

See https://stackoverflow.com/questions/2614984/sqlite-sqlalchemy-how-to-enforce-foreign-keys

(Data model) Relate log to session

Add session_id foreign key constraint to Log table to avoid orphaned logs.
Furthermore, having a foreign key constraint between logs and documents prevents logging prior to document initialization.

More documentation

Hi!
You recently contacted me on Reddit for part project management tutoring, so let's get the ball rolling (Filing issue #1)

Start by adding a README that documents:

• What this is
• Why this exists
• How to use it
• ( how to deploy it )
• How you want development to work

In the commit message, mention:
Issue #1 or Closes #1

(Event detection) Maximize button

Add maximize button to RegionPlotWindow

Explore possibility to integrate PR checks

CI (Travis) is already included. Explore possibility for code quality checks, such as coverage. See https://help.github.com/articles/about-status-checks/#checks

(Main window) Replace "Toggle Lock" with "Toggle Patient Lock"

Meta info locking in the main window is to be applied only to the patient field

x and y labels for plot axes

PlotWidget does not currently have axis labels

Main window with menu bar

Implement the main dialog as a QMainWindow with a menubar.

Visual event detection: Functionality for adding a specified number of events

Instead of having a single "+" / "Add" button for adding an event region to the region widget, there is a field where user inputs the number of events and clicks "Ok" / "Add".

• Add region count next to "Add" button
• Add "Remove all" button
• Change how regions are inserted: Instead of all regions spanning the first half of the region, insert them at uniform intervals

Session class

Methods:

• save(): saves the session on disk
• load(): load a session from disk
• as_document(): returns the session as a document that can be uploaded to a CouchDB database
• attachments(): returns the session attachments (i.e., data and log) as file-like objects

Attributes:

• id: uuid1 identifier
• patient_id: patient identifier (user input)
• datetime: timestamp of the start of data recording
• data: recorded data for the session as a pandas DataFrame
• log: software log for the session

Distraction event identifier to RegionEditWidget

User is able to view and edit event identifiers for each distraction event. Event identifier is autofilled (D001, D002, etc.).

Rename repository

Instead of a long "craniodistractor", use "cranio" or "CranIO". Compare the two:

from craniodistractor.core import Packet

vs

from cranio.core import Packet

The latter looks cleaner and is faster to write.

• Check that there are no existing projects in PyPi under the name "cranio"

• Rename github repo

• Update remote URL on local repositories

Relational data model

For simplicity sake, implement the data model as relational.

Invalid time stamps in dummy Sensor output

The figure illustrates a recording that lasted about 5 seconds. However, the x axis indicates that the recording lasted only for 0.2 seconds.

(Event detection) "Missed distraction" checkbox

Add missed distraction checkbox to RegionPlotWindow. Requires a "recorded" flag in the AnnotatedEvent data model.

Data model

Design a data model for storing recorded and meta data

Port from PyQt4 to PyQt5

Producer functionality

Implement a producer that is run in a separate process.

• producer can have multiple sensors

• sensor can have multiple channels (DAQ support)

• producer information (sensors and channels) is stored in a container

  • a consumer may utilize this information container

• implement Imada torque sensor

Producer can be started and stopped. Start reads data from all its sensors and puts it in a data pipe or queue until the producer is stopped.

(Data model) SensorInfo table

Implemented a sensor info table in the relational data model. Example:

CREATE TABLE SensorInfo{
    STRING serial_number,
    STRING hw_name,
    NUMERIC turns_in_full_turn,
    NUMERIC displacement_mm_per_full_turn
};

Default sensor: Imada torque sensor.

NOTE: Keep in mind when designing the model that there may be different types of sensors used by the application. In the typical use case there is only a torque sensor but in calibration there will also be a load cell.

Fix travis configuration

• Install PyQt5 instead of PyQt4
• Add daqstore to requirements (specify version?)

Update travis status URL in README

travis-ci.org\smomni\craniodistractor -> travis-ci.org\smomni\cranio

Use daqstore as the data store

Integrate daqstore to this project and add https://github.com/smomni/daqstore.git to dependencies

Distractor index as user input

The user inserts the distractor index into the measurement window. The distractor index is passed onto the event detection window when the measurement ends.

Distraction events are associated with the distractor index.

• Distractor index input field in measurement window
• Confirmation prompt: You entered <> as the distractor index. Are you sure you want to start the measurement?
• Distraction events are associated with the distractor index

Stub documentation

Write stub documentation using sphinx. A simple configuration example: https://medium.com/@richyap_13562/a-simple-tutorial-on-how-to-document-your-python-project-using-sphinx-and-rinohtype-177c22a15b5b

Related to #63

SessionMeta data collection

The following meta data is collected for each session (= every time software is started):

• Session id (autogenerated)
• Datetime (autogenerated)
• Patient ID (user input)

Patient ID prompt is shown on software startup. The prompt has two input fields: Patient and Session. Session input field is disabled and just shows the random generated id. The input fields have Help buttons for providing information to the user.

Log entries to database

Write a log handler that pushes log entries to a database

See https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/logging/sqlalchemy_logger.html

ETL script for old data

Extract: from old .txt files
Transform: (re)name columns
Load: to a simple database (sqlite)

• prototype notebook
• etl_old_data.py in scripts\

Annotated events -table

Instead of having an "event_id" column along the raw data, include the event information in a separate table. For example:

CREATE TABLE AnnotatedEvents(
    event_type VARCHAR(20),
    event_num INTEGER,
    document_id VARCHAR(20),
    event_begin NUMERIC,
    event_end NUMERIC,
    PRIMARY KEY (event_type, event_num, document_id),
    FOREIGN KEY (document_id) REFERENCES Document(document_id)
        ON UPDATE CASCADE
);

Patient creation widget

Implement a widget where the user can add patients to the database. The user must be able to see the existing patients.

Streamline measurement workflow

• Minimize the need for cursor movement (cursor is automatically moved to distractor_id input field so that user can just enter the number and hit enter to start the measurement)
• Add user prompt before starting the measurement (triggered by Enter / Start)
  • "You entered {patient_id} as patient and {distractor_id} as distractor. Are you sure you want to continue?"

Add unit testing to CI

Travis configuration does not yet include unit testing

• Add PyQt4 installation to travis configuration

• Add unit testing to travis configuration

Annotate training data for DED algorithm

Extract old data and annotate each distraction event. Write a script for graphical annotation by utilizing the region widgets implemented in cranio.

Graphical annotation

Data files are displayed in order. User adds regions to each file and clicks OK to accept the regions. The regions are updated to source data once the selected regions are accepted. User is able to navigate between files (Next / Previous).

Tasks

• Extract old data
• Write a graphical annotation script
• Annotate training data

Flat file database model

Design the structure for how documents are stored on disk as flat files.

Ideas

• commit(): commit all changes to disk
• load(): load from disk

Document workflow

I'd suggest a rather simple, but structured workflow:

• File issues for those features you're looking to get. They can be small or big, as long as they are solveable.
  you should be able to tell when something is done from reading the issue. Too open ended and it cannot be closed.

• Develop those issues

• Commits should touch one thing, preferrably, with a label that matches the code. IE, a change that reads "reformat foo" shouldn't add new features, etc.

• Open an Pull request /MR for your own review from the branch to master

• Try to keep the commits on an MR branch below a dozen.

• Keep the MR open for 24 hours, give people the chance to comment and look at it.

• Review that the changes in there look decent

It's a perfectly valid workflow to sit down and hack, realize you just built a feature, then file an issue for this feature. It's even commendable.

The goal is to get the habit in of referring to your issues, and opening code review points before merging directly to master.

"But I'm the only developer here?!" you may think, and yes. That's why it's good to practice now.

• Document your desired work flow, and open an MR for that documentation for a quick review.

Add version information

Add version attribute to cranio/init.py

Replace Float with Numeric

The database model contains Float (IEEE 754) columns which may lead to inaccuracies in decimal representation (conversion from base-10 to base-2). Replace the Float columns with Numeric.

Replace Enum with a more portable data type

From SQLAlchemy documentation:

"The Enum type will make use of the backend’s native “ENUM” type if one is available; otherwise, it uses a VARCHAR datatype and produces a CHECK constraint. Use of the backend-native enum type can be disabled using the Enum.native_enum flag, and the production of the CHECK constraint is configurable using the Enum.create_constraint flag."

Suggestion

Write a lookup table and populate it:

CREATE TABLE LogLevel(
    level VARCHAR(20) PRIMARY KEY,
    level_priority INTEGER NOT NULL,
);

INSERT INTO LogLevel VALUES ('NOTSET', 0), ('DEBUG', 10), ('INFO', 20), ('WARNING', 30), ('ERROR', 40), ('CRITICAL', 50);

CREATE TABLE Log(
    log_id INT PRIMARY KEY AUTO_INCREMENT,
    ...
    level VARCHAR(20),
    FOREIGN KEY (level) REFERENCES LogLevel(level)
        ON UPDATE CASCADE
);

The logging levels are the same as in the Python 3 logging module (see https://docs.python.org/3/library/logging.html).

VMultiPlotWidget show empty grid by default

VMultiPlotWidget shows nothing by default which displays as an empty widget before data is plotted. Instead of an empty widget, show a placeholder grid.

Document data model

Implement the document data model as specified in the attached image.

Distraction event detection (DED) algorithm

Design and implement a DED algorithm.

Design: find references from scientific literature

Move patient_id field from dim_session to dim_document

In case multiple patients are treated during a single session. Very unlikely but this is a very minor denormalization.

Update data model image

The data model image in docs/ is outdated

Type hinting

Add type hinting to the existing modules:

• core

• app.plot

• imada

• producer

For more information on type hinting, see PEP-484 and PEP-526.

Clean requirements

Remove old and/or redundant requirements from requirements.txt

(Main window) Replace session_id with distractor_id

Replace session_id with distractor_id in the main window. session_id is to be hidden from the user

Prototype for kick-off meeting

Improve encapsulation of PlotWidget

PlotWidget starts and stops producer processes which should not be included in PlotWidget's responsibilities. Either rename PlotWidget as ProducerPlotWidget or improve encapsulation in some other way.

Docstring style

• PEP-257 For consistency, always use """triple double quotes""" around docstrings. Use r"""raw triple double quotes""" if you use any backslashes in your docstrings.
• Sphinx markup for argument, return value and exception documentation
  • This style is also used by PyCharm comment autogeneration by default