icgc-argo/song-qc-module

A QC module for Song that performs icgc-argo specific validations of song submissions

song-qc-module

A QC module for Song that performs icgc-argo specific validations of song submissions

Creating a verifier program

There is a sample verifier program (written in Python) named verifier.py -- just edit it to create whatever verifier program you like.

verifier.py must define a single function, called verify(). The verify() function must take a string containing JSON as input, and must return a list of issues (strings describing problems with the input). If there are no issues, verify() must return an empty list.

Testing:

Files which contain sample inputs for which the verifier program should fail (produce issues) should be placed into the directory tests/failing. Files which contain sample inputs for which it should produce no issues should be placed into the directory tests/passing.

To run the tests, run

python3 -m pytest -s -v

To run the REST Server

When the Dockerfile is run, it creates a REST server on port 8080 which calls verifier.py to verify JSON data which is POSTED to it, and which returns the issues detected in JSON format.

To run the dockerfile, run

docker build . -t sidecar
docker run -p 8080:8080 sidecar

To use with SONG

Add the URL of your REST Server to the SONG setting verifierURLs in application.properties or application.yaml configuration file.

song-qc-module's Issues

Create Jenkins pipeline

• refer to https://wiki.oicr.on.ca/pages/viewpage.action?spaceKey=icgcargotech&title=ARGO+QC+Module+for+Song#ARGOQCModuleforSong-CI/CDSpecs
• Jenkinsfile update and autodeployment
• connect properly with github

Create simple test infrastructure

The goal is to create an easy way for the qc module to be tested with minimal coding.

One way to do this, is create 2 directories:

• passing
• failing

the passing directory should contain passing payloads. When these payloads are run against the handler, assert there are no errors

the failing directory should contain payloads that are invalidated by the qc module. When these payloads are run against the handler, assert errors of the expected type are thrown.

tests should be written using pytest

Implement handler and wrapper

https://wiki.oicr.on.ca/display/icgcargotech/ARGO+QC+Module+for+Song

• The wrapper should take the incoming request, extract http method, headers and body, and just pass the body to the verification function.

• the verification function takes the body, does whatever validation, and returns a list of strings. Each string describes one issue that was found during validation. If no issues were found, the function returns the empty list.

• If the verification function doesn't follow its contract (eg. it returns a non-list, or a non-string in the list, or throws an exception, the wrapper should return an error describing the problem.

The verification function should be the only function in the file. When bioinfo updates the QC validation, they should only really be touching 1 file and not concern them self with the house keeping.

Implement docker-compose

docker-compose should spin up song, song-db, ego and the QC module. You can follow what was done with https://github.com/overture-stack/genomic-data-playground docker-compose

This ticket depends on the feature in song being implemented: overture-stack/SONG#601