aced-idp/aced-idp.github.io

Documentation Site for the ACED Data Commons

Documentation Site for the ACED Data Commons

This mkdocs-based codebase deploys documentation to aced-idp.github.io

Local Development

# setup local environment
python -m venv venv
source venv/bin/activate
# install dependencies
pip install -r requirements.txt

To start the documentation server run the mkdocs serve command:

➜ mkdocs serve

INFO     -  Building documentation...
INFO     -  Cleaning site directory
INFO     -  Documentation built in 0.25 seconds
INFO     -  [13:45:40] Watching paths for changes: 'docs', 'mkdocs.yml'
INFO     -  [13:45:40] Serving on http://127.0.0.1:8000/

Running on a port other than 8000 is possible with the --dev-addr <IP:PORT> flag (e.g. mkdocs serve --dev-addr 8181 will start the server on localhost:8181).

Publishing to aced-idp.github.io

The site is automatically built and published on every push to the main branch (using the Github Actions workflow file in publish.yml).

To skip this workflow add [skip ci] (or any equivalent variation) anywhere in the commit message.

To manually update the site run the mkdocs gh-deploy --force command:

➜ mkdocs gh-deploy --force

INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /Users/beckmanl/code/aced-idp.github.io/site
INFO     -  Documentation built in 0.49 seconds
INFO     -  Copying '/Users/beckmanl/code/aced-idp.github.io/site' to 'gh-pages' branch and pushing to GitHub.
INFO     -  Your documentation should shortly be available at: https://aced-idp.github.io/

Terminal Image Sources

• ACED Upload
• ACED Download

aced-idp.github.io's Issues

quick use cases

reformat docs so there is one section for each use case that includes all commands to accomplish goal. with minimal text around it. links to other pages where more detailed explanation of commands are

Test sign gen3-client, update doc

Given we have the new signed chicago images, should we test and update docs?
https://ohsucomputationalbio.slack.com/archives/C043HPV0VMY/p1707773318523459

aced-idp.github.io/docs/requirements/gen3-client.md

Lines 13 to 45 in 278c015

	## Downloading gen3-client
	A binary executable of the latest version of the gen3-client should be downloaded from the following Table or from the [Release page on Github](https://github.com/ACED-IDP/cdis-data-client/releases). Choose the file that matches your operating system (Windows, Linux, or macOS).
	No installation is necessary. Simply download the correct version for your operating system and unzip the archive. The program is then executed from the command-line by running the command gen3-client <options>. For more detailed instructions, see the section below for your operating system.
	| Operating System | Gen3 Client | Checksum |
	| --------------------- | ---------------------------------------- | -------------------------- |
	| macOS (Apple Silicon) | [gen3-client-macos.zip][macos-arm] | [checksums.txt][checksums] |
	| macOS (Intel) | [gen3-client-macos-intel.zip][macos] | [checksums.txt][checksums] |
	| Linux (amd64) | [gen3-client-linux-amd64.zip][linux] | [checksums.txt][checksums] |
	| Windows (amd64) | [gen3-client-windows-amd64.zip][windows] | [checksums.txt][checksums] |
	[macos-arm]: https://github.com/ACED-IDP/cdis-data-client/releases/latest/download/gen3-client-macos.zip
	[macos]: https://github.com/ACED-IDP/cdis-data-client/releases/latest/download/gen3-client-macos-intel.zip
	[linux]: https://github.com/ACED-IDP/cdis-data-client/releases/latest/download/gen3-client-linux-amd64.zip
	[windows]: https://github.com/ACED-IDP/cdis-data-client/releases/latest/download/gen3-client-windows-amd64.zip
	[checksums]: https://github.com/ACED-IDP/cdis-data-client/releases/latest/download/checksums.txt
	!!! warning
	Do not try to run the program by double-clicking on it. Instead, execute the program from within the shell / terminal / command prompt. The program does not provide a graphical user interface (GUI) at this time; so, commands are sent by typing them into the terminal.
	### Checksum Verification
	In order to verify that the downloaded file can be trusted checksums are provided in [`checksums.txt`][checksums]. See below for examples of how to use this file.
	<details>
	<summary>Successful Verification</summary>
	To verify the integrity of the binaries on macOS run the following command in the same directory as the downloaded file:
	```sh
	$ shasum -c checksums.txt --ignore-missing

Organize documents around workflows

Jordan Lee
10:00 AM
In terms of the flow of the docs (not the functionality): I like that there are many tools/commands that cover different use cases, perhaps framing these tools into a workflow. Currently it feels like a page to search for key words (ex. “pull metadata” “push metadata”) rather than a step-wise workflow. an example being the step to convert fhir to tabular data is then followed by convert tabular data to fhir, where a user won’t need to do both of these conversions back to back (taking a second pass through all steps in the doc could be beneficial). this comment is regarding this:
https://github.com/ACED-IDP/aced-idp.github.io/blob/main/docs/workflows/metadata.md

Update documentation for new command line structure

This issue documents command line changes: ACED-IDP/gen3_util#34

The changes are implemented in this branch: ACED-IDP/gen3_util#39

This site should be updated to reflect those changes.

Documentation Updates

Proposed Changes

Not clear xargs
when to use xargs for all files in a directory
as opposed to individual files
http://localhost:8181/use-cases/use-case-3/#add-files-to-the-manifest

"Add this to mainfest"
Use this one for all files in directory
Use this one for specific ones

Replace 'Study' -> 'Project'

Sections:

• Upload
• Download

"Edge Cases"

• Restrict access

• Edit Project (adding files to project)

• Add steps for adding a user to the Getting Started section

• Update Usage tables with examples, steps for using env vars

• Add "General/Basic" file upload section (model after Second Use Case?)

• Update screenshots with https://aced-idp.org

• Test Microsoft/Linux steps (check source commands compatibility)

• Write plan on merging/deprecating our gen3-client fork when multipart updates are pulled in upstream

• Add link checker to mkdocs to verify links work (https://github.com/byrnereese/linkchecker-mkdocs)

• Investigate double checksums in gen3-client's checksums.txt?

Thanks @jordan2lee for the pointers and recommendations on this!

Data Model Documentation

From today's discussion:

Notes to bootstrap the documentation

ACED data model

Introduction

TODO: this whole section
Notes:

• We define all our metadata as FHIR resources.
• Metadata composition happens on the client
• There is a workflow

FHIR graph model

Examine resource definitions here:

• Details on uploaded files are captured as DocumentReference
• DocumentReference.subject indicates who or what the document is about:
  • Can simply point to the ResearchStudy, to indicate the file is part of the study
  • Can point to Patient, or Specimen, to indicate the file is based on them
• An Observation can point to any entity
• A Task can provide provenance on how the file was created

Each resource has at least one study controlled official Identifier. Child resources have Reference fields to point to their parent.

Key resources

ResearchStudy

A scientific study of nature that sometimes includes processes involved in health and disease.
More

ResearchSubject

A ResearchSubject is a participant or object which is the recipient of investigative activities in a research study.
More

Patient

Demographics and other administrative information about an individual or animal receiving care or other health-related services.
More

Specimen

A sample to be used for analysis.
More

DocumentReference

A reference to a document of any kind for any purpose.
More

Identifiers

A string, typically numeric or alphanumeric, that is associated with a single object or entity within a given system. Typically, identifiers are used to connect content in resources to external content available in other frameworks or protocols.

More

References

Many of the defined elements in a resource are references to other resources. Using these references, the resources combine to build a web of information about healthcare.

More

CodeSystems (Ontologies)

The Fast Healthcare Interoperability Resources (FHIR) standard defines a set of terminology resources that provide a mechanism to access clinical terminology in a standardised way. Many clinical terminologies are currently supported, including SNOMED CT, RxNorm and LOINC. Recently, there has been increasing interest in supporting terminologies from other domains. In genomics, for example, there is increasing use of the Human Phenotype Ontology (HPO)1 to capture information about rare diseases.

PMID: 32308861

SNOMED:
While there are many possible uses of snomed in the entire FHIR model, our focus is on these fields:

• Specimen.collection.bodySite
• Condition.bodySite
• Substance.code TODO: pubchem?

MONDO:

• Condition.code

Other: TODO:

• TODO

Happy path

• TODO As an ACED user, in order to contribute data, I need docs, code and examples on how to populate spreadsheets, validate and submit data.

Workflow

• TODO Use synthetic data as an example

docs/workflows/metadata.md Needs a bit more detail

There should be 2-3 more examples for how a researcher can add metadata to a project.
https://github.com/ACED-IDP/aced-idp.github.io/blob/feature/git-lite/docs/workflows/metadata.md

g3t add myfile.cram  --patient P0 --specimen P0-BoneMarrow --task_id P0-Sequencing
g3t utilities meta create 

^^ This is good but it's unclear what "g3t utilities meta create" does. Assuming that it creates ResearchStudy/Documentreferences if no metadata specific flags are added to g3t add myfile.cram but this should crystal clear for the end user reading the docs.

Also if diagnostic reports or observations can be added in same --patient P0 like style above, then this should also be communicated. This is probably not the case, but It is unclear to the end user weather this meta create command can also create observations and diagnostic reports since there is no example code showing that is can be done.

The end user should know exactly how the gen3_util can assist them with metadata creation, ie: what the gen3_util can do for them and what they must do for gen3.

expanded command docs

generate pages for each g3t command with in depth descriptions. includes

• command
• description of command in text
• what each parameter is, which parameters are optional/required
• returns
• raises
• example code

see drop function as an example of structure

Task vs DiagnosticReport

Use case:

As an aced user, when I upload files, I may need to document how those files were created.

Current documentation:

A Task, or DiagnosticReport can provide provenance on how the file was created

See doc

Discussion

The Task entity seems appropriate:

A task resource describes an activity that can be performed and tracks the state of completion of that activity. It is a representation that an activity should be or has been initiated, and eventually, represents the successful or unsuccessful completion of that activity.

However, it's output field contains a loosely typed array of value[x]

Where value[x] is a term for

In FHIR XML and JSON, the value[x] element has an actual name of "value" and then the TitleCased name of one of these defined types, and its contents are those as defined for that type. see https://build.fhir.org/extensibility.html

The concern is that this loose coupling will require more support, more explanation and perhaps more bugs as analysts maintain meta data

For inputs, there are untyped references:

• focus
• for

The DiagnosticReport is described a little more specifically:

The findings and interpretation of diagnostic tests performed on patients, groups of patients, products, substances, devices, and locations, and/or specimens derived from these.

Semantically, it would be better if they had named it AssayReport aka:

The findings and interpretation of assays performed on ...

Compared to Task, the edges are more complete and typed accordingly for these use cases:

• single or multiple individuals See Patient, Group
• multiple specimens
• multiple documents
• multiple observations

Submission Service

use case

As a ACED developer, in order to submit and maintain meta data, it would be useful to have an openapi defined service to allow server and client side engineers to develop services.

summary

Implement a very narrow subset of the FHIR REST API. i.e. POST [base]/Bundle

dependences

• FHIR bundle. Creating/updating/deleting a set of resources on a server as a single operation
• REST endpoint. The update interaction creates a new current version for an existing resource or creates an initial version if no resource already exists for the given id. The update interaction is performed by an HTTP PUT command, in effect an upsert

implementation notes

• Rule: For collections of type transaction or batch, all entries must contain request elements, and resources if the method is POST, PUT or PATCH

• The only type supported is transaction

• The ACED data model is a subset of the FHIR model, the server may reject the transaction if an unsupported resource is included. e.g. A Claim resource is included.

issues

• Semantics of fullUrl: only ids of form urn:uuid:XXXXX will be supported
• Each entry request should include the url [type]?identifier=$system|$value e.g. Patient?identifier=http:/example.org/fhir/ids|456456
• All resources should include an identifier - that identifier MUST not include PHI

examples

THIS IS STILL A DRAFT

• example openapi

openapi: 3.0.1
info:
  title: ACED Submission
  contact: {}
  version: 0.0.1
servers:
- url: https://aced-idp.org/Bundle
  description: ACED FHIR Bundle Implementtion
  /Bundle:
   post:
      tags:
      - Bundle
      summary: "create-type: Create a new Bundle instance"
      requestBody:
        content:
          application/fhir+json:
            schema:
              $ref: '#/components/schemas/Bundle'.  # type=transaction
            example: |-
              {
                "resourceType": "Bundle"
              }
      responses:
        "200":
          description: Success
          content:
            application/fhir+json:
              schema:
                $ref: '#/components/schemas/Bundle'. # type=transaction-response
              example: null
components:
  schemas:
    Bundle:  # see https://hl7.org/fhir/R5/bundle.schema.json.html 

• example bundle

---
resourceType: Bundle
id: bundle-transaction
type: transaction
entry:
- fullUrl: urn:uuid:61ebe359-bfdc-4613-8bf2-c5e300945f0a
  resource:
    resourceType: Patient
    active: true
    name:
    - use: official
      family: Chalmers
      given:
      - Peter
      - James
    gender: male
    birthDate: '1974-12-25'
  request:
    method: POST
    url: Patient
- fullUrl: http://example.org/fhir/Patient/123
  resource:
    resourceType: Patient
    id: '123'
    active: true
    name:
    - use: official
      family: Chalmers
      given:
      - Peter
      - James
    gender: male
    birthDate: '1974-12-25'
  request:
    method: PUT
- fullUrl: urn:uuid:74891afc-ed52-42a2-bcd7-f13d9b60f096
  resource:
    resourceType: Patient
    identifier:
    - system: http:/example.org/fhir/ids
      value: '456456'
    active: true
    name:
    - use: official
      family: Chalmers
      given:
      - Peter
      - James
    gender: male
    birthDate: '1974-12-25'
  request:
    method: PUT
    url: Patient?identifier=http:/example.org/fhir/ids|456456
- fullUrl: http://example.org/fhir/Patient/123a
  resource:
    resourceType: Patient
    id: 123a
    text:
      status: generated
      div: <div xmlns="http://www.w3.org/1999/xhtml">Some narrative</div>
    active: true
    name:
    - use: official
      family: Chalmers
      given:
      - Peter
      - James
    gender: male
    birthDate: '1974-12-25'
  request:
    method: PUT
    url: Patient/123a
    ifMatch: W/"2"
- request:
    method: DELETE
    url: Patient/234
- request:
    method: DELETE
    url: Patient?identifier=123456

`pip install gen3_util` appears in markdown + (overall QA of documentation)

Noticed pip install gen3_util appears here and here

Newest code now deployed on pypi as gen3_tracker

Latest version is 0.0.2rc6 not 0.0.14

Are there other inconsistencies?