d3b-center / clinical-data-flow Goto Github PK
View Code? Open in Web Editor NEW📓Project management and design artifacts for Clinical Data Flow
Home Page: https://handbook.d3b.io/docs/products/clinical-data-flow
License: Apache License 2.0
📓Project management and design artifacts for Clinical Data Flow
Home Page: https://handbook.d3b.io/docs/products/clinical-data-flow
License: Apache License 2.0
Questions
Tasks
Create a 1-click script with a docker-compose stack to bootstrap the CDF pipeline for developers.
We need to make it easier for other developers to spin up our CDF pipeline so that they can collaborate with us, test our our pipeline for demo purposes or use it for their own development needs.
DataOps team wants to copy over studies from Kids First postgres db to D3b postgres db.
Meen created a CLI tool to do this, but still needs to provide support to DataOps users, fix any bugs, etc.
Meen help DataOps deploy this as an automated (nightly?) process.
We need a script that can read in a TSV or spreadsheet file with the necessary inputs to define an Extension resource and generate the the JSON file with the Extension payload.
This script should live in kf-model-fhir/scripts
We need to start securing the FHIR server in preparation for production deployment. We also need to do this because netlify deploy preview links don't work without it - bad for PR reviews and in progress demos.
Short term:
Longer term (later sprints):
Create a file called lessons_learned.md
in https://github.com/d3b-center/clinical-data-flow/tree/master/docs. Make a PR against the clinical-data-flow repo. We can move this somewhere else if we find a better place for it.
The document should include issues with or modifications needed on the FHIR Phenopackets model. One goal of the document is to communicate the FHIR Phenopackets issues in the GA4GH phenopackets slack channel and create Github issues on the Phenopackets on FHIR repository.
Wrap up feature development so that we can begin using dashboard internally
Use the search parameter generation script in kf-model-fhir/scripts
to generate search parameters for all Participant, Biospecimen, and Phenotype Extensions.
Include documentation on the Warehouse schema so that users know how data is organized per study
Everyone should step through the guide on your own time. The earlier we all learn FHIR basics the better prepared we will be to design tools to work with FHIR.
Issues to be implemented are:
kids-first/kf-ui-fhir-data-dashboard#4
kids-first/kf-ui-fhir-data-dashboard#5
kids-first/kf-ui-fhir-data-dashboard#6
kids-first/kf-ui-fhir-data-dashboard#7
Convert a Kids First study (PCGC) to PFB to get a better understanding of its pros and cons.
See repo for [PFB Exporter] (https://github.com/d3b-center/d3b-lib-pfb-exporter)
Start prototyping a web app that gives someone a good visual overview of the data they have in a FHIR server. We still need to flesh out the scope of this, but putting this story here for tracking purposes
Per kids-first/kf-lib-data-ingest#342
Accounting implementation is memory intensive and takes too long. CDF team should consider doing this in one of the upcoming sprints.
The purpose of this story is for the responsible developer(s) to get an understanding of easy it is to map data to FHIR and load data in a FHIR server. Whatever is produced from this story will inform the longer-term ETL design from the warehouse to FHIR servers.
This story is here to capture any work we're doing on the ingest library to support data wranglers and ingestion of studies. It will likely keep moving into future milestones.
Every time a commit is pushed to the kf-ingest-packages
repository, run stages et
with warehouse loading for that package. This will tightly couple the version of the code with the version of the produced data and encourage developers to keep code updated in the repo.
This trigger could also be used to generate other automated processes such as accounting, data QA tests, etc. Or these things could also be triggered off of updates to the data in the warehouse.
Quick walkthrough of the warehouse, schema, etc.
Stub out the Python dict representing JSON payload for the Participant FHIR resource
Create a user per study
User has createdb and usage of ExtractStage, TransformStage schema permissions
Anyone who is working on ingest for that study needs those creds
Could manage all this through a users.yaml file in this repo at warehouse/users.yaml
Create a branch and PR on kf-model-fhir
. The SD JSON file should go in kf-model-fhir/site_root/source/resources
.
Make sure that any Extensions for Participant are included in the SD.
Create a branch and PR on kf-model-fhir. The SD JSON file should go in kf-model-fhir/site_root/source/resources.
Make sure that any Extensions generated for Biospecimen are included in the SD.
Aidbox has best search capabilities so it would be great for a staging service provided:
If any of the above turns out to be false then we will use Smile CDR with validation turned off as the staging service
By the end of this sprint we should at least have Study, Participant, Biospecimen, Phenotype, Outcome, Diagnosis resources from the PCGC data loaded into our FHIR server.
Now:
Near Term:
Add a tab to the Kids First Data Service to FHIR Mapping spreadsheet for the new Extensions that need to be created.
The first pass will consist of Extensions needed for Participant
and Biospecimen
.
The Extension spreadsheet will be fed into the script here #49 to generate the Extension StructureDefinitions.
Branch off of the kf-model-fhir/add-phenopackets-model
branch and create Pull Request against that branch
All of your conformance resource should go in:
https://github.com/kids-first/kf-model-fhir/tree/add-phenopackets-model/site_root/source/resources
term
-> StructureDefinition-disease-term.jsonAdd rows to the KF FHIR Model Mappings spreadsheet for Study and Diagnosis
Phenopackets FHIR model is loaded into our FHIR server, but it doesn't include any SearchParameter resources at the moment. This means we won't be able to search for resources using any of the Phenopacket attributes (FHIR extensions e.g. karyotypic-sex).
Would be great if we can autogenerate the SearchParameters given the extension StructureDefinition files.
An on-going story that may move from milestone to milestone.
Autogenerate SearchParameters for a given set of extensions
A user cannot search for resources by added attributes (extensions) unless SearchParameter resources have been created and loaded into the server for those extensions.
For example the Phenopackets FHIR model has an extension called karyotypic-sex
. The karyotypic-sex
extension defines a new attribute on the Patient resource. Users cannot do searches like this: /Patient?karyotypic-sex=XY
right now because the FHIR server does not have a SearchParameter for it.
Add a new CLI command to the kf-model-fhir CLI to generate SearchParameters for a file or directory containing extensions.
Make PR to add SearchParameter files to kf-model-fhir
kf-model-fhir/add-phenopackets-model
and make your PR against that branchLoad SearchParameters into FHIR server
Load the generated SearchParameters into the FHIR server
See kids-first/kf-model-fhir#71 for motivation and details
Create a markdown document describing:
Create a branch and PR for this on kf-model-fhir
Run the #49 script to generate extensions for Participant and Biospecimen.
Create a branch and PR on kf-model-fhir
. The Extension JSON file should go in kf-model-fhir/site_root/source/resources
.
Stub out the Python dict representing JSON payload for the Biospecimen FHIR resource
Every team member should have a practical understanding of basic FHIR concepts:
How are entities and their attributes represented in FHIR
How to model things - add/remove entity attributes, change entity relationships, and change constraints on relationships and attributes
How to make entity attributes searchable
How to represent entity attributes in various ontologies
Remove all of Phenopackets FHIR model
Load all of the following:
Participant StructureDefinition
Participant Extension StructureDefinition(s)
Participant SearchParameters for its Extension(s)
Biospecimen StructureDefinition
Biospecimen Extension StructureDefinition(s)
Biospecimen SearchParameters for its Extension(s)
Phenotype StructureDefinition
Phenotype Extension StructureDefinition(s)
Phenotype SearchParameters for its Extension(s)
We may also have to load the following into the server as well:
The FHIR data dashboard takes a long time to load results because it requires many calls to the FHIR server to aggregate data.
First investigate use of server side caching
If that is not enough then also investigate the following options:
Create a spreadsheet that captures which Kids First entities we are modeling in the MVP, which FHIR resources they map to, the name and types of the Kids First attributes, and which FHIR
attributes or extensions the they map to.
Something like this:
Participant tables:
Kids First Entity | Kids First Attribute | Kids First Attr Type | FHIR Base Type | FHIR Attr | FHIR Type |
---|---|---|---|---|---|
Participant | is_proband | boolean | Patient | None | N/A |
Participant | gender | enum | Patient | gender | code |
Phenotype | hpo_id | string | Observation | code.codings.0.code | CodeableConcept.Coding.code |
Need extensions for these attributes because they are not part of :
We need to know if extensions exist for these and where they are or if we need to create new extensions.
Kids First Entity | Kids First Attribute | Kids First Attr Type | FHIR Base Type | FHIR Extension | FHIR Type |
---|---|---|---|---|---|
Participant | ethnicity | boolean | Patient | None | N/A |
Stub out the Python dict representing JSON payload for the KF Phenotype FHIR resource
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.