cdcgov/prime-reportstream

ReportStream is a public intermediary tool for delivery of data between different parts of the healthcare ecosystem.

General disclaimer This repository was created for use by CDC programs to collaborate on public health related projects in support of the CDC mission. GitHub is not hosted by the CDC, but is a third party website used by CDC and its partners to share information and collaborate on software. CDC use of GitHub does not imply an endorsement of any one particular service, product, or enterprise.

Overview

The PRIME ReportStream project is the part of the Pandemic Ready Interoperable Modernization Effort that works with state and local public health departments. The project is a joint effort between the CDC and USDS. Currently, we are focusing on the problem of delivering COVID-19 test data to public health departments. Later, we will work on other tools to analyze and explore this data and different types of health data. See the PRIME ReportStream website (https://reportstream.cdc.gov) for further details.

PRIME ReportStream a sibling project to PRIME SimpleReport which is better way to report COVID-19 rapid tests.

PRIME ReportStream is a member of the Open CDC community.

Problem Scope

Public health departments (PHDs) rely on accurate, timely data to fulfill day to day-operations to make long-term strategic decisions. Between the time when they receive report files from reporting entities like laboratories and point of care centers, to taking action on this data, there are many barriers and challenges. ReportStream aims to provide the infrastructure and tools to address these challenges.

Our vision is to help public health systems make faster, more effective decisions. We want to both improve the speed of receiving data to action, as well as increase the quality and effectiveness of actions they can take.

Target users

• Senders of data
  • Healthcare institutions that generate data (e.g. reports) that needs to be reported to public health departments
  • Senders may include traditional (hospital, labs) and non-traditional (POC apps) organizations
• PHDs
  • Epidemiologists who rely on health data to take regular actions
  • Senior stakeholders who make executive decisions using aggregate health data
  • IT teams who have to support epidemiologists and external stakeholders integrating with the PHD
  • PHDs may include state, county, city, and tribal organizations

Public Domain Standard Notice

This repository constitutes a work of the United States Government and is not subject to domestic copyright protection under 17 USC § 105. This repository is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication. All contributions to this repository will be released under the CC0 dedication. By submitting a pull request you are agreeing to comply with this waiver of copyright interest.

License Standard Notice

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication. All contributions to this project will be released under the CC0 dedication. By submitting a pull request or issue, you are agreeing to comply with this waiver of copyright interest and acknowledge that you have no expectation of payment, unless pursuant to an existing contract or agreement.

Privacy Standard Notice

This repository contains only non-sensitive, publicly available data and information. All material and community participation is covered by the Disclaimer and Code of Conduct. For more information about CDC's privacy policy, please visit http://www.cdc.gov/other/privacy.html.

Contributing Standard Notice

Anyone is encouraged to contribute to the repository by forking and submitting a pull request. (If you are new to GitHub, you might start with a basic tutorial.) By contributing to this project, you grant a world-wide, royalty-free, perpetual, irrevocable, non-exclusive, transferable license to all users under the terms of the Apache Software License v2 or later.

All comments, messages, pull requests, and other submissions received through CDC including this GitHub page may be subject to applicable federal law, including but not limited to the Federal Records Act, and may be archived. Learn more at http://www.cdc.gov/other/privacy.html.

Records Management Standard Notice

This repository is not a source of government records, but is a copy to increase collaboration and collaborative potential. All government records will be published through the CDC web site.

Related documents

• Open Practices
• Rules of Behavior
• Thanks and Acknowledgements
• Disclaimer
• Contribution Notice
• Code of Conduct
• Development: Getting Started on the backend
• Development: Getting Started on the frontend

Additional Standard Notices

Please refer to CDC's Template Repository for more information about contributing to this repository, public domain notices and disclaimers, and code of conduct.

prime-reportstream's Issues

Short-term UI/UX fixes for AZ state POC Google Form

Background
After talking with Brenna Garrett at AZ state on 10/27, she's open to having folks take a look at their current Google Form for POC data entry and see where short-term improvements can be made. Ex: Make basic UI/UX updates to reduce repetitive tasks/make entry less time consuming. Could be a good opportunity to further build our working relationship and make some small improvements while the main app is in development.

To do

• Review existing form and draft recommendations
• Build example form and test with POC location
• Do it live

Prototype Router supports HL7

Story
As a PHD, I would like to receive messages in the HL7 format. As a prototype user, I would like to see this capability in action.

Assumptions

• HL7 2.5.1
• The prototype will not hit a real HL7 end-point

Tasks

• FHS, BHS
• MSH
• SFT
• ORC
• OBX
• OBX for AOE
• SPM
• NTE

Acceptance Criteria

• Each data element from the message is present in the HL7
• The HL7 message can verified by a tool like https://hl7inspector.com/

Master files for how to submit to Arizona, Florida, and Texas.

~~POC App needs to know what data is required to submit data to states/local counties so we can make sure to collect the required information in the app. The input app needs this information to lock down what data is collected.

Reworking this issue a bit ... collect flat files in one place for our partners.

Short-term UI/UX fixes for AZ state POC Google Form

Background
After talking with Brenna Garrett at AZ state on 10/27, she's open to having folks take a look at their current Google Form for POC data entry and see where short-term improvements can be made. Ex: Make basic UI/UX updates to reduce repetitive tasks/make entry less time consuming. Could be a good opportunity to further build our working relationship and make some small improvements while the main app is in development.

To do

• Review existing form and draft recommendations

If recommendations are ok:

• Build example form
• Test with POC location
• Handoff to AZ

AZ research week of 10/12-16

Pima county sessions & links to notes

• 10.14.20 Sasha Harb & Mary Derby - Analysis & reporting
• 10.14.20 Dr. Dr. Zach Gerberg & Matthew Christenberry - Contact tracing
• 10.16.20 Julie Kurdna & __ __ - Quality & cleaning

AZ - Data quality

10/22 2:30-3:30pm ET w/Amy Lai

Gather other pilot partner CSVs

Florida

https://cdc.sharepoint.com/:f:/t/USDSatCDC/Epok6OJFjiVOuonSB6jcrm0BY3rtMDqGqo3ltbxZlCGfNw?e=3gjRGi

Texas, Harris

• Mobile testing
• Nursing home
• ?

Texas, Houston

• Mobile testing
• Nursing home
• ?

The router supports sending HTTP

Create dummy test files for prototype CSV router

SendGrid: notifications of errors, receipts

When a flat file is received via the prototype router, some kind of receipt should be sent to confirm. Research options for sending a receipt to our initial POC locations in AZ. Connect w/Sarah Johnson.

Errors

• What kind of errors are common?
• What states do we include?

Receipts

• What does Pima require (if anything)?
• What type of receipt is acceptable by the POC location's standards? Ex: HCA will have lots of input here
• How does this integrate with Data Input?
  • If something like an email:
    • Who sends it? Hub or Input?
  • If something more tightly integrated:
    • What does a message look like on our end?
    • How will Input display it?
    • How will the POC save it?

AC

• How does Pima an AZ tell us something is wrong? What happens next?
• How do we handle this short term?
• How does this scale?

Prototype Router is in CDC GitHub

Story
As a developer, I would like to see the prototype in RickHawesUSDS in the data site

Assumptions

• The current prototype is a good place to start

Gather example CSV from Pima county, AZ & pilot POC location

Example CSV flat file:

• Pima county
• Initial POC/Nursing home location

---

For conversation with Eli in AZ, loop in Jim.

Documentation - Product requirements

FL research week of 10/12-16

Florida sessions & links to notes

• 10.15.20 Allison Culpepper & team - Ingesting & receiving, quality & cleaning

Documentation - Engineering/techincal

Acceptance

• Coding style
• Getting Started.md detailing how to setup a dev machine
• Contributors.md detailing code style and review expectations
• Architecture.ms detailing components and engineering choices
• Check in template

Whiteboarding - Houston TX org chart

Map out partner org to better understand team setup and find opportunities to collaborate.

Go live checklist for sending to AZ

• Establish relationship with AZ
• Get details to execute
  • Hosting #30
  • Authentication/Credentials (See SFTP #14 )
  • CLIA certification - not needed
  • Expectations on sending frequency
  • Timeline (2-4 weeks)
• Submit authentication form
• Get credentials
• Confirms we can connect to server, drop test file
• AZ reviews test files
• Generate multiple samples from PDI staging server, submitted to Data Hub, then exported to AZ's format
• Validate data is 100%
• Once several files are received without error through test system, then move to prod system

Todo other ...

• Get Hub engineers traveling next week up to speed with SimpleReport app/install locally. May need to provide on-site help.

Other questions
-We will start with "dual sending" on production. How to make sure the data sent is same? How do we transition off from dual sending?
-How to verify something is sent, was it received properly
-what is the difference test and prod, is it different credentials, folders, etc

End to end
-Be able to receive data from PDI app, agreement on schema [Jim, ask Tim] (#75)
-Write postman script to test end to end process [Chris] #94
-Wire the SFTP sending to latest build, support USDS test SFTP, AZ's test SFTP and AZ prod modes [Matt] #95
-Give Chris a SFTP docker container [Matt] #96

---

Via Elegante location details

• Starting w/ Tuscon Mountain location
• Expected testing volume: 140/week
• Heavy testing volume on Wednesday/Thursday/Sunday at 6am and 6pm MT
  • Corresponds with shift changes
• Uses BD and BinaxNOW machines

Whiteboarding - Pima org chart

Map out partner org to better understand team setup and find opportunities to collaborate.

Design and Build the Receive azure function

Done when we have a Receive azure Function that can receive a .csv file via http, and store it in a blob store.

No security. No API?

Run a router in a cloud service

Dependency

• Will get a hosting environment decision on Monday
• Will get access to the hosting environment soon after

Assumption

• Running Java Jars which are hand-built
• POC can deliver a CSV file to a common block store
• These environments will be torn down for MVP and reconstructed by automation

Acceptance Criteria

• Hosting container choice (mirth, function)
• A test and a prod instance runs
• Can be hand-built (automation comes later)
• Access to all devs

Follow-on tasks

• Deployment pipeline code and infrastructure

Prototype schema

Story
As a demostrator, I would like to talk about the PRIME data hub schema and its uses

Assumptions

• HL7 conversion information can come later
• FHIR conversion information can come later

Acceptance Criteria

• Can demo standard schema
• Can represent variations from PA, FL, PDI

FL research week of 10/12-16

Contact: Craig F (IT Director), Allison Culpepper (Bureau of Epidemiology)

Setup project board, determine agile stuff that works for us

Sprints

• Short 1st sprint, ends 10/30 to line up with Input board
• 2 week sprints thereafter, starting 11/2

Grooming, sprint planning, demos

• Regular grooming as part of joint PRIME grooming
• Sprint planning on every other Friday/last day of current sprint, joint with all PRIME projects 1-2pm ET/10-11am PT?
• Demos will become part of the overall PRIME project, do we want to do smaller ones as well?

Documentation & assets

• Short term, start following docs:
  • Engineering
  • Product
  • Design
• Ongoing:
  • Weekly research/design roundup to post in Teams. Top-level findings with any relevant design assets (Mural boards, etc.

Prototype Router for CSV

Story
As a prototype user, I can route a CSV with COVID-19 to PA and AZ standard CSV.

Assumptions

• CSV schemas in PRIME-Central's Translation Metadata
• Routing is state-based
• Sending to a folder

Acceptance Criteria

• File showed up in each state folder
• Some errors in input CSV result in well defined errors

Whiteboarding - Systems in use across pilots

FL

• Mirth
• Cloverleaf

Pima

• abc

Houston

• abc

Harris

• abc

PA

• abc

Lock down hosting!

Deciding on hosting environment for PRIME Data Hub.

Acceptance Criteria

• Agreement with CDC
• All developers have accounts
• Decision communicated

Prototype Router works in Mirth

Story
As a demonstrator, I would like to show that the prime router can work in a Mirth Connect channel to show that the data router can be used by standard healthcare integration engines.

Assumptions

• Mirth Connect 3.9 running in a Docker container
• Java 11
• Prototype type of volume and uptime

Acceptance Criteria

• A channel that accepts in a CSV file and routes it

Launch plan for sending with integration vendor

Approvals

• Amy/Leads sign off
• Internal CDC sign off

Product/operations

• Outreach to next ~10 states
• Figure out what are the eng changes are needed to send through 3rd party (#63)
• Testing process with 3rd party vendor

For user privacy, use deidentification Transform that complies with HIPPA

See the https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html#standard

Prototype Router support Prime Data Input Schema

Story
As a demonstrator, I would like to be able to use fake CSV data from the Prime Data Input app

Assumption

• Data hub schema and standard elements defined

Acceptance Criteria

• Ability to load a CSV with all PDI columns

Data Hub - Routing: Milestones and Features

Summary
The Data Hub - Routing project will provide the infrastructure to send data generated from the Data Input App. To achieve this, we will need to build the ability to send the data and to reformat the data to adhere to unique needs of each PHD.

Milestones to achieve an MVP

1. Fake data
2. Test data + Auth
3. MVP: Real data + Auth
   Post MVP milestones
4. Monitoring
5. Easier configurability

Sending and transforming
"Once data is collected by the Data Input app, the Routing app will send it to the correct PHD in the correct format."

• App can figure out where to send the data the proper jurisdiction
• App can transform the data to send the fields required by the target PHD, in the required format

Authentication and security
"The app securely connects with each PHD. Along the way, data is protected."

• App is able to save authentication information for each individual PHD
• App is able to authenticate with different types of configurations
• App has proper restrictions on what data is sent/accessible

Configuration: Pilot customers
"For our first pilot customers, we know the requirements to connect with each jurisdiction"

• What are the data requirements for each PHD
• What are the logistics/timeline for connecting with each of them

--

Monitoring
"As a human interacting with the app, I can get helpful updates about the status of data I expect to be sent"

• Receipts back to sender
  ...

Configuration: scalability
"As USDS/CDC works with more PHDs, we can easily add more jurisdictions to the Routing app"

• App can save settings for each unique jurisdiction
  ...

Updated research plan

Based on 10/19 discussion, update research plan to focus on 1. Routing prototype and 2. Analysis-related research for future follow-up/features

AZ research week of 10/12-16

Contact: [see notes: https://docs.google.com/document/d/1rK31CyqCPwY4cOspUhUeyheMQxWFoUaGDiXlwpVXPvY/edit?usp=sharing]

The router supports NHSN API

Story
As a nursing home that reports ELR to NHSN, I would like to use the POC app to handle this request, so I do not need to continue to report through NHSN.

Acceptance

• Connection to NHSN in test and prod
• NHSN schema in PRIME metadata
• NHSN component in router pipeline
• All fields available are transmitted

Whiteboarding - Harris TX org chart

Map out partner org to better understand team setup and find opportunities to collaborate.

AZ - ELR receiving

10/22 5-6pm ET

Agenda: Dive deeper into the process of setting up new organizations on ELR. In particular, we would love to learn more about how the process has evolved over the last few months, and also a deep dive on how data is received for CSVs, fax, and web

Talk to labs sending to multiple states

[X ] Color
[X ] Helix
[ ] Curative

Notes here - https://docs.google.com/document/d/1LuWxblkwwLdFYDd32TW1DnKKG-UWw-a7XcXVTifC-3U/edit

Could prototype router support email?

Investigate how a file could be shared securely via email.

• What does Pima consider secure?
• Does it adhere to SMIME standard?
• Depending on hosting, we could have some kind of drop box with temporary access (files not retained long-term)
• Ideal version for us is an SFTP to send to

Prototype Router supports SFTP

Story
As demonstrator, I would like show that the router can send via SFTP because most states accept SFTP connections.

Assumptions

• Fake accounts and passwords for SFTP
• Dummy

Acceptence Criteria

• SFTP option in the prime send and prime recievers.yml
• SFTP setup server in the demo script

Prototype Router demo script

Story

As a member of the team, I would like to be able to demo the router.

Assumptions

• Works on a mac laptop
• Laptop can install brew
• Laptop can install docker

Acceptance Criteria

• a setup instruction
• a script that is understandable

Test: Fix issues discovered by CSV file edge case tests

When I wrote the end-to-end tests of CSV files, I found a bunch of minor bugs in how the router handles CSVs. This story is done when those issues are fixed. I'll add them to the story, to flesh this out.

FL - onboarding follow up

10/27 w/ Allison Culpepper

Routing: Milestones and Features

Summary
The Data Hub - Routing project will provide the infrastructure to send data generated from the Data Input App. To achieve this, we will need to build the ability to send the data and to reformat the data to adhere to unique needs of each PHD.

Milestones to achieve an MVP

1. Fake data
2. Test data + Auth
3. MVP: Real data + Auth
   Post MVP milestones
4. Monitoring
5. Easier configurability

Sending and transforming
"Once data is collected by the Data Input app, the Routing app will send it to the correct PHD in the correct format."

• App can figure out where to send the data the proper jurisdiction
• App can transform the data to send the fields required by the target PHD, in the required format

Authentication and security
"The app securely connects with each PHD. Along the way, data is protected."

• App is able to save authentication information for each individual PHD
• App is able to authenticate with different types of configurations
• App has proper restrictions on what data is sent/accessible

Configuration: Pilot customers
"For our first pilot customers, we know the requirements to connect with each jurisdiction"

• What are the data requirements for each PHD
• What are the logistics/timeline for connecting with each of them

--

Monitoring
"As a human interacting with the app, I can get helpful updates about the status of data I expect to be sent"

• Receipts back to sender
  ...

Configuration: scalability
"As USDS/CDC works with more PHDs, we can easily add more jurisdictions to the Routing app"

• App can save settings for each unique jurisdiction
  ...

Who are our next set of senders? Who are we talking with next?

• Possibly TX, because of HL7
• Possibly FL

Whiteboarding - FL org chart

Map out partner org to better understand team setup and find opportunities to collaborate.

Data Hub - Routing: Milestones and Features

MOVED FROM PRIME CENTRAL

Summary
The Data Hub - Routing project will provide the infrastructure to send data generated from the Data Input App. To achieve this, we will need to build the ability to send the data and to reformat the data to adhere to unique needs of each PHD.

Milestones to achieve an MVP

1. Fake data
2. Test data + Auth
3. MVP: Real data + Auth
   Post MVP milestones
4. Monitoring
5. Easier configurability

Sending and transforming
"Once data is collected by the Data Input app, the Routing app will send it to the correct PHD in the correct format."

• App can figure out where to send the data the proper jurisdiction
• App can transform the data to send the fields required by the target PHD, in the required format

Authentication and security
"The app securely connects with each PHD. Along the way, data is protected."

• App is able to save authentication information for each individual PHD
• App is able to authenticate with different types of configurations
• App has proper restrictions on what data is sent/accessible

Configuration: Pilot customers
"For our first pilot customers, we know the requirements to connect with each jurisdiction"

• What are the data requirements for each PHD
• What are the logistics/timeline for connecting with each of them

--

Monitoring
"As a human interacting with the app, I can get helpful updates about the status of data I expect to be sent"

• Receipts back to sender
  ...

Configuration: scalability
"As USDS/CDC works with more PHDs, we can easily add more jurisdictions to the Routing app"

• App can save settings for each unique jurisdiction
  ...

Updated research plan for ELR receiving

Spec what it takes to receive data from HCA

Assumptions
-Data we receive from HCA is different from PDI. What are the product + operational changes we expect to make to handle this?

Working doc
https://docs.google.com/document/d/1IgJUFJXu74JP7PZ5WRcwF0agAu2vUDRr9ipT-RCnJtk/edit

Test: end2end from PDI --> AZ

Write end-to-end test cases that represent our initial use case - starting from Prime Data Input (PDI) data, and transforming it into the format Arizona's Public Health Dept (PDH) wants to receive.

See this prime-central ticket for info on the formats: CDCgov/prime-central#84

Scope out changes for large hospital group

When, with which hospitals, collecting what data?

• Define steps
• Review doc in progress @jlusds
• Determinte next steps/write tickets