The ef-cms from ustaxcourt

Define blackout window, release, and launch plan; and document it.

As the Court, so that we can effectively launch EF-CMS and operate the Court without losing any data, we need to minimize the time the Court’s systems are locked or inaccessible and effectively plan for launch.

Acceptance criteria:

The process for locking one system and running any final migrations in order to start operations in EF-CMS is documented.
The timeline needed for each step is known and documented.
Data is exported as csv and archived for record retention rules.

Notes:

Migrating and validating legacy data may be a good place to document this.

Determine a process for communicating about updates to EF-CMS.

As users of EF-CMS at the Court, so that I can understand how to use new features and understand how changes impact my work, I need to know when changes are made to EF-CMS.

Mike syncs with Jessica

Load test the Court's migration environment.

As the court, so that we can be assured that the migration environment can support expected capacity, we need to load test the migration environment.

Pre-Conditions

Use real legacy PDFs
System is loaded with lifelike cases & users & large PDFs
penetration testing can't happen simultaneously (coordinate with NSSPlus to ensure we aren't running this while they're doing pen testing)
ensure monitoring & alerting issue is complete (#250) so we can capture if/when the system fails - sync w/ @adunkman when things are nearly ready

Acceptance Criteria

Production environment load testing completed
Actionable items documented

Security Considerations

Does this work make you nervous about privacy or security?
Does this work make major changes to the system?
Does this work implement new authentication or security controls?
Does this work create new methods of authentication, modify existing security controls, or explicitly implement any security or privacy features?

Notes

Elasticsearch may not autoscale, so special attention should be paid to its performance during load testing. Adjustments to its clustering, autoscaling, and instance type may be needed.
Elasticsearch-specific items are also tracked on Flexion’s Production Prep backlog as Elasticsearch Performance - High.

Create plans for production launch and operations.

On the road to releasing EF-CMS to production, this epic represents the tasks needed to document and flesh out the launch and operations plans.

Teardown/rebuild and document steps for migration environment

This issue is to track the teardown/rebuild of the migration environment. We'll document steps taken and post a wiki.

Confirm w/ Eric that we're not missing anything.
Confirm with Mike/data migration team that we're ready to rebuild.
Run environment destruction script. npm run destroy:mig
Confirm that everything's torn down properly.
Kick off a CircleCI build for the migration environment.
Confirm that everything is set up properly upon successful CircleCI build.
Notify Mike/data migration team that the environment is back up
Confirm that the data migration team has the access needed to migrate cases.
Revise documents with findings from this activity: https://github.com/ustaxcourt/ef-cms/blob/staging/docs/SETUP.md / https://github.com/ustaxcourt/ef-cms/blob/staging/docs/ENVIRONMENT_DESTRUCTION.md

Do not rely on Serverless Framework for deployments.

Yesterday when releasing #210, we encountered an issue where Serverless could not modify the Court’s environment.

When reached for assistance, Flexion said that they had to destroy and re-create the CloudFormation stack in order to get it released.

We can’t have these kind of deployment issues when we are in production. I discussed with the Flexion team yesterday, and they have indicated that they don’t think we can use Serverless Framework, as this is a routine and regular issue that they have experienced.

The move away from CloudFormation would also significantly simplify our deployment scripts, as we’re currently working around a 200-resource limit for CloudFormation which may no longer be a relevant constraint. We may be able to continue to user Serverless Framework for local development and use sls package to generate a package which can be deployed using other tools.

Additionally, if we were to rely on Terraform exclusively for management, we’d be able to have higher confidence in execution plans using terraform plan.

Complete final accessibility testing.

Perform both automated and manual testing with users who are familiar users of assistive devices to ensure the site is not only meeting it’s legal obligations for accessibility but it is fully functional and useful.

Most interested in public-facing flows of an external user. May be able to get work flows from Kristen. Exclude accessibility of Court documents. 18F do microrequest to Accessibility Guild. Prioritize P1 (show stopper!), P2, etc. Add to this GitHub and tag with accessibility label.

Create and document the support and escalation system.

As a member of the public, so that I can access the Court’s systems and overcome setbacks, I need to know how to report issues and who to contact if I am stuck.

Acceptance criteria:

The way that system issues are troubleshooted is known, including who is the first-line of support, and how/when contacting the vendor is appropriate.

Determine the shape of a reusable test database.

We’re looking to create a test database seeded with test data that can be used across different test environments and forks.

It could be a script that generates a reusable DB, Flexion has a script already that creates something.
Unknowns that we’d like to figure out:
- How many cases are open vs. closed?
- What is the distribution of the number of practitioners?
- How many documents are there?

Complete load testing.

Prepare for having a lot of users in the system by performing load testing, specifically around search and areas of the site where volume matters in terms of return time.

Done when:

Tech lead is confident that the system can handle the expected traffic without issues, and that the system (including search) is scaled appropriately to reduce waste.

Public domain is set up

Court wants to use https://dawson.ustaxcourt.gov for production
Public domain is set up
Court's site points to the new domain

Lock all build dependencies to specific versions.

As the Court product owner, I need build dependencies locked to a specific version, so that we don't have unknown schedules/fails.

Acceptance criteria:

Instead of getting the latest version of API/library, we want to be specific about a version number, so we don't get changes when we deploy to a new environment.

Next steps:

Identify if there are dependencies in the code base that are not locked down
If there are dependencies that are not locked, specify where in the code base they exist (i.e. apiVersion = latest)
Specify which versions should be locked down to AWS CLI
Document environment variables in the README

There are at least some dependencies in the build process which are not version-locked, as recently discovered. This results in builds failing on unknown schedules, as other parties update these open source dependencies.

We’ll need to lock these dependencies to specific versions to prevent these automatic updates, as well as determining a strategy for keeping them updated.

This would be a good opportunity to examine the way tools are being used as well, to see if there’s an opportunity to simplify and reduce dependencies. We may want to use a tool like Artifactory to assist.

Capture error messages handled on AWS for Court's environments and route them

Capture error messages handled on AWS and route them to where they need to go
This is for court's environments

Training for new users

Training plan created: https://docs.google.com/spreadsheets/d/1GRLxRUVTQble0I4Lm5LEaPWdYKeZWUCK8_426KJTJ6s/edit?usp=sharing
Trainers identified (see above)
Training materials created: https://docs.google.com/spreadsheets/d/1fg1Gt4Jzz0y_FvZLW0lVvT6Y-7OcfLPldTyejsm958Y/edit?usp=sharing
Training dates scheduled: https://docs.google.com/spreadsheets/d/1KjKSpIM2plYzuT6jRgw9isTp4DJvi9FZiedq7wizQFY/edit?usp=sharing
~~Training location determined and scheduled~~
Trainees identified and scheduled (To be determined by trainers)
Identify all of the areas where there are work arounds and ensure that users have the information necessary. See MVP review doc for some of the work around ideas
- Create content (written or videos) that public users can reference for submitting a case and creating a petition
- Ensure that the Court's OIS staff is trained on system
- Ensure the court is trained on helpdesk process and triaging common system issues

Execute communication plans.

As previously planned for in the:

#239 Internal communications plan
#240 External communications plan

Use the EF-CMS domain for login and signup flows.

The Cognito-hosted pages (login, signup flows) are currently using AWS domains, for example:

auth-prod-ustc-efcms.auth.us-east-1.amazoncognito.com

Users should be skeptical of entering credentials for an application on a domain which doesn’t match the application.

Configure production AWS environment and destroy existing production environment.

As previously discussed and decided, the court will have a separate AWS account for the production environment.

Overall objectives:

Create a new environment in a new AWS account.
Destroy the existing prod environment in the previous/current AWS account.
Move away from using master as a branch name.
Avoid requiring immediate reconfiguration of Flexion’s prod environment (it should be a refactor task, not a immediately blocking task).

Checklist:

Acceptance criteria:

The production site has DNS records configured.
SPF record/DNS security is configured correctly.
Test/confirm DNS updates, from both inside the court and outside.

Separate application-specific vs environment-specific documentation.

We currently mix documentation about the application and the environment in which it runs, which causes confusion.

Type	Definition	Destination
Application-specific	Describes how the app is setup and run. This should remain in the codebase and will be applicable and useful to the Court, Flexion, and any entity that forks this repository and runs the application.	The docs folder.
Environment-specific	Describes the copies that the Court runs, and varies between the Court, Flexion, and any entity that may fork the repository.	The wiki.

Security testing complete

Any exercises and testing needed to deliver secure code to the court have been completed

Pen testing

Visualize deployment strategy

Our deployment strategy, once determined, should be visualized with a diagram and anything else appropriate.

Keep in mind that the order of operations is important to illustrate (workflow diagram?)

Determine what goals and metrics are relevant to EF-CMS’ launch.

As the Court, so that we can know if EF-CMS has launched successfully, we need to know what goals and metrics are relevant to track.

Acceptance criteria:

The expected load for the site is determined, so our load testing can be tuned appropriately.
The impact to Court staff, the public, and the level of service provided by the court is represented through the determined metrics.
Baselines for these metrics are established.

Notes:

What outcomes are we expecting, and what numbers can we record that will tell us if our expectations are real?
Do we have these numbers recorded today, so we’ll be able to compare after launch?
There are operational (AWS) metrics and business metrics.

Determine if/what adjustments to the deployment strategy are needed.

As the court, in order to have a successful deployment to the production environment with the latest code, we need to determine what our deployment strategy is.

Acceptance criteria:

The Court has determined what is acceptable downtime.
Any changes needed are flushed out into Flexion’s backlog.
The deployment strategy is visualized with a diagram or other appropriate choice, keeping in mind that order of operations is important to illustrate.
What is considered acceptable downtime and the rules around downtime are documented in the Site Operating Plan (#244).
Users should not see errors related to a deployment being in progress
Technical lead should be able to roll back to the previous deployment, including database (may be its own issue)
Technical lead be able to automatically validate that the deployment was successful (may be its own issue)

Notes:

The Court has previously discussed as acceptance criteria:

Downtime should be minimized as much as possible.
No downtime should be expected for routine updates.
- Any downtime should be scheduled in advance, notifications are sent to users, and it should happen outside of 9-5pm business hours in all US timezones.
- Scheduled downtime should not exceed 24 hours.

After the main development phase, updates in production are expected 1-2 times per quarter.

What kinds of changes cause downtime?
Should version numbers been included in footers for customer support?

Potentially related:

Pink/Purple Deploys

Modify Lansweeper for issue tracking.

As the Product Owner of EF-CMS, so that I can maintain a secure and functioning application, I need to track any defects reported so they can be prioritized and fixed.

Update references to the Court’s case management system.

We’ll need to update any place that references Blackstone and point it towards EF-CMS instead. (Jen & Scott)

https://ustaxcourt.gov/

Determine how users and roles will be managed by the Court

When an employee is hired at the Court, or an employee leaves the Court, somebody will need to be responsible for adding and removing their credentials. I assume that there are existing processes for this within the Court, since they need access to the Court's internal network and Blackstone. Is it feasible to bolt this process onto that?

And what about for role changes, such as when somebody gets a new job within the Court? Similar process and people doing the work?

Test the application’s failover, monitoring, alerting, outage, and recovery processes.

As the Court TL, I need to intentionally break the Court's environment, so I have assurances that the system (including people) can withstand outages, route correctly, and fix breaks in a timely manner.

Pre-conditions:

Develop training materials for Admissions section/OIS team on how to accept and route customer issues
Training to Admissions section/OIS team on how to accept and route customer issues

Acceptance criteria:

The test: When parts of the application fail, the Court knows how the application as a whole will respond.
The test: When parts of the application fail, the Court knows what kinds of alerts they will receive.
The test: The Court knows how the application will respond as these services come back online after an outage.
After: Any needed changes to monitoring and logging infrastructure are documented as backlogged work.
After: Any needed changes to alerting are documented as backlogged work.
After: Any needed changes to the application are documented as backlogged work.
The Court’s tech lead is confident that the system is resilient in the face of common issues and experiences outages in a predictable way under extraordinary issues.

Related to #412.

Draft a site operating plan.

As the Court, so that I can be prepared to maintain a production application, I need to develop plans for operating EF-CMS.

Acceptance criteria:

Covers:
- Ongoing maintenance necessary, how it's performed
- What actions to be taken in outage
- Process by which new administrative access is given to the system
- How to monitor a open source project, site up time
- The flow of users and data through and in the system
- Mechanism for ensuring change doesn't break the system

Notes:

Overall, the care and feeding of site in ongoing way.

Data/Entity change management strategy

Determine any issues we could encounter when making changes to data and validation schema and create a plan for avoiding data corruption, integrity issues, etc. This might include planning for how to communicate between dev teams and data administrators.

Complete final mobile testing.

Perform final mobile testing on public portion of the site on multiple devices.

Done when:

Product owner is confident that the system will function well on mobile devices for the public.

Create an operations dashboard.

Tech Lead and interested stakeholders can monitor system health, including uptime, who is logged in, how many cases in the system today, and other metrics which build confidence in the system.

Internal Registration: Create roles for Production Environment (Test run)

As a Court, so that we may ensure a successful first time login experience for all court users, we need to perform a test run on a small group of court users

Before the Court barrels down the path of creating users (internal, Practitioners, and Respondent) in Cognito and sending out mass emails, we would like to perform a test run on a small group of court users to make sure everything is working as expected. The test run should be on:

Jessica Marine
Deborah Kaio
Justine Miles
Santrisha Mapson
Joeliette Baldwin
Cheryl Crouse
Admissions staff
Trial Clerks
migrator account
Anyone who needs access to Admissions Database

Pre-Conditions

Acceptance Criteria

Set up a small group of users in the Court's production environment
Create passwords for users
Court users receive email notifying them that their account has been created
Users can successfully reset their passwords via Cognito "Forgot my password" and access the system
Also test with a fake practitioner?

Security Considerations

Does this work make you nervous about privacy or security?
Does this work make major changes to the system?
Does this work implement new authentication or security controls?
Does this work create new methods of authentication, modify existing security controls, or explicitly implement any security or privacy features?

Notes

Tasks

Definition of Done (Updated 8-28-19)

Product Owner

Acceptance criteria have been met

UX

Business test scenarios to meet all acceptance criteria have been written
Usability has been validated
Wiki has been updated (if applicable)
Story has been tested on a mobile device (for external users only)

Engineering

Prepare contract for 3rd party security scan (pen testing).

This will be to ensure the security of the Court's environments, including a security scan by a 3rd party company.

The running of tests is captured by #147.

Configure technical monitoring and alerting.

As the Court’s operating team, so that I can ensure EF-CMS is operating correctly, I need to be able to monitor the performance of the application’s components and be alerted to issues.

Acceptance criteria

Notifications are sent when:
- The application is partially or completely down
- There are usage patterns which indicate a security breach
- There is a spike in traffic which may cause unusually high AWS billing
- Routine processes are not occurring
- Elasticsearch is overloaded

Execute production launch.

Production launch tasks which will take EF-CMS into production.

First-Ever Merge to Master

Determine a strategy for user management in Cognito pools.

Petitioners

Data import: As the Court, in order to make sure that the legacy data with respect to more than two petitioners is not lost, we need a way to allow information for more than two petitioners to be migrated (where applicable). flexion#5430
Flag users that will need to log in: As a project team, in order to know which petitioners need access to the new system, Flexion needs the ability to identify which petitioners have eAccess to the Court's current system. flexion#5508
Allow users to login: As a petitioner who has open cases in the current CMS and e-Access, I need the ability to log in to the new CMS and see my open cases. flexion#5159

Already completed tasks:

New petitioners can self-register for an account.

Shelved questions:

How do petitioners who paper-file or (petitioners with open cases without e-Access) who later want e-Access get granted access to their case?
- May be the same solution as flexion#5159; will be re-addressed when picking up that story.

Notes:

Nothing needs to be done for legacy petitioners who don’t have e-Access in the old CMS or only have closed cases. Their data should be visible in EF-CMS, but they will not be able to log in.

Practitioners

Add bar numbers to practitioners: As the Court, in order to make sure our legacy practitioners get associated with the right cases, we need the data migration team to be able to transmit the bar nos. of the practitioners on each case as part of that case. flexion#5594
Link practitioners to cases via their bar numbers: As a practitioner (petitioner’s attorney or IRS attorney) who has cases in the current CMS, I need the ability to log in to the new CMS and see all cases that I’m associated with, including both open and closed cases. flexion#5595
Remove practitioners: As an Admissions Clerk, so that I may remove an inactive attorney from a case and limit their access to the system, I need the ability to change the attorney's status. flexion#4652

Already completed tasks:

Practitioners can be imported and granted user accounts via CSV import.
Practitioners can be added.

Court employees

As the Court, in order to keep up with employee turnover, we need the ability to manage users in the system. flexion#4386
As the Court’s Tech Lead, I need the ability to bulk load existing court users into the system, so I can ensure that they have access. [Script (already built) versus user interface?]
As the Court’s Tech Lead, I need the ability to add and remove court users into the system, so I can ensure that the proper employees have access. [Script versus user interface?]
As the Court’s Tech Lead, I need the ability to change a court user’s role in the system, so I can ensure that they have access to the right functionalities. [Script versus user interface?]

Developers

As a developer updating EF-CMS in production, I need to be confident that the new version is working. #250
As the Court, in order to maintain a secure system which does not disclose private information, I need to be sure that hardcoded user accounts with well-known passwords are not present.
- How can users continue to be well-known for development environments to enable developers to fully test all user roles?

Production readiness testing and hardening.

Testing to determine any issues needed before a production launch.

Create internal communications plan.

Create internal messaging for Court employees, including information about the rollout, timeline, key features, how the new system will make their jobs easier, what to expect during launch and after, how they can provide feedback or report a bug, how they can acquire training, triage process, etc

Add real users to production environment

As a court, in order for court users to access the production environment, we need to add all court users with their real email addresses to the system.

Acceptance Criteria

Messaging for internal users created (see comments below for sample)
Real users are loaded into the production environment with their real email addresses
Court users have verified their email addresses and logged in to the system
Court user email verification and login have been confirmed for all users

External communications prepared

Communications about new system is created, including name of the new system, benefits, etc.

System is named
Benefits list created

Add analytics tracking to the Court's production environment

As a Court, so that we may more easily track user activity, we need to add analytics tracing to the production site.

Pre-Conditions

Acceptance Criteria

Data analytics tracking added to production and staging site

Security Considerations

Does this work make you nervous about privacy or security?
Does this work make major changes to the system?
Does this work implement new authentication or security controls?
Does this work create new methods of authentication, modify existing security controls, or explicitly implement any security or privacy features?

Notes

Tasks

Definition of Done (Updated 8-28-19)

Product Owner

Acceptance criteria have been met

UX

Business test scenarios to meet all acceptance criteria have been written
Usability has been validated
Wiki has been updated (if applicable)
Story has been tested on a mobile device (for external users only)

Engineering

Launch Party for Internal Users

Launch party for internal users planned.

Date selected
Messaging determined
Use as an opportunity for those involved to share with others, asst w/adoption

Note - recommend sometime between formal training and launch

External communications plan.

Create external messaging and a plan to announce the launch of the new system, including benefits to the public, that it's a new feature, how it will save them time, how it is saving the government money, success of the project itself - timeframe, value, etc

Draft FAQ
Post FAQ to website
Email FAQs petitioners/practitioners [Jessica & Mike]
Alert Tax Press

Petitioner: Login/Authentication to Prod environment (Test Run)

As a Petitioner, in order to access the online filing system, I need to log in to the Court's production environment.

After registering for an account, Petitioner is able to log into the system with their username and password.

Pre-Conditions

Acceptance Criteria

Petitioner registers acct in new production environment [Note: Want to test this with as many domains as practicable; but especially hotmail]
Petitioner can log in

Security Considerations

Does this work make you nervous about privacy or security?
Does this work make major changes to the system?
Does this work implement new authentication or security controls?
Does this work create new methods of authentication, modify existing security controls, or explicitly implement any security or privacy features?

Notes

Tasks

Definition of Done (Updated 8-28-19)

Product Owner

Acceptance criteria have been met

UX

Business test scenarios to meet all acceptance criteria have been written
Usability has been validated
Wiki has been updated (if applicable)
Story has been tested on a mobile device (for external users only)

Engineering

Track permission changes to AWS IAM deployment user separately.

We currently have the permissions for the user used in deployment managed through code in this repository.

This means:

A manual step is required to run this script as an administrator.
There is a race condition on merging pull requests. When a pull request is merged with a change to these permissions, a deploy is started which will fail. An administrator then needs to update permissions for that user, and then the build needs to be restarted.

From a security perspective, this also means:

Permission changes are included in large pull requests, which means they may not receive the added scrutiny that changes to permissions should likely undergo.

A fix for this would be to track this user and its permissions in a separate location. We may want to rely on infrastructure which is court-wide to manage these permissions. This would:

Require this dependency to be specifically stated ("this pull request depends on permissions introduced in X")
Would separate these changes from larger application changes, giving them the added scrutiny as mentioned above
Allow them to be run automatically, if desired.

The downside to this approach would be that EF-CMS would need to reference another place to determine how to set up its deployment steps. I think this downside is worth the benefits above.

Configure DNS for production domain.

As a humanoid, so that I can access the court’s Dawson app, I need to access it when loading dawson.ustaxcourt.gov in my web browser.

Acceptance criteria:

The production site has DNS records configured.
SPF record/DNS security is configured correctly.
Test/confirm DNS updates, from both inside the court and outside.

All application content created, edited and approved

All content needed in the EF-CMS is created, edited and approved by the court

Determine application availability requirements

A question came up about whether there was ever a discussion between Flexion and USTC about application availability, specifically surrounding deployment windows. Requirements for application availability should be determined, and clear AC should be set.

Revisit machine-to-machine authentication strategy.

As the Court, so that we can provide programmatic access securely, we need to grant authorized programs access to data.

Notes:

There are two machine-to-machine authentication situations in play currently:

Data migration. The migration scripts need to authenticate with the migration endpoints in order to function. The current strategy, quoting from @michael-mcvicker:

Not sure how it got there, but a Cognito account was made for my email address. Could have been Waldo entering it by hand, or in the deployment script? The issue was clerkofthecourt role doesn't appear to have permissions to migrate cases, so the Cognito account created needs an "admin" role (albeit we still have to manually add a token). If we can forego Cognito and just use an API Key leaving Cognito accounts for actual users, that would be ideal.
Automated processes from the IRS. The IRS needs to pull documents into their system when served in order to effectively process cases. The current strategy:
- A Cognito user account exists with multi-factor authentication enabled.
- A human logs into that account once every 30 days to retrieve an access token and refresh token.
- Those tokens are handed off to an automated script, which uses those tokens to access the EF-CMS API when triggered by email notifications.
- Before 30 days, the refresh token must be regenerated.

Configure health monitoring alerts

As the Court, so that we can ensure we have a secure and available system, we need assurance that after code updates, known weak points are not vulnerable to leak sensitive data and the application is available.

Acceptance criteria

Alerts are configured when:

The UI is unavailable. Monitored by alerting on uptime/ping testing for dawson.ustaxcourt.gov and app.dawson.ustaxcourt.gov (and the equivalent in other environments). Implemented in #497.
The system health endpoints return red. Monitored by alerting on the system health endpoints (as implemented in flexion#6281). Blocked on flexion#6903. Implemented in #650.
The Elasticsearch cluster has a non-green status. Monitored by alerting on the cluster health status. Implemented in #539.

ustaxcourt / ef-cms Goto Github PK

ef-cms's Introduction

Electronic Filing / Case Management System (DAWSON)

DAWSON Environments

Documentation

Project status

Contributing

Public domain

ef-cms's People

Contributors

Stargazers

Watchers

Forkers

ef-cms's Issues

Pre-Conditions

Acceptance Criteria

Security Considerations

Notes

Acceptance criteria:

Notes:

Pre-Conditions

Acceptance Criteria

Security Considerations

Notes

Tasks

Definition of Done (Updated 8-28-19)

Petitioners

Practitioners

Court employees

Developers

Acceptance Criteria

Pre-Conditions

Acceptance Criteria

Security Considerations

Notes

Tasks

Definition of Done (Updated 8-28-19)

Pre-Conditions

Acceptance Criteria

Security Considerations

Notes

Tasks

Definition of Done (Updated 8-28-19)

Recommend Projects

Recommend Topics

Recommend Org