cityofboston/digital

City of Boston javascript webapps managed by DoIT Digital Team.

Digital Team

Public documentation and wiki for DoIT’s Digital team

Monorepo Installation

$ yarn install

Development

Organization

This repo is split into two broad categories: services and modules. Modules are JavaScript (and TypeScript) libraries that are shared across various apps. Services are individual apps.

Modules may depend upon each other (though not circularly). Services may only depend on modules. Nothing may depend on a service.

The majority of our services are written in JavaScript/TypeScript and are therefore in the services-js directory. We have a few legacy Ruby apps in services-ruby. These are stand-alone, without any dependencies on our own libraries. Because we’re not doing new development in Ruby, we didn’t put the effort into code-sharing infrastructure for it.

Compliation / Transpilation

Modules

This uses “watch” features to rebuild all of our modules as their source changes.

$ cd modules-js/react-fleet
$ yarn watch

Note: This may not necessarily capture recompiling depending modules when their dependencies rebuild.

Note: You may need to make heavy use of restarting any TypeScript server in your editor if types in these modules change.

Services

$ cd services-js/commissions-app
$ yarn dev

You also may want to run yarn watch-dependencies to recompile changes in any modules that the service depends upon.

Making a new JavaScript module

$ npx khaos create -d templates <template-name> <destination>
$ yarn install

Current templates

• js-server-module: Module for libraries that are only going to be required by server-side Node apps. Uses TypeScript for all compilation.
• js-browser-module: Module that’s optimized for libraries that are included via Webpack. Uses TypeScript for type checking but compiles with the browser.js Babel configuration.

Making a new JavaScript service

It’s best to copy from an existing one. See the New service setup documentation.

Making a new Ruby service

Please don’t. We have a few Ruby services that pre-date this repo, but want to focus future development entirely on JavaScript and Node.

Releasing

Staging

Staging instances are created by adding them into the Terraform templates. Typically, each service will have a default staging deploy and may have other “variants” for special circumstances.

Staging branch names follow the pattern:

• staging/<service-name>
• staging/<service-name>@<variant>

Where “<service-name>” is the service package’s name (minus any services-js. prefix).

To deploy to staging, first force-push to the staging branch. A GitHub webhook will fire and you’ll get a prompt in the #digital_builds Slack channel from Shippy-Toe, asking you if you want to deploy. Press the “Deploy” button to trigger the deployment, which is done via CloudBuild.

Tests are not run for the staging deploy so that you can put something on staging even if the tests don’t pass. (Tip: use the --no-verify flag to git push to keep it from trying to run the tests locally either.)

If you need to roll back a staging release, force-push an earlier commit and re-deploy.

Example:

Deploying the Commissions App service to staging:

$ git co my-feature-branch
$ git push --force --no-verify origin HEAD:staging/commissions-app

Now press “Deploy” in #digital_builds.

Check on the status of the deploy by following the link in Shippy-Toe’s Slack message to the CodeBuild run.

When the deploy completes, the new app will be available at: https://commissions-app.digital-staging.boston.gov/commissions

Production

We believe in continuous deployment, so anything that’s merged in to the develop branch is eligible for immediate deployment.

At the end of the develop Travis run, the script “deploys” by running deploy-tools’s report-updated-services tool. This uses Lerna to compare the develop branch with all of the production/* branches for our services. If develop contains any changes for a service—or one of its dependencies—that do not exist in that service’s production/<service-name> branch, it notifies the Shippy-Toe bot that a deploy is needed.

Just as with staging changes, Shippy-Toe will prompt in #digital_builds that there are services to deploy. Press the “Deploy” button to release them.

Note: If the internal-slack-bot service comes up as deployable, wait to deploy it last, since deploying it will wipe out the state of any running deployments. They’ll still complete, but you won’t get notification of their status.

Shippy-Toe will run the deployment by first pointing the production/<service-name> branch to develop, then run the CodeBuild deploy to release it.

If you need to roll back a production deploy, it’s cleanest to push a revert commit through Travis and re-deploy. You can alternately force-push to the appropriate production/<service-name> branch and run the CodeBuild deploy manually. However, as long as the production/* branch lags behind develop, Shippy-Toe will offer to deploy with every change that’s merged in.

Dev Notes

Testing

Jest

Most of our modules and services have tests written in Jest. Services that use Storybook also use Storyshots to integrate stories with Jest’s snapshot testing.

TestCafé

Some of our services also have TestCafé tests to do frontend/backend integration tests. We use these typically for “critical path” tests (like going through the Access Boston registration flow), especially those that are a pain to manually test.

These tests get run on a headless browser via the test package.json scripts. When developing, you can run yarn testcafe:dev to run TestCafé in your desktop browser (or even a browser on another machine, like the BrowserStack cloud).

TypeScript and Babel and Webpack and modules

We want to write the bulk of our code in TypeScript because type checking is good. Unfortunately the landscape of JavaScript modules, require vs. import, and client-side bundlers (such as Webpack) tend to complicate things. The same piece of code might be required server-side, used client-side by both Next’s webpack config and Storybook’s, and run in a Jest test.

Server-side Libraries

These tend to be the simplest to compile. We don’t need to worry about polyfills or bundle sizes.

This means that we can build directly with TypeScript, via tsc and tsc-watch, and get both typechecking and compiling at the same time.

The build target for server-side code is Node 8. This means ES2017 syntax and language features can be native (such as async/await), but the modules must be CommonJS. This is all set in config-typescript’s default.tsconfig.json config via "target": "es2017" and "modules": "commonjs".

Nevertheless, since we test with Jest and our Jest setup uses babel-loader, these packages still need a .babelrc file. It should just use the @cityofboston/config-babel/node and @cityofboston/config-babel/typescript presets.

(Babel is fully capable of processing TypeScript files into JavaScript, but it doesn’t do any typechecking on them. It just strips the type annotations away and compiles the rest.)

Client-side Libraries

These are libraries of frontend code, typically React components. We want to use Babel to build these libraries to take advantage of the frontend-specific Babel plugins (such as @babel/env, emotion, &c.) while still outputting browser-compatible (ES5) code.

These packages should load the @cityofboston/config-babel/browser preset from their .babelrc files, which uses @babel/env’s default to compile to ES5. They will likely also use @cityofboston/config-babel/typescript.

Despite having ES5 code, we want to export these libraries using ES2015 modules so that Webpack can better tree-shake and keep unused code out of our application bundles so that they’re smaller to download. See: Webpack 4 tree shaking guide

We do this by using a special esm value for BABEL_NODE when doing builds, which @cityofboston/config-babel/browser interprets to generate esm modules. We point package.json’s module property at the entry point for these modules (and set sideEffects to false).

However, we still need CommonJS files because these libraries won’t always be included via Webpack. This can be due to being included from server-side Node files or just used in a Jest test.

For Jest/Node to be able to handle imports of these modules from other packages, we still need a main entry that points at a CommonJS build. We use Rollup to convert the ESM build into a single .es5.js file that Jest can see.

Since we still want d.ts files so that there’s proper type checking when importing these packages, we run tsc --emitDeclarationOnly during build. This also has the side effect of doing the type checking that Babel does not.

NextJS Apps

These apps get tricky because they have server that runs in Node, client code that runs in the browser (and also on the server with SSR), and (optionally) shared library code that runs in both.

NextJS handles all of the web client code via Babel and Webpack. We use a .babelrc file to include Next and TypeScript presets. So far we have not had to make any of our own accommodations for the server-side rendering code.

Next and Webpack also handle hot-reloading of the client code when it changes.

For the server, we use tsc and tsc-watch but with a special tsconfig.server.json TypeScript configuration that is limited to the server-specific source directories. This keeps the server from restarting when we make client-only changes.

The default tsconfig.json still includes all the code so that tooling will see and typecheck everything.

In all cases the code is written in TypeScript. Do not use Babel plugins that enable syntax that tsc can’t understand. Since the client-side code is not typechecked when it’s compiled by Babel, we run tsc --noEmit on everything as a pretest script to do type checking.

Note: We may soon switch to just using Babel for the server code as well and avoiding tsc for these apps entirely (beyond typechecking). The latest version of babel-watch finally supports Babel 7. Previously it did not, so we used tsc-watch to automatically reload the TypeScript code.

Browser support and polyfills

As of this writing, Digital webapps support IE11 and the latest versions of the evergreen browsers. (See Browsers we support in the working agreement)

One exception to this is the public-notices app, which needs to run on the old version of Chrome that the digital display has installed.

Because we are always forgetting about polyfills, we’ve configured the babel-env plugin to "usage" mode for useBuiltIns. This causes it to automatically pull in core-js polyfills for functions as we use them. Note that it does not polyfill functions and classes that our dependencies may need, nor does it polyfill fetch.

We’ve added polyfills that our dependencies tend to need, as well as isomorphic-fetch, to the polyfills.js file in next-client-common. The withPolyfill mixin is used in next.config.js files to automatically include these polyfills before any other code.

We build all of our interfaces to be responsive, down to 320px wide.

Updates and Patches

• 2020.12.10: React-Fleet - Node-Fetch 1.6.9 > 2.6.1
  • Affected Apps
    • modules-js
      • react-fleat | []
    • services-js
      • 311 | []
      • 311-indexer | []
      • access-boston | []
      • group-mgmt | []
      • internal-slack-bot | []
      • payment-webhooks | []
      • permit-finder | []
      • public-notices | []
      • registry-certs | [x] | Docker > node:8.14-alpine

Deploys

• 2020.11.10: Reworking how AWS gets around Docker Hub Rate limiting

digital's Issues

311 error pages should prompt to call if the error keeps happening

Forgot password flow

• Get new app configuration
• Authentication that differentiates between login and forgot password
• Forgot password HTML form
• GraphQL mutation for forgot password
• Non-JS support for forgot password

Email to Policy Office after submitted an application

After an applicant successfully submits an application, we need to send an email to [email protected] notifying them of the new application.

The subject should be:

New City Clerk Boards and Commissions Application

The body text should read:

Hello,

A new application was received for the following Boards and Commissions:

• Zoning Board of Appeals
• <additional boards/commissions applied to>

View the application details.

"View the application details" should be hyper linked to the application in the boards/commissions database. Example link: http://zpappweb01/cityclerk/commissions/account/Login?ReturnUrl=%2Fcityclerk%2Fcommissions%2Fapplications%2Fview.aspx%3Faid%3D1158

Cache service descriptions in the client

Since removing the generic loopbackGraphql cache, the 311 webapp is making more calls to get the details for a service.

We should cache those results on the client to reduce traffic.

Build out Birth Certs email templates

• Request has been received
• Request processed / invoice
• Addt'l information request
• Information request reminder
• Request cancelled

Birth certs 1st milestone punch list

• Quantity dropdown on review page
• Date input field mask / validation
• Remove "X" on review page, add "cancel and start over" link

Imported cases have incorrect closed date (salesforce change manager 000103)

International Shipping

From @joshuagee on January 24, 2018 22:24

Birth certificates will have requesters who need to ship internationally.

Currently, the fields in our shipping address only handle domestic shipping. This is a potential 2.0 feature (when we handle births).

Copied from original issue: CityOfBoston/registry-certs#287

Change password flow

Need a flow in Access Boston to change passwords for logged-in users.

"Client" landing page

When user selects “client” for the question ”how are you related?”, this information is displayed and the request flow ends.

Fix single logout

Logout is not working.

• Update UI to redirect the user to https://sso-dev.boston.gov/idp/startSLO.ping rather than creating a SAML logout request
• Figure out why the logout success URL is 404ing

Forgot password page has active button w/o input

When no fields are filled out, the forgot password page still has an active submit button.

Fade out / undo for removing from cart

From @finneganh on March 12, 2018 13:25

Clicking the "X" for removing from the cart is instantaneous. If someone double-clicks they may remove more than one thing, and there's no way to undo.

We should change the UI so that if you remove an item it shows a "removed" line while you're still on the cart page, so things don't bounce around and you can easily add it back in.

Copied from original issue: CityOfBoston/registry-certs#356

Implement new question flow designs

When CityOfBoston/creative-team-workflow#931 is done, implement the new UI changes in registry-certs.

Change from email when emailing applicants

Got confirmation that we can change the "From" email when emailing applicants to [email protected].

Language on success page for application form

After an applicant successfully submits an application, we should show a success page with the following language:

Thank you for applying to serve on a Board or Commission. We appreciate your willingness to make a difference in the City of Boston. If you have any questions about your application, contact [email protected].

Pressing enter on boards and commissions search page

When users search a term and press enter, the search results should update, but in testing the app on staging it seems like that functionality isn't working.

Implement birth certificates confirmation page after successful orders

Update commissions search app to work with D8

The commissions search app is designed to ingest settings that are created by Drupal. In Drupal 7, those settings lived at window.Drupal.settings.bos_commissions_search.bos_commissions_search_graphql_endpoint
window.Drupal.settings.bos_commissions_search.bos_commissions_search_graphql_api_key

That path is different in D8. It is now: window.drupalSettings.bos_commissions.bos_commissions_search.graphql_endpoint
window.drupalSettings.bos_commissions.bos_commissions_search.api_key

Ideally we could push an update to this app that would allow it to function in both D7 + D8.

I assigned @jessicamarcus and @mmcgowanBOS as I think you're best suited to take this over.

Move Commissions Search Page to Digital repo

Put source for commissions search page in its own folder here: https://github.com/CityOfBoston/digital/tree/develop/services-js

Move John’s apply form changes from commissions repo to digital repo

Boards and commissions application form required/helper text

We should use the following text for the required field on the boards and commissions application form:

• First Name: "The first name field is required."
• Last Name: "The last name field is required."
• Street Address : "The street address field is required."
• Unit: helper text should just read "Unit" instead of "Unit or Apartment #"
• City: "The city field is required."
• State: "The state field is required."
• Zip Code: "The ZIP code field is required."
• Email: "The email field is required."
• Confirm email: "Email confirmation is required."

Update "Order Details" dropdown in birth checkout flow

Also update "Order Details" on the Review page to not just say "1 copie(s)"

Map birth cert request data to fulfillment DB

We need a specific plan on how to put the responses to the birth certificate request questions into the orders database.

Make "Record may have an access restriction" be a error-ish box that pops up when "Don't know" / "no" are selected in "were your parents married"

Updates/fixes to search page

• add a little more space between filter options so the separation between "Policy Area" and "Open Seats" sections on mobile and desktop is clearer
• make filter drawer automatically close when "Apply" is hit on mobile
• pressing enter on search doesn't do anything. This was fixed last week on the testing instance (https://boston-gov-commissions-search.digital-staging.boston.gov/commissions-search) but seems to be broken again
• it looks like the search page is pulling from the dev database on the testing instance since its not using updated URLs. This was also fixed/working last week

Email language update

Change last sentence in email to applicant to:
If you have any questions about your application, please contact [email protected].

The last sentence currently says:
If you have any questions in the meantime, please contact [email protected].

Typo on boards and commissions application page

We have a small typo on the boards and commissions application page. Where we tell people what boards they've selected to apply for we have "You're" instead of "You've".

Make email option less obvious (Register MFA Page)

IAM would like to encourage users to use Phone MFA over Email. Let's make the Email option less obvious to users visually.

Here's what we're currently doing:

Here's a visual option they liked (as seen on the RegisterMfaPage > DeviceVerificaitonModal > incorrect code preview):

Remove find polyfill

From @finneganh on April 2, 2018 15:6

Now that Stencil is updated we can remove the find polyfill for IE11 in _document.js.

Copied from original issue: CityOfBoston/registry-certs#370

Add an identification verification step to the questions flow

The parentalInformation step includes the question ”were your/their parents married at the time of your/their birth?”. Any answer other than yes should take the user to an id verification step, otherwise the question flow ends and the user proceeds to /birth/review.

The actual ID verification functionality will be a discrete component, since there is also a freestanding id verification page to support cases where a user submits a request, and is later asked for ID verification by the Registry.

user-facing:

• explanatory content
• Upload ID => proceed to id scan => success => proceed to /birth/review
• No ID? => provide messaging and “Request help” button, then end request flow

requirements:

• add a step to the overall progress if id verification is required
• addt'l logic for back/next buttons

Identify and split out shared functionality for outgoing emails

• Refactor sendReceiptEmail method of Email service to accept subject line and email template data as parameters
• Split out email headers and footers into Partials to be used for any registry-related email template

Logout not working on AccessBoston test

The "logout" SSO feature on the AccessBoston test instance does not log out of the SSO session.

Application form alignment issues

QA testing revealed some alignment oddities:

1. with the external links when viewing the application form in Chrome on an iPhone 6+ (IOS 11)
2. with the text boxes that are on the same line when the page is viewed in landscape mode

More screenshots can be found in the QA testing doc I've shared.

Block any City email domains for registering in MFA

We need to prevent any City email domains from registering in MFA.

See 2nd tab in Portal Content spreadsheet: https://docs.google.com/spreadsheets/d/1OiH2HBLE55X8MzGaB7H9VC_a2WpQQcjV4-eyNpJGTPY/edit?ts=5bbf4804#gid=0

Email to applicant after submitting application

After an applicant submits an application, we should send an email to them with the following language:

Subject: We’ve received your application
Body text:
Serving on a board or commission is one of the most impactful ways Bostonians can become active in their community - thank you for your application(s) to:

- Board/Commission 1
- Board/Commission 2
- etc.

If you have any questions in the meantime, please contact [email protected].

Thank you,
City of Boston

Fix GraphQL API key checking

The prod commissions-search app seems to be able to fetch GraphQL without an API key: https://apps.boston.gov/commissions-search/

It should give a 403.

Use new Forgot Password token

Ping is now sending a "userAccessToken" during login assertions for the forgot password endpoint.

We need to save that in the session and send it back as "Token" when triggering the forgot password workflow.

"403:Unknown client" errors in 311-indexer (salesforce change manager 000127)

We’re getting errors from the Salesforce streaming API every 5–20 minutes about an "unknown client". We're currently restarting so things work, but this indicates there may be a deeper problem that should be resolved.

{
    "clientId": "2st8w3gricczecj1a6kdr718go6m",
    "advice": {
        "interval": 0,
        "reconnect": "none"
    },
    "channel": "/meta/connect",
    "id": "8",
    "error": "403::Unknown client",
    "successful": false
}

ID verification - add/remove files in DB

Binary files provided by the user will be immediately uploaded to the db server, separate from the other information collected during the questions workflow.

Functionality to delete a file on the server must be present for cases when the user wants to replace or delete a file before proceeding with their request order.

The server will return a unique attachmentKey on success.

The following is no longer applicable:

This page supports the case of a Registry-initiated request for id verification to support an existing order. The user will land on this page and see the order number and name on the requested record, and the functional component.

Page will handle uploading files to the db, and showing errors and/or success confirmation to the user, while the upload/photo functionality is handled by VerifyIdentificationComponent.

Questions:

• should we display name of person who ordered in addition to the name on requested record and order id#?
• need text content for submission success
• should we get already-submitted images/files from server and display those assets as already selected in the “upload front/back of id” and “supporting documents” inputs?

Functional requirements:

• submit files to db
• show error messaging
• show success messaging

Feeback button work on commissions apply form

Make feeback button work on commissions apply form

"Operation is insecure" Safari errors

It appears that Safari (at least on iOS) throws SecurityException errors when registry-certs attempts to access window.localStorage and the user’s cookie settings are set to deny all.

Since all we need localStorage for is saving address information, we could catch the errors and not show the "Save contact info on this computer" checkbox.

Search by year

From @joshuagee on January 24, 2018 22:28

For a potential 1.1 release: Allow users to search by year or death date.

Copied from original issue: CityOfBoston/registry-certs#291

UI updates to application form:

• take prefix field off
• change "Other Information" in Education and Experience section" to be "Relevant Work Experience"

Recommendations from design (result of this issue):

• put the two lists of boards/commissions (those with open seats/those with_out_ open seats) into separate drawers
• have the first list (boards and open seats) default to open and the second list (boards without open seats) default to closed
• in each of these lists on larger screen sizes (anything above mobile) put the list of boards/commissions in two columns
• lower the font sizes of the board/commission names a little bit

Set up prod and staging deploys for commissions search page

Get staging and production deploys for commissions search app set up.

UI for ID upload step in questions flow

Component to enable a user to upload images and documents for identification verification.

This component will be used in the questions flow verifyInformation step, as well as a freestanding page; when the Registry manually requests id verification from a user for an existing order, this page is where that user is directed.

Waiting on:

• detailed instructions for the user
• visual design for interface
• allowed file types for documents

Functional requirements:

• user must submit front and back images of their id
• user can add addt’l supporting documents
• user can review their images before submitting
• user can use their device’s camera to take a photo of documents, if possible

Back button doesn't work in checkout flow

Ideal back button behavior, per @fionawhim and @jessicamarcus:

• Clicking "back" takes you to the previous page
• Data on the previous page should be preserved
• Clicking back then forward should leave you in the same place
• Clicking back, modifying form, then clicking forward w/o submitting should not have the form modifications
• Hacking the URL to bypass steps should show errors on the "review" page w/ missing info (alternate: redirect to first missing place instead?)
• Reloading a page should preserve submitted data (e.g. on "Review" page) but clear out any changes.

Changes to make:

• Change text fields to not write to the Order unless submitted
• Add in URL parameters so Next routes w/ pushState instead of replaceState
• Switch order storage from instance variable on CheckoutPage to session-backed storage
• Add error state on review page for missing shipping / billing / empty cart
• Add tokenized card state to payment page (w/ "clear" link to make CC fields come back)

Checkout flow for Birth Certs

Have a flow at /birth/checkout that goes through shipping, payment, and review.

(This is likely / definitely based on components already written for death certificate checkout.)

Update to newest web components

From @finneganh on June 29, 2018 14:59

Web component JS is now at: https://patterns.boston.gov/web-components/fleetcomponents.js

Copied from original issue: CityOfBoston/registry-certs#387

Wiki link to Budget doesn't go anywhere

The link to 'FY19 Budget Site' on https://github.com/cityofboston/digital/wiki doesn't go anywhere.