suffolklitlab/docassemble-alweaver

A tool to help quickly generate draft interviews from an existing document (pdf or DOCX) for the docassemble platform.

Assembly Line Weaver: Suffolk LIT Lab Document Assembly Line

The Assembly Line Project is a collection of volunteers, students, and institutions who joined together during the COVID-19 pandemic to help increase access to the court system. Our vision is mobile-friendly, easy to use guided online forms that help empower litigants to access the court remotely.

Our signature project is CourtFormsOnline.org.

We designed a step-by-step, assembly line style process for automating court forms on top of Docassemble and built several tools along the way that you can use in your home jurisdiction.

This package contains an automation and rapid prototyping tool to support authoring robust, consistent, and attractive Docassemble interviews that help complete court forms. Upload a labeled PDF or DOCX file, and the Assembly Line Weaver will produce a runnable, clean code, draft of a Docassemble interview that you can continue to edit and refine.

Read more on our documentation page.

Related repositories

• https://github.com/SuffolkLitLab/docassemble-AssemblyLine
• https://github.com/SuffolkLitLab/docassemble-ALMassachusetts
• https://github.com/SuffolkLitLab/docassemble-MassAccess
• https://github.com/SuffolkLitLab/docassemble-ThemeTemplate
• https://github.com/SuffolkLitLab/EfileProxyServer

Documentation

https://suffolklitlab.org/docassemble-AssemblyLine-documentation/

History

See the CHANGELOG for more information.

Authors

Quinten Steenhuis, [email protected]
Michelle
Bryce Willey, [email protected] Lily
David Colarusso
Nharika Singh

Installation requirements

Using auto drafting mode

As of June 2023, the Weaver includes auto drafting mode.

To use the automatic field grouping feature of auto drafting mode, you need to install either:

1. The en_core_web_lg model on your server, or
2. An API token for tools.suffolklitlab.org.

You can request an API token by emailing [email protected]. If you prefer to install your own copy of the en_core_web_lg model, you can access it the first time you select to use auto drafting mode when logged in as an administrator.

docassemble-alweaver's Issues

No longer turn any variable that we don't explicitly handle into an attribute of an object

This is leading to errors with intrinsic names.

E.g.:

defendants_name_middle_initial is turned into defendants[0].name_middle_initial. This breaks if you have a question that looks for users[0].name_middle_initial. I think there are workarounds for that, but it's probably better to just let people decide to make them attributes if they want to, but not default to that.

The preview / final download screens should always show the PDF version of the attachment

PDF field has underscores replaced by spaces in logic

This Trello card:
https://trello.com/c/sV287tk1/90-209a-affidavit-for-filing-out-of-state-protective-order

What I did:

1. I run that PDF through the current wizard

What do I expect:

No spaces in docassemble variable names in the logic/questions

What happens

1. OUTOFSTATE PROTECTIVE ORDER appears in the logic controller block.

Interview generator: Unit is not included when you use address.on_one_line()

This is documented in Docassemble documentation for this method. We need to change our uses to always include unit explicitly.

address.on_one_line(include_unit=True)

Built-in fields need some more metadata to help generating review screens

Every built-in field needs a few placeholders:

1. A label
2. An editable variable name (e.g., users[0].name.first)
3. A display value (for showing current value in review screen. E.g., users[0].address.block() )

These will then be usable when we generate the placeholder review screen. We could also use to improve the logic list and the PDF attachment.

Remove arbitrary attribute feature. Create plain variable names.

Currently, we have pre-determined core words that the wizard identifies in the user-created PDF field names. Ex: plaintiff. We've determined some standard attributes for those core words that we account for, but we also transform any arbitrary characters associated with a core word into an arbitrary attribute for the core word. For example, the PDF field label plaintiff_has_children_yes is turned into plaintiffs[0].has_children_yes.

It's possible there would be a small future benefit to this, but in the meantime it's causing problems. If you have the time at some point, @nonprofittechy , I'd love to have a more detailed description of those problems. An example of the intrinsic name problem could especially be useful.

Instead of forcing arbitrary values, we should use the PDF field labels as they are.

Reduce the size of the interview_metadata block so it only outputs useful information

Combine this with the PDF scanner and run it as the first step in this tool

Add placeholder ids to each question that are meaningful. Use question text as default

Handle `attorney_bbo` for attorney

Basic questions asking for nonexistent `preferred_court` value

Basic questions is asking for a preferred_court value, but no value is given in any of the interview files, just in the meta data of the interview.

add code to set interview subtitle to user v other_party in interview order block

limit currency values to 0

When word_to_markdown adds a newline in the middle of a field label, wizard incorrectly adds \n to field name

For example, if the text were:

some 80 characters {{
some_var }}

cdc_mtd.docx

cdc_mtd.md.txt

Add support for Docx files to the interview generator

Add send_email() button on the placeholder download screen

Or move the download screen into basic-questions.yml

Create addendum placeholder code

Unhelpful validation message

When an invalid date is put in (like 11/31), the validation message says, 'you need to fill in this field'.

Some variable names that are not correctly handled

• _Form_14 (will be renamed to 646 Form 14)
• Form_14_s_647 (will be renamed to Form 14 s 647)
• plaintff1_address_street
• plaintff1_signature
• county_name (doubled)

Here's a file that exposes many bugs:
Tenant_Petition_Form_ready_for_upload.pdf

Move some "magic" code out of basic-questions and into the wizard output

This will make it easier for interview authors to reason about the code and what it is doing without clicking "source".

Look into how/where to insert a code block like this: addresses_to_search = [users[0].address]. In the interview order block, or standalone?

If it's in the interview_order block, it needs to be after users[0].address.address is defined.

Figure out to eliminate the need to have two attachment blocks with their own set of fields

Most promising idea so far:

make the attachment variable a DADict. It will have two keys: my_attachment['final'] and my_attachment['preview']. We can control whether certain fields are displayed by checking if i=="final". That will also work for fields like editable. We now only need one attachment block. Also: use fields code to make the attachment variable more concise. This means it's all Python, instead of Mako and regular text. We almost never use regular text.

other ideas:

• use the preview attachment as the final attachment's source, with pdf template file: code. just add in the signature field (this seems the most promising approach)
• use the Docassemble data block to create list of variables. reconsider it on the attachment block. add in the signature field just for the final attachment block. Problem: this won't work with skip undefined
• build the list of fields in a Python block. Copy the list, and add the attachment variable. Will this cause problems of timing for when list is built?
• Use an if statement for the signature field, keep the rest as a YAML list. Problem: we need to undefine the attachment variable to display the signature in the final version. This requires a very specific order of operations to keep working, but we use the attachment in a few different places and may not have that level of control over the timing for when it gets turned into a PDF, e.g.
• Keep the structure basically the same, but simplify to the version that doesn't use Mako tags so it's a little easier to read
• Send all of the interview variables to the PDF. Use a convention or make sure that all variables use the same name as in the PDF. Problem: PDFs can't handle duplicate variable names. So we already know that we'll need to handle variables like user's name in two places--it's not a 1-1 map.
• Use a hybrid approach: create a function that processes the list of variables "just in time" and removes the signature block.

Add default decimal places: 2 to PDF attachment blocks

automatically generate progress at a few intervals (count # of questions and divide)

query: do this in the code block or in the question modifier?

Create a cucumber test that verifies you can run the wizard through and download a working interview

Reference to nonexistent `courts[0].name`

Noticed that in the basic-questions file, it says it sets courts[0].name, but later asks for courts[0].

Variable name underscores get replaced with spaces unexpectedly.

Splitting up bugs from #17 so each can be dealt with appropriately.

With Tenant_Petition_Form_ready_for_upload.pdf these variable name changes have been observed:

1. _Form_14 should stay _Form_14 but becomes 646 Form 14
2. Form_14_s_647 should stay Form_14_s_647 but becomes Form 14 s 647

Let someone re-run the wizard with an uploaded YAML file that went through the wizard before

Support `choices` in the wizard with both `checkboxes` and `radio` datatypes

A natural way to do this is to show an area field and allow the user to enter the choices separated by newlines. perhaps also with display: value

We should handle mail_address as the user's mailing address

Make the logic block the only mandatory block

Old design made sense for umbrella interview--one mandatory block is much cleaner default.

Set the properties on the DAFile so that they persist across interview sessions?

I'm not sure if we need to do that or not

Identify radio button fields in PDFs

Detect radio buttons in PDF

default limit birthdates to dates in the past

Add metadata block to set form title

Create placeholder section labels. Include some basic ones we can predict, like user/other party/address

These can be linked to the placeholder review screen as well.

Time of day values are not validated to be time of day values

Generate a template review screen along with the wizard-generated YAML

Create a version of the PDF scanner that is more of a "preflight" check tool. Combine that with the wizard tool

This could replace the PDF scanner, but the use case for the wizard is slightly different for now. This should be more "green for go" or STOP--fix these issues before running the wizard.

Format all numbers in the attachment to have 2 decimal places, or none (probably none is a safe default)

Move the metadata title block **after** include statement so it overrides

Look over the API and check to see if any variables that an interview needs to define should be added to the wizard

• interview_short_title

See also all of the steps here to see what can be automated: https://trello.com/c/unTtUH4b

`plaintiffs` and `defendants` should show up as .gather() in the interview order block

Don't let people edit court name / court county in the auto-generated review screen

Improve the order of blocks in the generated YAML file

Replace "mark_unfilled_fields_empty" with use of `skip undefined`

Currently, we set every field to empty at the end of the code block before trying to assemble a PDF. We should experiment with the use of the skip undefined option instead, which should be much safer to use.

This may spark some discussion but I think the key to making this a safe choice is to explicitly trigger every screen/needed definition in the logic block; not rely on the attachment block to implicitly seek the definition of a variable.

In the wizard, limit the length of text fields to the width of their PDF field

The wizard doesn't handle a form without at least one field that isn't in basic-questions

Objects need to escape internal single quotes

or whatever will break a string value.

Some fields are doubled

Splitting up bugs from #17 so each can be dealt with appropriately.

With Tenant_Petition_Form_ready_for_upload.pdf this happens:

1. county_name should appear only once, but does appear twice

Variable transformed that should not be

Actual result:

      {'variable': 'attorney_of_record_address_on_one_line',
       'transformed_variable': 'attorneys[0].of_record_address_on_one_line',
      'field_type': 'text',
      'field_data_type': 'text',
      },

Expected result:

• attorney_of_record_address_on_one_line is not transformed at all