openoakland/disclosure-alert

Prototype of email alerts whenever campaign finance disclosures come in

openoakland

This is a Github page, so check out the gh-pages branch

disclosure-alert's Issues

Convert "Filed At" timestamp to Pacific Time

Define a helper method that is used in the email template, which converts the UTC filing.filed_at timestamp into the timezone for America/Los_Angeles.

Actually stop sending messages to people who are inactive past a certain point

Add email that goes out to users that are going inactive

If we stop sending to a user, let's give them a warning and a chance to stay subscribed.

Some 497 filings report "Contributions made" as "Contributions received"

e.g. https://netfile.com/Connect2/api/public/image/205218456

Links to candidate page and view contributions not working

From Karen Ivey - On 2 alerts just now I’ve tried to follow a link to OpenDisclosure and gotten a 404. One was a link to the Candidate page for Derreck Johnson and one was a link View Contributions to this committee for Oakland Youth Vote.

Links from email to committess with no contributions is 404

Some links from the alert email for committees that have filed a 410 Statement of Organization point to committee pages on the website that do not exist. It's unclear to me why the pages do not exist. Could it be because they don't have any contributions yet? Some committees without contributions do have pages on opendisclosure.io, such as Elias Ferran's https://www.opendisclosure.io/committee/1426949/. Is it because the committees are brand new, rather than carried over from a previous year? Marchon Tatmon has run before, but his committee page is 404.

This issue is not critical, but probably looks bad to users. The solution might not belong to this repo but instead to one of the Open Disclosure website repos.

Example from yesterday's alert email: https://www.opendisclosure.io/committee/1427385/, Faye Taylor City Council District 3

Do user testing with journalists

• What journalists work this beat?

Combining 496 forms should not sum together the same contribution to multiple candidates

For example,

This represents the combination of two 496 forms, presumably because the 496 form only allows you to say a contribution is in support of a single candidate. So - since this is probably a single expense in support of two candidates, the contributions should also be combined so it does not read "Total Independent Expenditures Made: $3,680" but rather "Total Independent Expenditures Made: $1,840"

Fix new layout "Other Forms" section

• add a 410 form so we can handle these forms appropriately
• fix labels for "Other Forms" section

Investigate pulling Lobbyist Quarterly Report PDF data

The Oakland PEC is updating their process for Lobbyist Disclosures to be a digital process. The filings in Netfile for the new process look like this:

https://netfile.com/Connect2/api/public/image/189594727

That document is a PDF with fillable fields containing the values for the person. The values can be extracted with the pdftk command like this:

» wget -O189594727.pdf https://netfile.com/Connect2/api/public/image/189594727
» pdftk 189594727.pdf dump_data_fields | grep FieldValue
FieldValue: Reynaldo A. Fuentes
FieldValue: The Partnership for Working Families
FieldValue: 1305 Franklin St Suite 501
FieldValue: Oakland, CA 94612
FieldValue: [phone]
FieldValue: [email protected]
FieldValue: East Bay Alliance for a Sustainable Economy
FieldValue: Emergency Paid Sick Leave
FieldValue: East Bay Alliance for a Sustainable Economy
FieldValue: Paid Sick Leave Enforcement
FieldValue: East Bay Alliance for a Sustainable Economy
FieldValue: Department of Workplace and Enforcement Standards
FieldValue: East Bay Alliance for a Sustainable Economy
FieldValue: Department of Workplace and Enforcement Standards
FieldValue: April 26, 2020
FieldValue: Choice2
FieldValue: Choice3
FieldValue: Support
FieldValueDefault:
FieldValue: Support
FieldValueDefault:
FieldValue: Policy Development
FieldValueDefault:
FieldValue: Informational Briefing
FieldValueDefault:
FieldValue:
FieldValueDefault:
FieldValue:
FieldValueDefault:

Is there an easy way to get this data out of the PDF with ruby? Maybe a gem that wraps PDFtk?

Use mailgun open tracking instead of Ahoy

Open tracking is key to understanding whether we should continue sending to a user. Since it's removed from Ahoy 2, let's reimplement via Mailgun.

• Add new model to replace Ahoy
• Let it run for a while
• Backfill old data
• Update admin metrics to use new table
• Remove Ahoy

Sort and group email filings in a way that puts more important stuff at the top

Rough thoughts on the hierarchy of importance:

1. 497 reports are most important
2. 460 is next
3. group all 410's into a tiny section
4. all other forms should be in an "other" section

Ideas for features that would be nice for candidates

• something that checks for excessive contributions
• when there are a lot of IE's (above threshhold) in a race, that means they can spend more (more than $25k in total, or $x,000 by a single entity)

Better design for filings from committees supporting/opposing multiple candidates

For example

https://netfile.com/Connect2/api/public/image/192739405
https://netfile.com/Connect2/api/public/image/192739536
https://netfile.com/Connect2/api/public/image/192739591

They're very similar filings, and I'm not sure what the best way to go about this is. Maybe we should combine all 496's for a given filer together?

e.g. merge this:

with this:

Improve formatting of Form 700 position titles

Here is an example of a Form 700 for a fire chief:

There are a couple issues:

1. No line-break before "Other Positions"
2. Large gap below the person's name, which ideally would be more closely related to their primary position
3. To me the bullet point distracts from the "No interests reported on any schedule" which is the real important bit. We should de-emphasize the other position title by moving it out of a bulleted list.

The idea of this issue is to make the form 700 look like this:

---

Robert Lipp
Battalion Chief, City of Oakland
Assistant Fire Chief, City of Oakland

No reportable interests on any schedule

[View original filing]

---

Rewrite in JavaScript

Prob makes sense to move away from Ruby given the team's strengths.

Upgrade Rails 5.2 -> 6.1

May need to upgrade to 6.0 first? Ideally upgrade all the way to Rails 7, but in that case 6.1 will definitely be an intermediate step.

Upgrade ahoy_email to the 2.x branch

It changes its DSL away from track.

Consolidate multiple amendments on the same day together

In the 2/21/20 email there were multiple amendments from the same committee. This creates some noise for users, and for the user they probably just want to know in comparison to the previous filing.

Email readability

Improve email styling:

• Make it easier to see the form number
• Add some description to the header/footer (create a visual identity)

Add confirmation email to subscription process

By sending the user a confirmation email in order confirm their subscription:

• We will make sure there aren't invalid emails on our list
• We will teach the user what emails from our system look like
• We will give Gmail an opportunity to realize that these emails are important to the user who opens the confirmation email.

Upgrade Ruby 2.7.6 -> 3.0.4

Add ability to receive email weekly instead of daily

• Add column to AlertSubscriber that tracks the last filed_at date that the user has seen
• Update sending logic to send according to that date
• Add ability to change this via the link in the email footer (in addition to unsubscribe)

Perhaps this should automatically be enabled for users between elections.

Show amended return differences with the originals

Since 3b372eb we have been downloading the original filings as well as the amended ones. Let's show what was changed in the amended filings.

e.g. for these ones, what changed (2/21/20 email)?

Add support for SF

I live in SF now and want to follow the election here.

Here's how to I'll plan to add support:

• Remove hardcoding of "COAK" Netfile agency and replace with a table that stores agencies
• Create database columns to keep track of which subscribers are subscribed to which Netfile agencies
• Create database columns to keep track of which filings came from which Netfile agencies
• Update alert mailer to only show subscribers the filings from their agency
• Download SFO and COAK agencies nightly
• Subscribe myself to it for some time
• Remove OpenDisclosure links when viewing a non-COAK agency
• Update copy for anything that is Oakland specific
• Release to a small set of beta users
• Create a new way to subscribe via the homepage
• Add support for custom SF form types

Future work:

• Unified subscription between the two agencies?

Keep track of which filings have been sent via email to send any missed ones the next day

Upgrade heroku stack from heroku-18 -> heroku-22

Before May 1, 2023.

Better design for Form 460

We currently list only Total Contributions Received and Total Expenditures Made. What more could we do here?