18f/2015-foia

Please check out https://github.com/18F/foia-hub/issues to track our work. This repo is for project wide discussion, blogging, and scratch space for 18F's FOIA modernization team.

2015-foia's Issues

An agency-facing Request Tracking API

We're highly likely to be sending emails to some agencies when requests come in. But we also want to provide the tools for integration that works better for both sides, especially for agencies who prefer or already use a non-email based system.

I had some ideas for this today, and so here's one proposal. There's 3 components to that I can think of, with some possible URLs for them (ignoring an /api/ prefix or subdomain or whatever):

• /requests - A GET here provides requests awaiting an agency. agency access to every single detail.
• /request/:id/status - A PUT here allows an agency to update the status of a request, by a unique identifier. This could be its original tracking number, or other valid identifiers (more on that next).
• /request/:id/tracking - A PUT here allows an agency to add additional tracking identifiers that a request can be referred to with. This allows agencies to manage a crosswalk between our tracking numbers and theirs, on their terms. This endpoint would return a 4XX code if the value was not unique. Whether the ID needs to be unique across all agencies, or across the submitting agency, would need to be decided.

Authentication is clearly needed here, the simplest solution being a shared secret for each integrating agency. How this ties into our set of contacts would need some implementation choices on our part.

API check-in

Hi folks. Just wanted to check in on a few questions. No hurry but appreciate any thoughts when you've a second. Thanks!

1. Are you following https://github.com/18f/api-standards/ ? (Assume so but just double checking)
2. Are you planning to integrate api.data.gov?
3. Are you planning to participate in an API usability session before your initial release?
4. Do you plan to stay with readthedocs.org for your api documentation? If not, are you planning to use github pages for them? If so, have you considered repurposing an existing model? https://bit.ly/govapidocs
5. When do you expect the FOIA APIs to be 'public'?

Make design previews and mockups easy to find

Right now the ReadMe points to a different "design" repo -- which is "retired" and points back to this repo!

It should be easy for any new user -- technical or non-technical -- to check out what the product/website looks like (or what the most recent alpha/beta looked like).

List of all agencies' FOIA libraries and materials

One doesn't exist right now, for US federal agencies. This is a pretty good discrete task that just about anyone could do: find them and put them in a spreadsheet.

Fields worth collecting for each agency (or other discrete FOIA publishing unit):

• URL of the library or libraries
• Whether an API or bulk data is officially provided (probably doesn't exist anywhere)
• Difficulty of scraping it (e.g. unofficial API <--> paginated URLs with permalinks <--> non-permalinked form navigation)
• How complete is the library?
• Updated automatically, or with careful discretion?
• Is metadata about the original request/response included alongside the document (and document-level metadata)?

Common FOIA Regulations

As part of 18F's work on FOIA, I thought it would be helpful to flag efforts to create common FOIA regulations. In other words, a single set of regulations that would govern how all agencies implement many aspects of FOIA.

The effort to develop model FOIA regulations was outlined in the administration's second national action plan (See item #3, bullet point #2). The Department of Justice's Office of Information policy is coordinating the multi-agency multi-stakeholder process.

A group of non-profit organizations that regularly litigate on FOIA have put together model common FOIA regulations, which hopefully will serve as a starting point for agency/stakeholder deliberations.

GSA missing statistics data

It's not in the data files, so it's not in the site. Not clear why it's being missed -- but we should figure out whether the root cause affects other data files too.

Create first draft of general diagram of components

Create general diagram of components as they relate to each other.

Discuss FOIA project deployment & create tickets to make it happen

We should talk about how we want to handle deployment for this project.
Ideas very very briefly discussed:

• Separate repo to deploy components to deploy all?
• Ansible? Salt?

/cc @konklone
b/c @NoahKunin might also be interested.

Figure out the data repo situation

We discussed this in passing, but we need to figure out the general setup of repo organization to make sure that the place where data is being referenced is consistent.

Example where structure will start to matter:
https://github.com/18F/foia-hub/blob/cfb88236eb57984845dbdc6747287029f9397489/foia_hub/management/commands/load_agency_contacts.py#L5

This is a quick dev discussion.

The latest mockup?

Hi hi! Heard in the meeting today there's a new mock. Would love to see it!

Create first round of tickets in repos

Create tickets from conversations in the following repos...
https://github.com/18F/foia-portal/issues
https://github.com/18F/foia-core/issues
https://github.com/18F/foia-search/issues

Milestone: MVP
Tags:

• 1_in_queue if being worked on now.
• 0_next_in_queue if queued up as one of the next items.

@konklone and I are working on this together.

Update numbering in all repos for ticket lables

1_in_queue --> 0_in_queue
0_next_in_queue --> 1_next_in_queue

Intro to GitHub

This will cover:

• signing up for a GitHub account,
• "Watch"-ing a GitHub project,
• beginning a new discussion thread on a GitHub project, and
• commenting on an existing discussion thread on the web and via email.

Signing up for a GitHub account

Visit https://github.com and fill out the signup form. Pick a "handle" (all lower case, avoid hyphens) and use your government email address.

You will be taken to a plan selection page -- just click the "Finish sign up" button to stick with the free plan.

"Watch"-ing a GitHub project

When you "Watch" a project on GitHub, you will receive email notifications for every new discussion or comment that appears in the project.

Visit https://github.com/18f/foia. The top of the page should show the name of the organization ("18F"), the name of the project ("foia"), and three buttons on the right: "Watch", "Star", and "Fork".

Click the "Watch" button, and then from the dropdown that appears, click "Watching".

Starting a new discussion

Visit https://github.com/18f/foia/issues to visit the list of open discussion threads.

(You can also get there by visiting https://github.com/18f/foia and clicking the "Issues" link on the righthand side.)

To open an issue, click the "New issue" button. This will take you to a form to create a new issue. Like an email, you will need to give it a title, and a body.

Click "Submit new issue" to create the discussion thread. This will send an email to all people who are "Watch"-ing the repository.

Commenting on an existing discussion

From the web: visit the issue you wish to comment on, and scroll to the bottom of the discussion. You can enter your comment in the empty form at the bottom of the discussion thread.

From your email: you can also respond to a discussion from your email inbox, by replying as you normally would. (The screenshot below shows Gmail, but you can reply to a discussion from Microsoft Outlook, or any other email program.)

I hope that's helpful in getting started with GitHub!

FOIA Url not working

The following url https://18f.gsa.gov/site/18F/18f.gsa.gov/tags/foia/
gives a 404 error

Add city/state FOIL portals

Worth including mentions of the FOIL trackers around.

Example: http://records.oaklandnet.com/

FOIA request intake API

I'd love community input, now and ongoing, on where we should take our FOIA request intake API should look.

Our current FOIA request intake API, as deployed, looks like this:

/api/request/

Accepts POSTs with POST bodies containing JSON like this:

{
  "first_name": "Eric",
  "last_name": "Mill",
  "email": "[email protected]",
  "body": "Body of the request",
  "documents_start": "September 9, 2014",
  "documents_end": "September 11, 2014",
  "fee_limit": 20,
  "agency_fields": {
    "region": "3A",
    "contract": "GS-12345"
  },
  "agency": "general-services-administration",
  "office": "headquarters"
}

Thoughts on the request:

• The schema is obviously going to be expanding as we add fields, and agency-specific fields. The schema will be more loosely defined for agency-specific fields.
• We need to get documentation up: what fields are required and optional, what type a field needs to be, and any constraints and enumerations on field values.
• agency_fields rubs me the wrong way aesthetically for an API - agency_specific or even just agency (where the current agency and office fields get moved into this object) would be better for API clients, I think.

For successful requests, it returns a response with a 201 Created status code, and a JSON body like this:

{
  "fee_limit": 0,
  "custom_fields": {
    "region": "3A",
    "contract": "GS-12345"
  },
  "date_end": "2014-09-11T00:00:00",
  "status": "O",
  "requester": 4,
  "tracking_id": 2,
  "date_start": "2014-09-09T00:00:00",
  "request_body": "Testing"
}

Thoughts on the success response:

• The response uses inconsistent field names from the request, which should be changed.
• Some of them: date_start, date_end, request_body, custom_fields, fee_limit, and even status (assuming it's always O for new requests) are not needed here.
• It's not obvious what the requester field is.
• Our tracking_id should be something besides a database autoincrement (and be a string instead of a number).

For an unsuccessful response, we return a 500 status code and a JSON body including a Django stack trace. We should update that to handle errors in a graceful, predictable way.

'NoneType' object has no attribute 'string'

I was trying to update the manual data when I got this error message. I reset from master, but the error persisted.

INFO:root:[Treasury] Already downloaded.
Traceback (most recent call last):
  File "scraper.py", line 391, in <module>
    save_agencies()
  File "scraper.py", line 360, in save_agencies
    save_agency(agency)
  File "scraper.py", line 338, in save_agency
    data = parse_agency(abb, BeautifulSoup(text))
  File "scraper.py", line 210, in parse_agency
    departments.append(parse_department(elem, dept_name))
  File "scraper.py", line 187, in parse_department
    for key, value in find_bold_fields(ps):
  File "scraper.py", line 144, in find_bold_fields
    value = strong.next_sibling.string
AttributeError: 'NoneType' object has no attribute 'string'

Common data model for FOIA requests?

I don't know if this is the venue or wildly out of scope, but I would love to see mature data modeling/standardization around FOIA data.

This would include public-facing data (requester, date of request, nature of request, etc.), process data (response time, exemptions invoked, etc.), and agency-facing data (user, search terms used, etc.).

Even having a public conversation about what types of FOIA data should be collected, maintained, would be a helpful start.