mysociety/alaveteli

Provide a Freedom of Information request system for your jurisdiction

alaveteli's Issues

Alerts are sent for old events, if the event status changes

A user reported that she had received an email alerting her that a response had been made to a request for which she has an alert set up, even though the response was actually sent in September last year (i.e. nine months ago).

On investigation, what happened is:

• The status of this request was wrongly set (it was set to waiting_response, and should have been not_held).
• An admin corrected the status, which updated the last_described_at field on the response event.
• This triggered a new alert to be sent the user (who had an alert set up for the relevant public body).

The proposed fix is to use the created_at field rather than the last_described_at field to trigger alerts, with the exception of alerts that query on the status which should use the last_described_at field as at present, to pick up events whose status has been changed so they now match the query.

Ability to make private requests

We're often asked about the ability to make requests in private, by journalists or companies.

This could be a premium, for-money feature, partly for revenue, but also to discourage private requests.

Requests could remain private permanently, or could have an embargo date on them after which we pester the user to put them public.

Some notes on the wiki at https://github.com/sebbacon/alaveteli/wiki/Embargo-requests-feature-requirements

Obviously, this feature requires much more discussion...

Payment could be handled with http://www.activemerchant.org/

Allow users to take copy of response

Sometimes, especially if a response involves a number of attachments, or is oddly formatted it would be useful if the requester (or perhaps anyone) could obtain a copy of the raw response.

One option would be a button allowing a requestor to send themselves a copy of the reply by email. The potential problem with this might be requestors replying to that message and unintentionally then taking the correspondence off-site; there would also need to be caution to avoid exposing the request addresses.

Another option would be a download of a .zip archive containing the correspondence text and any attachments.

Ability to Sort Requests

A user has requested the ability to sort their Whatdotheyknow FOI requests by status, so all Successful replies are grouped together and same for Long overdue.

This kind of thing might be possible already with carefully crafted search terms; but ought be a single click via a link/button.

Anything done for an individual's own requests could presumably also be done on the authority page.

Handle bounce messages from alerts

We could copy the "bounce" semantics from software such as mailman. When a user is considered to be hard bouncing, disable alerts to that user. Also alert them when they next log in that their email address is bouncing.

BCCd responses not matched to right request

Examples of incoming mails that aren't automatically redirected to the right request:

1. 1 WDTK request email address, 1 non-WDTK address.
   https://www.whatdotheyknow.com/admin/raw_emails/156083

To: Andrew Greenhalgh <[email protected]>, FOI <[email protected]>

2. request blind-copied (BCC:) to request. This example was intended to be BCC'ed to 7 diff't requests...
   https://www.whatdotheyknow.com/admin/raw_emails/153576

I'll add more in the comments as they arrive

allow unsubscribe from request emails

There is currently no way for a user that submits a request to unsubscribe from emails relating to that request.

"I do see that the emails such as "You have a new response",
"Was the response any good", "Clarify your request" and "Delayed
response" do not have any way of unsubscribing - is this deliberate? If
someone wants to come to the site manually to see if their requests have
updated and not get any emails, should they not be able to? Certainly we
need to provide an opt-out facility for every type of email the site
sends to comply with Hotmail's anti-spam guidelines, amongst others."

Ask people for annotation immediately after they submit request

To improve request metadata (and involve user more), ask for any comments and additional information about their request immediately after they submit a new request (or possibly at the same time?)

Special annotations - for links to articles based on responses, summaries of correspondence, or admin notes

Administrators could have an additional check-box when they make an annotation; to check if the annotation is being made in an official capacity as an administrator of the site.

This could
i/ Highlight the annotation eg. via a different background colour
ii/ Cause an alert box to appear at the top of the request thread, and on HTML conversions, and if possible on downloaded documents, suggesting the reader looks at the annotations.

Administrator annotations can be important eg. drawing attention to known inaccuracies in material released, noting where and why redaction by administrators has occurred, or explaining inconsistencies in the timeline - if admin intervention has occurred.

Display Should Reflect History for Name Change for a Body

Currently when a body changes its name the requests from both before and after the change carry the new name of the body.

Ideally requests / correspondence from before the change ought be associated with the old name; and those after the change with the new.

Allow Users to Access Auto-Redacted Email Addresses in Responses

Emails are redacted from responses to prevent harvesting by spammers.

These could be revealed to all users who prove they're human via a CAPTCHA.

Currently there are regular support requests to the WhatDoTheyKnow asking for such addresses to be revealed.

Better content discovery through cross-linking

We should expose tags to end users (#2302), consider allowing them to add tags themselves, and use them to cross-reference possibly related requests.

Tags can also be automatically generated (for english, at least) - see https://github.com/sebbacon/alaveteli/issues/19

We can also consider cross-linking authority names where they appear in the text body, and could perhaps also geotag entries based on OpenCalais city / county data.

Address perception that website is adversarial

There is a concern (from FOI officers) that the site fosters an "us vs. them" attitude which is not helpful.

It has been said that requests via whatdotheyknow tend to be less courteous that those received by other means (which means almost exclusively email).

This issue needs further research with FOI officers.

The alert-tracks script should have a daemon mode

The alert-tracks script sends out any daily email reminders that are due. The way it works (currently) is that reminders become due 24 hours after the previous one was sent. But because site usage has peaks at particular times of day, the script takes longer to run at some times of day than others. This makes it awkward to run from cron, because we don’t want to have more than one instance running at a time, but it is hard to predict how long it’s going to take.

One solution would be for the script to have an option to run in daemon mode, where it processes due alerts continually, and sleeps – either with an exponential back-off, or just until the next email is due – when there are no more to process.

Bulk Requests

Feature allowing a user to make requests to a set of public bodies.

Needs careful consideration; perhaps needs to be developed along with a system for collecting responses (Allowing hidden links to Google Forms / Survey Monkey or similar).

Protection against mis-use would be required.

Better alphabetisation

At the moment, if the name of a body starts with "The", it gets filed under /body/list/t, which is unhelpful. It would be better to ignore a leading "The". Something more configurable might be better, but I can't think of any other obviously ignorable prefixes.

poor rendering of some incoming emails

Incoming emails from authorities are often multipart/alternative, and we display the text part. Often the HTML part is much more readable because of the extra formatting. Usually the writers of the emails don't even know or think about the text part and so it can be very hard to read.

Directly inlining the HTML might be problematic because it could play havoc with the layout etc of the request thread, but it would be good to use it somehow.

quoted phrase search no longer working

Search box: Quotes from phrase search stripped out, and search does boolean AND search instead.

Steps to reproduce:

1. homepage
2. enter following into search box, including quotes, and submit: "safe and effective public health measure"
   (should match 1 FOI request)

Actual results:
3. search URL returned: http://www.whatdotheyknow.com/search/safe%20and%20effective%20public%20health%20measure
=> 163 results

Expected results:
3. http://www.whatdotheyknow.com/search/%22safe%20and%20effective%20public%20health%20measure%22
=> 1 result

Spurious line-breaks in requests typed in on iPhone

A user has let us know that when using an iphone with an external keypad to send FoI requests there are undesired breaks in the lines of text appear when displayed on WhatDoTheyKnow

An example appears to be :

http://www.whatdotheyknow.com/request/implementation_of_the_30_mile_ru

Allow requesters to see spam-protected email addresses

We currently hide email addresses in responses, but often these are important, and the requester needs to email the team to find them out. We should display these if the person viewing the response is the original requester. (And possibly under other circumstances too: e.g. if you've earned the rights, somehow, but that's for another day. Let's make the easy fix first.)

The update-xapian-index job is noisy

The update-xapian-index job spits out a variety of error messages that seem to be related to the processing of attachments, specifically PDF files. There is nothing we can do about these, since we don’t control the PDF files, so there should at least be an option to suppress such messages when the thing is run from cron.

Search Result Text When One User Only Found

Currently the search results page states:

"People 1 to 1 of 1 for [Searchterm] "

It has been brought to our attention that this is pretty much nonsense and could be written better.

Alerts sometimes appear to be several weeks late

Alerts are only sent out to users when a request state changes. This means that a request that got a reply several weeks ago will only turn into an email alert when someone marks it as "successful" (or whatever).

For alerts which ask for all updates from a particular body or a particular request, we should change the logic to alert on a new incoming request, rather than a state change.

Notes from Francis:

For queries which include the status, it obviously can't do that.

Right now the alert engine uses a search query internally. Very few
alerts though are free text so there are two options:

a) For certain kinds of alerts (like on a particular public body)
special case it to use the created at rather than described at.
Continue to use described at for all free text search alerts.

b) Do as in a), but also somehow parse the free text alert.
Maybe via a tree you get back from Xapian. Use that to see if it
is referring to the status of the request, in which case use
described at rather than created at.

Might need care as to how the rest of the alerts are structured in how
they decide what to alert on and used described vs. created.

This fits in a bit with how you want to do a permission system as
well.

In the current code, I was deliberately driving lots of things through
search, which is weird to people who love relational database, but to
my mind is more general (you always need full text search in the end)
and leaves less complex code. My plan was to do permissions via the
search as well.

However, for other reasons, you might want to do more things via
database queries. But keep in mind that it might make the code more
complex when you add search features or permissions.

Add OpenCalais-generated metadata to requests

See https://github.com/sebbacon/alaveteli/wiki/automate-metadata-with-OpenCalais

Add dynamic linkage to machine tags

WTDK uses machine tags like ch:_, where _ is replaced by the company number, which is eight characters long and consists of optional upper-case letters followed by digits. They are listed at http://foiwiki.com/foiwiki/index.php/WhatDoTheyKnow.com_Tags

There should be an admin-configurable way of setting the target of machine tags so they can link through to related online resources automatically.

Privileges for Public Authority Users

Users with an email address domain matching that of the public authority domain (or users to which we've manually assigned the privilege level) ought get an improved user experience and access to more features.

(Careful to avoid giving excess privileges to those on webmail / ISP domains / those on council/gov subdomains)

Subsets of this issue are:

#37 - Allow authorities to update status
#40 - Resend request button for authorities
#231 - Allow authorities to edit their request email address
#237 - Extend information we hold about public bodies

Users often complain there's no submit button

We regularly get users writing to us saying they've entered text either as a request or a reply but can't see how to send it.

Presumably they are not realising they have to "preview" first.

"My Requests" link on the NavBar

The "My Requests" page has now become more of a user page.

Often we get asked where users should go to change their account details; unsubscribe from alerts, etc.

I suggest changing the text; perhaps to "My Account" or "My User Page" or something along those lines.

Better Confirmation Request Has Been Sent

After a request has been sent it takes a few minutes for it to appear in search results on the site, or on a users' "my requests" page.

This results in users wondering if it has got lost (and writing to us to ask) or in some cases making the same request again.

Ideas:

• A warning that the request might not appear in searches / on the user page for a while.
• Fixing the slow search indexing problem (even if just for the relevant user and body pages)
• Providing users a link to their new request; perhaps in a flash message; perhaps even by email (for their records)

Censor Rules Cannot be Multi-Line

Currently a multi-line censor rule doesn't work.

This means administrators often have to create tens of single-line censor rules to censor large blocks of text. This can be time consuming and annoying.

Allow authorities to update request status

We are often contacted by authorities about responses miscategorised by users. In most cases the authorities are correct. We should offer authorities a direct mechanism to reclassify requests if they choose to.

I suggest it defaults to off and we enable it on a case-by-case basis, with the implicit understanding that we would withdraw it if the authority misused it.

We may need some mechanism to deal with "categorisation fights", though those can in principle happen now between users and administrators and I'm not aware of any.

Show clearly how to make follow-up to body's main FOI address

Users regularly want to send a "reply" not to the person who sent the last message in the thread, but to the main FOI email address for the public body.

eg. When the reply is an auto-responder from someone who's moved on / is away

This is currently possible, by clicking "send follow-up" at the bottom of the original request message; however this behaviour isn't made clear to users. I suggest an additional link at the bottom of the page explicitly offering the option to reply to the main FOI email address for the public body.

Sorting this out and making clear to users how this can be done would save 2-4 support requests per week, and no-doubt help more users who don't get in touch with us.

Occasional blank pages

I just (22 Dec 2010) hit preview on an internal review request and got a blank page; thought it might be worth reporting that as while I've not experienced it before and we do get a fair number of people reporting things along those lines.

http://www.whatdotheyknow.com/request/54822

Time 21:56 ish

Allow search for requests made to a tagged set of public authorities

The tag search function only returns records with variety:authority. You should be able to search for requests made to a set of authorities that have a specified tag, allowing alerts or a feed to be set up for requests made to bodies in your local area. eg:

http://www.whatdotheyknow.com/search/tag:islington%20variety:sent

Ability to attach files to requests

Use case: a company that works with data on empty properties / rateable values etc makes FOI requests where they send an Excel spreadsheet to a council containing the details of the properties they're interested in and the council fills in the blanks.

Improve document conversion

See https://github.com/sebbacon/alaveteli/wiki/Improved-document-conversion

Covers:

• nicer-looking conversions
• OCR of scanned documents
• thumbnails in request threads
• annotations of source documents

Not urgent but would make things look and feel smoother

Print Style Sheet / Print to PDF

A user has requested an ability to print (and/or create a PDF) of an FOI request without the extraneous content in the side bar (related requests etc.).

Perhaps related to the other ways users wish to "take a copy" of a response as requested at:
https://github.com/sebbacon/alaveteli/issues#issue/11

Correcting a typo in a URL ought not to break links to the uncorrected one

Sometimes a word is mistyped in the title of a request, and if we correct it the URL changes, potentially breaking any links to the mistyped version.

"Make Request" Link on Front Page

On the front page of WhatDoTheyKnow.com the "Make Request" link points to the front page; the effect is that clicking it does nothing.

Given the number of users who ask us how to make a request I think this might be a problem.

I'm not sure what the solution is, but some ideas:

• Removing this link from the front page.
• Showing something on mouseover maybe some text, maybe an arrow to the instructions for making a request.

Public Authority Users - Resend Request Button

Users with a public authority email address ought be able to request a re-send of any message themselves without contacting the support team.

Unable to re-withdraw requests

I had a request marked "withdrawn", which the public body then responded to; as a normal user my options for categorising didn't let me put it back in the status of "withdrawn".

Outstanding Actions Pending Alert Notification on All Pages

Currently users get a list of their outstanding actions (uncategorised requests) when they try and make a new request or if they visit their user page.

When users have outstanding actions pending there should be a note to this effect on all pages.

(something along the lines of a flash message at the top saying "you have outstanding actions" and offering a linking to the user page where the details will be.)

Address performance problems

Some ideas here [https://github.com/sebbacon/alaveteli/wiki/Performance-issues]

Email address checker/censor doesn't check FOI subject

Email from user:

I have just made a request that included an email address in both the subject line and the body of the request: http://www.whatdotheyknow.com/request/response_times_for_enquiries_to

While I (correctly) got a warning for the email address in the body of the request, there was no warning for the email address in the subject. While this isn't a problem for me, I just thought that I would alert you to it in case you think that it is something that you need to include in your checks.

.....

Might be worth extending the check & on-screen censoring, in order to avoid spam harvesters.

improve workflow for dealing with holding pen emails

I think this is quite an important area to improve to reduce to overhead of running an alavateli site.

I'm making one issue rather than several different issues for the different points because I think it's best to have a single place to discuss the UI in general. Might be worth spinning off separate tickets for actual implementation if it gets that far.

• The heuristics for guessing the request could be improved. One possibility would be to consider the hash against all existing requests - since they number in the tens of thousands that should be feasible via recomputation or a cache. That would catch cases where the id has been corrupted but the hash is intact, which are quite common.
• The guessed request should have a button next to it to just accept that guess. Right now you have to cut and paste to another box.
• If the request text associated with the guessed request was available inline via an expander or similar then it would be easier to check that it's really the right request.

Show request history on followup form

When following links such as "I'm about to send clarification", a
form appears into which the reply can be typed. However, the
previous correspondence in that thread is not shown.

The solution is something like:
Make it so when you make followups the whole request is shown on the page.
Remove all show_response URLs, and replace with a special version of the
request URL with a new input box at the bottom and a hash link to it

Include full text of requests & responses in RSS

Provide a version of the RSS feeds to Include the full text of requests with links to the attachments

500 error for non-existent pages

The site returns a 500 error, rather than a 404, for non-existent paths.

More Admin Links

On a request page, for super users, the only admin link is to the request admin page.

It would be useful to have direct admin links to the requestor's user admin page and the relevant body's admin page as well.

This would regularly save a click / page load for many admin transactions.

Moderation - prevent new annotations being made to requests

Requests which attract lots of comments can descend into chaos (eg OT comments, trolling, flame wars etc), so it wold be nice to have a "stop new annotations" admin function to prevent new comments from being added.

• admin Edit Request - allow annotations y/n dropdown / radio
• request screen - display suitable message saying eg "no more comments allowed on this request"
• request screen - remove Add Annotation link to prevent user from adding new annotations.
• request screen - (visible to admins only) new link "close request for new annotations" | "open request for new annotations"