BCCd responses not matched to right request

[mysociety]

BCCd responses not matched to right request

[skenaja]

another bcc mail: https://www.whatdotheyknow.com/admin/raw_emails/189024

[hsenag]

It's not clear why this one wasn't redirected. Only odd thing I can see about it is a slightly malformed To: line (starting '<' character): https://www.whatdotheyknow.com/admin/raw_emails/194685

[bjh21]

For Bccs, my suspicion is that Alaveteli doesn't get to see the envelope destination address at all, and is relying on parsing the "To" header instead. This is obviously wrong, and it would be better if it got to see the envelope destination and used that instead, since that's much more likely to reflect the intended destination of the message. This may require some reconfiguration of the MTA, particularly since the envelope destination seems to get lost when messages are passed from majestic to wildfire.

[RichardTaylor]

Another one with an errant "<" in the To line:

https://www.whatdotheyknow.com/admin/raw_emails/226021

[hsenag]

This message seems to use a weird charset in the To: line:

https://www.whatdotheyknow.com/admin/raw_emails/235526

[hsenag]

Another bcc mail: https://www.whatdotheyknow.com/admin/raw_emails/356158

Problems like those in this thread, where correctly addressed messages are nonetheless not delivered, are more serious than where the authority has got the address wrong, because the requester can end up being wrongly told that the authority has missed the deadline etc.

[hsenag]

A weirdly malformed to line: https://www.whatdotheyknow.com/admin/raw_emails/365506

To: "request-148954-xxxxxxxx@"
<whatdotheyknow.com [email protected]>

[I've replaced the actual hash, which was correct, with xxxxxxxx]

[hsenag]

Just dealt with another BCC'd mail that from the content could easily have been sent to multiple WDTK addresses. The only sign of the target was in the received headers and I'm not sure if I'd have been able to get hold of all the intended recipients if it had been sent to multiple recipients.

http://www.whatdotheyknow.com/request/commissioning_92#incoming-366281

Just to echo Ben's point above about the envelope information being the One True Way. I guess the relatively low frequency of problems means it's not that urgent, but I feel uncomfortable with having this weakness in the site whereby entirely valid email from the authority can be silently lost.

[hsenag]

Neath Port Talbot council seem to have adopted a standard response to FOI requests that is always BCC'd - seen a few of them lately. Still just a minor irritant though.

[hsenag]

Just been hit with a BCCed email that is at least somewhat important to deliver to the right requests (only) and I can't easily deduce them.

[hsenag]

Looks like DECC may be sending BCC'd auto-acknowledgements: https://www.whatdotheyknow.com/request/renewable_energy_dilemma#incoming-469519

[hsenag]

Seeing BCC'd acknowledgements from Land Registry and Merseyside Police

[crowbot]

I've renamed this issue to be specifically about BCC'd responses.

[dracos]

Exim on our servers is putting the envelope destination address in the "Envelope-To" header of the email. It would be possible to use that regardless of whether To/Cc/Bcc was used by the sender, if not for the fact that the email first goes via the redirect router which loses the local part suffix. I have altered our exim to add an "X-Delivered-Suffix" header to the email at this point, which can then be read to reliably have the fetched suffix. I couldn't see any other easy way to carry the suffix through the redirect lookup from request-*@wdtk to foi@internalserver.

[hsenag]

@dracos: Thanks a lot for this - it is very useful even without alaveteli interpreting the header, as we can look at the new header in the holding pen.

One issue - I just tried this with a test message delivered to multiple recipients and only the first envelope recipient ends up in the header. See https://www.whatdotheyknow.com/admin/raw_emails/556879 for the test.

[dracos]

Hmm, yes, exim considers that as the redirect (internal) recipient is the same for both BCCed addresses that it only needs to deliver it once - http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_redirect_router.html#SECTdupaddr - and so only one copy goes through the redirect router (and so has the header added). The only total solution I can see is to not have these addresses going via the redirect router, as I can't see a way to handle this case with it, but handled directly in some way; but perhaps someone with greater exim knowledge can see another way out.

[hsenag]

Another authority sending BCC'd acknowledgments: https://www.whatdotheyknow.com/body/peterborough_and_stamford_hospitals_nhs_foundation_trust

One incentive for fixing this properly is that we could consider turning off the holding pen and just bouncing all incoming misaddressed messages, which should lead to the senders checking the spelling of the request address.

Would something like 'batch_max' help with the single copy problem? (http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_pipe_transport.html) I haven't looked at exactly how the delivery from exim to alaveteli works so that may be a red herring.

Is there a "right" design for software that really needs to handle its own mail delivery? CC @Flupsy

[olineham]

Have had this issue in NZ too - authority emailed "To:" themselves and BCC'd several requests (and probably media - was a high profile story). Therefore there was no request information at all to work with in the headers. We're using postfix. I will look to get X-Envelope-To prepended to make this easier in future (I had to read our postfix log to work out the requests) but would be good to have Alaveteli use this header itself.

[garethrees]

I'm going to close this now.

The feedback so far is that we don't get Bcc responses that often, and the additional check in the holding pen seems like a good idea. #4529 makes it easier to identify multiple Bcc recipients.

We're explicitly not auto-delivering Bcc responses. It wouldn't be too difficult to do so if we wanted to, but it doesn't seem like its a big enough problem to spend time on.

[garethrees]

I've just merged #4529 which extracts a bunch of headers so that its easier to spot these:

This will be out on WDTK soon.