twisted-infra/lists.twistedmatrix.com

Deployment repository for lists.twistedmatrix.com

lists.twistedmatrix.com's Issues

some messages don't have a message ID, fix it during ingest

Traceback (most recent call last):
  File "/appenv/site-packages/twisted/internet/defer.py", line 651, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/appenv/site-packages/twisted/internet/defer.py", line 1355, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/appenv/site-packages/twisted/internet/defer.py", line 1297, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/appenv/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/appenv/site-packages/klein/storage/_sql.py", line 136, in sql
    result = yield callable(txn)
  File "/appenv/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = g.send(result)
  File "/appenv/site-packages/txlists/app.py", line 153, in do
    id=message['message-id'].decode('charmap'),
exceptions.AttributeError: 'NoneType' object has no attribute 'decode'

Make lists discoverable from the index page

Right now I think you have to read the source code to find a URL to load that has any content on it.

Put a list of lists with archive links on /

Recipients are identified incorrectly

Currently recipients are determined by looking at the To and CC header fields. While in most cases these fields do include the recipient address(es), they actually have nothing to do with mail routing, so this will fail if a mesage is BCCed or bounced, or otherwise generated with a mismatch here.

We should instead be looking at "recipient", but the Mailgun docs are a little fuzzy about how this field works (for example, the JSON version has "recipients" but the rfc822 version has "recipient"; can this field still hold multiple values?), so someone should probably look at some real stored messages to figure it out.

Document the dogfood deployment of this code

Since dogfood is the preferred testing methodology, make it obvious to everyone how to do that.

Point to the deployment, describe how it gets updated (sounds like every push to any branch gets deployed?), maybe mention some interactions that are supposed to work (and ... continue to work?).

make ingestion idempotent enough that it's safe to run it repeatedly

right now restarting the server and re-ingesting produces duplicate messages