yunojuno-archive/django-inbound-email

An inbound email handler for Django.

Django Inbound Email

This package now supports Python3 and Django1.11 or above. For previous version support please refer to the Python2 branch.

An inbound email handler for Django.

Current Status

We have a working implementation, with SendGrid, Mailgun and Mandrill backends. (Both SendGrid and Mandrill have been used in production environments.)

The test_app is deployed to Heroku and any emails sent to [email protected] will be parsed by a live version of the test_app.

This test app just does a bounceback - any incoming email to that address is sent back to the sender's address. This makes it easy to test 'real-world' email examples. Please feel free to send email to that address - and if you're using Outlook 97 on Windows XP, in China, then we'd really like to hear from you. You don't hear that much on Github.

NB The app deployed on Heroku currently sends the bouncebacks using the Mailtrap app, so you won't actually receive the email, but I can give limited access to contributors so that they can see them all. I'll truncate the Mailtrap inbox at irregular intervals.

What?

A Django app to make it easy to receive inbound emails from users via a hosted transactional email service (e.g. SendGrid, Postmark, Mandrill, etc.)

Contained within this project is the django app itself, together with a working Django project that uses the app, and is separately deployable to Heroku for testing purposes. The app has good test coverage, but it's really very hard to test inbound emails without having real data, and that requires a public endpoint that you can use to hook up your preferred email provider's webhooks.

Why?

If your project accepts inbound emails, you are probably using one of the big transactional email providers.

These services all provide a mechanism for receiving inbound emails which involves them (the service) parsing the inbound email and then posting the contents to an HTTP endpoint in your project. This is a great service, but it can often be fiddly to integrate into your app and it reinforces service lock-in, as each service's callback is slightly different.

There is also a significant SMTP-HTTP 'impedance mismatch'. You send emails through Django's (SMTP) mail library, which provides the EmailMessage and EmailMultiAlternative objects, but you receive emails as an HTTP POST. Wouldn't it be nice if you could both send and receive Django objects?

This app converts the incoming HttpRequest back into an EmailMultiAlternatives object, and fires a signal that sends both the new object, and the original request object. You simply have to listen for this signal, and process the email as your application requires.

The mail parser handles non-UTF-8 character sets (so those pesky PC Outlook emails don't come through all garbled), and file attachments.

How?

Although this is Django app, it contains (for now) no models. Its principle component is a single view function that does the parsing. There is a single configuration setting - INBOUND_EMAIL_PARSER, which must be one of the supported backends.

This setting is expected to be available to the app from django.conf.settings, and the app will raise an error if it does not exist.

The default URL for inbound emails is simply '/inbound/'.

The flow through the app is very simple:

1. The app view function receive_inbound_email is called when a new email POST is received from your service provider.
2. This function looks up the INBOUND_EMAIL_PARSER, loads the appropriate backend, and parses the request.POST contents out into a new django.core.mail.EmailMultiAlternatives object.
3. The email_received signal is fired, and the new EmailMultiAlternatives instance is passed, along with the original HttpRequest (in case there's any special handling that you require - e.g. DKIM / SPF info, if your provider passes that along).

If an email is unacceptable in some way (e.g. an attachment is too large), then the email_received_unacceptable signal is fired instead. This signal has an argument exception describing the problem.

Installation

For use as the app in Django project, use pip:

$ pip install django-inbound-email

For hacking on the project, pull from Git:

$ git pull [email protected]:yunojuno/django-inbound-email.git
$ cd django-inbound-email
django-inbound-email$
# use virtualenvwrapper, and install Django to allow tests to run
django-inbound-email$ mkvirtualenv django-inbound-email
(django-inbound-email) django-inbound-email$ pip install django

Usage

Your main concern, after installing and configuring the app, is handling the email_received signal:

# This snippet goes somewhere inside your project,
# wherever you need to react to incoming emails.
import logging
from inbound_email.signals import email_received

def on_email_received(sender, **kwargs):
    """Handle inbound emails."""
    email = kwargs.pop('email')
    request = kwargs.pop('request')

    # your code goes here - save the email, respond to it, etc.
    logging.debug(
        "New email received from %s: %s",
        email.from_email,
        email.subject
    )

# pass dispatch_uid to prevent duplicates:
# https://docs.djangoproject.com/en/dev/topics/signals/
email_received.connect(on_email_received, dispatch_uid="something_unique")

Handling file attachments as FileField properties

There is one gotcha in the handling of file attachments. The email object that is sent via the signal has an attachments property, but this contains a list of 3-tuples [(name, contents, content_type),], not a list of file objects. In order to store the attachments against a model as a FileField, you'll need to convert the tuples back into something that Django can deal with.

from django.core.files.uploadedfile import SimpleUploadedFile
from django.db import models

from inbound_email.signals import email_received


def get_file(attachment):
    """Convert email.attachment tuple into a SimpleUploadedFile."""
    name, content, content_type = attachment
    return SimpleUploadedFile(name, content, content_type)


class Example(models.Model):
    """Example model that contains a FileField property."""
    file = models.FileField()


def on_email_received(sender, **kwargs):
    """Handle inbound emails."""
    email = kwargs.pop('email')
    for attachment in email.attachments:
        # we must convert attachment tuple into a file
        # before adding as the property.
        example = Example(file=get_file(attachment))
        example.save()

Tests

There is a test django project, test_app that is used to run the tests.

(django-inbound-email) django-inbound-email$ python manage.py test

Configuration

• Install the app
• Add the app to INSTALLED_APPS
• Add INBOUND_EMAIL_PARSER setting
• Update your provider configuration to point to app URL

# the fully-qualified path to the provider's backend parser
INBOUND_EMAIL_PARSER = 'inbound_email.backends.sendgrid.SendGridRequestParser'

# if True (default=False) then log the contents of each inbound request
INBOUND_EMAIL_LOG_REQUESTS = True

# if True (default=True) then always return HTTP status of 200 (may be required by provider)
INBOUND_EMAIL_RESPONSE_200 = True

# add the app to Django's INSTALL_APPS setting
INSTALLED_APPS = (
    # other apps
    # ...
    'inbound_email',
)

Mandrill Features

If you wish to check the X-Mandrill-Signature header that Mandrill provides in the requests, you will need to set the INBOUND_MANDRILL_AUTHENTICATION_KEY setting to your Mandrill authentication key. When the key is supplied Inbound Email will check the signature supplied versus the one calculated from the request.

If signatures don't match, the system will send the signal email_received_unacceptable with the exception describing the problem.

Features

Things it will do:

• Parse HTTP requests into EmailMultiAlternatives objects
• Pluggable backends (SendGrid, Mailgun and Mandrill currently supported)
• Handle character encodings properly
• Handle attachments, including if they are too large

Things it (probably) won't do:

• Handle email reply parsing - use https://github.com/zapier/email-reply-parser

django-inbound-email's Issues

Support multiple encodings

Emails can have different encodings set on different fields, and these may well be different from the encoding used for the HTTP request - e.g. and email sent from Outlook running on Windows may have the 'to' and 'subject' fields encoded in 'windows-1252', which is then forwarded on via the backend provider in an HTTP request encoding with UTF-8. Different backends will deal with this in their own way, but as an example, SendGrid sends a 'charsets' value in the request, which is a JSON-formatted string that contains the specific encoding for each field. This can then be used by the backend to decode the values sent.

Imap / pop backend?

Hi,

I'm looking for a solution similar to this project but compatible with IMAP/POP instead of request-based.

Would a PR with IMAP & POP backends + management command (to check the inbox periodically) be of interest to you for this project or would you rather not deal with "regular" e-mail checking.

Cheers

Exceptions should be called Exceptions, not Errors?

Should we consider checking email address validity before handing email on?

Configurable option to reject emails where the from-address does not match an email-address regex (it'd have to include unicode chars). This would prevent email from deliberately fake senders (or accidentally misconfigured clients) making it into the project's data stores

This could be added to every backend, or everywhere via a base, perhaps

Incorrect env var used to set a setting

In https://github.com/yunojuno/django-inbound-email/blob/master/settings.py#L30

we see

INBOUND_MANDRILL_AUTHENTICATION_KEY = environ.get('INBOUND_EMAIL_PARSER')

which isn't the right env var name. This is made impossible to work around (eg set it to '' so it's ignored) because INBOUND_EMAIL_PARSER is used a few lines above.

The fix is to change L30 to

INBOUND_MANDRILL_AUTHENTICATION_KEY = environ.get('INBOUND_MANDRILL_AUTHENTICATION_KEY') and set that as an env var

Support for Mailgun

Self-explanatory - we need a mailgun.py backend. Docs here - http://documentation.mailgun.com/quickstart.html#receiving-and-parsing-email

Does this package support Python 3?

Thanks.

Adding a url pattern for Sendgrid Post

I am trying to use this package and have followed the setup guide but I am getting an
HTTP POST /inbound/ 404
My sendgrid url is set up as the default /inbound/.

Do I need to specifically add an url pattern in the urls.py file. If so how do I do it?
If not how should I solve this issue.

Tests fail on Django 1.9

Unable to import HttpResponse, because it's coming sneakily from the decorators module not from django.http

Will fix this along with #20

Support whitelist or blacklist for allowed attachment types

This would mean one could enforce similar consistency to form-based file uploads in one's app. In YunoJuno's specific example, we may want to limit conversation attachments to not be executables, for instance, and the reply-by-email feature for conversations should also enforce this too.

Thoughts on this welcome

Support for Postmark

As described in title. There is an inbound lib we could use as a dependency to make this easy

Mandrill backend blows up hard if settings.INBOUND_MANDRILL_AUTHENTICATION_KEY is missing

TBC whether that's a deliberate design decision or not (the example config implies it should be added, but the readme for the project makes it sound a bit optional.

A fix would simply be to change https://github.com/yunojuno/django-inbound-email/blob/master/inbound_email/backends/mandrill.py#L122

from

        if settings.INBOUND_MANDRILL_AUTHENTICATION_KEY:

to

        if hasattr(settings, 'INBOUND_MANDRILL_AUTHENTICATION_KEY'):

Messages with only HTML (and no plain text) content from Sendgrid are rejected

Hi there,

It's took me a bit of time to track down this problem with the package. In the sendgrid.py backend, text content is required, while HTML is optional:
text = _decode_POST_value(request, 'text')
html = _decode_POST_value(request, 'html', default='')

This means that any message with only HTML encoded text is rejected by the server. If text is changed to include a default, like this:
text = _decode_POST_value(request, 'text', default='')

Then any type of message can be received.

Not sure if it's easier for me to do a PR for this, or just make the quick change when you're doing other tests, but I'd be happy to. Not sure if this is an issue the other backends.

Cheers

Support Mandrill's Authentication

Mandrill supplies a header X-Mandrill-Signature to authenticate their requests. Inbound Email should be able to support it by adding one setting to determine the webhook url.

Mandrill's signature calculation is as follows:

1. Create a string with the webhook's URL, exactly as you entered it in Mandrill (including any query strings, if applicable). Mandrill always signs webhook requests with the exact URL you provided when you configured the webhook. A difference as small as including or removing a trailing slash will prevent the signature from validating.
2. Sort the request's POST variables alphabetically by key.
3. Append each POST variable's key and value to the URL string, with no delimiter.
4. Hash the resulting string with HMAC-SHA1, using your webhook's authentication key to generate a binary signature.
5. Base64 encode the binary signature
6. Compare the binary signature that you generated to the signature provided in the X-Mandrill-Signature HTTP header.

PR to follow.

Support for Mandrill

Self-explanatory - we need a mandrill.py backend - docs here - http://help.mandrill.com/entries/22092308-What-is-the-format-of-inbound-email-webhooks-

Support email attachments

This is the main missing feature - support for file attachments in emails. SendGrid supports this, and I assume most other backends would too.

Sender of signals should not be None?

Could use the class of the backend instance in use.

Shared (inherited) `clean_FOO` methods for common message fields

Each backend service has a unique implementation that extracts the relevant data as part of their implementation of the parse method. In reality, although each backend is unique, they are all extracting strings from an HTTP request object, and there are some shared rules around how to treat this data - e.g. the various email RFCs.

Proposed solution is to have a set of clean_FOO methods in the base RequestParser class that the the implementation can call that implement these basic rules.

e.g.

import email

class RequestParser()
    def clean_to(self, value):
        """Parse string and return list of (name, <address>) tuples."""
        return email.utils.getaddresses(value)

    def parse(self, request):
        raise NotImplementedError(u"Must be implemented by inheriting class.")

# and in the implementation
class MyRequestParser(RequestParser):
    def parse(self, request):
        to = self.clean_to(request.POST.get('recipient')
        [...]

Suggested methods:

• clean_sender - parse email address
• clean_recipients - parse email addresses
• clean_html_message - parse HTML, mark safe,
• clean_txt_message - parse text, convert to UTF-8 encoded unicode
• clean_subject - parse subject, convert to UTF-8 encoded unicode
• clean_cc_recipients - parse email addresses
• clean_bcc_recipients - parse email addresses
• clean_attachments - handle file type restrictions, max/min sizes etc.

Exceptions should be defined in their own module

Importing them from the root module of django_inbound_email.backends is unexpected.

Mandrill backend doesn't handle some email addresses correctly

Emails in this format

"Jalim, Steve" <[email protected]>

don't get parsed properly.

It's because "Jalim, Steve" loses its quotes when being parsed from the payload sent to the Mandrill webhook.

(PR to follow later)

Filename is not being passed through correctly.

The name that gets passed to the email.attach method is the request form element name, and not the actual name of the file. This means that we end up with attachments being saved that are called 'attachment1' instead of 'my_file.png'.