Amazon SES

Anymail integrates with Amazon Simple Email Service (SES) using the Boto 3 AWS SDK for Python, and includes sending, tracking, and inbound receiving capabilities.

New in version 2.1.

Installation

You must ensure the boto3 package is installed to use Anymail’s Amazon SES backend. Either include the “amazon_ses” option when you install Anymail:

$ pip install django-anymail[amazon_ses]

or separately run pip install boto3.

To send mail with Anymail’s Amazon SES backend, set:

EMAIL_BACKEND = "anymail.backends.amazon_ses.EmailBackend"

in your settings.py.

In addition, you must make sure boto3 is configured with AWS credentials having the necessary IAM permissions. There are several ways to do this; see Credentials in the Boto docs for options. Usually, an IAM role for EC2 instances, standard Boto environment variables, or a shared AWS credentials file will be appropriate. For more complex cases, use Anymail’s AMAZON_SES_CLIENT_PARAMS setting to customize the Boto session.

Limitations and quirks

Hard throttling
Like most ESPs, Amazon SES throttles sending for new customers. But unlike most ESPs, SES does not queue and slowly release throttled messages. Instead, it hard-fails the send API call. A strategy for retrying errors is required with any ESP; you’re likely to run into it right away with Amazon SES.
Tags limitations
Amazon SES’s handling for tags is a bit different from other ESPs. Anymail tries to provide a useful, portable default behavior for its tags feature. See Tags and metadata below for more information and additional options.
No merge_metadata
Amazon SES’s batch sending API does not support the custom headers Anymail uses for metadata, so Anymail’s merge_metadata feature is not available. (See Tags and metadata below for more information.)
Open and click tracking overrides
Anymail’s track_opens and track_clicks are not supported. Although Amazon SES does support open and click tracking, it doesn’t offer a simple mechanism to override the settings for individual messages. If you need this feature, provide a custom ConfigurationSetName in Anymail’s esp_extra.
No delayed sending
Amazon SES does not support send_at.
No global send defaults for non-Anymail options
With the Amazon SES backend, Anymail’s global send defaults are only supported for Anymail’s added message options (like metadata and esp_extra), not for standard EmailMessage attributes like bcc or from_email.
Arbitrary alternative parts allowed
Amazon SES is one of the few ESPs that does support sending arbitrary alternative message parts (beyond just a single text/plain and text/html part).
Spoofed To header and multiple From emails allowed
Amazon SES is one of the few ESPs that supports spoofing the To header (see Additional headers) and supplying multiple addresses in a message’s from_email. (Most ISPs consider these to be very strong spam signals, and using either them will almost certainly prevent delivery of your mail.)
Template limitations
Messages sent with templates have a number of additional limitations, such as not supporting attachments. See Batch sending/merge and ESP templates below.

Tags and metadata

Amazon SES provides two mechanisms for associating additional data with sent messages, which Anymail uses to implement its tags and metadata features:

  • SES Message Tags can be used for filtering or segmenting CloudWatch metrics and dashboards, and are available to Kinesis Firehose streams. (See “How do message tags work?” in the Amazon blog post Introducing Sending Metrics.)

    By default, Anymail does not use SES Message Tags. They have strict limitations on characters allowed, and are not consistently available to tracking webhooks. (They may be included in SES Event Publishing but not SES Notifications.)

  • Custom Email Headers are available to all SNS notifications (webhooks), but not to CloudWatch or Kinesis.

    These are ordinary extension headers included in the sent message (and visible to recipients who view the full headers). There are no restrictions on characters allowed.

By default, Anymail uses only custom email headers. A message’s metadata is sent JSON-encoded in a custom X-Metadata header, and a message’s tags are sent in custom X-Tag headers. Both are available in Anymail’s tracking webhooks.

Because Anymail tags are often used for segmenting reports, Anymail has an option to easily send an Anymail tag as an SES Message Tag that can be used in CloudWatch. Set the Anymail setting AMAZON_SES_MESSAGE_TAG_NAME to the name of an SES Message Tag whose value will be the single Anymail tag on the message. For example, with this setting:

ANYMAIL = {
    ...
    "AMAZON_SES_MESSAGE_TAG_NAME": "Type",
}

this send will appear in CloudWatch with the SES Message Tag "Type": "Marketing":

message = EmailMessage(...)
message.tags = ["Marketing"]
message.send()

Anymail’s AMAZON_SES_MESSAGE_TAG_NAME setting is disabled by default. If you use it, then only a single tag is supported, and both the tag and the name must be limited to alphanumeric, hyphen, and underscore characters.

For more complex use cases, set the SES Tags parameter directly in Anymail’s esp_extra. See the example below. (Because custom headers do not work with SES’s SendBulkTemplatedEmail call, esp_extra Tags is the only way to attach data to SES messages also using Anymail’s template_id and merge_data features, and the merge_metadata cannot be supported.)

esp_extra support

To use Amazon SES features not directly supported by Anymail, you can set a message’s esp_extra to a dict that will be merged into the params for the SendRawEmail or SendBulkTemplatedEmail SES API call.

Example:

message.esp_extra = {
    # Override AMAZON_SES_CONFIGURATION_SET_NAME for this message
    'ConfigurationSetName': 'NoOpenOrClickTrackingConfigSet',
    # Authorize a custom sender
    'SourceArn': 'arn:aws:ses:us-east-1:123456789012:identity/example.com',
    # Set Amazon SES Message Tags
    'Tags': [
        # (Names and values must be A-Z a-z 0-9 - and _ only)
        {'Name': 'UserID', 'Value': str(user_id)},
        {'Name': 'TestVariation', 'Value': 'Subject-Emoji-Trial-A'},
    ],
}

(You can also set "esp_extra" in Anymail’s global send defaults to apply it to all messages.)

Batch sending/merge and ESP templates

Amazon SES offers ESP stored templates and batch sending with per-recipient merge data. See Amazon’s Sending personalized email guide for more information.

When you set a message’s template_id to the name of one of your SES templates, Anymail will use the SES SendBulkTemplatedEmail call to send template messages personalized with data from Anymail’s normalized merge_data and merge_global_data message attributes.

message = EmailMessage(
    from_email="[email protected]",
    # you must omit subject and body (or set to None) with Amazon SES templates
    to=["[email protected]", "Bob <[email protected]>"]
)
message.template_id = "MyTemplateName"  # Amazon SES TemplateName
message.merge_data = {
    '[email protected]': {'name': "Alice", 'order_no': "12345"},
    '[email protected]': {'name': "Bob", 'order_no': "54321"},
}
message.merge_global_data = {
    'ship_date': "May 15",
}

Amazon’s templated email APIs don’t support several features available for regular email. When template_id is used:

  • Attachments are not supported
  • Extra headers are not supported
  • Overriding the template’s subject or body is not supported
  • Anymail’s metadata is not supported
  • Anymail’s tags are only supported with the AMAZON_SES_MESSAGE_TAG_NAME setting; only a single tag is allowed, and the tag is not directly available to webhooks. (See Tags and metadata above.)

Status tracking webhooks

Anymail can provide normalized status tracking notifications for messages sent through Amazon SES. SES offers two (confusingly) similar kinds of tracking, and Anymail supports both:

  • SES Notifications include delivered, bounced, and complained (spam) Anymail event_types. (Enabling these notifications may allow you to disable SES “email feedback forwarding.”)
  • SES Event Publishing also includes delivered, bounced and complained events, as well as sent, rejected, opened, clicked, and (template rendering) failed.

Both types of tracking events are delivered to Anymail’s webhook URL through Amazon Simple Notification Service (SNS) subscriptions.

Amazon’s naming here can be really confusing. We’ll try to be clear about “SES Notifications” vs. “SES Event Publishing” as the two different kinds of SES tracking events. And then distinguish all of that from “SNS”—the publish/subscribe service used to notify Anymail’s tracking webhooks about both kinds of SES tracking event.

To use Anymail’s status tracking webhooks with Amazon SES:

  1. First, configure Anymail webhooks and deploy your Django project. (Deploying allows Anymail to confirm the SNS subscription for you in step 3.)

Then in Amazon’s Simple Notification Service console:

  1. Create an SNS Topic to receive Amazon SES tracking events. The exact topic name is up to you; choose something meaningful like SES_Tracking_Events.

  2. Subscribe Anymail’s tracking webhook to the SNS Topic you just created. In the SNS console, click into the topic from step 2, then click the “Create subscription” button. For protocol choose HTTPS. For endpoint enter:

    https://random:random@yoursite.example.com/anymail/amazon_ses/tracking/

    Anymail will automatically confirm the SNS subscription. (For other options, see Confirming SNS subscriptions below.)

Finally, switch to Amazon’s Simple Email Service console:

  1. If you want to use SES Notifications: Follow Amazon’s guide to configure SES notifications through SNS, using the SNS Topic you created above. Choose any event types you want to receive. Be sure to choose “Include original headers” if you need access to Anymail’s metadata or tags in your webhook handlers.

  2. If you want to use SES Event Publishing:

    1. Follow Amazon’s guide to create an SES “Configuration Set”. Name it something meaningful, like TrackingConfigSet.

    2. Follow Amazon’s guide to add an SNS event destination for SES event publishing, using the SNS Topic you created above. Choose any event types you want to receive.

    3. Update your Anymail settings to send using this Configuration Set by default:

      ANYMAIL = {
          ...
          "AMAZON_SES_CONFIGURATION_SET_NAME": "TrackingConfigSet",
      }
      

Caution

The delivery, bounce, and complaint event types are available in both SES Notifications and SES Event Publishing. If you’re using both, don’t enable the same events in both places, or you’ll receive duplicate notifications with different event_ids.

Note that Amazon SES’s open and click tracking does not distinguish individual recipients. If you send a single message to multiple recipients, Anymail will call your tracking handler with the “opened” or “clicked” event for every original recipient of the message, including all to, cc and bcc addresses. (Amazon recommends avoiding multiple recipients with SES.)

In your tracking signal receiver, the normalized AnymailTrackingEvent’s esp_event will be set to the the parsed, top-level JSON event object from SES: either SES Notification contents or SES Event Publishing contents. (The two formats are nearly identical.) You can use this to obtain SES Message Tags (see Tags and metadata) from SES Event Publishing notifications:

from anymail.signals import tracking
from django.dispatch import receiver

@receiver(tracking)  # add weak=False if inside some other function/class
def handle_tracking(sender, event, esp_name, **kwargs):
    if esp_name == "Amazon SES":
        try:
            message_tags = {
                name: values[0]
                for name, values in event.esp_event["mail"]["tags"].items()}
        except KeyError:
            message_tags = None  # SES Notification (not Event Publishing) event
        print("Message %s to %s event %s: Message Tags %r" % (
              event.message_id, event.recipient, event.event_type, message_tags))

Anymail does not currently check SNS signature verification, because Amazon has not released a standard way to do that in Python. Instead, Anymail relies on your WEBHOOK_SECRET to verify SNS notifications are from an authorized source.

Note

Amazon SNS’s default policy for handling HTTPS notification failures is to retry three times, 20 seconds apart, and then drop the notification. That means if your webhook is ever offline for more than one minute, you may miss events.

For most uses, it probably makes sense to configure an SNS retry policy with more attempts over a longer period. E.g., 20 retries ranging from 5 seconds minimum to 600 seconds (5 minutes) maximum delay between attempts, with geometric backoff.

Also, SNS does not guarantee notifications will be delivered to HTTPS subscribers like Anymail webhooks. The longest SNS will ever keep retrying is one hour total. If you need retries longer than that, or guaranteed delivery, you may need to implement your own queuing mechanism with something like Celery or directly on Amazon Simple Queue Service (SQS).

Inbound webhook

You can receive email through Amazon SES with Anymail’s normalized inbound handling. See Receiving email with Amazon SES for background.

Configuring Anymail’s inbound webhook for Amazon SES is similar to installing the tracking webhook. You must use a different SNS Topic for inbound.

To use Anymail’s inbound webhook with Amazon SES:

  1. First, if you haven’t already, configure Anymail webhooks and deploy your Django project. (Deploying allows Anymail to confirm the SNS subscription for you in step 3.)

  2. Create an SNS Topic to receive Amazon SES inbound events. The exact topic name is up to you; choose something meaningful like SES_Inbound_Events. (If you are also using Anymail’s tracking events, this must be a different SNS Topic.)

  3. Subscribe Anymail’s inbound webhook to the SNS Topic you just created. In the SNS console, click into the topic from step 2, then click the “Create subscription” button. For protocol choose HTTPS. For endpoint enter:

    https://random:random@yoursite.example.com/anymail/amazon_ses/inbound/

    Anymail will automatically confirm the SNS subscription. (For other options, see Confirming SNS subscriptions below.)

  4. Next, follow Amazon’s guide to Setting up Amazon SES email receiving. There are several steps. Come back here when you get to “Action Options” in the last step, “Creating Receipt Rules.”

  5. Anymail supports two SES receipt actions: S3 and SNS. (Both actually use SNS.) You can choose either one: the SNS action is easier to set up, but the S3 action allows you to receive larger messages and can be more robust. (You can change at any time, but don’t use both simultaneously.)

    • For the SNS action: choose the SNS Topic you created in step 2. Anymail will handle either Base64 or UTF-8 encoding; use Base64 if you’re not sure.
    • For the S3 action: choose or create any S3 bucket that Boto will be able to read. (See IAM permissions; don’t use a world-readable bucket!) “Object key prefix” is optional. Anymail does not currently support the “Encrypt message” option. Finally, choose the SNS Topic you created in step 2.

Amazon SES will likely deliver a test message to your Anymail inbound handler immediately after you complete the last step.

If you are using the S3 receipt action, note that Anymail does not delete the S3 object. You can delete it from your code after successful processing, or set up S3 bucket policies to automatically delete older messages. In your inbound handler, you can retrieve the S3 object key by prepending the “object key prefix” (if any) from your receipt rule to Anymail’s event.event_id.

Amazon SNS imposes a 15 second limit on all notifications. This includes time to download the message (if you are using the S3 receipt action) and any processing in your signal receiver. If the total takes longer, SNS will consider the notification failed and will make several repeat attempts. To avoid problems, it’s essential any lengthy operations are offloaded to a background task.

Amazon SNS’s default retry policy times out after one minute of failed notifications. If your webhook is ever unreachable for more than a minute, you may miss inbound mail. You’ll probably want to adjust your SNS topic settings to reduce the chances of that. See the note about retry policies in the tracking webhooks discussion above.

In your inbound signal receiver, the normalized AnymailTrackingEvent’s esp_event will be set to the the parsed, top-level JSON object described in SES Email Receiving contents.

Confirming SNS subscriptions

Amazon SNS requires HTTPS endpoints (webhooks) to confirm they actually want to subscribe to an SNS Topic. See Sending SNS messages to HTTPS endpoints in the Amazon SNS docs for more information.

(This has nothing to do with verifying email identities in Amazon SES, and is not related to email recipients confirming subscriptions to your content.)

Anymail will automatically handle SNS endpoint confirmation for you, for both tracking and inbound webhooks, if both:

  1. You have deployed your Django project with Anymail webhooks enabled and an Anymail WEBHOOK_SECRET set, before subscribing the SNS Topic to the webhook URL.

    (If you subscribed the SNS topic too early, you can re-send the confirmation request later from the Subscriptions section of the Amazon SNS dashboard.)

  2. The SNS endpoint URL includes the correct Anymail WEBHOOK_SECRET as HTTP basic authentication. (Amazon SNS only allows this with https urls, not plain http.)

    Anymail requires a valid secret to ensure the subscription request is coming from you, not some other AWS user.

If you do not want Anymail to automatically confirm SNS subscriptions for its webhook URLs, set AMAZON_SES_AUTO_CONFIRM_SNS_SUBSCRIPTIONS to False in your ANYMAIL settings.

When auto-confirmation is disabled (or if Anymail receives an unexpected confirmation request), it will raise an AnymailWebhookValidationFailure, which should show up in your Django error logging. The error message will include the Token you can use to manually confirm the subscription in the Amazon SNS dashboard or through the SNS API.

Settings

Additional Anymail settings for use with Amazon SES:

AMAZON_SES_CLIENT_PARAMS

Optional. Additional client parameters Anymail should use to create the boto3 session client. Example:

ANYMAIL = {
    ...
    "AMAZON_SES_CLIENT_PARAMS": {
        # example: override normal Boto credentials specifically for Anymail
        "aws_access_key_id": os.getenv("AWS_ACCESS_KEY_FOR_ANYMAIL_SES"),
        "aws_secret_access_key": os.getenv("AWS_SECRET_KEY_FOR_ANYMAIL_SES"),
        "region_name": "us-west-2",
        # override other default options
        "config": {
            "connect_timeout": 30,
            "read_timeout": 30,
        }
    },
}

In most cases, it’s better to let Boto obtain its own credentials through one of its other mechanisms: an IAM role for EC2 instances, standard AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables, or a shared AWS credentials file.

AMAZON_SES_SESSION_PARAMS

Optional. Additional session parameters Anymail should use to create the boto3 Session. Example:

ANYMAIL = {
    ...
    "AMAZON_SES_SESSION_PARAMS": {
        "profile_name": "anymail-testing",
    },
}

AMAZON_SES_CONFIGURATION_SET_NAME

Optional. The name of an Amazon SES Configuration Set Anymail should use when sending messages. The default is to send without any Configuration Set. Note that a Configuration Set is required to receive SES Event Publishing tracking events. See Status tracking webhooks above.

You can override this for individual messages with esp_extra.

AMAZON_SES_MESSAGE_TAG_NAME

Optional, default None. The name of an Amazon SES “Message Tag” whose value is set from a message’s Anymail tags. See Tags and metadata above.

AMAZON_SES_AUTO_CONFIRM_SNS_SUBSCRIPTIONS

Optional boolean, default True. Set to False to prevent Anymail webhooks from automatically accepting Amazon SNS subscription confirmation requests. See Confirming SNS subscriptions above.

IAM permissions

Anymail requires IAM permissions that will allow it to use these actions:

  • To send mail:
    • Ordinary (non-templated) sends: ses:SendRawEmail
    • Template/merge sends: ses:SendBulkTemplatedEmail
  • To automatically confirm webhook SNS subscriptions: sns:ConfirmSubscription
  • For status tracking webhooks: no special permissions
  • To receive inbound mail:
    • With an “SNS action” receipt rule: no special permissions
    • With an “S3 action” receipt rule: s3:GetObject on the S3 bucket and prefix used (or S3 Access Control List read access for inbound messages in that bucket)

This IAM policy covers all of those:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["ses:SendRawEmail", "ses:SendBulkTemplatedEmail"],
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": ["sns:ConfirmSubscription"],
    "Resource": ["arn:aws:sns:*:*:*"]
  }, {
    "Effect": "Allow",
    "Action": ["s3:GetObject"],
    "Resource": ["arn:aws:s3:::MY-PRIVATE-BUCKET-NAME/MY-INBOUND-PREFIX/*"]
  }]
}

Following the principle of least privilege, you should omit permissions for any features you aren’t using, and you may want to add additional restrictions:

  • For Amazon SES sending, you can add conditions to restrict senders, recipients, times, or other properties. See Amazon’s Controlling access to Amazon SES guide.

  • For auto-confirming webhooks, you might limit the resource to SNS topics owned by your AWS account, and/or specific topic names or patterns. E.g., "arn:aws:sns:*:0000000000000000:SES_*_Events" (replacing the zeroes with your numeric AWS account id). See Amazon’s guide to Amazon SNS ARNs.

  • For inbound S3 delivery, there are multiple ways to control S3 access and data retention. See Amazon’s Managing access permissions to your Amazon S3 resources. (And obviously, you should never store incoming emails to a public bucket!)

    Also, you may need to grant Amazon SES (but not Anymail) permission to write to your inbound bucket. See Amazon’s Giving permissions to Amazon SES for email receiving.

  • For all operations, you can limit source IP, allowable times, user agent, and more. (Requests from Anymail will include “django-anymail/version” along with Boto’s user-agent.) See Amazon’s guide to IAM condition context keys.