Amazon SES¶
Anymail integrates with Amazon Simple Email Service (SES) using the Boto 3 AWS SDK for Python, and includes sending, tracking, and inbound receiving capabilities.
New in version 2.1.
Installation¶
You must ensure the boto3 package is installed to use Anymail’s Amazon SES backend. Either include the “amazon_ses” option when you install Anymail:
$ pip install django-anymail[amazon_ses]
or separately run pip install boto3
.
To send mail with Anymail’s Amazon SES backend, set:
EMAIL_BACKEND = "anymail.backends.amazon_ses.EmailBackend"
in your settings.py.
In addition, you must make sure boto3 is configured with AWS credentials having the
necessary IAM permissions.
There are several ways to do this; see Credentials in the Boto docs for options.
Usually, an IAM role for EC2 instances, standard Boto environment variables,
or a shared AWS credentials file will be appropriate. For more complex cases,
use Anymail’s AMAZON_SES_CLIENT_PARAMS
setting to customize the Boto session.
Limitations and quirks¶
- Hard throttling
- Like most ESPs, Amazon SES throttles sending for new customers. But unlike most ESPs, SES does not queue and slowly release throttled messages. Instead, it hard-fails the send API call. A strategy for retrying errors is required with any ESP; you’re likely to run into it right away with Amazon SES.
- Tags limitations
- Amazon SES’s handling for tags is a bit different from other ESPs.
Anymail tries to provide a useful, portable default behavior for its
tags
feature. See Tags and metadata below for more information and additional options. - Open and click tracking overrides
- Anymail’s
track_opens
andtrack_clicks
are not supported. Although Amazon SES does support open and click tracking, it doesn’t offer a simple mechanism to override the settings for individual messages. If you need this feature, provide a custom ConfigurationSetName in Anymail’s esp_extra. - No delayed sending
- Amazon SES does not support
send_at
. - No global send defaults for non-Anymail options
- With the Amazon SES backend, Anymail’s global send defaults
are only supported for Anymail’s added message options (like
metadata
andesp_extra
), not for standard EmailMessage attributes likebcc
orfrom_email
. - Arbitrary alternative parts allowed
- Amazon SES is one of the few ESPs that does support sending arbitrary alternative message parts (beyond just a single text/plain and text/html part).
- Spoofed To header and multiple From emails allowed
- Amazon SES is one of the few ESPs that supports spoofing the To header
(see Additional headers) and supplying multiple addresses in a message’s
from_email
. (Most ISPs consider these to be very strong spam signals, and using either them will almost certainly prevent delivery of your mail.) - Template limitations
- Messages sent with templates have a number of additional limitations, such as not supporting attachments. See Batch sending/merge and ESP templates below.
Tags and metadata¶
Amazon SES provides two mechanisms for associating additional data with sent messages,
which Anymail uses to implement its tags
and metadata
features:
SES Message Tags can be used for filtering or segmenting CloudWatch metrics and dashboards, and are available to Kinesis Firehose streams. (See “How do message tags work?” in the Amazon blog post Introducing Sending Metrics.)
By default, Anymail does not use SES Message Tags. They have strict limitations on characters allowed, and are not consistently available to tracking webhooks. (They may be included in SES Event Publishing but not SES Notifications.)
Custom Email Headers are available to all SNS notifications (webhooks), but not to CloudWatch or Kinesis.
These are ordinary extension headers included in the sent message (and visible to recipients who view the full headers). There are no restrictions on characters allowed.
By default, Anymail uses only custom email headers. A message’s
metadata
is sent JSON-encoded in a custom
X-Metadata header, and a message’s tags
are sent in custom X-Tag headers. Both are available in Anymail’s
tracking webhooks.
Because Anymail tags
are often used for
segmenting reports, Anymail has an option to easily send an Anymail tag
as an SES Message Tag that can be used in CloudWatch. Set the Anymail setting
AMAZON_SES_MESSAGE_TAG_NAME
to the name of an SES Message Tag whose value will be the single Anymail tag
on the message. For example, with this setting:
ANYMAIL = { ... "AMAZON_SES_MESSAGE_TAG_NAME": "Type", }
this send will appear in CloudWatch with the SES Message Tag "Type": "Marketing"
:
message = EmailMessage(...) message.tags = ["Marketing"] message.send()
Anymail’s AMAZON_SES_MESSAGE_TAG_NAME
setting is disabled by default. If you use it, then only a single tag is supported,
and both the tag and the name must be limited to alphanumeric, hyphen, and underscore
characters.
For more complex use cases, set the SES Tags
parameter directly in Anymail’s
esp_extra. See the example below. (Because custom headers do not
work with SES’s SendBulkTemplatedEmail call, esp_extra Tags is the only way to attach
data to SES messages also using Anymail’s template_id
and merge_data
features.)
esp_extra support¶
To use Amazon SES features not directly supported by Anymail, you can
set a message’s esp_extra
to
a dict
that will be merged into the params for the SendRawEmail
or SendBulkTemplatedEmail SES API call.
Example:
message.esp_extra = { # Override AMAZON_SES_CONFIGURATION_SET_NAME for this message 'ConfigurationSetName': 'NoOpenOrClickTrackingConfigSet', # Authorize a custom sender 'SourceArn': 'arn:aws:ses:us-east-1:123456789012:identity/example.com', # Set Amazon SES Message Tags 'Tags': [ # (Names and values must be A-Z a-z 0-9 - and _ only) {'Name': 'UserID', 'Value': str(user_id)}, {'Name': 'TestVariation', 'Value': 'Subject-Emoji-Trial-A'}, ], }
(You can also set "esp_extra"
in Anymail’s global send defaults
to apply it to all messages.)
Batch sending/merge and ESP templates¶
Amazon SES offers ESP stored templates and batch sending with per-recipient merge data. See Amazon’s Sending personalized email guide for more information.
When you set a message’s template_id
to the name of one of your SES templates, Anymail will use the SES
SendBulkTemplatedEmail call to send template messages personalized with data
from Anymail’s normalized merge_data
and merge_global_data
message attributes.
message = EmailMessage( from_email="[email protected]", # you must omit subject and body (or set to None) with Amazon SES templates to=["[email protected]", "Bob <[email protected]>"] ) message.template_id = "MyTemplateName" # Amazon SES TemplateName message.merge_data = { '[email protected]': {'name': "Alice", 'order_no': "12345"}, '[email protected]': {'name': "Bob", 'order_no': "54321"}, } message.merge_global_data = { 'ship_date': "May 15", }
Amazon’s templated email APIs don’t support several features available for regular email.
When template_id
is used:
- Attachments are not supported
- Extra headers are not supported
- Overriding the template’s subject or body is not supported
- Anymail’s
metadata
is not supported - Anymail’s
tags
are only supported with theAMAZON_SES_MESSAGE_TAG_NAME
setting; only a single tag is allowed, and the tag is not directly available to webhooks. (See Tags and metadata above.)
Status tracking webhooks¶
Anymail can provide normalized status tracking notifications for messages sent through Amazon SES. SES offers two (confusingly) similar kinds of tracking, and Anymail supports both:
- SES Notifications include delivered, bounced, and complained (spam) Anymail
event_type
s. (Enabling these notifications may allow you to disable SES “email feedback forwarding.”) - SES Event Publishing also includes delivered, bounced and complained events, as well as sent, rejected, opened, clicked, and (template rendering) failed.
Both types of tracking events are delivered to Anymail’s webhook URL through Amazon Simple Notification Service (SNS) subscriptions.
Amazon’s naming here can be really confusing. We’ll try to be clear about “SES Notifications” vs. “SES Event Publishing” as the two different kinds of SES tracking events. And then distinguish all of that from “SNS”—the publish/subscribe service used to notify Anymail’s tracking webhooks about both kinds of SES tracking event.
To use Anymail’s status tracking webhooks with Amazon SES:
- First, configure Anymail webhooks and deploy your Django project. (Deploying allows Anymail to confirm the SNS subscription for you in step 3.)
Then in Amazon’s Simple Notification Service console:
Create an SNS Topic to receive Amazon SES tracking events. The exact topic name is up to you; choose something meaningful like SES_Tracking_Events.
Subscribe Anymail’s tracking webhook to the SNS Topic you just created. In the SNS console, click into the topic from step 2, then click the “Create subscription” button. For protocol choose HTTPS. For endpoint enter:
https://random:random@yoursite.example.com/anymail/amazon_ses/tracking/
- random:random is an
ANYMAIL_WEBHOOK_SECRET
shared secret - yoursite.example.com is your Django site
Anymail will automatically confirm the SNS subscription. (For other options, see Confirming SNS subscriptions below.)
- random:random is an
Finally, switch to Amazon’s Simple Email Service console:
If you want to use SES Notifications: Follow Amazon’s guide to configure SES notifications through SNS, using the SNS Topic you created above. Choose any event types you want to receive. Be sure to choose “Include original headers” if you need access to Anymail’s
metadata
ortags
in your webhook handlers.If you want to use SES Event Publishing:
Follow Amazon’s guide to create an SES “Configuration Set”. Name it something meaningful, like TrackingConfigSet.
Follow Amazon’s guide to add an SNS event destination for SES event publishing, using the SNS Topic you created above. Choose any event types you want to receive.
Update your Anymail settings to send using this Configuration Set by default:
ANYMAIL = { ... "AMAZON_SES_CONFIGURATION_SET_NAME": "TrackingConfigSet", }
Caution
The delivery, bounce, and complaint event types are available in both SES Notifications
and SES Event Publishing. If you’re using both, don’t enable the same events in both
places, or you’ll receive duplicate notifications with different
event_id
s.
Note that Amazon SES’s open and click tracking does not distinguish individual recipients. If you send a single message to multiple recipients, Anymail will call your tracking handler with the “opened” or “clicked” event for every original recipient of the message, including all to, cc and bcc addresses. (Amazon recommends avoiding multiple recipients with SES.)
In your tracking signal receiver, the normalized AnymailTrackingEvent’s
esp_event
will be set to the
the parsed, top-level JSON event object from SES: either SES Notification contents
or SES Event Publishing contents. (The two formats are nearly identical.)
You can use this to obtain SES Message Tags (see Tags and metadata) from
SES Event Publishing notifications:
from anymail.signals import tracking
from django.dispatch import receiver
@receiver(tracking) # add weak=False if inside some other function/class
def handle_tracking(sender, event, esp_name, **kwargs):
if esp_name == "Amazon SES":
try:
message_tags = {
name: values[0]
for name, values in event.esp_event["mail"]["tags"].items()}
except KeyError:
message_tags = None # SES Notification (not Event Publishing) event
print("Message %s to %s event %s: Message Tags %r" % (
event.message_id, event.recipient, event.event_type, message_tags))
Anymail does not currently check SNS signature verification, because Amazon has not
released a standard way to do that in Python. Instead, Anymail relies on your
WEBHOOK_SECRET
to verify SNS notifications are from an
authorized source.
Note
Amazon SNS’s default policy for handling HTTPS notification failures is to retry three times, 20 seconds apart, and then drop the notification. That means if your webhook is ever offline for more than one minute, you may miss events.
For most uses, it probably makes sense to configure an SNS retry policy with more attempts over a longer period. E.g., 20 retries ranging from 5 seconds minimum to 600 seconds (5 minutes) maximum delay between attempts, with geometric backoff.
Also, SNS does not guarantee notifications will be delivered to HTTPS subscribers like Anymail webhooks. The longest SNS will ever keep retrying is one hour total. If you need retries longer than that, or guaranteed delivery, you may need to implement your own queuing mechanism with something like Celery or directly on Amazon Simple Queue Service (SQS).
Inbound webhook¶
You can receive email through Amazon SES with Anymail’s normalized inbound handling. See Receiving email with Amazon SES for background.
Configuring Anymail’s inbound webhook for Amazon SES is similar to installing the tracking webhook. You must use a different SNS Topic for inbound.
To use Anymail’s inbound webhook with Amazon SES:
First, if you haven’t already, configure Anymail webhooks and deploy your Django project. (Deploying allows Anymail to confirm the SNS subscription for you in step 3.)
Create an SNS Topic to receive Amazon SES inbound events. The exact topic name is up to you; choose something meaningful like SES_Inbound_Events. (If you are also using Anymail’s tracking events, this must be a different SNS Topic.)
Subscribe Anymail’s inbound webhook to the SNS Topic you just created. In the SNS console, click into the topic from step 2, then click the “Create subscription” button. For protocol choose HTTPS. For endpoint enter:
https://random:random@yoursite.example.com/anymail/amazon_ses/inbound/
- random:random is an
ANYMAIL_WEBHOOK_SECRET
shared secret - yoursite.example.com is your Django site
Anymail will automatically confirm the SNS subscription. (For other options, see Confirming SNS subscriptions below.)
- random:random is an
Next, follow Amazon’s guide to Setting up Amazon SES email receiving. There are several steps. Come back here when you get to “Action Options” in the last step, “Creating Receipt Rules.”
Anymail supports two SES receipt actions: S3 and SNS. (Both actually use SNS.) You can choose either one: the SNS action is easier to set up, but the S3 action allows you to receive larger messages and can be more robust. (You can change at any time, but don’t use both simultaneously.)
- For the SNS action: choose the SNS Topic you created in step 2. Anymail will handle either Base64 or UTF-8 encoding; use Base64 if you’re not sure.
- For the S3 action: choose or create any S3 bucket that Boto will be able to read. (See IAM permissions; don’t use a world-readable bucket!) “Object key prefix” is optional. Anymail does not currently support the “Encrypt message” option. Finally, choose the SNS Topic you created in step 2.
Amazon SES will likely deliver a test message to your Anymail inbound handler immediately after you complete the last step.
If you are using the S3 receipt action, note that Anymail does not delete the S3 object.
You can delete it from your code after successful processing, or set up S3 bucket policies
to automatically delete older messages. In your inbound handler, you can retrieve the S3
object key by prepending the “object key prefix” (if any) from your receipt rule to Anymail’s
event.event_id
.
Amazon SNS imposes a 15 second limit on all notifications. This includes time to download the message (if you are using the S3 receipt action) and any processing in your signal receiver. If the total takes longer, SNS will consider the notification failed and will make several repeat attempts. To avoid problems, it’s essential any lengthy operations are offloaded to a background task.
Amazon SNS’s default retry policy times out after one minute of failed notifications. If your webhook is ever unreachable for more than a minute, you may miss inbound mail. You’ll probably want to adjust your SNS topic settings to reduce the chances of that. See the note about retry policies in the tracking webhooks discussion above.
In your inbound signal receiver, the normalized AnymailTrackingEvent’s
esp_event
will be set to the
the parsed, top-level JSON object described in SES Email Receiving contents.
Confirming SNS subscriptions¶
Amazon SNS requires HTTPS endpoints (webhooks) to confirm they actually want to subscribe to an SNS Topic. See Sending SNS messages to HTTPS endpoints in the Amazon SNS docs for more information.
(This has nothing to do with verifying email identities in Amazon SES, and is not related to email recipients confirming subscriptions to your content.)
Anymail will automatically handle SNS endpoint confirmation for you, for both tracking and inbound webhooks, if both:
You have deployed your Django project with Anymail webhooks enabled and an Anymail
WEBHOOK_SECRET
set, before subscribing the SNS Topic to the webhook URL.(If you subscribed the SNS topic too early, you can re-send the confirmation request later from the Subscriptions section of the Amazon SNS dashboard.)
The SNS endpoint URL includes the correct Anymail
WEBHOOK_SECRET
as HTTP basic authentication. (Amazon SNS only allows this with https urls, not plain http.)Anymail requires a valid secret to ensure the subscription request is coming from you, not some other AWS user.
If you do not want Anymail to automatically confirm SNS subscriptions for its webhook URLs, set
AMAZON_SES_AUTO_CONFIRM_SNS_SUBSCRIPTIONS
to False
in your ANYMAIL settings.
When auto-confirmation is disabled (or if Anymail receives an unexpected confirmation request),
it will raise an AnymailWebhookValidationFailure
, which should show up in your Django error
logging. The error message will include the Token you can use to manually confirm the subscription
in the Amazon SNS dashboard or through the SNS API.
Settings¶
Additional Anymail settings for use with Amazon SES:
AMAZON_SES_CLIENT_PARAMS
Optional. Additional client parameters Anymail should use to create the boto3 session client. Example:
ANYMAIL = { ... "AMAZON_SES_CLIENT_PARAMS": { # example: override normal Boto credentials specifically for Anymail "aws_access_key_id": os.getenv("AWS_ACCESS_KEY_FOR_ANYMAIL_SES"), "aws_secret_access_key": os.getenv("AWS_SECRET_KEY_FOR_ANYMAIL_SES"), "region_name": "us-west-2", # override other default options "config": { "connect_timeout": 30, "read_timeout": 30, } }, }
In most cases, it’s better to let Boto obtain its own credentials through one of its other mechanisms: an IAM role for EC2 instances, standard AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables, or a shared AWS credentials file.
AMAZON_SES_SESSION_PARAMS
Optional. Additional session parameters Anymail should use to create the boto3 Session. Example:
ANYMAIL = { ... "AMAZON_SES_SESSION_PARAMS": { "profile_name": "anymail-testing", }, }
AMAZON_SES_CONFIGURATION_SET_NAME
Optional. The name of an Amazon SES Configuration Set Anymail should use when sending messages. The default is to send without any Configuration Set. Note that a Configuration Set is required to receive SES Event Publishing tracking events. See Status tracking webhooks above.
You can override this for individual messages with esp_extra.
AMAZON_SES_MESSAGE_TAG_NAME
Optional, default None
. The name of an Amazon SES “Message Tag” whose value is set
from a message’s Anymail tags
.
See Tags and metadata above.
AMAZON_SES_AUTO_CONFIRM_SNS_SUBSCRIPTIONS
Optional boolean, default True
. Set to False
to prevent Anymail webhooks from automatically
accepting Amazon SNS subscription confirmation requests.
See Confirming SNS subscriptions above.
IAM permissions¶
Anymail requires IAM permissions that will allow it to use these actions:
- To send mail:
- Ordinary (non-templated) sends:
ses:SendRawEmail
- Template/merge sends:
ses:SendBulkTemplatedEmail
- Ordinary (non-templated) sends:
- To automatically confirm
webhook SNS subscriptions:
sns:ConfirmSubscription
- For status tracking webhooks: no special permissions
- To receive inbound mail:
- With an “SNS action” receipt rule: no special permissions
- With an “S3 action” receipt rule:
s3:GetObject
on the S3 bucket and prefix used (or S3 Access Control List read access for inbound messages in that bucket)
This IAM policy covers all of those:
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["ses:SendRawEmail", "ses:SendBulkTemplatedEmail"], "Resource": "*" }, { "Effect": "Allow", "Action": ["sns:ConfirmSubscription"], "Resource": ["arn:aws:sns:*:*:*"] }, { "Effect": "Allow", "Action": ["s3:GetObject"], "Resource": ["arn:aws:s3:::MY-PRIVATE-BUCKET-NAME/MY-INBOUND-PREFIX/*"] }] }
Following the principle of least privilege, you should omit permissions for any features you aren’t using, and you may want to add additional restrictions:
For Amazon SES sending, you can add conditions to restrict senders, recipients, times, or other properties. See Amazon’s Controlling access to Amazon SES guide.
For auto-confirming webhooks, you might limit the resource to SNS topics owned by your AWS account, and/or specific topic names or patterns. E.g.,
"arn:aws:sns:*:0000000000000000:SES_*_Events"
(replacing the zeroes with your numeric AWS account id). See Amazon’s guide to Amazon SNS ARNs.For inbound S3 delivery, there are multiple ways to control S3 access and data retention. See Amazon’s Managing access permissions to your Amazon S3 resources. (And obviously, you should never store incoming emails to a public bucket!)
Also, you may need to grant Amazon SES (but not Anymail) permission to write to your inbound bucket. See Amazon’s Giving permissions to Amazon SES for email receiving.
For all operations, you can limit source IP, allowable times, user agent, and more. (Requests from Anymail will include “django-anymail/version” along with Boto’s user-agent.) See Amazon’s guide to IAM condition context keys.