Preventing AutoReply Storms (Part 1)

If you would like to read the next part in this article series please go to Preventing AutoReply Storms (Part 2).

Introduction

An e-mail storm is normally associated with a big spike in the number of e-mails being sent due to the Reply All functionality on a large Distribution List [DL]. These storms start when multiple members of the DL reply to the entire list around the same time in response to the original e-mail followed by other members also replying asking to be removed from the DL or answering someone’s question. When enough members reply to these messages, a chain reaction of e-mails is triggered, and this dramatically increases the generated traffic and can bring e-mail servers to a halt.

For this article we will focus on storms not created by DLs but by an AutoReply rule in Outlook (not to be confused with Out-Of-Office [OOF] messages).

A Microsoft Outlook AutoReply rule is a feature only available through Outlook and Exchange that allows users to send automated replies to incoming messages. The biggest difference between an AutoReply and an OOF e-mail is that the OOF only replies once per sender when they first e-mail the user, while the AutoReply rule will reply to every single e-mail received by the mailbox.

This rule is often used on generic mailboxes to inform senders that they will receive a response shortly or that the mailbox is no longer in use, for example, and is created using the have server reply using a specific message action:

Figure 1.1: Creating an AutoReply rule in Outlook

Generating a Storm

The problem with an AutoReply rule is when both the sender and the recipient have one enabled. Imagine UserA and UserB:

• UserA sends an e-mail to UserB;
• The AutoReply rule in user’s B mailbox sends an automatic reply to UserA;
• When Exchange receives this e-mail for UserA, it automatically sends a reply back as UserA also has an AutoReply rule in place;
• UserB receives the automatic reply from UserA and sends another automatic reply;
• And so on…

As you can imagine, this can generate a huge number of e-mails per minute and the only way to stop them is for one of the users to temporarily disable the rule… This is known as an AutoReply Storm.

The good news is that Exchange is smart enough to prevent these storms from happening when both sender and recipient are internal users. However, if one of them is external, Exchange will not stop these as every e-mail received from an external sender is seen as a “brand new” e-mail without any association with previous e-mails even though they are from the same sender and with the same subject.

This is a common problem faced by Exchange Administrators and many organizations chose to simply remove the functionality altogether. To do this, we can block AutoReply e-mails at a global level which would prevent everyone, with no exception, from AutoReplying e-mails to outside the domain:

Set-RemoteDomain Default –AutoReplyEnabled $False

If you haven’t created any additional remote domain (for a partner company, for example), the Default domain will match everything outside your domain, thus blocking any AutoReply to the outside world.

However, because for most organizations it is not an option to prevent AutoReply e-mails, I developed a script that tries to detect and prevent storms caused by them.

Detecting Storms

As these storms generate multiple identical e-mails per minute, the script basically looks for e-mails sent from a sender to the same recipient with the exact same subject in a specific timeframe. If it sees a large number of these e-mails in a short period of time, it is fair to assume it is a storm.

To start off with, we will set the script to run every 15 minutes and look for users who sent 25 or more e-mails in the last 15 minutes:

[Int] $intTime= 15

[Int] $intNumEmails= 25

[DateTime] $dateStartFrom= (Get-Date).AddMinutes(–$intTime)

Note:
Because every environment is different, and just because the script is 100% accurate so far for me, it doesn’t mean it will be for you! You might have to fine-tune it by playing around with these parameters or even exclude some mailboxes from “inspection”.

Now let’s analyze the Message Tracking Logs. To do this, we search for EventID of SEND and Source of SMTP as we are looking for e-mails sent outside our organization. We only want those users that sent 25 or more e-mails so let us group all e-mails by sender and filter them based on the number of e-mails sent. If there are none, then we simply exit:

$logEntries= Get-TransportServer | Get-MessageTrackingLog -ResultSize Unlimited -Start $dateStartFrom -EventId SEND | ? {$_.Source -eq“SMTP”} | GroupSender | ? {$_.Count -ge$intNumEmails}

If ($logEntries-eq$null) { Exit }

Note:
if you have multiple Active Directory sites or are running in a co-existence scenario with multiple Exchange versions, you might want to include -and $_.Recipients -notmatch “your_domain” to make sure you only include e-mails sent to recipients outside your domain. This is because the previous search will capture internal e-mails if your environment matches these conditions. For example, if you are running both Exchange 2007 and 2010, e-mails sent from a user in 2007 to a user in 2010 will be sent over SMTP from Exchange 2007 to 2010. To ignore these, let’s update our search:

$logEntries= Get-TransportServer | Get-MessageTrackingLog -ResultSize Unlimited -Start $dateStartFrom -EventId SEND | ? {$_.Source -eq“SMTP”-and$_.Recipients -notmatch“letsexchange.com”} | GroupSender | ? {$_.Count -ge$intNumEmails}

We now have in $logEntries a list of all users who sent 25 or more e-mails within the last 15 minutes. However, we need to search the Transport Logs again for each of these users in order to check if they sent multiple e-mails to the same recipient and with the same subject. To achieve this, the easiest way I found was to use a Hash Table, a data structure that maps values known as keys (e.g., a person’s name) to their associated values (e.g., their telephone number). We will go through all the e-mails found in our search and add them to the hash table where each key will be a string composed of the sender’s e-mail address, the recipient’s e-mail address and the message subject. When we first process an e-mail, the initial value for each key will be one; if there is more than one e-mail with the exact same subject sent to the same recipient from the same sender, the value will be incremented by one. At the end, the hash table will give us a list in the following format:

Figure 1.2: Content of the Hash Table

In the above picture we can see that [email protected] sent 33 e-mails with 3 different subjects. Out of these 3, we are only interested in the one with the subject of “Trying to Generate Storm” because it represents more than 25 e-mails in 15 minutes which indicates a possible storm.

The 19 e-mails sent from [email protected] also indicate a possible storm but because it didn’t cross the 25 threshold, the script will ignore it.

To build this hash table, we need to go through all the e-mails saved into $logEntries so we can group them into a string made of “sender_address, recipient_address, email_subject” and add them to our hash table:

# Initialize HashTable

$hashEmails= @{}

# For each sender, check all the e-mails they sent

ForEach ($logEntryin$logEntries)

{

      ForEach ($emailin (Get-TransportServer | Get-MessageTrackingLog -ResultSize Unlimited -Start $dateStartFrom -Sender $logEntry.Name -EventId SEND | ? {$_.Source -eq“SMTP”-and$_.Recipients -notmatch“letsexchange.com”} | Select MessageSubject, Sender, Recipients))

      {

            $hashEmails[“$($email.Sender), $($email.Recipients), $($email.MessageSubject)”] += 1

      }

}

This will ensure that $hashEmails will contain data in the format that we want (Figure 2). However, please note that because an e-mail can be sent to more than one recipient at a time, the key in the hash table might be in the format of “sender_address, recipient_address, recipient2_address, recipient3_address, …, email_subject” but because when a storm happens it will only be between one sender and one recipient, we don’t need to worry about these situations.

Now that we have our hash table populated, we need to check if any of the entries have a value of 25 or higher. If yes, then we send an e-mail to the administrator. Because all the information we want is in big strings (the keys), we will need to split them in order to compose the e-mail. This can easily be done with the Split method:

$arrDetails= [Regex]::Split($storm.Name, “, “)

In this case we are splitting the string by commas, and saving it to the $arrDetails array. The problem with this is that sometimes the subjects of e-mails contain commas, so the best option is to use a character that we know users will not use. I will be using the ô character because it is not part of the English alphabet but you can use any character you want.

(…)

$hashEmails[“$($email.Sender) ô $($email.Recipients) ô $($email.MessageSubject)”] += 1

(…)

ForEach ($stormin ($hashEmails.GetEnumerator() | ? {$_.Value -ge$intNumEmails} | SortName))

{

      $arrDetails= [Regex]::Split($storm.Name, ” ô “)

      $strEmailBody+=“`n`nSender:    “, $arrDetails[0]

      $strEmailBody+=“`nRecipient: “, $arrDetails[1]

      $strEmailBody+=“`nSubject:   “, $arrDetails[2]

      $strEmailBody+=“`n# e-mails: “, $storm.Value

      Send-MailMessage-From“[email protected]”-To“[email protected]”-Subject“Storm Detected!”-Body$strEmailBody-SMTPserver“smtp.letsexchange.com”-DeliveryNotificationOptiononFailure

}

A couple of notes about this code:

• To be able to sort/filter a hash table, we need to transform its content into individual objects by using the GetEnumerator method;
• If in the hash table we have more than 1 case of storms, individual e-mails will be sent instead of everything in just one e-mail (we will change this in the next part of this article).

If a storm is detected, the e-mail will contain its details which match what we have in the hash table:

Figure 1.3: Storm Detected E-mail

Conclusion

In this first part of the article, we explored the core of the script, including how to detect storms and inform administrators. In the second part we will improve this e-mail, learn how we can schedule the script to run every 15 minutes and see how we can automatically put an end to a storm once it is detected!

If you would like to read the next part in this article series please go to Preventing AutoReply Storms (Part 2).