Configuring a Spam and Attachment Filtering SMTP Relay on the ISA Server 2000 Firewall – Part 2: Configuring the Server Publishing Rules and SMTP Filter and Message Screener

Configuring a Spam and Attachment Filtering SMTP Relay on the ISA Server 2000 Firewall
Part 2: Configuring the Server Publishing Rules and SMTP Filter and Message Screener


By Thomas W Shinder M.D.

http://www.msexchange.org/articles/messagescreeneronfirewallpart1.html

In part 1 of this two part article on configuring the ISA Server 2000 firewall as a spam and attachment filtering SMTP relay, we discussed the issues of spam and attachment control and anti-spam Defense in Depth. Detailed step by step instructions were provided on how to install and configure the IIS SMTP service on the ISA Server 2000 firewall, disable socket pooling for the SMTP service and create remote domains for your email domains.

In this, part 2 of this two part series, we go over the details of configuring the Server Publishing Rules and the SMTP Message Screener.

Procedures discussed in this article are:

The next step after installing and configuring the SMTP service on the ISA Server firewall is to install ISA Server 2000 with the SMTP Filter and Message Screener on to the Windows Server 2003 computer. Please refer to the ISA Server 2000 installation instructions included on the CD and http://support.microsoft.com/default.aspx?scid=kb;en-us;331062 for information on running ISA Server 2000 on Windows Server 2003.

You use a Server Publishing Rule to make your SMTP relay available to external users. One of the main advantages of using a Server Publishing Rule is that it exposes the incoming connections to buffer overflow protection features included with the SMTP filter.

Perform the following steps to create the SMTP Server Publishing Rule:

1. Open the ISA Management console, expand the Servers and Arrays node and then expand the server node. Expand the Publishing node and click on the Server Publishing Rules node. Right click on the Server Publishing Rules node, point to New and click on Rule.

2. Enter a name for the Server Publishing Rule in the Server publishing rule name text box on the Welcome to the New Server Publishing Rule Wizard page. Click Next.
3. On the Address Mapping page, enter the IP address of the internal interface of the ISA Server firewall in the IP address of internal server text box. Click the Browse button to the right of the External IP address on ISA Server and select an address on the external interface of the ISA Server firewall that you want the accept the incoming SMTP messages. Select the IP address in the New Server Publishing Rule Wizard dialog box and then click OK.

4. Click Next on the Address Mapping page.

5. On the Protocol Settings page, click the down arrow on the Apply the rule to this protocol drop down list box and select the SMTP Server entry. Click Next.

6. On the Client Type page, select the Any request option. Click Next.
7. Review your selections on the New Server Publishing Rule Wizard page and click Finish.
8. The details of the new Server Publishing Rule appear in the right pane of the ISA Management console.

The ISA Server firewall and SMTP relay are now ready to accept incoming connections from external SMTP servers. All SMTP email messages destined for the remote domains you’ve configured on the SMTP relay will forward these messages to the Exchange Server on the internal network and the messages will appear in the users’ mailboxes.

The SMTP filter and SMTP Message Screener configuration uses the same interface, which can be found in the SMTP Filter Properties dialog box. However, the SMTP filter and SMTP Message Screener are two distinct entities. It is possible to use the SMTP filter and not use the SMTP Message Screener and it is possible to use the SMTP Message Screener and not use the SMTP filter.

For example, you can use the SMTP Filter without using the SMTP Message Screener by simply not installing the SMTP Message Screener. The SMTP filter then protects the published SMTP server against buffer overflow attacks, including the SMTP server co-located on the ISA Server firewall.

You can use the SMTP Message Screener and not the SMTP Filter by using an SMTP packet filter to allow inbound access to the SMTP relay. The SMTP Message Screener examines the incoming SMTP messages when they are accepted by the IIS SMTP service. The SMTP Filter will not be able to protect against buffer overflow attack because incoming SMTP messages accepted via a packet filter are not exposed to the SMTP filter.

Perform the following steps to configure the SMTP filter and SMTP Message Screener components:

1. Open the ISA Management console, expand the Servers and Arrays node and expand your server name. Expand the Extensions node and click on the Application Filters node. Right click on the SMTP Filter entry in the right pane of the console and click on the Properties command.
2. The General tab is the first thing you see when the SMTP Filter Properties dialog box opens. You can enable or disable the filter by adding or removing the checkmark in the Enable this filter checkbox. Click on the Keywords tab.

3. You can enter a prioritized list of keywords to filter on the Keywords tab. The SMTP Message Screener mediates the keyword filtering function. The SMTP filter does not examine SMTP messages for keywords. Click the Add button to add a keyword.

4. Confirm the there is a checkmark in the Enable keyword rule checkbox. Type in a keyword that you want the SMTP Message Screener to look for in the Keyword text box. Note the SMTP Message Screener does not search for whole words; it only looks at text strings.

Select one of the following options in the Apply action if keyword is found in frame:

Message header or body

If the keyword is found in either the message header or message body, then the Action you configure for the rule will be applied.

If the keyword is found in the header (subject line), then the Action you configure for the rule will be applied.

If the keyword is found in the body of the message, then the Action you configure for the rule will be applied

Click the down arrow for the Action drop down list box. You have the following options:

The email message is deleted without being saved or informing anyone that it has been deleted.

The SMTP message is held in the BADMAIL directory in the SMTP service’s folder hierarchy. You can view components of the held message, but the message is not saved in a format that you can easily forward to the recipient.

The SMTP message is forwarded to an email address you configure in this rule. Each rule can have a different email address that the message is forwarded to.

Click OK on the Mail Keyword Rule dialog box after entering a keyword and action.

5. The keyword rule appears in the keywords list on the Keywords tab. Click on the Users / Domains tab.

6. You can configure the SMTP Message Screener to block messages based on the sender’s user account or email domain on the Users / Domains tab. Enter a user email account in the Sender’s name text box and click Add. The senders email address appears in the Rejected Sender’s list. Enter an email domain in the Domain name text box and click Add. The email domain appears in the Rejected Domains list.

Email messages processed by the SMTP Message Screener matching email addresses or email domains found in these lists are deleted. These messages are not stored anywhere on the server, nor are they forwarded to any user or administrator. If a message from a rejected sender or rejected domain also contains a keyword matching a keyword rule, and that keyword rule is configured to hold the message, the message will not be held because it is rejected before the keyword search begins.

Click Apply and then click OK. Click on the Attachments tab.

7. You can block messages with certain types of attachments on the Attachments tab. Click Add to add an attachment rule.

8. Confirm that there is a checkmark in the Enable attachment rule checkbox on the Mail Attachment Rule dialog box. You have three options in the Apply action to messages containing attachments with one of these properties frame:

Attachment name

Select this option and enter a name for the attachment, including file name and file extension. Use this option when you do want to block a specific file name and you don’t want to block all attachments with a particular file extension. For example, you do not want to block all .zip files, but you do want to block a file named exploit.zip.

It is more common to block all files with a specific file extension. For example, if you want to block all attachments with the exe file extension, select this option and then type in either exe or .exe in the text box to the right of this option.

You can also block attachments based on their size. Select this option and type in the size of the file extension you want to block.

Click the down arrow for the Action drop down list box. You have the following options:

The SMTP message is deleted without being saved or informing anyone that it has been deleted.

The SMTP message is held in the BADMAIL directory in the SMTP service’s folder hierarchy. You can view components of the held message, but the message is not saved in a format that you can easily forward to the recipient.

The SMTP message is forwarded to an email address you configure in this rule. Each rule can have a different email address that the message is forwarded to.

In this example we’ll select the Forward message to option so that you can see how to enter the forwarding address.

9. When you select the Forward message to option, a text box appears allowing you to enter an email address to forward the message. However, the ISA Server must be able to resolve the address of the mail domain of this user.

For example, in the figure below we have entered the email address [email protected]. The ISA Server 2000 firewall must be able to access an MX record for the internal.net domain. The ISA Server firewall forwards the message to the SMTP server responsible for internal.net mail based on the information in the MX record.

In this example the firewall is configured with an address of an internal network DNS server that can resolve both internal and external network names. The message is forwarded to the internal address of the Exchange server. You must configure a split DNS infrastructure if the internal.net domain is available to both internal and external users.

Click OK in the Mail Attachment Rule dialog box. Click on the SMTP Commands tab.

10. The settings on the SMTP Commands tab are mediated by the SMTP filter component. The SMTP Message Screener does not evaluate SMTP commands and it does not protect against buffer overflow conditions. The commands in the list are limited to a pre-defined length. The connection is dropped if an incoming SMTP connection sends a command exceeding the allowed length. In addition, if a command not on this list is sent over the SMTP channel is, it is dropped.

Click the Add button to add an SMTP command to the list.

11. A command you may want to enter into the list of allowed SMTP commands is the AUTH command. This is required if you want to allow external users to authenticate with an SMTP server published via an SMTP Server Publishing Rule. Users will not be able to authenticate with a SMTP server published via an SMTP Server Publishing Rule if the AUTH command is not added to the list and the SMTP filter is enabled.

Confirm that the Enable an SMTP command checkbox is checked. Type AUTH in the Command Name text box. Type 1024 in the Maximum Length Bytes text box. Click OK in the SMTP Command Rule dialog box.

12. The new command appears in the list of SMTP commands on the SMTP Commands tab (figure 44). Click Apply and then click OK.

13. Close the ISA Server Management console.
14. Restart the ISA Server 2000 machine.

The ISA Server firewall/SMTP server is now ready to filter SMTP messages based on the parameters you set for the SMTP filter and SMTP Message Screener.

ISA Server 2000 is an advanced application aware firewall that examines application layer content of packets moving through it. You can use ISA Server 2000’s advanced application layer filtering to block spam and viruses from endangering your network. You can use the ISA Server 2000 firewall as a spam and attachment filtering SMTP relay to offload processing from a dedicated spam filtering gateway on the internal network or a spam filtering service co-located on the Exchange Server. This anti-spam Defense in Depth approach increases the overall efficiency of your anti-spam infrastructure and reduces total cost of operation for spam control.