Configuring the ISA Firewall as an Outbound Filtering SMTP Relay

In my article Configuring the ISA Firewall as an Inbound Filtering SMTP Relay, I discussed procedures you can use to make the ISA firewall (ISA Server 2004) an inbound filtering SMTP relay to help offload some processing from your dedicated spam filtering solution. The ISA firewall’s built-in SMTP Message Screener, while not a complete anti-spam and e-mail anti-virus solution, can go a long way at improving the performance of your current e-mail hygiene solution by performing basic keyword and attachment filtering duties.

The benefits of filtering inbound e-mail is obvious; every network hosting its own SMTP services is under constant attack from spammers and e-mail virus writers who send malicious inbound SMTP messages. However, you can also benefit from filtering outbound e-mail connections. For example, if one or more of your users are infected with an e-mail virus, they may send messages with a pre-defined attachment name or type, or there may be key words in the message body or subject. You can block those messages from leaving your network by configuring the ISA firewall to filter outbound SMTP messages. This not only prevents your network from spreading these exploits, it also reduces the overall bandwidth consumption on your Internet links.

We will build on the configuration established in the last article, which you can find at http://isaserver.org/articles/2004inboundsmtprelay.html. We will discuss the following procedures required to complete the configuration to make the ISA firewall an inbound and outbound SMTP filtering relay:

• Configure Exchange Server to use the ISA firewall as its smart host
• Configure the SMTP Server Publishing Rule to listen on the internal interface of the ISA firewall
• Configure the SMTP service on the ISA firewall to allow outbound relay from the Exchange (or any other e-mail) SMTP service
• Configure System Policy on the ISA firewall to allow outbound SMTP from the Local Host Network
• Confirm the that e-mail message was filtered
• Check the e-mail headers on the message sent out via the SMTP relay that was allowed outbound

Configure the Exchange Server to use the ISA Firewall as its Smart Host

A smart host is a machine that takes over the duty of name resolution and forwarding of SMTP messages for another SMTP server. In this example of making the ISA firewall an outbound SMTP relay for the Exchange Server, the ISA firewall’s SMTP service will act as a smart host for the Exchange Server. It then becomes the responsibility of the ISA firewall’s SMTP service to resolve e-mail domain name and forward the SMTP messages to the appropriate SMTP server for delivery.

Perform the following steps on the Exchange Server to configure it to use the ISA firewall’s SMTP service as its smart host:

1. At the Exchange Server, open the System Manager.
2. In the Exchange System Manager console, expand the Servers node and then expand the server name. Expand the Protocols node and then expand the SMTP node.
3. Right click the Default SMTP Virtual Server and click Properties.
4. In the Default SMTP Virtual Server Properties dialog box, click the Delivery tab.
5. On the Delivery tab, click the Advanced button.
6. In the Advanced Delivery dialog box, enter the IP address of the listener the SMTP service uses to listen for SMTP connections. In this example, the ISA firewall’s SMTP service listens for connections on 10.0.0.1 on its Internal Network interface, so we enter that value into the text box. Make sure to surround this value in straight brackets, so that it looks like [10.0.0.1] in the text box. Click OK.

7. Click Apply and then click OK in the Default SMTP Virtual Server text box.
8. Restart the SMTP service on the Exchange Server.

Configure the SMTP Server Publishing Rule to Listen on the Internal Interface of the ISA Firewall

In the last article we configured the SMTP Server Publishing Rule to listen for incoming mail on the external interface of the ISA firewall and forward to the internal IP address of the ISA firewall on which the SMTP service is configured to listen. We can configure the SMTP Server Publishing Rule to also listen for mail arriving on the internal interface of the ISA firewall. The nice thing about the new ISA firewall is that we don’t have to create a new Server Publishing Rule; we can reconfigure the existing Server Publishing Rule to accept outgoing messages from the corporate network.

Perform the following steps to configure the SMTP Server Publishing Rule to accept SMTP messages arriving on the internal interface of the ISA firewall:

1. In the Microsoft Internet Security and Acceleration Server 2004 management console, expand the server name and then click the Firewall Policy node. In the Firewall policy node, double click the SMTP Relay Server Publishing Rule you created to allow inbound SMTP relay.
2. In the SMTP Relay Properties dialog box, click the Networks tab. On the Networks tab, put a checkmark in the Internal checkbox. Click Apply and then click OK.

Configure the SMTP Service on the ISA Firewall to Allow Outbound Relay from the Exchange (or any other e-mail) SMTP Service

The next step is to configure the SMTP service on the ISA firewall to accept outbound messages for relay to any e-mail domain. This makes the SMTP service on the ISA firewall an “open relay”. It’s referred to an “open relay” because the machine is able to relay to any e-mail domain. However, we can make this open relay more restrictive by allowing only the Exchange Server machine on the corporate network to relay through the ISA firewall’s SMTP service.

Note that the relay characteristics for inbound and outbound messages are quite different. The SMTP service on the ISA firewall is configured to allow inbound relay from all SMTP servers to the specific domain or domains that we host. Mail arriving for any other domain is rejected. In contrast, we will allow the Exchange Server to relay mail to all e-mail domains. Only the Exchange Server will be allowed outbound relay to all e-mail domains; no other host will be able to relay through the ISA firewall, with the exception for mail destined to the e-mail domains we host.

Perform the following steps to configure the SMTP service on the ISA firewall to allow the Exchange server to relay outbound to all e-mail domains:

1. On the ISA firewall device, open the Internet Information Services (IIS) Manager console from the Administrative Tools menu.
2. In the Internet Information Services (IIS) Manager console, expand the server name and click the Default SMTP Virtual Server. Right click the Default SMTP Virtual Server node in the left pane of the console and click Properties.
3. In the Default SMTP Virtual Server Properties dialog box, click the Access tab.
4. On the Access tab, click the Relay button in the Relay restrictions frame.
5. In the Relay Restrictions dialog box, confirm that the Only the list below option is selected. Click the Add button.
6. In the Computer dialog box, select the Single computer option and enter the IP address of the Exchange Server on the corporate network. In this example the Exchange Server has the IP address 10.0.0.2, so we enter that into the text box. Click OK.

7. In the Relay Restrictions dialog box we now see the IP address of the Exchange Server in the Computers list. Note that we do not select the Allow all computers which successfully authenticate to relay, regardless of the list above option. This prevents spammers from authenticating with the SMTP service on the ISA firewall and use it as an authenticating SMTP relay. We could enable this option and configure a very strong password for an account that the Exchange Server could use to authenticate to send outbound mail. The problem with this configuration is that we would prefer to make the ISA firewall a member of the Active Directory domain to take advantage of the superior level of security you obtain with the ISA firewall’s domain membership. If the ISA firewall is a member of the domain, spammers could take advantage of relatively weak passwords users set for themselves. Because we don’t control user’s passwords in the domain, we prefer to not allow authenticated users to relay through the ISA firewall. Click OK.

8. Click Apply and then click OK in the Default SMTP Virtual Server Properties dialog box.
9. Restart the IIS SMTP service.

Configure System Policy on the ISA Firewall to Allow Outbound SMTP from the Local Host Network

The default System Policy settings on the ISA firewall is to allow SMTP messages to be sent by the ISA firewall to SMTP servers on the default Internal Network. This allows the ISA firewall to send SMTP messages in response to Alerts. In order to support outbound SMTP messages to be sent to any location outside the default Internet Network, then we need to configure the ISA firewall’s System Policy to allow outbound messages from the firewall’s Local Host Network to the default External Network.

Perform the following steps to configure System Policy:

1. In the Microsoft Internet Security and Acceleration Server 2004 management console, expand the server name and then click the Firewall Policy node.
2. On the Firewall Policy node, click the Tasks tab in the Task Pane. On the Tasks tab, click the Show System Policy Rules link.
3. In the list of System Policy Rules, right click on the Allow SMTP from ISA Server to trusted server rule and click Edit System Policy.
4. In the System Policy Editor, confirm the red arrow is pointing at SMTP and click the To tab. On the To tab, click the Add button.
5. In the Add Network Entities dialog box, click the Networks folder and double click the External entry. Click Close.
6. The To tab should now show both the External and Internal Networks. Click OK.

7. Click Apply to save the changes and update the firewall policy.
8. Click OK in the Apply New Configuration dialog box.

Confirm the that E-mail Message was Filtered

Now let’s test the filtering configuration. In the example we configured in the last article, we set the SMTP Message Screener to block messages containing attachments with the .pif file extension. From a Outlook MAPI client (note that we’re doing this from the Outlook MAPI client, since the Exchange Server’s SMTP service isn’t configured to relay mail from Internal network hosts connecting to it via SMTP) we’ll send a message with the dangerous attachment.

Attached to the message is a file with the .pif extension. After sending the message, we can check the \Inetpub\mailroot\Badmail folder on the ISA firewall and see that that there are three files associated with the filtered message. We went over the details of these files in the last article. This confirms that the message was not sent out, but instead was filtered by the ISA firewall’s SMTP Message Screener.

We can check the SMTP filter’s log file and see an entry for this message. The log file entry provides details on why the message was blocked.

Check the E-mail Headers on the Message Sent out via the SMTP Relay that was Allowed Outbound

Now let’s test the configuration for sending legitimate e-mail messages outbound. Send an e-mail message from your Outlook MAPI client that’s connected to the Exchange Server.

In this example I sent a test message to my hotmail account. The salient e-mail headers are shown below.

Received: from ISALOCAL ([24.1.226.66]) by mc9-f6.hotmail.com with Microsoft SMTPSVC(5.0.2195.6824); Sun, 26 Dec 2004 08:13:14 -0800
Received: from EXCHANGE2003BE.msfirewall.org ([10.0.0.2]) by ISALOCAL with Microsoft SMTPSVC(6.0.3790.0); Sun, 26 Dec 2004 10:13:18 -0600
Received: from EXCHANGE2003BE ([10.0.0.2]) by EXCHANGE2003BE.msfirewall.org with Microsoft SMTPSVC(6.0.3790.0); Sun, 26 Dec 2004 10:13:11 -0600
X-Message-Info: JGTYoYF78jG+oarT45PqEpoyA3I3W9mg
X-MSMail-Priority: Normal
Return-Path: [email protected]
X-OriginalArrivalTime: 26 Dec 2004 16:13:11.0865 (UTC) FILETIME=[CBEB8290:01C4EB65]

We see in the message shows that the last hop outbound through our network is from the ISA firewall’s SMTP service, showing the name of the ISA firewall’s SMTP service and the IP address on the public interface of the NAT device in front of the ISA firewall.

Summary

In this article we went over the concepts and procedures required to make the ISA firewall an outbound SMTP filtering relay. We built on the procedures we carried out in my previous article, Configuring the ISA Firewall as an Inbound Filtering SMTP Relay to make the ISA firewall both an inbound and outbound SMTP filtering relay.