Configuring the ISA Firewall as an Inbound Filtering SMTP Relay

Configuring the ISA Firewall as an Inbound Filtering SMTP Relay

By Thomas W Shinder MD, MVP

Got Questions? Go to:
http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=22;t=000182 and ask!

A popular configuration for the ISA firewall is to use it as an inbound SMTP filtering relay. You can setup the ISA firewall as an inbound SMTP relay and leverage the built-in SMTP filter and SMTP Message Screener to offload some of the spam and attachment filtering duties from your dedicated spam whacking device or Exchange Server located on an ISA firewall Protected Network. While the ISA firewall’s SMTP Message Screener isn’t a full-fledged spam whacking and e-mail anti-virus solution, it can perform some initial processing on incoming messages, which takes some heat off your dedicated e-mail scrubbing devices.

Get the New Book!

You need to make the ISA firewall’s primary IP address your SMTP domain’s authoritative SMTP server This is accomplished by first creating a DNS Host (A) record for your mail server and then creating an MX record based on that Host (A) record. Incoming mail then arrives on the external interface of the ISA firewall and is accepted by the SMTP service installed on the ISA firewall device. The SMTP Message Screener statefully inspects the e-mail message and then forwards those that pass inspection to the mail server or spam whacker on an ISA firewall protected network.

In this article we will discuss the following procedures:

  • Configure DNS to support your SMTP Server Publishing Rule for incoming mail
  • Install the SMTP service on the ISA firewall machine
  • Configure the SMTP service on the ISA firewall machine
  • Install the ISA firewall software on the ISA firewall machine
  • Create the Server Publishing Rules on the ISA firewall
  • Configure the SMTP Filter and Message Screener on the ISA firewall
  • Test the configuration
  • The sample network is shown in the figure below.

    Configure DNS to Support Your SMTP Server Publishing Rule

    While not required in order to perform the tasks in the test deployment discussed in this article, its worth going over the basic procedures required to configure DNS to support your SMTP Server Publishing solution. When you make the ISA firewall the authoritative SMTP server for your domain, you make the IP address on the external interface of the ISA firewall used in the SMTP Server Publishing Rule the IP address responsible for accepting incoming mail to your mail domain. In order for this to work, you need to configure your public DNS server with the appropriate entries.

    You have two options for hosting your public DNS records:

  • Your public access domain information is hosted by your ISP
  • You host your own DNS server and publish the DNS server through the ISA firewall
  • If you already have your ISP hosting your public DNS information, then there is no reason to switch over to publishing your own records. The only thing you need to do is create the Host (A) and MX records in your public DNS zone.

    If you are hosting your own public DNS, then you should use a Server Publishing Rule to publish that DNS server on a ISA firewall Protected Network. Some things to consider when hosting your own public DNS server:

  • Do not put your public DNS server or DNS resources records on the same machine that hosts your internal domain DNS information.
  • Place the DNS server on a anonymous access DMZ segment. This DMZ segment does not require authentication in order to access the resources located on this segment. Other services you might put on this segment are your anonymous access Web and FTP servers.
  • Use a split DNS. A split DNS will greatly simplify life for your users and make publishing secure Web servers and RPC over HTTP Exchange servers much easier for you. For your users, life will be much much simpler because they’ll never need to use different names to access resources based on their location. Regardless if they’re on the corporate network or somewhere else and connecting through the Internet, they’ll always use the same names to connect to the same services
  • Do not put the DNS server hosting your public records on the ISA firewall itself. If you want to put a caching-only DNS server on the ISA firewall and use that as a forwarder for your internal network DNS server, that’s fine. But don’t put an authoritative DNS server on the ISA firewall device itself
  • Creating the appropriate DNS records is easy. In this example we’ll show how to do it on a Windows Server 2003 DNS server, but the same principles apply to any brand of DNS server. There are two steps: create the Host (A) record and then create the MX record mapping to the Host (A) record.

    Get the New Book!

    Perform the following steps on a Windows Server 2003 DNS server to create the appropriate records:

    1. In the DNS console, expand the server name and then expand the Forward Lookup Zone node in the left pane of the console.
    2. Right click your domain name and click New Host (A).
    3. In the New Host dialog box, enter a name for your authoritative mail server in the Name (uses parent domain name if blank) text box. A common name for mail servers is mail, so we’ll enter that into the text box. In the IP address text box, enter the IP public IP address on the external interface of the ISA firewall that you’re using in the Server Publishing Rule to publish the SMTP relay on the ISA firewall itself. Enter your public IP address there. We will add 111.111.111.1 as an example.

    1. Click Done.
    2. Right click your forward lookup zone and click New Mail Exchanger (MX).
    3. In the New Resource Record dialog box, leave the Host or child domain text box empty. In the Fully qualified domain name (FQDN) text box, enter the FQDN of the mail server. If you’re not sure how it works, click the Browse button.
    4. In the Browse dialog box, double click your server name in the Records list, then double click the Forward Lookup Zones entry in the Records list. Now double click your forward lookup zone name in the Records list. Double click the Host (A) record that you just created from the list of records in the Records list. Click OK.

    1. The FQDN of your mail server now appears in the Fully qualified domain name (FQDN) text box. Click OK.

    1. Your records should now appear in the right pane of the console. Restart the DNS Server service and then click the DNS console.

    Note in this example we used a public address on the external interface of the ISA firewall, because in most cases the ISA firewall device will be located behind a packet filtering router or a simple stateful packet filtering firewall (like a PIX). The address on the external interface of the ISA firewall will be one of the organization’s public block.

    However, you can still publish an SMTP server behind a NAT device and not use a public address on the external interface of the ISA firewall. For example, suppose you run a small business and use a DSL connection to the Internet that require PPPoE. PPPoE is a problematic protocol for Windows-based devices so its best not to use PPPoE connections on the ISA firewall device itself. Instead, put a simple NAT “router” in front of the ISA firewall and then forward all incoming connections to the ISA firewall’s external IP address. The details on how to do this vary with the brand of NAT device you use.

    However, when you create your DNS records, you do not use the IP address on the external interface of the ISA firewall, since that will be a private IP address. Instead, you use the public address bound to the WAN interface of the NAT device in front of the ISA firewall.

    Get the New Book!

    Install the SMTP Service on the ISA Firewall Machine

    The first step, before installing the ISA firewall software, is to install the SMTP service on the ISA firewall. However, if you have already installed the ISA firewall software, you can still install the SMTP service; the procedures are just a bit different. If you have already installed the ISA firewall software, you can carry out the following procedures for installing and configuring the SMTP service. Later I’ll let you know where your path diverges from that we focus on in the scenario discussed in this article.

    On the machine that will become the ISA firewall device, perform the following steps:

    1. Click Start and point to Control Panel. Click Add/Remove Programs.
    2. In the Add or Remove Programs window, click the Add/Remove Windows Components button.
    3. In the Windows Components dialog box, click the Application Server entry in the list of Components and click Details.
    4. In the Application Server dialog box, click the Internet Information Services (IIS) entry in the Subcomponents of Application Server list and click Details.
    5. In the Internet Information Services (IIS) dialog box, put a checkmark in the SMTP Service checkbox and click OK.
    6. Click OK in the Application Server dialog box.
    7. Click Next on the Windows Components page.
    8. Click OK in the Insert Disk dialog box.
    9. In the Files Needed dialog box, point the installation wizard to the i386 folder from the Windows Server 2003 disk and click OK.
    10. Click Finish on the Completing the Windows Components Wizard page.
    11. Close the Add or Remove Programs window.

    Configure the SMTP Service on the ISA Firewall Device

    The next step it to configure the SMTP service to support our inbound SMTP relay configuration. Perform the following steps to configure the SMTP service:

    1. Click Start, and point to Administrative Tools. Click Internet Information Services (IIS) Manager.
    2. In the Internet Information Services console expand your server name and then expand the Default SMTP Virtual Server. Right click Default SMTP Virtual Server and click Properties.

    1. In the Default SMTP Virtual Server Properties dialog box, click the General tab. On the General tab, select the internal IP address on the ISA firewall from the IP address list. In this example, the internal address on the ISA firewall is 10.0.0.1, so we will select that address.

     

    1. Click the Access tab. On the Access tab, click the Relay button in the Relay restrictions frame.

    1. In the Relay Restrictions dialog box, remove the checkmark from the Allow all computers which successfully authenticate to relay, regardless of the list above option. This protects the SMTP relay on the ISA firewall from being victimized by spammers who launch dictionary attacks against the SMTP relay. Click OK.

    1. Click Apply and then click OK in the Default SMTP Virtual Server Properties dialog box.
    2. In the left pane of the console, right click the Domains node, point to New and click Domain.
    3. On the Welcome to the New SMTP Domain Wizard page, select the Remote option and click Next.
    4. On the Domain Name page, enter the name for which you want to accept inbound mail for relay to your mail server or spam whacker on the ISA firewall Protected Network. In this example, we want to accept incoming mail to the domain msfirewall.org, so we enter that into the Name text box. Click Finish.

    1. Double click the msfirewall.org entry in the right pane of the console.
    2. On the General tab of the remote domain’s Properties dialog box, put a checkmark in the Allow incoming mail to be relayed to this domain checkbox. In the Route domain frame, select the Forward all mail to smart host option and enter the IP address of the mail server or dedicated spam whacker you’re using. In this example we’re forwarding the mail directory to the Exchange Server on the Internal Network, so we enter the Exchange Server’s IP address, with straight brackets, in the text box as [10.0.0.2]. Click Apply and then click OK.

    1. Close the Internet Information Services (IIS) Manager console.

    Get the New Book!

    Install the ISA Firewall Software on the ISA Firewall Machine

    Now we’re ready to install the ISA firewall software onto the firewall device. If you have already installed the ISA firewall software, then you should use the Add/Remove Programs applet in the Control Panel and update the ISA Server 2004 software settings using the Modify option in the setup routine. Then when you get to the setup options page, select the SMTP Message Screener as the add-on component you want to install.

    Perform the following steps to install the ISA firewall software onto the firewall device:

    1. Place the ISA Server 2004 CD into the firewall. In the autorun menu, click the Install ISA Server 2004.
    2. Click Next on the Welcome to the Installation Wizard for Microsoft ISA Server 2004.
    3. On the License Agreement page, read the license agreement and select the I accept the terms in the license agreement option and click Next.
    4. Enter your customer information on the Customer Information page and click Next.
    5. On the Setup Type page, select the Custom option and click Next.
    6. On the Custom Setup page, accept the default selections, Firewall Services, Advanced Logging and ISA Server Management. In addition, click the Message Screener option and click the This feature, and all subfeatures, will be installed on local hard drive and click Next.

    1. On the Internal Network page, click the Add button.
    2. On the address ranges page, click the Select Network Adapter button.
    3. On the Select Network Adapter page, remove the checkmark from the Add the following private ranges… checkbox. Confirm that there is a checkmark in the Add address ranges based on the Windows Routing Table checkbox. The place a checkmark in the checkbox for the Internal interface. Click OK. Click OK in the Setup Message dialog box informing you that the Internal Network was defined based on the information in the routing table.

    1. Click OK on the addresses page.
    2. Click Next on the Internal Network page.
    3. On the Firewall Client Connection Settings page, do not put a checkmark in the Allow computer running earlier versions of the Firewall Client software to connect checkbox and click Next.
    4. Click Next on the Service page.
    5. Click Install on the Ready to Install the Program page.
    6. Click Finish on the Installation Wizard Completed page. Restart the firewall if the wizard instructs you to do so. Then log onto the machine as Administrator and open the firewall management console.

    Get the New Book!

    Create the Server Publishing Rules on the ISA firewall

    In this example we will create two Server Publishing Rules: one for the DNS server on the Internal Network, and then one for the SMTP server. We need to publish the DNS server on the Internal Network in this example because the public records for our domain are hosted on the domain DNS server.

    In your production environment, you might want to host your own DNS sever on an anonymous access DMZ segment, or have your ISP host your public DNS. If you host your own DNS server, you can create a DNS Server Publishing Rule to publish the DNS server on your anonymous access DMZ segment.

    Perform the following steps to create the DNS Server Publishing Rule:

    1. In the Microsoft Internet Security and Acceleration Server 2004 management console, expand your server name and then click the Firewall Policy node.
    2. On the Firewall Policy node, click the Tasks tab in the Task Pane. Click the Create a New Server Publishing Rule link.
    3. On the Welcome to the New Server Publishing Rule Wizard page, enter a name for the rule in the Server Publishing Rule name text box. In this example we’ll call is DNS Server and click Next.
    4. On the Select Server page, enter the IP address of the DNS server in the Server IP address text box. In this example, the IP address of the DNS server is 10.0.0.2 so we will enter that into the text box. Click Next.
    5. On the Select Protocol page, select the DNS Server protocol from the Selected protocol list. Click Next.

    1. On the IP Addresses page, put a checkmark in the External checkbox. Click Next.
    2. Click Finish on the Completing the New Server Publishing Rule Wizard page.

    Next, perform the following steps to create the SMTP Server Publishing Rule:

    1. In the Microsoft Internet Security and Acceleration Server 2004 management console, expand your server name and then click the Firewall Policy node.
    2. On the Firewall Policy node, click the Tasks tab in the Task Pane. Click the Create a New Server Publishing Rule link.
    3. On the Welcome to the New Server Publishing Rule Wizard page, enter a name for the rule in the Server Publishing Rule name text box. In this example we’ll call is SMTP Relay and click Next.
    4. On the Select Server page, enter the IP address of the SMTP relay on the ISA firewall machine in the Server IP address text box. In this example, the IP address of the co-located SMTP relay is 10.0.0.1 so we will enter that into the text box. Click Next.
    5. On the Select Protocol page, select the SMTP Server protocol from the Selected protocol list. Click Next.

    1. On the IP Addresses page, put a checkmark in the External checkbox. Click Next.
    2. Click Finish on the Completing the New Server Publishing Rule Wizard page.
    3. Click Apply to save the changes and update the firewall policy.
    4. Click OK in the Apply New Configuration dialog box.

    If you’re read our book Configuring ISA Server 2004, you might wonder if we need to create an Access Rule allowing outbound access from the ISA firewall’s Local Host Network to the SMTP server on the Internal Network in order for the SMTP relay on the ISA firewall to forward the messages.

    OK, if you read the book, you already know the answer. We don’t need to create the Access Rule because there is a System Policy Rule allowing the ISA firewall to forward SMTP messages from the Local Host Network to servers on the default Internal Network. However, if the server you want to forward the messages to is not on the default Internal Network, then the System Policy Rule would not work for you, and you would need to create an Access Rule allow SMTP from the Local Host Network to the host you want the message forwarded to.

    Configure the SMTP Filter and Message Screener on the ISA Firewall

    At this point we’re ready to configure the SMTP Message Screener. Actually, you could have configured the SMTP Message Screener before creating the SMTP Server Publishing Rule. In the Microsoft Internet Security and Acceleration Server 2004 management console, expand the server name and then expand the Configuration node. Click the Add-ins node. In the details pane of the console, double click the SMTP Filter entry.

    There are five tabs on the SMTP Filter Properties dialog box. We won’t go into all the details of how to configure each of the options on each of the tabs, as we’ve done that in Configuring ISA Server 2004. However, we do want to test our configuration, so we’ve configure the SMTP Message Screener to block attachments with the .pif file extension. There no reason for ever receiving e-mail with this file extension, so we can offload some processing from our e-mail spam whack/AV server by capturing these messages at the network perimeter.

    Click the Attachments tab. On the Attachments tab, click the Add button. Select the Attachment extension option and enter the extension in the text box, which in this example is .pif. In the Action drop down list you have three options: Delete message, Hold message and Forward message to. In a production environment you would most likely want to delete the message, since the chance of a legitimate e-mail message containing such an attachment is low. However, in order to show you how things work, we’ll select the Hold message option. This allows the ISA firewall to save the message on the server. You also have the option to Forward message to an account that will receive all the messages. Click OK.

    Click Apply and then click OK on the SMTP Filter Properties dialog box.

    Click Apply to save the changes and update the firewall policy. Click OK in the Apply New Configuration dialog box.

    Get the New Book!

    Test the Configuration

    Now we’ll test the configuration using our external client machine. This external client machine is a Windows XP Professional computer that is configured with a DNS server address that matches the IP address used in the DNS Server Publishing Rule. Outlook Express is configured on this machine to use the IP address of the external interface of the ISA firewall as its SMTP server.

    From the external client I’ll send a email message that has a file with a file attachment with the .pif file extension. The message won’t ever reach the Exchange Server because the SMTP Message Screener whacks it. Since we choose the option to hold the messages, we can find them in the C:\Inetpub\mailroot\Badmail folder. For each held message you’ll see three entries: a BAD file, a BDP file and a BDR file. The .BAD file shows the actual contents of the message. The BDP file shows a bunch of garbage, and the BDR file shows an error message indicating that the message could not be delivered.

    For more information about the activity of the SMTP Message Screener, you can look in the SMTP Message Screener Log.

     

    Summary

    In this article we went over the concepts and procedures involved with making the ISA firewall an inbound SMTP relay for your corporate network. We began by installing the SMTP service on the ISA firewall device and then installed the ISA firewall software. We then configured the required Server Publishing Rules and finished up with configuring the SMTP Message Screener to block incoming files with the .pif extension. This article focused on inbound SMTP relay only. The ISA firewall can also act as an outbound SMTP relay. The SMTP Message Screener will also screen outgoing messages when you configure it for outbound SMTP relay. If you’re interested in making the ISA firewall both an inbound and outbound SMTP relay, let me know by writing me at [email protected].

    I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=22;t=000182 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks! –Tom

    If you would like us to email you when Tom Shinder releases another article on ISAserver.org, subscribe to our ‘Real-Time Article Update’ by clicking here. Please note that we do NOT sell or rent the email addresses belonging to our subscribers; we respect your privacy.

    About The Author

    Leave a Comment

    Your email address will not be published. Required fields are marked *

    This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

    Scroll to Top