Troubleshooting SMTP Server Publishing Rules

Troubleshooting SMTP Server Publishing Rules

By Thomas W Shinder MD, MVP

Got Questions? Go to:
http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=22;t=000168 and ask!

ISA firewall Web and Server Publishing Rules are typically used to allow inbound access from remote hosts to an ISA firewall Protected Network situated server. ISA firewall Protected Networks are those Networks for which the ISA firewall has a definition. Any address that isn’t included in an ISA firewall Network definition is considered to be part of the default External Network.

Get the New Book!

One of the most common Server Publishing Rule scenarios is for SMTP servers. SMTP Server Publishing Rules allow you to publish SMTP servers on an ISA firewall Protect Network. The SMTP server can be a dedicated SMTP relay, or it can be the endpoint of the inbound e-mail messages, such as you Exchange Server. The SMTP Server Publishing Rule allows inbound connections to TCP port 25 through the ISA firewall to the SMTP server on the ISA firewall Protected Network.

However, unlike the simple stateful packet inspection (stateful filter) hardware firewall, the ISA firewall performs both stateful filtering and stateful application layer inspection on the incoming SMTP messages. The ISA firewall performs stateful application layer inspection using two features:

  • The SMTP filter
  • The SMTP Message Screener
  • The SMTP filter is always enabled for SMTP Server Publishing Rules. The SMTP Message Screener must be installed on the ISA firewall, on an SMTP relay or even the Exchange Server itself (I don’t recommend that you install the SMTP Message Screener on the Exchange Server). The SMTP filter screens for buffer overflow attacks against your SMTP server and the SMTP Message Screener actually looks at the e-mail messages themselves and blocks them based on keywords, file attachments, or source or destination address.

    ISA firewall SMTP server publishing is popular, but along with its popularity comes a lot of troubleshooting issues. In this article we’ll take a look at one approach to troubleshooting SMTP Server Publishing Rules.

    We’ll go over the following topics:

  • The sample network
  • Using Telnet
  • Using Network Monitor on the ISA Firewall and Exchange Server
  • The Sample Network

    The figure below shows the basic configuration of our simple sample network. There is a single ISA firewall with an internal and external interface and an Exchange Server with a single interface connected to the default Internal Network. There is also an external network client that we’ll use to troubleshoot our Server Publishing Rule problem. The ISA firewall and Exchange Server are on Windows Server 2003 and the external test client is running Windows XP.

    We created an SMTP Server Publishing Rule that publishes the Exchange Server at the Exchange Server’s IP address 10.0.0.2. The listener on the ISA firewall for the SMTP Server Publishing Rule uses IP address 192.168.1.70. DNS entries are in place with an A record that points to 192.168.1.70 and an MX record for our e-mail domain that points to this A record. Everything should be working, but users inform us that they are not receiving e-mail and when we try to send e-mail from a Hotmail account, it doesn’t arrive to the Exchange Server.

    This is a common scenario and one I see on the www.isaserver.org Web boards and mailing list everyday. Solving the problem is relatively simple, it just requires a methodical approach. Let’s start our investigations by using Telnet

    Get the New Book!

     

    Using Telnet

    The first thing we want to do is make sure it’s not a DNS issue. DNS issues are the most common problems when it comes to Web and Server Publishing Rules. We can quickly check if this is the case in this situation by using Telnet on an external host and trying to connect to the Exchange Server’s SMTP service using the IP address for the Server Publishing Rule’s listener address.

    Open a command prompt on the external client machine. At the command prompt enter the following:

    telnet 192.168.1.70 25

    and press ENTER. Replace 192.168.1.70 with the IP address you’re using for the listener on your SMTP Server Publishing Rule. If you see what appears in the figure below, you know that it’s not primarily a DNS problem that’s causing the connectivity issues. That isn’t to say that there can’t also be a DNS problem, but at this point we know for sure that there is a problem that isn’t DNS related.

    Using the ISA Firewall’s Real-Time Log Viewer

    Now we have to wonder if the Telnet connection even made it to the external interface of the ISA firewall. We can quickly check this out by using the ISA firewall’s real-time log viewer. Perform the following steps to configure the real-time log viewer to focus on the connections of interest:

    1. In the Microsoft Internet Security and Acceleration Server 2004 management console, expand the server name and click the Monitoring node.
    2. On the Monitoring node, click the Logging tab. While on the logging tab, click the Tasks tab in the Task Pane. Under the Logging Tasks list, click the Edit Filter link.
    3. In the Edit Filter dialog box, click the down-arrow in the Filter by drop-down list. Click the Protocol entry. Click the down-arrow for the Condition drop-down list and click the Equals entry. Click the down-arrow in the Value drop-down list and select SMTP Server. Click the Add to List button.

    1. Click the Start Query button.
    2. Telnet to the ISA firewall again. In the ISA firewall’s real-time log viewer, you’ll see something like the figure below. This proves that your telnet connection actually made it to the external interface of the ISA firewall but the connection failed. The Rule column shows that a Server Publishing Rule named SMTP Server allowed the inbound connection to the SMTP server. Notice the Client IP address is 192.168.1.90, which is the IP address of the host on the external network, and the Destination IP is the address of the SMTP server on the Internal Network.

    At the point we know the connections are making it to the ISA firewall from our test client on the external network and we know that the Server Publishing Rule is correctly configured. The next step is to use Network Monitor to see what might be happening to these connections.

    Get the New Book!

     

    Using Network Monitor on the ISA Firewall and Exchange Server

    We can use Network Monitor on the ISA firewall and on the Exchange Server to arrive at a solution for this problem. You’ll find that using Network Monitor can solve the vast majority of ISA firewall and related problems you encounter.

    The first step is to install Network Monitor, which is not installed by default. You can install Network Monitor using the Add/Remove Programs applet in the Control Panel. If you haven’t already done so, install Network Monitor on both the ISA firewall and the Exchange Server.

    Perform the following steps on the ISA firewall after installing Network Monitor:

    1. On the ISA firewall machine, click Start and point to All Programs. Point to the Administrative Tools menu and click Network Monitor.
    2. The first thing you’ll see is a request for you to specify a network for Network Monitor to listen on. Click OK in the Microsoft Network Monitor dialog box.

    1. In the Select a network dialog box, expand the Local Computer node in the left pane. Under the Local Computer node you’ll see a list of network interfaces on the ISA firewall machine. Notice that the interfaces in this example have meaningful names. The names are those I gave the interfaces by renaming the interfaces in the Network and Dial-up Connections window. If you didn’t rename the interfaces, then you’ll see something like Local Area Connection 1, Local Area Connection 2, etc. You’ll then need to figure out which of the Local Area Connections represent which interface. In this example we want Network Monitor to listen on the LAN interface. Click the interface representing the network interface closest to your Exchange Server and click OK.

    1. Click the Capture menu and click the Buffer Settings command.
    2. In the Capture Buffer Settings dialog box, enter 100 in the Buffer Size (MB) text box. This will help prevent missing packets we’re interested in because we ran out of buffer space for the capture. Click OK.

    1. Start the capture by clicking the Start Capture button, which looks like a right-pointing arrow in the button bar.
    2. Go back to the test client on the external network and run the telnet command again.
    3. Return to the ISA firewall and click the Stop and View Capture button in the console’s button bar. This button looks like a square with a pair of eye-glasses above the square.
    4. If you have a busy firewall, your capture file can get quite large. You can create a display filter to help simplify things. In the [Capture 1: (Summary)] window, click the Display menu and then click Filter.
    5. In the Display Filter dialog box, click the Protocol == Any entry and click Edit Expression.

    1. In the Expression dialog box, click the Property tab. On the Property tab, scroll down to find the TCP entry in Protocol Property list. Double click TCP and click the Destination Port entry. Click the = entry in the Relation list. Select the Decimal option and then enter 25 in the Value (Word) text box. Click OK.

    1. Click OK in the Display Filter dialog box.

    1. In the figure below you can see the filtered list of SMTP connections. The figure shows that our external host at 192.168.1.90 is attempting a connection to IP address 10.0.0.2. This is similar to the information we gathered from the real-time log viewer. However, what is significant here is that these packets are leaving the internal interface. In contrast, what we saw in the real-time log viewer was the connection attempt on the external interface. Since the Network Monitor is configured to listen on traffic on the internal interface, we only see traffic handled by that interface. In this case, we see traffic leaving the internal interface and destined to the Exchange Server’s IP address. This indicates the ISA firewall’s Server Publishing Rule is doing what it’s supposed to do.

    Since the ISA firewall’s Server Publishing Rule is working correctly, the next step is to run Network Monitor on the Exchange Server. Install Network Monitor on the Exchange Server and then perform the following steps:

    1. At the Exchange Server, click Start and point to All Programs. Point to Administrative Tools and click Network Monitor.
    2. Click OK in the dialog box asking you to configure a network on which you want to capture data.
    3. In the Select a network dialog box, expand the Local Computer node in the lane pane and click the Local Area Connection entry (since the Exchange Server would only have one NIC, there’s no need to rename the interfaces). Click OK.
    4. Click the Capture menu and then click the Buffer Settings command.
    5. In the Capture Buffer Settings dialog box, enter 100 in the Buffer Size (MB) text box. Click OK.
    6. Click the Start Capture button in the Network Monitor button bar.
    7. Run the telnet command on the external client.
    8. Return to the Exchange Server and click the Stop and View Capture button.
    9. Since we’re not really sure what we’re looking for here, we won’t filter this capture. You’ll see something like what appears in the figure below. The capture shows that the incoming connections are arriving to the Exchange Server. After each connection we see an ARP broadcast seeking the MAC address of 10.0.0.10. These ARP Request messages don’t have any match ARP Reply messages. This indicates that host 10.0.0.10 is either offline or doesn’t exist. The question you need to ask yourself at this point is “why is the Exchange Server trying to find this IP address”?

    The reason why there are no ARP replies in this example is that there is no host with the IP address 10.0.0.10. So why is the Exchange Server looking for 10.0.0.10 each time the external host connects to the Exchange Server? The Exchange Server is trying to reply to the external client by connecting to its default gateway. The response must go to the Exchange Server’s default gateway because the destination IP address of the test client is on a different network ID than the Exchange Server. In this example the Exchange Server was configured with 10.0.0.10 as its default gateway, probably because of a typo when configuring the default gateway setting on the Exchange Server’s network interface.

    Get the New Book!

     

    Correcting the Problem

    There are two ways we can solve this problem:

  • Configure the Exchange Server as a SecureNAT client of the ISA firewall by fixing the default gateway address
  • Configure the SMTP Server Publishing Rule to use the ISA firewall’s IP address as the client source IP address
  • The default configuration for Server Publishing Rules requires the published server to be a SecureNAT client. This is similar to what we saw with ISA Server 2000. However, you have a choice with the new ISA firewall: you can preserve the source IP address of the remote client (the default setting), or you can replace the remote client’s IP address with the IP address of the ISA firewall’s internal interface. When you choose the latter option, the published server does not need to be a SecureNAT client.

    Let’s see what happens when we change the default gateway on the Exchange Server to be the IP address on the internal interface of the ISA firewall. After making the change, we can see that the incoming connections make it to the Exchange Server and the outgoing responses are returned to the test host on the external network. The responses made their way outbound because the new default gateway configured on the Exchange Server enabled it to return responses to the host on a different network ID.

    At the test client on the external network, you’ll see something like what appears below in the Telnet window.

    Another option is to configure the Server Publishing Rule to replace the client IP address. To do this, double click the SMTP Server Publishing Rule and click the To tab. On the To tab, select the Requests appear to come form the ISA Server computer option. Click Apply and then click OK. Then click OK to save the changes to the firewall policy. Click OK in the Apply New Configuration dialog box.

    Repeat the telnet connection again while running Network Monitor on the Exchange Server. Now you see the client address sending the SMTP message to the Exchange Server as 10.0.0.1, which is the IP address on the internal interface of the ISA firewall. The Exchange Server responds by sending the replies to 10.0.0.1, since that is the client IP address seen by the Exchange Server. In this case, there is no need to make the Exchange Server a SecureNAT client, because the Exchange Server is not responding to a host on an external network.

    The advantage of replacing the client IP address with the address of the internal interface of the ISA firewall is that you don’t need to make the Exchange Server a SecureNAT client, and you don’t have to make any changes in your routing infrastructure to support SecureNAT clients that are on networks remote from the ISA firewall. The downside of this configuration is that you won’t have the actual remote client’s IP address in the SMTP log, although it does appear in the ISA firewall’s Firewall Service log.

    Get the New Book!

     

    Summary

    In this article we discussed how you can use the Telnet, the ISA firewall’s real-time log viewer, and Network Monitor to troubleshoot an SMTP Server Publishing Rule. These same methods and principles can be used to troubleshoot any other type of Server Publishing Rule. In the example discussed in this article, the Exchange Server was not configured as a SecureNAT client, so its replies could not be returned to the external client. We went over two solutions to this problem: one was to make the Exchange Server a SecureNAT client and the second was to configure the Server Publishing Rule to change the client IP address to the address used on the internal interface of the ISA firewall.

    I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=22;t=000168 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks! –Tom

    If you would like us to email you when Tom Shinder releases another article on ISAserver.org, subscribe to our ‘Real-Time Article Update’ by clicking here. Please note that we do NOT sell or rent the email addresses belonging to our subscribers; we respect your privacy.

    Leave a Comment

    Your email address will not be published. Required fields are marked *

    This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

    Scroll to Top