ISA Server SMTP Server Support.

How to configure ISA Server to support internal SMTP servers is a really popular subject on the mailing list and web boards. Making SMTP Servers work with ISA Server is really quite easy; you just need to know a few tricks. Once you know the tricks, your mail servers will be up and running in no time.

In this article I will go over what you need to do so that your internal mail servers can send outbound mail and how your internal mail servers can receive inbound mail. We’ll go over the following topics:

• Configuring DNS to support ISA Server
  
• Configuring Protocol Rules to support the internal SMTP and DNS Servers
  
• Configuring the SMTP Server Publishing Rule
  
• Checklist: Putting it all together

Once you understand and apply these sure-fire techniques, you’ll never have to worry about your ISA Server/SMTP Server configuration again.

Configuring the Internal DNS Servers

If there’s one subject that will cause a stir, its configuring DNS. However, you’ve got to know how to configure your internal DNS server to support DNS queries for internal network clients.

First, you must have a DNS server on your internal network. If you are running a Windows 2000 domain, you already have one installed. If you are running a Windows NT 4.0 domain, you may not have a DNS server installed because you didn’t need one. If you don’t have a DNS server installed, install one now.

Now that you have the DNS server installed, you need to configure it so that it is able to resolve Internet host names. There are a couple of ways you can do this:

• Ensure that the Root Hints file contains information on the Internet Root Servers
  
• Configure the DNS Server to use a Forwarder for domains that is it not authoritative

The Root Hints file contains the names and IP addresses of the Internet Root Servers. The DNS server uses this information as a starting place to perform iterative queries that allows it resolve the names for foreign domains (domains for which the DNS server is not authoritative). To find your Root Hints file information, perform the following steps:

1. Open the DNS management console from the Administrative Tools menu.
   
2. Right click on your server name and click Properties.
   
3. Click on the Root Hints tab.

Once you apply the fix, you will be able to configure the DNS Server to use a Forwarder.

Once the DNS Server is setup, you must configure the SMTP Server to use the internal DNS server for name resolution. The internal SMTP Server must be a SecureNAT client. SecureNAT clients are not able to use the ISA Server to resolve names on their behalf. Even if you do not want to publish the SMTP server, you should make it a SecureNAT client.

After configuring the SMTP server to use the internal DNS server, test your configuration by using the nslookup command. Perform the following steps to test your DNS configuration:

1. Open a command prompt and type nslookup and press [ENTER]


2. Type set type=mx and press [ENTER]


3. Type microsoft.com. (be sure to include the trailing period) and press [ENTER]


4. Your display should look like what appears below.

WARNING:

Don’t perform this test until you have a Protocol Rule that supports outbound access for DNS queries.

So, to sum up your DNS configuration:

1. You will have an internal DNS Server configured on the internal network


2. You will configure the internal DNS Server to use a Forwarder


3. You will configure your SMTP server as a SecureNAT client and configure it to use your internal DNS server


4. You will test your DNS configuration by using the nslookup utility

NOTE:
If you want to avoid all this DNS stuff, you can configure your SMTP servers to use a Smart Host. When configured to use a Smart Host, the SMTP server will forward all mail to the Smart Host. It then becomes the responsibility of the Smart Host to resolve the mail domain names.

Configuring Protocol Rules to Support the Internal SMTP and DNS Servers

You have to create Protocol Rules to support your SMTP and DNS servers:

• A Protocol Rule that allows outbound access to the SMTP server port TCP 25


• A Protocol Rule to support outbound DNS queries

The Protocol Rule to support outbound access to SMTP server port TCP 25 is required for SMTP servers that resolve their own mail domain names, and for those that use Smart Hosts. The internal SMTP server must be able to send mail to external SMTP servers through TCP port 25.

To create the protocol rule, perform the following steps:

1. In the ISA Management console, expand the Access Policy node and right click on the Protocol Rules node. Click New and then click Rule.


2. Name the rule Outbound SMTP and click Next.


3. For the Rule Action select Allow and click Next.


4. On the Protocols page, click the down-arrow and select the Selected Protocols option. Scroll down the list and place a checkmark in the checkbox for SMTP and click Next.


5. Set the Schedule for Always and then click Next.


6. For the client type, make the appropriate selection for your network’s access policy. If you want to allow all machines on the network access to outbound SMTP, then you can select Any request. However, if you want to limit outbound SMTP to the SMTP server, then you should first create a Client Address Set that contains the IP addresses of your internal SMTP servers. After the Client Address Set is configured, select the Specific computers (client address set) option and use your SMTP servers client address set. In this example, we’ll assume that everyone has access to outbound SMTP. Make your selection and click Next.


7. Confirm your settings and click Finish.

The next step is to configure a Protocol Rule that allows outbound DNS queries. Normally, DNS queries use UDP port 53. However, DNS will use TCP port 53 if the entire query will not fit into a single UDP datagram. Also, the IIS 5.0 SMTP service needs to use TCP 53 to perform DNS queries. Although your SMTP server will be sending DNS queries directly to the internal DNS server, it is wise to create protocol rules to support both TCP and UDP DNS query requests.

To create the DNS rules, perform the following steps:

1. In the ISA Management console, expand the Access Policy node and right click on the Protocol Rules node. Click New and then click Rule.


2. Name the rule Outbound DNS and click Next.


3. For the Rule Action select Allow and click Next.


4. On the Protocols page, click the down-arrow and select the Selected Protocols option. Scroll down the list and place a checkmark in the checkbox for DNS Query and DNS Zone Transfer. Click Next.


5. Set the Schedule for Always and then click Next.


6. For the client type, make the appropriate selection for your network’s access policy. If you want to allow all machines on the network access to outbound DNS queries, then you can select Any request. However, if you want to limit outbound DNS queries to the DNS server, then you should first create a Client Address Set that contains the IP addresses of your internal DNS servers. After the Client Address Set is configured, select the Specific computers (client address set) option and use your DNS servers client address set. In this example, we’ll assume that everyone has access to outbound DNS. Make your selection and click Next.


7. Confirm your settings and click Finish.

To sum up your Protocol Rules:

1. You will create a Protocol Rule that allows the SMTP server outbound access to TCP port 25


2. You will create a Protocol Rule that allow the internal DNS server outbound access to TCP and UDP port 53

Configuring the SMTP Server Publishing Rule

If you are running an internal SMTP server, you probably want to allow both inbound and outbound access to and from the machine. We have allowed outbound access by creating the SMTP Protocol Rule. However, in order for external SMTP servers to access your internal SMTP server (and also external SMTP mail clients), then you must publish the server.

I am assuming that you are publishing an SMTP server on the internal network, and that you are not trying to publish or run the SMTP service on the ISA Server itself. Please never run IIS services such as SMTP, WWW, NNTP or FTP on the ISA Server itself. You purchased ISA Server as a firewall to protect your internal network from attack. When you run additional services on the firewall, you open it up for a number of attacks that can disable your firewall security.

If you must run some IIS services on the ISA Server itself, be sure to disable the SMTP service on the ISA Server. To disable the IIS SMTP service, perform the following steps:

1. From the Administrative Tools menu, click the Services command.


2. Scroll down the list of services and find the Simple Mail Transport Protocol (SMTP) entry. Double click on it now.


3. On the General tab, change the Startup type setting to either Manual or Disabled. I usually choose Manual on test machines, but on production machines you’re best off by disabling the service.


4. Click Apply and then click OK.

Now that I’m off that soapbox, we can get to publishing the SMTP server. ISA Server includes a Secure Mail Server Publishing Wizard that can simplify the server publishing process. However, unless you understand what the Wizard is doing, you might find that things don’t work the way you expect them to. I find it much easier to publish the SMTP server by creating a Server Publishing Rule manually.

To create the SMTP server publishing rule, perform the following steps:

1. In the ISA Management console, expand the Publishing node and then right click on the Server Publishing node. Click on the New command and then click Rule.


2. Name the rule Internal Mail Server and click Next. If you wish to publish more than one mail server, you might use the name of the server in the name of the rule.


3. On the Address Mapping page, enter the IP address of the internal SMTP server and the IP address on the external interface of the ISA Server to which your mail domain resolves to. Remember that you must have a DNS entry on a publicly available DNS server that includes an MX record for your mail domain. Then click Next.


4. On the Protocol Settings page, select the SMTP Server protocol and click Next.


5. On the Client Type page, select Any request and click Next.


6. On the last page of the Wizard, confirm your settings and click Finish.

Before depending on your rule, confirm the port is open and that a firewall session exists between the SMTP and the ISA Server. To confirm that the port is open, open a command prompt and type netstat -na (or, you can type in netstat -an). You should see that only your external IP address is listening on port 25. If you see that 0.0.0.0 is listening on port 25, then you forgot to whack the IIS SMTP service. To confirm that a Firewall session is active between the SMTP server and the ISA Server, check the Sessions node in the ISA Management console.

To sum up your SMTP Server Publishing Rules:

1. You will disable the IIS SMTP service on the ISA Server


2. You will not use the Secure Publishing Wizard


3. You will manually publish your internal SMTP server using a Server Publishing Rule


4. You will confirm that the Server Publishing Rule worked by using the netstat -na command from the command prompt and insure that there is a firewall session active between the ISA Server and the SMTP server.

Checklist: Putting it All Together

To make it easy for you to remember what you need to do, here’s a checklist of what we talked about in this article:

1. You will have an internal DNS Server configured on the internal network


2. You will configure the internal DNS Server to use a Forwarder


3. You will configure your SMTP server as a SecureNAT client and configure it to use your internal DNS server


4. You will test your DNS configuration by using the nslookup utility


5. You will create a Protocol Rule that allows the SMTP server outbound access to TCP port 25


6. You will create a Protocol Rule that allows the internal DNS server outbound access to TCP and UDP port 53


7. You will disable the IIS SMTP service on the ISA Server


8. You will not use the Secure Publishing Wizard


9. You will manually publish your internal SMTP server using a Server Publishing Rule


10. You will confirm that the Server Publishing Rule worked by suing the netstat -na command from the command prompt and insure that there is a firewall session active between the ISA Server and the SMTP server.

Summary

Configuring ISA Server to get along with your internal SMTP servers is a snap, but you must make sure your network configured correctly. In this article, I presented a method you can use to assure that everything works. This is not the only way to accomplish the task, but I can guarantee this method is sound. You will run into problems if you do things like using the IIS SMTP service on the ISA Server. It you don’t try to force ISA Server to do things it doesn’t want to do, you’ll be in Fat City!