Publishing SMTP Servers: Supporting SMTP Authentication, Part 2

Publishing SMTP Servers: Supporting SMTP Authentication, Part 2

By Thomas W Shinder M.D.

In part 1 of this article, I covered the topic of authenticating with SMTP servers on the internal network and how you can use the new SMTP filter included with Feature Pack 1 to make this happen. In this article, I discuss some of the specifics of how to configure the corporate user SMTP server, the Internet relay SMTP server and the Exchange server.

Get the Book!

Configure the Corporate SMTP Server

The corporate SMTP server is actually on the same physical machine as the Internet mail relay server. SMTP Virtual Server 1 is the corporate user SMTP server. Requirements for this server include:

  • Configuring the authentication method
  • Configuring a bogus domain name for the virtual server
  • Configuring relay for only clients who authenticate
  • Configuring the server to use the Exchange Server a smart host
  • Configuring the Authentication Method

    You must prevent the SMTP server from becoming an open mail relay. The best way to do this is to require clients to authenticate with the server. Perform the following steps to configure authentication:

    1. Open the Internet Information Services console from the Administrative Tools menu.
    2. Expand the server name and right click on the virtual SMTP server name. Click on the Properties command.
    3. Click on the Access tab in the virtual server Properties dialog box. On the Access tab, click on the Authentication button.

    1. In the Authentication dialog box, you have the options to require Basic authentication or Windows security package authentication. We all understand what Basic security is about. It passes user name and password information in the clear. I highly recommend that you do not use Basic authentication, since anyone can sniff that information on the client or server network. It would be realistic to use Basic authentication if you could use TLS encryption, but TLS encryption is not supported when the SMTP filter is enabled. A better option is the Windows security package. I’d like to tell you that I knew for sure what this was, but I don’t have to Rosetta stone that can translate this classic Help file description (I assume it’s the same as NTLM, but who knows?):

      Select this option to enable the standard Security Package security mechanism that is provided with Microsoft Windows 2000. This security feature makes it possible for businesses to provide secure logon services for their customers. Virtual servers that already use Windows Security Package in an internal system can benefit by using a single, common security mechanism. Windows Security Package authentication uses a cryptographic technique for authenticating users and does not require the user to transmit actual passwords across the network. For more information, see the Microsoft Internet Information Services documentation. Note: Using Windows Security Package authentication requires a mail client that supports this authentication method. Microsoft Outlook Express supports Windows Security Package authentication.

    1. Click OK after selecting your authentication protocol. Click Apply and then OK in the server Properties dialog box.

    Configuring the Virtual Server and Bogus Domain

    When you create a new virtual SMTP server, you need to assign that virtual domain a local domain name. This domain is the domain for which this SMTP server is the endpoint. Since the corporate user SMTP is not the endpoint of any mail messages, you want to assign the server a bogus local domain name. Let’s create a new Virtual Server to illustrate the process:

    1. In the Internet Information Services console, right click on the Default SMTP Virtual Server node in the left pane of the console, point to New and click Virtual Server.
    2. On the Welcome to the New SMTP Virtual Server Wizard page, type in a descriptive name of the virtual server. In this example we’ll call it CorpServer, to denote the fact that this server is used by remote corporate users. Click Next.

    1. On the Select IP Address page, click the down arrow in the drop down list box and select the IP address you want the virtual server to listen on. You’ll need to bind multiple IP address to this machine’s interface. You need one IP address for each virtual server you create. Make sure that you disable the Default SMTP Virtual Server and that you do not allow any virtual server on this machine to listen on all IP addresses. Click Next.

    1. On the Select Home Directory page, type in a path, or browse, to the directory that you want to become the root of the virtual server’s folder hierarchy. You can type in a path and the Wizard will create the folder for you. Click Next.
    2. On the Select Default Domain name. Type in a bogus domain in the Type the default domain for this virtual server text box. In this example, I’ll type in Bogus1. Click Finish.


    Configure the Server to Relay Only for Authenticated Users

    The corporate user virtual SMTP server only relays for users who authenticate. While we could configure this server to relay all mail (because the server does not accept anonymous connections), its better to increase the security of the server by restricting relay to users who authenticate.

    Right click on the CorpServer virtual server and click the Properties command. Click on the Access tab and click the Relay button. In the Relay Restrictions dialog box, put a checkmark in the Allow all computers which successfully authenticate to relay, regardless of the list above checkbox. Select the Only the list below option. These two options set the server to delay mail relay to everyone except those who successfully authenticate.


    Configure the Virtual Server to Use Exchange as its Smart Host

    The final thing we need to take care of on the corporate user virtual server is to have it use the Exchange Server as its smart host. This is something that may or may not be required depending on your environment, depending on how your have you DNS infrastructure set up. You do not need to configure the corporate user virtual server to use the Exchange Server as a smart host if you’ve configured a split DNS infrastructure. If you’ve configured a split DNS infrastructure, then mail your external users send to your own mail domains will be sent to the Exchange Server. If you haven’t configured a split DNS infrastructure, then the mail will be relayed to the external interface of the ISA Server and you’ll end up with the “isotropic bounce” phenomenon described by Jim Harrison. You’ll see this bounce when SecureNAT clients attempt to “loop back” through the external interface of the ISA Server to access internal network resources.


    For a thorough explanation of the issues involved with configuring relay in these various scenarios, check out the mail publishing sections in ISA Server and Beyond .

    I’ll assume that you didn’t configure a split DNS. In this case, you should configure the Exchange Server as the corporate user SMTP server’s smart host. Right click on the virtual SMTP server and click the Properties command. In the virtual server’s Properties dialog box, click the Delivery tab. On the Delivery tab, click on the Advanced button.

    This takes you to the Advanced Delivery dialog box. In the Smart host text box, type in the IP address of the Exchange Server and enclose that address in brackets. It is critical that you enclose the IP address in brackets. If you don’t, it won’t work. You can also enter a FQDN if your DNS infrastructure is appropriately configured.

    Once you configure the corporate user SMTP relay server, you’ll have what we appear in the figure below.

    1. The external corporate user sitting in his hotel room connected to a broadband connection provided by the hotel sends an SMTP message to the external IP address on the ISA Server used for the corporate user SMTP server publishing rule.
    2. The ISA Server forwards the SMTP message to SMTP Virtual Server 1, which is the corporate user SMTP server.
    3. The corporate user SMTP server uses the Exchange Server as its smart host. The Exchange Server is configured to allow the corporate user SMTP server to relay messages through it.
    4. The Exchange server relays messages for all domains that it is not responsible for. If mail addressed to users for you own mail domains is received by the Exchange Server, the Exchange server will route it to the appropriate user’s mailbox.

    The scenario is a bit different when you don’t configure the corporate user SMTP virtual server use the Exchange Server as its relay (that is to say, you’ve configured a split DNS). The figure below shows what happens in that scenario.

    Step 3 shows the path for the mail domains until your administrative control. You’ll need MX records in your split DNS infrastructure that point to the Exchange Server for those mail domains under your control. Step 4 shows the path for messages for all other domains. Mail for external domains must go out through the ISA Server (via an SMTP Protocol Rule) to Internet SMTP servers.

    Get the New Book!

    Configure the Internet SMTP Relay Server

    The Internet SMTP relay server is used to accept mail anonymously from Internet SMTP servers. You don’t have to use an Internet mail relay server; you can allow your Exchange server to accept inbound mail for mail domains until your administrative control. I prefer, and so do most security administrators, to put a device between the Exchange server and Internet host. This prevents Internet servers from directly communicating with the Exchange server.

    Configuration requirements for the Internet SMTP relay server include:

  • Preventing open relay
  • Configuring remote domains for your organizations mail domains
  • Configuring the remote domains to use your Exchange server as a smart host
  • Preventing relay is especially critical for the Internet SMTP relay server because no authentication is required to relay mail through the server. You must disable open relay and allow relay only for your own mail domains. You create a remote domain for each mail domain under your control. For example, if you want to receive mail for, then you create a remote domain for and then configure that remote domain to relay messages to the Exchange server on the internal network.

    Let’s look at the path of the messages moving through the Internet SMTP mail relay server.

    1. Internet SMTP server sends an SMTP message to the ISA Server’s external IP address that is being used by the Server Publishing Rule listening for messages destined to the Internet SMTP server.
    2. The ISA Server forwards the message SMTP Virtual Server 2, which is the Internet SMTP relay server.
    3. The Internet SMTP relay server examines the SMTP message and evaluates the destination domain. If the server has a remote domain configured that matches the domain in the message, the message will be forwarded to the smart host for that domain (which is the Exchange Server). With this setup, the smart host is the endpoint of the message. If there is no remote domain on the Internet SMTP relay server that matches the destinatino domain in the message, then the message is dropped.

    If you’ve not configured an Internet SMTP relay server, check out my article Mail Relay Scenario Using Mail Essentials 6.0 for SMTP Gateways. If you’d like a step by step on the configuration, check out mail relay scenario configuration examples in the ISA Server and Beyond book.

    Configuring the Exchange Server

    Microsoft Exchange is a massive piece of software, and we can’t get into the full installation and configuration details in this article. However, there are configuration steps that you can carry out fairly easily, which apply to both Exchange and non-Exchange SMTP servers. Things you need to take into consideration include:

  • Preventing SMTP relay except for authorized computers
  • Configuring Exchange to forward mail for Internet mail domains
  • Consider using a smart host
  • Consider using external DNS servers for name resolution
  • Note that we’re focusing only on the SMTP configuration issues. There are many more configuration issues you may have to consider on the Exchange Server, depending on what roles you want the server to perform. I’ve included a ton of details on Exchange Server configuration in ISA Server and Beyond.

    Get the New Book!

    Preventing Mail Relay

    As with all the SMTP servers we’ve covered so far, you don’t want the Exchange SMTP service to be able to relay for all users. You don’t have to worry about allowing relay for your internal Outlook clients that are using MAPI to connect to the Exchange Server. If you have internal network clients that need to use SMTP send mail to the Exchange Server, you have two options: you can have these clients send mail to the Internet SMTP mail relay server, or you can configure the Exchange Server to allow relay for authenticated clients or clients from certain IP addresses. The best solution is to require all inbound mail from your internal SMTP clients to go through the Internet SMTP relay server.

    The figure below shows the relay configuration on the Exchange Server’s SMTP service. In this case, only the corporate user SMTP server is allowed to relay through the Exchange Server. Remember that the external corporate users must be able to relay so that they can send mail to mail domains not under your control.


    Configure the Exchange Server to Relay Mail to non-Local Domains, Using a Smart Host, and Mail Domain Name Resolution

    The Exchange Server must be configured so that it can forward mail to Internet mail domains. The Exchange Server must forward mail for all domains that it isn’t responsible for. In order to accomplish this task, the Exchange Server must be able to resolve the Internet mail domain name, and once the mail domain name is resolve, it must be able to forward the SMTP message through the ISA Server.

    There are several ways the Exchange SMTP service can resolve Internet mail domain names:

  • Names can be resolved using an internal DNS Server
  • Names can be resolved using an external DNS Server
  • Name resolution can be handled by the smart host
  • If your internal network’s DNS infrastructure is well designed, you will have configured DNS servers to resolve Internet host names. These DNS servers can be configured to resolve Internet host names via a forwarder or by performing recursion themselves. I prefer to use a forwarder for security reasons. When you use a forwarder, your DNS server only communicates with the forwarder. In contrast, when your DNS server performs recursion itself, it must contact and communicate with a number of DNS servers on the Internet. This has the potential of opening up your DNS servers to attacks that you would not otherwise have to worry about if you used a forwarder.

    Regardless if you use a forwarder or not, you need to create two DNS Protocol Rules to support these DNS servers. These are the DNS Query and the DNS Zone Transfer Protocol Rules. The Zone Transfer Protocol Rule is required because the MX queries require messages that don’t fit into a single UDP packet.

    However, you might not want to depend on your internal DNS infrastructure to resolve Internet mail domain names. In that case, you can configure the Exchange Server to use an external DNS server. Take a look at the figure below. Notice that in the Advanced Delivery dialog box that you can click the Configure button to get the Configure dialog box to configure external DNS servers for the virtual server. This allows the SMTP service to bypass the DNS configuration on the Exchange Server and use the servers you list here.

    If you want to almost completely bypass name resolution issues, you can use a smart host. Forwarding all external mail messages to a smart host offloads the name resolution chores to the smart host. The Exchange SMTP service doesn’t have to do any name resolution (except for maybe the name of the smart host, if you decide to use a FQDN to forward the message, if you use an IP address, no name resolution for each hosts is required). The smart host is typically your ISPs SMTP server. You’ll have no problems using a smart host if you have a good ISP that knows how to manage its Internet services infrastructure.

    Get the Book!


    ISA Server Feature Pack 1 includes a new and improved SMTP filter that allows you to authenticate with published SMTP servers while the filter is enabled. Now that you can authenticate with the SMTP server, you can put up an SMTP server for your external network users to send SMTP mail to. I also covered some network infrastructure issues and how you can configure SMTP virtual servers to provide SMTP relay for both your corporate users and anonymous Internet SMTP servers. Using the techniques described in parts 1 and 2 of this article, you can safety publish your SMTP servers and not worry about spammers taking advantage of an open SMTP relay.

    I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to;f=6;t=001391 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks! –Tom

    About The Author

    Leave a Comment

    Your email address will not be published. Required fields are marked *

    This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

    Scroll to Top