Publishing SMTP Servers: Supporting SMTP Authentication, part 1

Publishing SMTP Servers: Supporting SMTP Authentication, part 1

By Thomas W Shinder M.D.

Black Flag, a band that used to play in my home town a lot when I was an adolescent, had a song named “TV Party”. If you’ve ever heard that song, you might recall the memorable lyrics “TV News shows what its like out there. It’s a scare! You can go out if you want, we don’t care!” I try to stay close to home, but sometimes I get dragged out of our sleepy hollow and I need to stay in touch with my Exchange mailbox. You probably have traveling users who need to connect to your Exchange Server while they’re on the road too.

Get the Book!

I prefer to configure Exchange Servers to use Exchange RPC publishing. This allows users with the “big” Outlook client (Outlook 2000/2002) to connect to an Exchange server in the same they connect to the server when the clients are on the internal network. There are a lot of advantages to this configuration. If you’ve never checked it out, read my article Using the Exchange RPC Filter to Publish Exchange 2000.

The problem is that not everyone uses big Outlook, especially when on the road. Many users are stuck with Outlook Express or some other convention POP3/SMTP client. You need to make an SMTP and POP3 server available to these users. Publishing POP3 and SMTP servers is very easy, but until recently, there was an important snag that made supporting external SMTP clients problematic. Publishing IMAP4 was a solution, because while you could use IMAP to get mail from the Exchange Server, you still have to use SMTP to send the mail.

For example, suppose you’re in Las Vegas and you have a laptop computer with only Outlook Express. Even though you’re in Vegas, there’s still work to be done! You need to pull your mail from the POP3 server, and you need to send your mail to an SMTP server. It’s very easy to publish a POP3 server and connect to that server from anywhere in the world. It’s also easy to publish an SMTP server and connect to that server from anywhere in the world.

But, there’s the problem. When you publish an SMTP server, you must make sure that server is not configured as an open SMTP relay. An open SMTP relay allows anyone to send mail destined to anywhere from it. This is why spammers love open SMTP relays. They can send spam through the relay and you get the blame. Spammers especially like it when they can connect to your private network via a non-secured wireless LAN and forward mail through an open relay.

There’s an easy fix for this problem: require that users authenticate before they can connect to your SMTP server. Once the user authenticates, they can send SMTP mail to the server and the server will relay that to any mail server in the world. This isn’t an open relay because if the user cannot authenticate, the user will not be able to send mail. It’s unlikely that the spammer will be privy to your network’s user names and passwords.

The only problem was that ISA Server did not support the SMTP AUTH command when the SMTP filter was enabled. Sure, you could disable the SMTP filter, but then you would miss out on the benefits of the filter, such as SMTP command screening and the SMTP Message Screener. There was no way to have the best of all possible worlds.

That is until now. When you install the Microsoft ISA Server Feature Pack 1, you benefit from a new and improved SMTP Filter. You won’t see any changes in the filter interface, but you’ll notice improved Message Screener performance and you’ll also be able to AUTH through to your published SMTP Servers.

Let’s look at an example of how you can leverage SMTP AUTH to provide SMTP servers for your external corporate users and SMTP servers that relay mail from anonymous Internet mail servers. The figure below shows the basic setup.

The Corp User with the laptop uses a conventional POP3/SMTP client. The Internet SMTP Server needs to relay mail for your email domains through your SMTP server. SMTP Virtual Server 1 is located on an IIS 5.0 SMTP server and is used by your external users to send SMTP messages to all domains. SMTP Virtual Server 2 is an SMTP relay, that accepts mail only for your email domains and drops all other mail. The Exchange Server can be an Exchange 2000 or Exchange 5.5, or it can be any other SMTP server that your organization might use.

In order to make this work, you need to take care of the following tasks:

  • Configure the ISA Server
  • Configure the corporate user SMTP server to require authentication
  • Configure the SMTP relay to relay mail only for your email domains
  • Configure the Exchange Server (or whatever SMTP server you’re using)
  • In this part of the article we’ll go over some of the important ISA Server settings. We’ll cover the last three bullet points in the second part of this article.

    Get the New Book!

    Configure the ISA Server

    The ISA Server doesn’t require any special or unusual configuration steps. Most of these are included in our typical ISA Server setups. The minimal requirements include:

  • The machine must have an external and an internal interface
  • You must use integrated or firewall mode
  • You must have the same number of IP addresses bound to the external interface as the number of SMTP servers you want to publish; then create the SMTP Server Publishing Rules
  • You must disable the SMTP service on the ISA Server, or disable socket pooling and bind the SMTP service to the internal interface.
  • You must create Protocol and Site/Content Rules to support any servers that require outbound access
  • Install ISA Server Feature Pack 1
  • Enable and Configure the SMTP Filter
  • The Machine Must Have an Internal and External Interface

    The machine must have an internal and external interface because the only supported unihomed configuration is the Web cache mode ISA Server. You can’t publish servers when the ISA Server has only a single interface. The unihomed ISA Server doesn’t have a concept of internal and external networks, so it has no way of assessing how to pass messages from a untrusted (external) to an trusted (internal) network.

    Install ISA Server in Integrated or Firewall Mode

    Server Publishing Rules can be created on ISA Servers installed in Integrated or Firewall mode. You can’t create Server Publishing Rules when ISA Server is installed in Cache Mode. If you notice that you only have four Protocol Definitions and that you can’t create Server Publishing Rules, then you know you’ve installed ISA Server in cache mode. If that happens, go to the Add/Remove Programs applet in the Control Panel and reinstall ISA Server in the appropriate mode.

    Configure the Appropriate Number of IP Addresses on the External Interface

    As with all Server Publishing Rules, you must have the same number of IP addresses bound to the external interface of the ISA Server as the number of SMTP servers you want to publish. For example, if you want to publish two SMTP servers on the internal network, then you must have at least two IP addresses bound to the external interface. In addition, if you want to run the SMTP service on the ISA Server itself and allow it accept SMTP messages (such as when you want to run the SMTP Message Screener on the ISA Server), then you want to have three IP addresses bound to the external interface. Remember that you must never allow socket contention. No service or server publish rule can listen on the same socket as another service or Server Publishing Rule.

    Disable Socket Pooling or Disable IIS Services on the ISA Server

    The most common reason for Server and Web Publishing Rules to fail is IIS on the ISA Server machine. There are two ways to handle the IIS problem. The best solution is to disable all IIS services on the ISA Server machine. Its better to disable the IIS services instead of uninstall IIS because there are issues with installing and uninstalling ISA Server when IIS is removed from the machine. You can disable the IIS service in the Services console in the Administrative Tools menu.

    There are times when you might want to run some IIS services on the ISA Server itself. For example, if you have a powerful ISA Server machine, you might want to run the SMTP Message Screener on the ISA Server. A less palatable example is when you are running ISA Server on a SBS machine, and your license limits you to installing the ISA Server on the SBS server that lies on the edge of the network. In that case, you will have to run the WWW, FTP, SMTP and NNTP services on the ISA Server.

    If you must run IIS services on the ISA Server, then you must disable socket pooling for these services. The procedures are slightly different depending on the service you want to disable socket pooling for. You can find out just about everything you ever wanted to know about socket pooling by reading my article The Misery of IIS 5.0 Socket Pooling  After you disable socket pooling, make sure the services are listening on the internal IP address only. You do not want the services to listen on the external IP address because you need to use those addresses for your Server Publishing Rules.

    Create Site and Content Rules and Protocol Rules to Support Internal Servers

    You must create Protocol Rules and Site and Content Rules to support services that require outbound access. In our current example, Virtual Server 1 needs to create new outbound connections so that it can relay mail to external SMTP servers. Virtual Server 2 is used by Internet SMTP servers to relay mail to your Exchange Server, so it does not need a Protocol Rule to allow it outbound access to TCP port 25. However, the Exchange Server does need outbound access to TCP port 25.

    You need to create a Site and Content Rule in addition to the SMTP Protocol Rule that allows Virtual Server 1 and the Exchange Server outbound access. The Site and Content Rule should allow these servers outbound access to all sites (content doesn’t matter because content is evaluated only by the Web Proxy service). Since these servers usually won’t have logged on users, you should create a client address set that includes the IP addresses of Virtual Server 1 and the Exchange server. Then configure the Site and Content Rule so that it applies to this client address set.

    Get the New Book!

    Install ISA Server Service Pack 1

    You must install ISA Server Feature Pack 1. Feature Pack 1 is required if you want to support authenticating with the SMTP server while the SMTP filter is enabled, and we always enable the SMTP filter when publishing SMTP servers. While you don’t have to enable the SMTP filter when publishing SMTP servers, there’s no reason not to now that Feature Pack 1 allows us to authenticate through the ISA Server. Download the feature pack at Remember that need to install ISA Server Service Pack 1 before you install the Feature Pack. If you don’t have ISA Server SP1 installed, head on over to and download it.

    Enable the SMTP Filter

    Finally, you need to enable the SMTP filter. The SMTP filter is not enabled by default. You can enable the filter by going into the ISA Management console. Expand your server name, then expand Extensions node. Click on the Application Filters node and click Enable. You’ll be asked if you want to restart the Firewall service. Go ahead and let the ISA Server restart the Firewall service.

    Double click on the SMTP Filter entry in the right pane of the console and click the Properties command. In the SMTP Filter Properties dialog box, click on the SMTP Commands tab. You should see the AUTH command with a maximum length of 500. Jim Harrison has informed me that he’s heard from several people who, for some reason, do not get the AUTH command added automatically to their SMTP Commands list after installing the Feature Pack. You’ll have to add the command manually if you’re one of these people. Just click the Add button on the SMTP Commands tab and add AUTH to the Command Name text box and the Maximum Length of 500.

    Now it’s fine and good to configure the SMTP Filter and configure the SMTP service on the IIS server (which we’ll cover in part two of this article), but how do we know it actually works and that authentication is taking place? By using a packet sniffer, of course. The next two figures show frames 25 and 30 of a capture taken while mail was sent from an Outlook Express client to the internal network SMTP server through the ISA Server while the SMTP filter is enabled.


    In this article we went over the advantages of being able to authenticate through the ISA Server. With the release of ISA Server Feature Pack 1, you can now authenticate with a published SMTP Server through the ISA Server while the SMTP filter is enabled. You could not do this prior to ISA Server Feature Pack 1. You can leverage this authentication ability to provide an SMTP server your external users can use to relay mail to any domain, including your own email domains.

    In part two of this article we go over the details of the corp user SMTP server, the anonymous Internet relay SMTP server and the Exchange Server configuration. Stay tuned!

    Get the Book!

    I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to;f=6;t=001388 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks! –Tom

    About The Author

    Leave a Comment

    Your email address will not be published. Required fields are marked *

    This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

    Scroll to Top