Configuring an Inbound and Outbound SMTP Relay to Complement ISA 2004 Firewall Protection for Exchange Servers

I’m a big proponent of the SMTP relay concept. A properly configured SMTP relay can protect your Exchange Server by preventing untrusted SMTP servers on the Internet from directly communicating with your Exchange server. An SMTP relay doesn’t require a significant amount of system resources and you can install the IIS SMTP service without incurring the resource or security overhead you would have if you installed the IIS W3SVC (World Wide Web service).

An SMTP relay is a machine that can accept incoming and outgoing SMTP messages and forward them to their appropriate location. The machine is an SMTP relay because it is not the endpoint of the SMTP message. You can think of the SMTP relay as an SMTP router. The SMTP relay routes SMTP messages to the appropriate location. No messages are stored on the SMTP relay as it is not authoritative for any SMTP mail domains.

Most people who implement an SMTP relay think of using it for incoming mail. The SMTP relay machine accepts mail only for the domains you host. For example, if you host an e-mail domain named maildomain.com, then the relay accepts messages for that domain and routes them to the appropriate location. The appropriate location may be an Exchange Server on the internal network, or it may be another SMTP relay that is responsible for spam whacking and virus checking. Mail destined to domains you do not host is dropped by the SMTP relay. This helps to prevent spammers from using your Exchange Server to relay spam to other mail domains on the Internet.

The SMTP relay can also handle also relay outbound mail from your organization. For example, if a user on your network sends mail to microsoft.com, the Exchange Server on your internal network must be able to forward that message to the mail server responsible for the microsoft.com mail. The Exchange Server on the Internal network can forward the mail directly to the microsoft.com SMTP server by resolving the MX domain name of the microsoft.com mail server, or the Exchange Server can forward the SMTP message to your outbound SMTP relay and then the SMTP relay becomes responsible for relaying the message to the microsoft.com SMTP server.

One of the major advantages of the outbound SMTP relay is that it can also be configured as a message screener that can remove attachments. Many prevalent viruses and worms include attachments and flood the Internet with bogus mail with viral payload. The outbound SMTP relay can strip the attachments or block the messages from leaving your network. You can use a full featured e-mail antivirus application like GFI’s MailSecurity or you can use the built-in ISA Server 2004 SMTP Message Screener.

The SMTP relay can be configured on a separate IIS 6.0 server or on the ISA Server 2004 firewall machine itself. The advantage of putting the SMTP relay on a separate machine is that if the SMTP relay machine is attacked by a DoS attack, only that machine is adversely affected. In contrast, if the ISA Server 2004 firewall is subjected to an e-mail DoS attack (such as a DDoS spam attack), there is the potential of bringing down the firewall by filling up the log files. However, even with that possibility in mind, if you keep close eye on your available disk space using an SNMP application (or any other method you choose), putting the SMTP relay on the ISA Server 2004 firewall is a viable alternative.

The challenge in this entire scheme is configuring the SMTP service on the SMTP relay machine to support both inbound and outbound relay. There are several requirements for our SMTP relay:

• Inbound SMTP relay is allowed only for domains you host
• Outbound SMTP relay is allowed to any domain
• The machine must not be an open SMTP relay that can be leveraged by criminal spammers
• Remote users can forward mail to domains you do not host by authenticating with the SMTP relay machine

Inbound SMTP Relay is Allowed Only to Domains You Host

The SMTP relay machine should only accept mail for relay to domains you host. For example, if you host the domain maildomain.com, the SMTP relay should only accept mail for that domain and reject mail for any other domain. (At least, this should be the case for unauthenticated connections). The reason for this is that if external SMTP servers and clients can send mail to other domains through your SMTP relay, then those machines will be able to relay spam through your SMTP server. You want to reject mail sent to the SMTP relay for domains you do not host.

This is a critical configuration. I’ve talked to a lot of ISA Server 2004 and Exchange administrator’s who have been put on dreaded RBLs because they did not correctly configure their SMTP relays to prevent unauthenticated relay. If you’ve been victimized by one or more of these RBLs, you know that you might as well try to come out a winner in an IRS audit. For this reason I highly recommend that you do not publish SMTP relays to the Internet until you have a good understanding on how to configure the relay to prevent spammers from abusing it.

The IIS SMTP service allows you to create remote domains and then configure those remote domains to relay mail to a server on your internal network. The SMTP relay is configured with remote domains for the domains you host and relay mail to your Exchange Server or to another SMTP relay on your internal network. The default IIS and Exchange SMTP server configuration blocks mail relay while allowing relay for mail addressed to the remote domains you configure.

You can see that the IIS SMTP service blocks relay by default by viewing the Properties dialog box of the SMTP server, as seen in the figure below. The option Only the list below is selected and there are no entries in the Computers list on the Relay Restrictions dialog box. Notice that I have unchecked the Allow all computers which successfully authenticate to relay, regardless of the list above checkbox. I removed this option because I do not need to use this SMTP relay for outbound relay for authenticated users on external networks. However, you may wish to enable this feature for your remote users. I’ll talk more about this option later in this article.

The figure below shows a list of remote domains.

Each domain is configured to relay to a specific host on the internal network. An example for the shinder.net domain appears in the figure below. Note that there is a checkmark in the All incoming mail to be relayed to this domain checkbox. This must be enabled so that mail addressed to users in this domain can be relayed. Otherwise, the mail would stop at the SMTP relay machine.

Outbound SMTP Relay is Allowed to Any Domain

If you choose to take advantage of outbound SMTP relay, you need to configure the SMTP relay computer to allow outbound relay to any to any mail domain. You must allow relay to any mail domain because you cannot predict what domains your internal network users are going to send mail to. You can configure the Exchange Server with a list of domains to which you do not want users to send mail to if you want to create an outbound block list. But in general, the SMTP relay machine should be able to send outbound mail to any domain.

One important function the outbound SMTP relay must carry out is resolving names of Internet mail domains included in the outbound SMTP messages. For example, when a user sends an outbound message to [email protected], the outbound SMTP relay must be able to resolve the microsoft.com e-mail domain to the IP address of a server that can accept incoming mail for the microsoft.com domain. This requires that the SMTP relay have outbound access to the DNS (UDP) query and DNS (TCP) zone transfer protocols. The SMTP relay will use these protocols to query DNS servers when resolving the MX domain name for the destination mail server.

You can test the SMTP relay’s ability to query an SMTP server by using the nslookup command. Just do this:

1. Open a command prompt and enter nslookup and press ENTER.
2. At the nslookup command prompt, enter set type=MX and press ENTER.
3. Enter the domain name that you want to test. In this example we’ll test microsoft.com. Enter microsoft.com. (including the trailing period) and press ENTER.
4. You’ll see that Microsoft has a number of mail servers that accept mail for their domain.
5. You can exit the nslookup prompt by entering exit at the prompt and pressing ENTER.

The Machine Must Not Be an Open SMTP Relay for Spammers

As I mentioned earlier, it’s absolutely critical that you do not allow spammers to relay through your SMTP relay machine. This is the most common reason for getting placed on an RBL. RBLs are pretty bad, as they try to act like law enforcement agencies without doing the prerequisite research to determine the details of the crime or criminal. RBL “censors” place SMTP servers on their lists without checking with the mail administrator of the domain. It’s the classic paradox where the criminal spammers aren’t affected while hapless and harried e-mail server administrators, who have no criminal intent at all, are victimized by RBL perpetrators.

OK, so much for my rant regarding RBLs. The bottom line is that you must prevent your SMTP relay from being used by spammers. The default configuration on Microsoft Exchange and Microsoft IIS 5.0 and 6.0 SMTP servers prevents relay from unauthenticated users. However, if you’re not familiar with the configuration, you might check or uncheck something that could leave you open. If you’re not sure if you’re running a properly configured SMTP relay, you can use a nice utility provided by the good folks at www.zonedit.com to test your configuration.

For example, suppose I want to check if my SMTP relay to www.tacteam.net is configured as an open relay. First, I go to http://www.zoneedit.com/smtp.html and enter the information you see in the figure below. I’ve entered the name of the SMTP relay in the Enter the host name or IP address of the e-mail server text box (this is a name that resolves to an IP address on the external interface of the ISA Server 2004 firewall that publishes this SMTP relay). I enter a random user name in the Enter From: e-mail address text box. It doesn’t matter what name you enter here, unless you have configured the relay to block access from certain addresses or mail domains. In the Enter To: e-mail address text box, I’ve entered [email protected]. Since I don’t host mail for the aol.com domain, my SMTP relay should never accept mail for that domain. This is how we test for open relay. If the SMTP server rejects the message, its configured correctly. If it accepts the message, its misconfigured and you’ll end up on some dreaded RBL.

Click the Begin Test button. You should see output similar to what appears in the figure below. Notice the line 550 5.7.1 Unable to relay for [email protected]. This indicates that the SMTP relay machine is correctly configured and spammers will not be able to send mail to domains you do not host through the SMTP relay machine.

You can see the SMTP log entries for this connection attempt if you have enabled logging on the SMTP virtual server. The 550 code is the SMTP code for Requested action not taken: mailbox unavailable. The reason for the unavailability is that relay to the aol.com is not allowed.

2004-05-24 14:45:29 64.21.143.22 edit.dnsvr.com SMTPSVC1 FILEWAY 192.168.1.21 0 HELO – +edit.dnsvr.com 250 0 34 19 0 SMTP – – – –

2004-05-24 14:45:29 64.21.143.22 edit.dnsvr.com SMTPSVC1 FILEWAY 192.168.1.21 0 MAIL – +FROM:<[email protected]> 250 0 47 34 0 SMTP – – – –

2004-05-24 14:45:29 64.21.143.22 edit.dnsvr.com SMTPSVC1 FILEWAY 192.168.1.21 0 RCPT – +TO:<[email protected]> 550 0 46 24 0 SMTP – – – –

2004-05-24 14:45:29 64.21.143.22 edit.dnsvr.com SMTPSVC1 FILEWAY 192.168.1.21 0 QUIT – edit.dnsvr.com 0 691 46 24 150 SMTP – – – –

One thing I see a often is people commenting that they’ve configured their SMTP servers to prevent relay but they are still receiving spam. Correctly configuring the SMTP relay will not prevent spam from being sent to domains you host. It only prevents spammers from using your SMTP relay to send spam to mail domains that you do not host.

Allowing Relay for Authenticated Users

While it’s always good to block spammers from relaying to external domains through your SMTP relay server, you may want to allow your users the ability to relay to external domains through that same SMTP server. This can be accomplished by requiring your users to authenticate with the SMTP server. Users can then send mail to the corporate domain, or to external domains after they successfully authenticate.

The figure below shows what we can accomplish by allowing relay for authenticated users:

1. When a spammer tries to send spam from his spam server through your SMTP relay, the relay attempt fails because the SMTP relay will not forward mail for domains you do not host. In this example, the spammer is trying to relay spam to the aol.com domain through the SMTP relay. The relay blocks the messages.
2. When a corporate user tries to send mail to a user at [email protected], the user will first authenticate with the SMTP relay via his SMTP client application’s configuration interface. After the user authenticates, he sends an e-mail message to [email protected]. The SMTP relay allows the mail to be relayed to the aol.com domain because the user logged on successfully before attempting to send the mail.
3. Another user wants to send mail from his SMTP client application to a user at [email protected]. You host the mail domain maildomain.com on your Exchange Server. The SMTP relay is configured with a remote domain that forwards messages destined to the maildomain.com domain to the Exchange Server on the internal network. Note that this user does not need to authenticate with the SMTP relay in order to send mail to the maildomain.com domain because you allow anonymous inbound mail to be sent to domains that you host.

Configuring Inbound and Outbound Relay on an IIS 6.0 SMTP Server with GFI MailEssentials

That’s all there is to it. Well, almost. As network and firewall administrators, you know that the devil is always in the details. Next week (after I return from TechEd in San Diego) I’ll provide you the step by step details on how to configure the inbound and outbound SMTP relay on a separate IIS 6.0 machine. We will also go over the details of the ISA Server 2004 firewall configuration that publishes the SMTP relay, allows the SMTP relay outbound access to the SMTP and DNS protocols, and how to install and configure GFI’s MailEssentials to prevent spam from clogging the Exchange Server.

The figure below shows the sample network that I’ll use when we go over the details.

The steps involved with getting this set up include:

• Installing the SMTP service on the SMTP relay
• Requesting a certificate for the SMTP service on the SMTP relay – this is used to allow secure connections for authenticated users
• Configuring the SMTP service for inbound secure/authenticated and anonymous relay
• Configuring the SMTP service to resolve Internet mail domain names
• Configuring the Exchange Server to use the SMTP relay for outbound relay
• Configuring the ISA Server 2004 Firewall to allow the SMTP relay outbound access to the DNS (TCP and UDP) protocols
• Configuring the ISA Server 2004 firewall to allow the SMTP relay outbound access to the SMTP protocol
• Creating an SMTP Server Publishing Rule that allows inbound connections to the SMTP relay on the Internal network
• Configuring Public DNS entries to support the SMTP Server Publishing Rule
• Configuring authoritative DNS server entries with your domain Registrar – this is only required if you decide to host your own DNS server

I’ll see you next week for the detailed step by step. If you have any questions about the setup before then, come on by the Security Cabana at TechEd 2004 and I’ll answer them there!

I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=23;t=000058 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks! –Tom