Mail Relay Scenario Using GFI Mail essentials 6 for SMTP Gateways

Mail Relay Scenario Using GFI Mail essentials 6 for Gateways

Part 1: Design and Rationale

By Thomas W Shinder MD

We’ve covered the subject of SMTP Server publishing and mail relay here in the Shinder area before, but I thought it would be a good idea to share with you a recent SMTP relay and spam filtering solution we put together for a client recently. You can use this example as a baseline for your own Exchange Server solution and modify it as you see fit.

Get the Book!

The client had a T1 link to the Internet and he wanted to host his own Internet services, including his own Exchange Server that would receive mail for his domain. We decided to put together a back to back ISA Server DMZ configuration as seen in the figure below:

We decided to put two IIS SMTP mail relay servers on the DMZ segment. The advantage of putting more than one mail relay server is that you can take one or more of them down and the servers that remain up can handle all the incoming mail for the domain. An MX record was created for each SMTP mail relay, with Relay 1 having a higher priority than Relay 2.

The client was very interested in minimizing the amount of spam entering his network. This guy hated spam with a passion and was willing to do whatever we could to bring it down to an absolute minimum. With this in mind, we decided to configure a third SMTP mail relay on the internal network. We installed GFI software’s Mail essentials for SMTP Gateways 6 on this server. We’ve had nothing but excellent results with the Mail essentials spam filter and we’re happy to recommend this solution to the client.

The Exchange 2000 Server was located on the same internal network segment as the SMTP mail relay server that did the spam filtering. The Exchange Server received mail for the corporate domain from the spam filtering SMTP mail relay server. All internal network clients connected to the Exchange 2000 Server using Microsoft Outlook 2000 MAPI client. No client system on the internal network used POP3 or SMTP to access external network servers. The internal ISA Server blocked outbound access to POP and SMTP from all clients; only the spam filtering SMTP mail relay server was allowed inbound and outbound access to SMTP.

Why We Did It This Way

Here’s the explanation for why we did things the way we did:

Two Mail Relay Servers on the DMZ Segment

We decided to put mail relay servers on a DMZ segment. My belief is that no Internet host should have direct access to any internal network client if at all possible. In this case, SMTP servers on the Internet needed to have direct access to the SMTP mail relay servers in order to send mail to the client’s mail domain. Since Internet hosts must have direct interaction with the SMTP mail relay servers, we decided to keep them out of the internal network and put them on a DMZ segment.

Any server on a DMZ segment should be considered a bastion host and its expected that a bastion host will be compromised at some time. It’s a matter when, not if. Since we expect these servers to be compromised, its important to have fault tolerance so that the inbound mail can keep flowing inbound. Installing a second mail server allows mail to be delivered if one of the servers becomes unavailable. Note that these servers have been hardened to the extent that they are virtually unusable for anything but SMTP mail relay. Essentially the only way these servers can be compromised is if the SMTP service is compromised. The external ISA Server had the SMTP filter enabled to help prevent SMTP buffer overflow conditions.

The DMZ SMTP Servers we’re configured to accept mail only for the customer’s mail domain. We did this by configuring a remote domain on the IIS SMTP servers and then configuring the Remote Domains to relay mail to the internal spam filtering SMTP relay mail server. The internal ISA Server was configured with a single SMTP Server Publishing Rule and a client address set was configured so that only the DMZ SMTP servers could use the SMTP Server Publishing Rule on the internal ISA Server.

A Dedicated Spam Filtering Mail Relay Server

You might be asking yourself why we decided to bring a third mail relay into the mix. After all, you can install GFI Mail essentials on Exchange 2000. This customer receives a relatively large volume of mail, and the processing requirements to filter the spam would be significant.

The Exchange 2000 Server would already be working near the limits of its capacity. Adding the processing overhead of spam filtering would have put the Exchange 2000 Server over the top and users would have noticed a precipitous decrement in the Exchange Server’s performance if we had put GFI Mail essentials onto the Exchange Server. I also prefer not to add software to Exchange Servers unless its absolutely required. We were fortunate that the customer was more than happy to absorb the addition hardware and software cost to implement a separate server for spam filtering.

The spam filtering SMTP relay server was configured with a Remote Domain that matched the customer’s mail domain name. The Remote Domain was then configured to relay mail for this Remote Domain to the Exchange Server. The Exchange Server was the end of the line for the incoming SMTP messages.

What About Remote Access?

We did bring up the issue of users needing remote access to their corporate mail account. The nature of this customer’s business didn’t lend itself to a great deal employee travel or work from home. However, our analysis of the situation is that there would be an absolutely maximum of 10 users requiring remote access to their mail at any one time. Options we consider for remote access included:

  • Publishing the Exchange Server Secure POP3 through both ISA Servers
  • Publishing the Exchange Server Secure IMAP Service through the ISA Servers
  • Publishing Exchange Server RPC through the ISA Servers
  • Creating a VPN connection to the internal ISA Server
  • An “out of band” ISDN connection to the internal network for VPN access
  • An “out of band” ISDN connection to the Exchange Server for VPN access

  • We didn’t have to worry about publishing the SMTP service because the users ISP could handle outbound SMTP.

    We decided to use Exchange RPC publishing to allow the client’s Outlook clients inbound access to the Exchange Server. Yes! You can use Exchange RPC publishing in a back to back ISA Server DMZ environment. The Outlook clients were configured to secure the data connection. This is a very secure option and I highly recommend Exchange RPC publishing when you want to allow external Outlook MAPI clients access to an internal Exchange Server.

    The other option we gave serious attention to was Secure IMAP. The Secure IMAP link uses TLS (SSL 3.1) to secure the connection between the IMAP client and the Exchange Server. This requires that you set up the Exchange Server with a certificate and that the client trusts the certificate installed on the Exchange Server. Actually, the client does not need to trust the certificate on the Exchange Server, but you will get an informational message telling you that the certificate isn’t in the trusted store. When you ignore the message, the client successfully uses Secure IMAP to connect to the Exchange Server.

    How the Mail Flows

    Let’s take another look at the graphic and follow the email path inbound and outbound.

    Step 1:

    The ISA Server receives the mail from the Internet SMTP server. Two SMTP Server Publishing Rules are configured on the external ISA Server that point to either Relay 1 or Relay 2. MX records are configured on a DNS server authoritative for the mail domain and a preferred and secondary SMTP server are setup based on preference values in the MX records. The SMTP filter on the external ISA Server is enabled to help protect against SMTP buffer overflow attacks on Relay 1 and Relay 2.

    We configured the SMTP Message Screener on both Relay 1 and Relay 2. You might ask yourself why would we install the Message Screener on the DMZ relays when we have GFI Mail essentials on the spam filtering relay on the internal network? The reason for this is that the Message Screener allows you to filter out messages on a per user address basis, a feature not available in GFI Mail essentials. Mail essentials allows you to filter out mail from entire domains, but not a particular address within a mail domain.

    Step 2:

    Relay 1 and Relay 2 are configured with Remote Domains for the customers mail domain name. The Remote Domains are configured to forward (relay) mail for the customer’s domain to the internal spam filtering SMTP relay server. All mail for domains other than then the customer’s domain is dropped by Relay 1 and Relay 2. The servers then forward the customer’s mail to Relay 3. The internal ISA Server is configured with an SMTP Server Publishing Rule that allows only Relay 1 and Relay 2 access to the SMTP Server Publishing Rule. The SMTP Server Publishing Rule on the internal ISA Server forwards mail received from Relay 1 and Relay 2 to Relay 3 on the internal network. The SMTP filter on the internal ISA Server is configured to prevent SMTP buffer overflow attacks on Relay 3. While its unlikely that such an attack would take place, we figured why not take advantage of this feature when its so easily configured?

    Step 3:

    The spam filtering SMTP mail relay server, Relay 3 accepts the mail from the DMZ mail relays. Relay 3 is configured with a Remote Domain for the customer’s email domain and all mail accepted by the Remote Domain is relayed to the Exchange Server. Of course, the spam filter chews up and spits out the spam so only legitimate mail hits the Exchange Server.

    Step 4:

    The users on the internal network use the Outlook 2000 MAPI client to connect to the Exchange Server. The Exchange Server handles mail sent between users on the internal network internally. The Exchange Server must forward mail destined to an external mail domain. We configured the Exchange Server to use the spam filtering SMTP relay server Relay 3 as a Smart Host. The default SMTP virtual server on Relay 3 was configured to allow relay to any domain from the Exchange Server only.

    This configuration allowed for virus checking and content inspection for outgoing mail as well as incoming mail. This is a nice trick you can use to prevent your internal network users from making your mail servers a conduit for AOL user like behavior. The same keywords are used for both inbound and outbound access.

    Step 5:

    We configured Relay 3 to use the ISPs mail server as a Smart Host. This offloaded the DNS mail domain name resolution off of Relay 3 and onto the ISPs mail server. This usually works well because the ISPs DNS cache is a lot larger than the private network’s DNS cache.

    Get the Book!


    This multiple relay configuration worked well for the customer. The most time consuming aspect of the configuration was coming up with the appropriate key word filters on the Mail essentials machine. There is no “one size fits all” keyword list! Each industry has its own language that might be considered spam in another industry. It took about two weeks to come up with a good list of key words so that spam was whacked but legitimate mail was allowed through. The company’s security officer was responsible for reviewing the spam and determining whether a false positive was noted. All employees were informed that a security officer might view their inbound or outbound mail if the message was caught by the spam filter. Employees then had to sign off on the announcement to confirm that they read it and agreed with the policy.

    In the second and final part of this article, I’ll review the specifics of the configuration on each of the servers. The configuration details are not that complex, but if you haven’t done this sort of thing before you might find it helpful to see the exact steps we went through to make this work.

    If you have any questions or comments on this article, please feel free to write! Send a message with the title of this article in the subject line to [email protected] and we’ll see what we can do to get you fixed up. Thanks! –Tom.

    About The Author

    Leave a Comment

    Your email address will not be published. Required fields are marked *

    This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

    Scroll to Top