Configuring the SMTP Message Screener.

A subject that gets a lot of discussion on the Web boards and mailing list is the SMTP Message Screener. The reason for this is that the Message Screener takes a bit of tweaking to get working right. The SMTP Message Screener does provide functionality that you would otherwise have to obtain from third party solutions. The good news is that it does indeed work!

Your challenge is to figure out how to make it work.

I think one of the major stumbling blocks people run into when configuring the SMTP Message Screener is that they try to make it work by installing it on the ISA Server itself. I consider this a bad security policy. You purchased ISA Server to be your firewall and Web caching server. What you didn’t do is purchase ISA Server to be part of your server consolidation project! Adding extra services to the firewall just provides the bad guys more portals of attack.

The Keep it Simple Principle

The biggest problem people have is that they try to implement the Message Screener in an overly complex configuration. Keep it simple! The SMTP Message Screener configuration I find least problematic requires that you use three servers:

Configuring ISA Server 2000 : Building Firewalls for Windows 2000
By Deb and Tom Shinder

  • An ISA Server at the edge of the network
  • An IIS 5.0 SMTP Server on the internal network
  • An SMTP/POP3 server on the internal network (e.g., Exchange or any other mail server you like to use).

The IIS 5.0 SMTP server will act as a mail relay and forward mail for your domain to the mail server. The ISA Server and the IIS 5.0 SMTP server (with the Message Screener installed) communicate with each other via DCOM to provide Message Screener functionality.

Not Simple Configurations
Note that the above isn’t the only configuration option. Other possibilities include:

  • Install the IIS 5.0 SMTP service and Message Screener on the ISA Server
  • Install the Message Screener on your Exchange Server
  • Install Exchange and the Message Screener on the ISA Server

While these configurations are possible, making them work quickly and easily is not.

In this article we’ll go over the procedures you need to carry out to make the SMTP Message Screener work using the simple configuration.

Install the SMTP Message Screener
The SMTP Message Screener needs to be installed on the internal networks IIS 5.0 SMTP server. This is the server that will be used to relay mail to your SMTP/POP3 or Exchange mail server.

  1. Install the ISA Server Message Screener. Put the ISA Server CD into the CD drive and let it autorun. If you do not have the CD click the ISAautorun.exe file.
  2. Start the installation of ISA Server. Choose the Custom installation option. Remove the check mark from the ISA Services check box. Click the Administration Tools option, and click Change Option. Put a check mark in the ISA Management check box. Remove the check mark from the H.323 Gatekeeper Administration Tool check box. Click OK.
  3. Place a check mark in the Add-in Services check box, and click Change Option. Remove the check mark from the check box for Install H.323 Gatekeeper Service. Place a check mark in the check box for Message Screener. Click OK and click Continue.
  4. Setup installs the Message Screener which will be used by IIS 5.0. Restart the computer after installing the Message Screener

Enabling the ISA Server SMTP Application Filter
The SMTP Application Filter on the ISA Server is disabled by default. Therefore, before you can take advantage of the filter’s features, you’ll have to manually enable it. Fortunately, its easy.

  1. At the ISA Management console, expand your server name and then expand the Extensions node. Click on the Application Filters node.

  2. Right click on the SMTP Filter and click Enable.

We’re not going to go into the configuration of the SMTP application filter in this article. We’ll cover that in a future article at the Web site.

Configuring the Internal IIS 5.0 SMTP Server
Now its time to configure the internal network’s IIS 5.0 SMTP Server that will run the Message Screener. This server will be published using server publishing rules. When mail for your internal network’s domain is received by the external interface of the ISA Server, it will be forwarded to this SMTP server.

This SMTP server will be configured with a Remote Domain that will only accept mail for your mail domain. This prevents spammers from using your server as a relay. The Remote Domain will be configured to relay mail to your Exchange or other SMTP server. The Exchange Server will need to be configured to accept mail from Message Screener SMTP Server.

  1. Install the IIS 5.0 SMTP service on a Win2k machine on the internal network.

  2. The network interface configuration isn’t important unless you want to publish multiple virtual SMTP servers on this machine. Therefore, you can let the SMTP service listen on all interfaces. Otherwise, you will have to disable socket pooling.

  3. Create a Remote Domain to support your incoming messages. Open the Internet Services Manager from the Administrative Tools menu. Expand the Default SMTP Virtual Server and right-click the Domains node. Click the New command, and then click Domain.

  4. The New SMTP Domain Wizard appears. Select the Remote option, and then click Next.

  5. On the Select Domain Name page type in the domain name for which your mail server will accept mail. For example, if you wanted the IIS 5.0 SMTP server to accept mail sent to only, you would create a remote domain for Messages destined for other domains are rejected. This prevents the server from being used as a relay. Click Finish.

  6. Double click the remote domain, Select the Forward all mail to smart host option. Type in the IP address of your internal mail server. Surrounded the address with straight brackets. Select the Allow incoming mail to be relayed to this domain option. Click OK. Stop and Start the SMTP service.

Configure Permissions with the SMTPCred.exe Tool

The Message Screener must be configured with an account that has permissions to access the ISA Server. An Enterprise Administrator has the appropriate permissions. You should also be able to use the Local Administrator account.

  1. Search the CD-ROM for the file SMTPCred.exe. Copy that file to your hard disk, and then double-click it.
  2. Enter the name of the ISA Server. You can leave the default time period that the remote server uses to retrieve settings. Enter a Username/Domain and password that has administrator access to the ISA Server. The SMTP server will use these credentials to communicate with the ISA Server. Click OK.

Configuring DCOM Permissions
ISA Server using DCOM. You will have to use the Dcomcnfg.exe tool to configure the proper permissions on the ISA Server.

  1. Click Start, click Run and type dcomcnfg.exe in the Open text box, and then click OK.
  2. Click the Applications tab, click the VendorData class entry, and then click the Properties button.

  3. On VendorData Class Properties page click on the Security tab. Select the Use custom access permissions option . In addition, select the Use custom launch permissions option button. Finally, select the Use custom configuration permissions option.

  4. For each of these options, click the Edit button. You will see the Registry Value Permissions dialog box. For each configuration permissions, you add the Everyone group by clicking Add and then selecting the Everyone group. Click OK then click OK again.

  5. Restart both the ISA Server and the IIS 5.0 SMTP Server. I suggest restarting the ISA Server first.

The SMTP Message Screener is relatively simple to setup when you use the configuration covered in this article. If you wish to setup other configurations, you may have more complications and issues that need to be attended to, but they are possible. Just keep in mind that ISA Server should not be a part of your server consolidation plan.

About The Author

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top