ISA Server Security Checklist – Part 1: Securing the Operating System and the Interface

ISA Server is all about security. ISA is about securing network access into and out of the internal network. But after you’ve done all of your configuring, how do you know that you’ve done an adequate job of securing the internal network and the system that ISA Server is running on?

Gary Andersen, a real ISA Server sharpie, asked if I could put together a checklist of some kind that ISA Server administrators can use to see if their configuration is a secure one. A great idea! Of course, security, like beauty, is in the eye of the beholder. Some people feel financially secure if they have $5.00 in their pocket; other don’t feel secure if they have $1,000,000. In this article I’ll focus on what I think constitutes a secure setup.

Configuring ISA Server 2000 : Building Firewalls for Windows 2000
By Deb and Tom Shinder

We’ll focus on three main areas you need to attend to when securing ISA Server:

In part 1 of this two part series, we’ll cover operating system and interface security. Part two will cover issues in creating a secure ISA Server configuration.

Note: If you don’t want to read all the background, you can jump to part 2 of this series and go to the end of that article for the security checklist.

Securing the Operating System

A lot of material has been written about securing the Windows 2000 operating system. In fact, there is an excellent book on this subject, Securing Windows NT/2000 Servers on the Internet by Stephen Norberg . Its an excellent book, and I recommend that everyone get a copy. Its very short and you can read it in a long evening. Note that some of the recommendations that Norberg makes in the book don’t apply to systems running ISA Server because these suggestions will prevent ISA Server services from working properly.

Do Not Install Applications and Services on the ISA Server

The number one rule for securing the system that the ISA Server is installed on is to insure that no other applications or services run on the system except for the ISA Server itself. This is a rather broad statement, but represents a basic truth: the more services or applications running on the ISA Server, the more vectors of attack are available to hackers. Every service and application has a weakness. The more services and applications you add to the ISA Server machine, the more weaknesses you introduce.

Examples of services and applications that you should avoid putting on the ISA Server:

  • DHCP
  • DNS
  • WINS
  • Certificate Services
  • IIS Services: WWW, SMTP, NNTP, FTP
  • Mail Servers, such as Exchange
  • Third party FTP servers
  • Mail clients, such as Outlook and Outlook Express
  • Web browsers
  • Anything else you can think of, such as NetMeeting

  • The only application or service on this list that you can’t realistically do without is the Web browser. You will probably need the Web browser to access hotfixes for the operating system or for the ISA Server components. However, it’s a realistic expection that you could use another machine to download software and copy that software to CD, and then install using the CD. The point is that you don’t want to run anything you don’t have to on the ISA Server machine.


  • Do not install services and applications on the ISA Server

  • Harden the Windows 2000 OS

  • Disable Built-in Windows 2000 Applications and Services

    You should also minimize the number of Windows 2000 built-in services that run automatically after installing ISA Server. Some of those that you don’t need include:

  • Alerter

  • Clipbook

  • Computer Browser

  • Distributed File System

  • Fax Service

  • File Replication

  • Indexing Service

  • Internet Connection Sharing

  • Intersite Messaging

  • Kerberos Key Distribution Center

  • License Logging Service

  • Messenger

  • NetMeeting Remote Desktop Sharing

  • Network DDE

  • Print Spooler

  • QoS RSVP

  • Removable Storage

  • RunAs

  • Telnet

  • There may be other services running on the machine that you do not need. Experiment by disabling services one or two at a time, and then test ISA Server functionality. This is very much a trial and error game at this time because there is no official Microsoft explication of the services required to support all of the ISA Server services. Once you do find a configuration that works, be sure to use the Security Analysis and Configuration snap-in to save your settings so that you can apply them again if you want to install ISA Server on another machine, or restore these settings to the same ISA Server in the event that they are changed.

    SBS users much have the browser service running because the SBS server is the DC for the organization. The DC acts as the domain master browser which is required for the browser service to work correctly in a multisegment internal network.


  • Disable all services that aren’t required by the base operating system and ISA Server

  • Do Not Install ISA Server on a Domain Controller

    There are a lot of reasons to not install ISA Server on a domain controller. First, the configuration can get complex and confusing for someone that doesn’t have a lot of experience with Windows 2000 networking and ISA Server. Second, you’re putting your family jewels on the edge of the network.

    While it’s unlikely that the ISA Server will ever be compromised, you can bet that it’s going to be the ISA Server machine everyone is trying to break into. If someone does compromise the ISA Server that’s running on a domain controller, they now have access to the most important component of your enterprise. It would be relatively easy to just delete your Active Directory database or corrupt it (in fact, you don’t even need a hacker to get a corrupted Active Directory J ).

    The exception to this rule is when you decide to put the ISA Server or an array of ISA Servers in their own domain. This is a good security configuration because even if someone compromises an enterprise administrator account in the ISA Server domain, that account will have no influence over the internal network’s operations. In order to allow for user/group based access controls, you would create an external trust between the ISA Domain/Forest and the internal network’s Domain/Forest. Note that you want to put the ISA Server(s) in a completely different forest. Otherwise, a compromised enterprise administrator account can wreck havoc over your entire organization.


  • Do not install ISA Server on a domain controller unless it’s a dedicated ISA Server domain and forest

  • Securing the Interfaces

    You’ll usually have at least two interfaces on the ISA Server: an internal interface and an external interface. The internal interface is connected to the internal, private network. The external interface is typically connected directly to the Internet, although it can be connected to any untrusted network, such as the backbone network within your corporate network. You may also have a DMZ interface, which is a second public interface that is attached to the ISA Server, but is not directly connected to the Internet. The DMZ interface should be configured in the same way as the external interface because it is also connected to an untrusted network.

    Configuring the External Interface

    The external interface of your ISA Server is a war zone. This is where the enemy is going to hit you over and over again. You need to make sure that the interface is configured so that unexpected holes are not left open in your configuration.

    When you install a new interface on a Windows 2000 computer, the default settings are designed for an internal network interface. The level of security provided by the default interface settings is very low because it is assumed that communications over the corporate LAN can be trusted. While that may be more or less true, it is definitely not the case when the interface is directly connected to the Internet.

    The following graphics show the options seen on the Interface properties dialog box:

    When a new interface is installed and detected by Windows 2000, two potentially dangerous settings are enabled: File and Printer Sharing for Microsoft Networks and Client for Microsoft Networks. The former allows the machine to share SMB/CIFS resources and the latter allows the machine to access SMB/CIFS resources. These options enable NetBIOS and/or Direct Hosting ports, both of which are used for conventional file sharing and access on Microsoft networks. You definitely do not want these options enabled.

    Note: Even if you disable File Sharing on an interface, the Direct Hosting Port (TCP 445) will still remain open and apparent on your netstat -na printouts. If you wish to completely abolish this port, you must disable the TCP over NetBIOS driver (nbt.sys). This can be done in the Devices applet by showing the hidden devices and disabling the NetBIOS over TCP/IP entry.

    If you double click on the Internet Protocol (TCP/IP) entry and click on the WINS tab, you’ll see the following dialog box:

    The default setting is to Enable NetBIOS over TCP/IP (in spite of what the pop-up help text says about this selection). You should choose the Disable NetBIOS over TCP/IP option because the external interface does not need to be configured as a WINS client, does not need to issue NetBIOS broadcasts, does not need to send out browser service announcements, and does not need to access NetBIOS resources on the Internet. Not only will disabling NetBIOS on the external interface make your computer more secure, it will also prevent large numbers of NetBIOS name resolution broadcast entries from littering your packet filter logs.

    You should also disable the LMHOSTS lookup option. Windows 2000 and all subsequent Windows operating systems will use DNS preferentially for computer name resolution. As the world of NetBIOS dependent applications slips away, the need for NetBIOS name resolution methods does so to. You definitely do not need this enabled on the external interface of the ISA Server.

    If you click on the DNS tab, you see what appears in the following graphic:

    Our gaming guru, Jamie Pirnie, informed me of a possible weakness in the way that unqualified names are handled for inbound Web requests. By default, the Append primary and connection specific DNS suffixes option is selected. That means if the external interface receives a request for http://www, the primary DNS suffix for this computer would be appended to the request. So, if this computer was a member of the internal network domain,, the request would resolve to If a Web Publishing Rule included an entry for in its Destination Set, the request would be passed to that server on the internal network.

    The problem with this is that some of the NIMDA and Code Red like Worms are beginning to use www instead of an IP address in the HTTP request header. That means that even if you don’t have a Destination Set for www, the unqualified www request turns into a FQDN by virtue of the DNS interface settings. This can then make a requests that would be rejected by the ISA Server become accepted. And that is definitely not a good thing.

    Note: This is the default setting on ISA Server. I consider this an insane default setting, since the entire reason to use Destination Sets is so that the Web Publishing Rules can match exactly what is contained in the HTTP header with a Destination Set entry.

    You might have noticed that the Network Monitor Driver in the interface configuration. It is important that you know how to use the Network Monitor so that you can watch conversations that go on between the external interface and external network hosts. There is a lot of information contained in so Network Monitor traces that can not only help you figure out which protocols you need to enable in order to get a particular application working, but also to determine if traffic arriving or leaving the external interface is legitimate or not.

    One solution is to select the Append these DNS suffixes (in order) option, and then enter a bogus domain name (such as bogus.local). Now when that unqualified request for www hits the external interface of the ISA Server, it becomes a query for www.bogus.local, which you surely don’t have any entries for in any of your Destination Sets.

    Some arguments can be made that this a bad thing, since most applications use DNS name resolution preferentially for resolving names. This can lead to some problems if you make the ISA Server a domain controller, because the NetBIOS names for the domain may resolve to the bogus address. Of course, this isn’t an issue for our secure server, which would never have Active Directory installed on it. In fact, you *never* want the ISA Server to resolve unqualified names, since even in the case of Firewall and Web Proxy clients, the client side resolver on the host on the internal network will always append a domain name for unqualified requests, and the browsers will *always* be configured to bypass the ISA Server when an unqualified request is made!

    However, the best solution to this problem is to configure the ISA Server to stop resolving unqualified requests. Make the following registry changes:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W3Proxy\Parameters\SkipNameResolutionForPublishingRules : DWORD : 1

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W3Proxy\Parameters\SkipNameResolutionForAccessAndRoutingRules : DWORD : 1


  • Disable File and Printer sharing on the external interface

  • Disable Client for Microsoft Networks on the external interface

  • Disable NetBIOS over TCP/IP on the external interface

  • Change the method for resolving unqualified names by choosing the Append these DNS suffixes (in order) option in the DNS tab in the Advanced TCP/IP settings Properties dialog box

  • Configuring the Internal Interface

    Configuring the internal interface is a bit more problematic. Because the internal interface is a LAN interface, you could just accept the default settings. In most cases, this is probably what you will want to do.

    You will need to allow File and Printer sharing on the internal interface if you want internal network clients to access the Firewall client software. You do have the option of putting the Firewall client software on another machine on the internal network, but you need to be careful that the mspclnt.ini file settings in this alternate location are up to date, and you have to make sure that the settings indicate the correct ISA Server.

    Note: This isn’t a problem if you have your wpad network configuration in place and working.

    This isn’t too difficult to do, but you need to keep in mind that when you make changes to the mspclnt.ini file via the ISA Management console, these settings will not be written to the alternate location and therefore will not be made immediately available to client when the Firewall client is installed from the alternate location. The machine will obtain up-to-date settings after the first refresh, which is 6 hours by default.

    The Client for Microsoft networks setting will also need to be enabled if you wish to access resources on the internal network. There may also be circumstances when you need this enabled for authentication purposes. Therefore, I recommend that you leave this setting enabled unless you’ve tested it in the disabled state and find no adverse affects.

    Even though NetBIOS over TCP/IP isn’t required on pure Windows 2000 networks, you might find that if you turn off NetBIOS on the internal interface that things won’t work the way you expect them to work. The reason for this is that Windows 2000 machines still contain some vestiges of NetBIOS, and almost all legacy Windows client and server applications require NetBIOS.

    You can immediately forget about disabling NetBIOS if you have any legacy clients on your internal network that need to access the mspclnt share on the ISA Server to access Firewall client software. Windows 2000 network hosts don’t require NetBIOS to access the mspclnt share because they can use DNS and Direct Hosting to access the share. But downlevel clients (such as Win9x and NT) require NetBIOS to access shared network resources. I ran into this problem once when a Win98 client refused to install the Firewall client software. It turned out that I had disabled NetBIOS on the internal interface because I had recently worked on a Windows 2000 only network.

    Note: If you disable nbt.sys, NetBIOS will be completely disabled on all interfaces.


  • Determine whether your network infrastructure requires enabling the Microsoft Client, File and Printer sharing, and NetBIOS on the internal interface. If you do not require these features, try turning them off.

  • Summary

    In part 1 of our ISA Server security checklist series, we covered things you can do to secure the base operating system and the network interfaces. In the second part of the this article, you’ll see things you can do optimize your ISA Server configuration. The compiled security checklist will round out part 2 of this series. So, check it out.

    About The Author

    Leave a Comment

    Your email address will not be published. Required fields are marked *

    This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

    Scroll to Top