Configuring NLB for Inbound Access to Published Servers
By Thomas W Shinder and Kai Wilke
Network Load Balancing (NLB) is a really cool tool that you can use to improve the uptime for your ISA Server solution. NLB allows you to configure one or more servers in an NLB cluster, any of which can take over for another server in the cluster in the event that an cluster member becomes unavailable. This allows users to connect to an ISA Server cluster member for either inbound or outbound access, even if one or more of the ISA Servers in the cluster becomes unavailable.
We've noticed there isn’t much useful documentation on NLB. If you look in TechNet, you won’t find anything there that really helps you with the step by steps of implementing NLB. You’ll find a lot of information about how it works, where it fits into the enterprise, and a lot of blah blah blah about performance and scaling. While all that information is interesting, it doesn’t help you out too much if you can’t get the dreaded thing to work!
A lot of people ask questions about how to get NLB to work for both inbound and outbound access. NLB can be used to provide load balancing and fault tolerance for outbound access so that your internal network SecureNAT and Firewall clients will never be without a connection to an ISA Server. NLB can also be used for inbound access; so that your published servers can still be available, even if one of the ISA Servers in the NLB cluster goes down.
When we talk about NLB clusters, we are not talking about Web Caching arrays. If you configure a Web caching array and configure the clients to use the Autoconfiguration script, the Web Proxy clients are automatically configured to take advantage of the load balancing and fault tolerance mechanisms built into the Web Proxy service and CARP. However, NLB will allow for true fault tolerance and load balancing for your Web Proxy arrays. You can get more information on caching and CARP at http://www.microsoft.com/isaserver/techinfo/planning/cachingwp.asp
Another thing to keep in mind is that NLB clusters do not allow you to use different ISPs on multiple external interfaces. The NLB virtual IP address must be on the same network ID as the non-virtual addresses on the external interfaces of the ISA Servers in the cluster. Load balancing and fault tolerance for the ISA Servers is quite different than load balancing and fault tolerance for your Internet links. If you need fault tolerance for your Internet links, you might want to look into hardware solutions that provide this feature. Many high-end routers provide this feature, or you might want to look at F5’s BigIP. Also note that you can use multiple ISPs for outbound NLB if you are using routers in front of the ISA Server.
In this article we’ll focus on configuring the ISA Servers to use NLB on the external interface. Windows 2000 supports NLB on the internal or external interface, but not both. However, you have something to look forward to if you need NLB on both interfaces, as Microsoft .Net Server does provide this functionality.
Steps to Configuring NLB on ISA Server
Enabling NLB on the external interface of the ISA Server is useful for providing fault tolerance and load balancing for servers that you’ve published using Server Publishing Rules. I’ll only cover the use of Server Publishing in this article, but you can use NLB to load balance your Web Publishing Rules too. If there's enough interest in an article on using NLB with Web Publishing Rules, Kai and I will do one on that too.
Steps involved with configuring NLB on the external interface are:
· Install Windows 2000, then install SP2 or higher
· Configure NLB on the external interface of the NLB cluster members
· Install ISA Server on each cluster member
· Install ISA Server SP1 on each cluster member
· Configure the Registry to support ISA Server Publishing
· Create the Server Publishing Rules
· Cross you fingers!
We'll only cover steps 2 and 5 in this article, as you should already be aware of how to do this other steps for you reading of the book.
Configure NLB on the External Interface of the ISA Server
Configuring the NLB can be a bit tricky. I’ll take you through the steps and show you how it done.
1. Go to your first NLB cluster member computer and log on as an administrator.
2. Right click on the My Network Places object on the desktop and click the Properties command.
3. Click the Advanced menu in the Network and Dial-up Connections window, and then click the Advanced Settings command. On the Adapters and Bindings tab, make sure your internal interface is listed on the top of the Connections list. Just select the internal interface and click the up arrow if it isn’t on the top of the list. Then click OK.
4. Right click on your external interface and click the Properties command.
5. On the General tab of the connection’s Properties dialog box, place a checkmark in the Network Load Balancing checkbox and then click the Properties command.
6. This brings up the Network Load Balancing Properties dialog box. The Primary IP address is the virtual NLB IP address. The Primary IP address will be the SAME on all NLB cluster computers. The Subnet Mask is the appropriate subnet mask for the IP address you’re using for your Primary IP address. Note that the Primary IP address must be valid for the network ID that the external interface is connected to. You can’t just put in a random IP address in this box because external network clients need to know the route to your Primary NLB IP address. The Full Internet name is the FQDN that you’ve registered for the NLB address. Since external network clients will reach your published servers using this IP address, you should register a domain name and create the appropriate Host (A) entry for your NLB cluster. Its likely that you'll have multiple Host records pointing to the NLB interface. Place a checkmark in the Multicast support checkbox if you are using a single NIC for each cluster member. If you are using a multiNIC configuration, clear this checkbox and consult the Windows 2000 help file for configuration details for Unicast Mode. There can be issues with interposed switches learning the cluster MAC address that will prevent NLB from working correctly. Check out Q247297 for information on how to configure your routers/switches to work with NLB multicasting and put a high-speed hub between the cluster and your switch. This will avoid switch flooding issues and the switch will successfully learn the cluster MAC address. The password text boxes are used for remote management using the wlbs.exe tool; if you want to remotely manage the cluster, enter a password for a minor level of security.
7. Click the Host Parameters tab. The Priority (Unique Host ID) number The number is only used to determine the default NLB cluster node (node with the lowest number). The default cluster node is responsible for generating and distributing a host list of all hosts which are successfully joined to the cluster. The list changes when nodes go into offline/drain state or you add new host to the cluster; this is referred to as "cluster convergence". This ensures that all nodes are using the same NLB hash formulas for calculating NLB routing. Each NLB cluster member must have a different Priority number. If you give two hosts the same priority, things are not going to work right! The Dedicated IP address and Subnet mask are those of the non-virtual IP address on the external interface of the ISA Server NLB cluster member. Note that the Dedicated IP address is not the NLB cluster’s virtual IP address. This will get a bit tricky, as you’ll see in later steps. Click OK in the Network Load Balancing Properties dialog box.
8. In the connections’s Properties dialog box, click on the Internet Protocol (TCP/IP) entry and then click the Properties button.
9. In the Internet Protocol Properties (TCP/IP) Properties dialog box, click on the Advanced button.
10. Click the Add button and add the IP address that you’re using for your virtual NLB IP address. Now, here is where things get tricky. You need to make sure that the virtual IP address is not at the top of the list, so that it becomes one of the secondary IP addresses on the external interface. This is important if you have internal network clients making requests to Internet servers that need to initiate secondary inbound connections. An example would be PORT FTP requests from internal network clients. The secondary connection needs to be established with the non-virtual IP address on the ISA Server's external interface. After you’ve added the virtual IP address to the list of IP addresses bound to the external interface, click OK. Click OK again, then click OK one more time.
11. If you see an error message at this time (or any time) regarding duplicate IP addresses being detected, don’t worry about it. The problem will go away by the time you’re done setting things up and restarting the servers.
12. Now go to the other members of the NLB cluster and perform the same steps. Make sure that each NLB cluster member uses the same virtual IP address and that each NLB cluster member is configured to use a different Priority ID. After all the servers configured, restart them one at a time. Wait for the previous server to restart before restarting the next one.
13. You should no long see any error messages regarding duplicate IP addresses after the servers are restarted. If you continue to see the error, check out your configuration and make sure you did it correctly. You might also want to check out Q264645 for more information. Check out the Event Viewer on each of the servers to confirm that the NLB cluster has configured itself properly. It should look something like what you see below. If you have more than 2 hosts, you’ll see the host IDs for each of the hosts in the Event Properties dialog box.
When you see the cluster has converged, you’ll know that everything was setup correctly. Congratuations!
The next step is to install ISA Server. In this lab I installed ISA Server in integrated mode and included all the components. The installation should go fine without any special configuration options. After you install ISA Server, restart the computer and then download and install ISA Server SP1. You must install SP1 in order to create the Registry entry required to use Server Publishing Rules. Restart the computer after SP1 is installed.
Configuring the Registry According to Q311777
Because of problems related to setting the default gateway on the publishing servers you, could not use NLB on the external interface of the ISA Server to publish internal network servers prior to SP1. Let's go over the details of what the problems was before SP1.
Take a look at the figure below. The servers on IP addresses 192.168.1.10, 192.168.1.11 and 192.168.1.12 are all published by ISA Server and use the ISA NLB server with the internal interface 192.168.1.1 as their default gateway.
When you use Server Publishing Rules (on a simple network) to publish internal network servers, you need to set the publishing servers up to use the internal interface of the ISA Server as their default gateway. The problem with this is that the SecureNAT engine on the ISA Server leaves the original source IP address on the packet intact. The publishing server sees the original IP address in the packet and therefore needs to route the packet back to the Internet using a gateway or series of gateways that can get the packet back to the Internet host that made the request. On a simple, single segment network, the servers are configured to use the internal interface of the ISA Server as their default route to get packets back to hosts on Internet network IDs.
The problem with this is that if the publishing servers have a default gateway address assigned, they will always return the packets to their default gateway address (since the source IP address in the requests they receive are valid Internet addresses). This breaks NLB because the members of the NLB cluster are not state aware; if the packet was passed to the publishing server from the NLB ISA Server that is not configured as the Publishing server’s default gateway, the packet will be dropped by the ISA Server.
For example, suppose someone with the IP address 18.104.22.168 sends a request to one of your published mail servers. The virtual IP address on the ISA NLB cluster is 22.214.171.124. The IP address of the internal network server is 192.168.1.10 and that server is configured to use 192.168.1.1 as its default gateway.
The packet (with a source address of 126.96.36.199) is accepted by the ISA NLB server with an internal IP address of 192.168.1.2. That ISA NLB server passes the packet to the internal network server 192.168.1.10. The internal network server sees that the packet originated from 188.8.131.52 and so when it responds, it sends the packet to its default gateway (since the address is a non-local subnet address). The default gateway for 192.168.1.10 is 192.168.1.1, so it sends its reply to 192.168.1.1 so that it can be properly routed to the Internet host. When the packet is received by 192.168.1.1, the ISA Server has no idea what to do with it (since the NLB cluster members do not share session state information) and therefore it drops the packet.
Take a look at the figure below to see a frame in a network trace on a published DNS server (note that the traces come from various networks, not necessarily from those we're talking about in the text; that's why the IP addresses and network IDs may vary). Notice the packet source address is a public address, not the private IP address of the internal interface of the ISA Server. When this published DNS server replies to the request, it must send the response to a gateway that can route the packet to the Internet host that made the request.
The solution to this problem is to configure the ISA Server to include the private IP address of the ISA Server as the source IP address of the packet it passes to the published server. When the source IP address is the internal interface of the ISA Server, you don’t need a route to the Internet to return the response to the Internet host making the request. The ISA Server will keep track of the host IP address making the request and forward the response to the appropriate Internet host.
On a simple, single segment network (below), this allows you to not configure a default gateway on the published servers on your internal network. You can put in a default gateway if you like, but its not required to publish the server, because the published server does not require a default gateway when the internal interface of the ISA Server is on the same network ID as the publishing server.
If you have a complex network, you need to set a default gateway to allow published servers remote from the internal interface of the ISA Server to get to your various internal network IDs. But the default gateway address won’t cause a problem, because the source address in the packet forwarded by the ISA NLB cluster member is in the packet received by the published server. That way, the published server can return the response to the ISA NLB server that forwarded the packet to it in the first place.
Look at the figure below to see what happens when you configure the ISA Server to insert its own internal IP address as the source IP address on packets it forwards to published servers. Note the source address is 10.0.0.1, which is the internal interface of an ISA NLB cluster member. The published FTP server will send the response to the 10.0.0.1, and the ISA NLB cluster member will forward the response to the host that made the request.
Q311777 covers the Registry configuration. Just do the following:
1. Go to the Services node in the ISA Management console and stop the Firewall service.
2. Click Start an the click the Run command.
3. In the Open text box, type regedit
4. Navigate to:
5. Add the following REG_DWORD Value:
Value name: UseISAAddressInPublishing
Data type: REG_DWORD
Value data: 1
6. After you add the value, restart the Firewall service. I prefer to restart the entire ISA Server machine, but its up to you, as its not required.
That’s all there is to it. I highly recommend that you restart all ISA NLB cluster members after making the Registry change.
The final step is to create the Server Publishing Rules. You will see the virtual IP address for the NLB cluster in the IP address list for the external interface in the Server Publishing Rule Wizard. Configure all servers in the NLB cluster to use the virtual IP address when publishing servers. You can then test connectivity to the cluster after creating the Server Publishing Rule. Use netstat or Network Monitor to determine which server is accepting the connection and then take down that server. You’ll be able to reconnect in about 10-12 seconds.
Limitations to using NLB on the External Interface and Q311777
There are some limitations inherent in this method of using NLB on the external interface of the ISA Server. These include:
- Does not work for outgoing connections
- Source IP address is the internal interface of the ISA Server
The first problem should be immediately obvious. Since we are configuring NLB on the external interface only, the outgoing connection will not be load balanced or fault tolerant. Internet network servers and client that need to establish new outbound connections will not be able to take advantage of NLB when its configured on the external interface only. This could be a significant limitation if the publishing server is an Exchange Server that needs fault tolerance for inbound and outbound access. In this situation, you would need to configure two ISA Server NLB clusters, one with NLB configured on the external interface, and another on the external interface. You would also need to configure an SMTP relay on the internal network to accept the incoming connections to make the most of this configuration.
The second problem may be even more important to many Web server administrators. One of the drawbacks of using Web Publishing Rules to publish Web Servers on the internal network is that the source IP address that appears in the Web Proxy logs is the IP address of the internal interface of the ISA Server. This makes it very difficult to get information about client access to your Web server. One method you can use to get around this issue is to publish Web servers using Server Publishing Rules. But if you implement the Q311777 method, you will not longer get the original source IP address in the Web server logs.
Solutions to the NLB Problem
There are some solutions to the problems we have with NLB and ISA Server. In Windows .Net Server, ISA Server can be installed (with ISA SP1) and then be configured to use something called "bi-directional affinity". When bi-directional affinity is enabled, two different hash algorithms for the internal and external interfaces are used to determine which cluster node services a particular request. These different algorithms will synchronize the two NLB instances and avoid asynchronous routing. Bi-directional affinity allows you to configure NLB on both the internal and external interface.
Another solution is StoneBeat Full Cluster for ISA Server. This software add-on allows the ISA NLB cluster to use NLB on the internal and external interfaces on Windows 2000 machines. . From what we understand, StoneBeat Full Cluster provides features similar to those you'll see in Windows .Net Server. This is an interesting product and we'll review StoneBeat Full Cluster in the near future.
Both .Net Server bi-directional affinity and Stonebeat Full Cluster get around the problem with the private IP address showing up in the Web server logs
You can use Network Load Balancing Clusters to improve uptime for access to published servers on the internal network. In this article we went through how to configure NLB and how to configure the Registry to support ISA NLB clusters and Server Publishing. We also went over some of the problems inherent with implementing NLB and some possible fixes. You should now have a better idea of how the ISA NAT service handles packets, and how the default ISA NAT configuration creates problems with NLB and Server Publishing.
We would like to thank Zach Gutt (MS) and Steve Riley (MS) for inspiring us to do this article. Zach and Steve did a joint presentation at the recent TechNet conference in New Orleans and it was interesting and entertaining. If you get the chance, make it a point to see them do this presentation, its well worth your time! If you have any questions or problems with the information presented in this article, please let me know by writing to me at [email protected]. We'll see what can be done to fix you up. Thanks! --Tom