Configuring ISA/VPN Servers to use Network Load Balancing – Part 1

Configuring ISA/VPN Servers to use Network Load Balancing, Part 1

By Thomas W Shinder M.D.

One of the reasons why you would want to use ISA Server over another firewall is ISA Server’s great VPN integration. Unlike other firewalls solutions, you don’t have to pay one penny extra to have a virtually unlimited number of VPN client connections to your ISA Server. ISA Server’s combination of firewall, Web Proxy and VPN features make it a favorite in the Microsoft and non-Microsoft shops alike.

If your company uses ISA Server and depends on high availability for VPN client connections, then you might want to consider using Windows 2000 Advanced Server’s Network Load Balancing (NLB) feature. While the cost of Windows 2000 Advanced Server can be high compared to Windows 2000 Server, you might be able to legitimize the cost when you assess how much money VPN downtime would cost you and how much a competitor’s firewall/VPN solution with load balancing and fault tolerance would cost. You might just find that Windows 2000 Advanced Server, ISA Server and NLB end up saving you’re a load of cash!

Windows 2000 does not support NLB for incoming L2TP/IPSec connections, but it does fully support incoming PPTP connections. Windows 2003 supports both PPTP and L2TP/IPSec connections. However, there are some serious limitations regarding what client operating systems can access a Windows 2000 server configured as both a VPN and ISA Server. We’re go into those details in a little bit.

I won’t go into the configuration details of VPN clients and servers, as I’ve done that in a number of articles over at www.isaserver.org/shinder and in Configuring ISA Server and ISA Server and Beyond. I’ll assume that you’re already knowledgeable about VPNs and want to take your existing VPN solution to the next level by implementing NLB for incoming VPN connections.

In the rest of this two part article I’ll go over the following subjects:

The configuration I recommend for your ISA/VPN Server will limit the type of VPN client you can use to connect to the VPN server. Windows 9x, NT, and Windows XP SP1 VPN clients will not be able to connect to the ISA/VPN server because the NLB dedicated address and primary NLB VIP are different. Downlevel and Windows XP SP1 VPN clients expect the VPN server to respond on the same IP address that the VPN client sent the connection request to. That can’t happen with our multicast mode ISA/VPN server because we’re using a single network interface for the NLB dedicated and virtual addresses. The problem of the VPN server answering an responding on different addresses is described in the following KB article: http://support.microsoft.com/default.aspx?scid=kb;en-us;271731 Something a bit disappointing is that Windows XP did not have this problem until Windows XP SP1 was released. If you do not install Windows XP SP1, you will still be able to use Windows XP as a VPN client.

The bottom line is that if you want to create a PPTP connection to your ISA/VPN server, you will need to use Windows 2000 or Windows XP pre-SP1.

Let’s look at the problem more detail, as it will help you understand how NLB works with ISA Server and also help you explain the problems to your boss or client. The figure below shows an ideal ISA/VPN server configuration. There is a single IP address bound to the external interface of the ISA Server and this is also the NLB primary address; no dedicated IP address is configured on the external interface. This configuration works great for VPN client connections to the ISA/VPN Servers and it also allows both PPTP and L2TP/IPSec clients to connect to the ISA/VPN server (although L2TP connections will be dropped during convergence events). The VPN client connects to the VIP and responses are returned from the VIP.

If we only wanted to use the ISA/VPN Server as a VPN server and not allow internal clients outbound access through the same server, then everything would be great. Unfortunately, most shops probably aren’t that interested in using ISA Server just for its VPN services. Most people want to use ISA Server as a firewall too. This is where we start to run into problems.

Take a look at the figure below and follow the steps:

1. Internal client makes a request for Internet resources through the ISA Server.
2. The ISA Server has a single IP address bound to the external interface. It sends a request to the Internet using this IP address as the source IP address.
3. The Internet server sends its response to the IP address used as the source address in step 2. The NLB driver determines that the frames from the Internet server should be handled by the ISA Server that sent the request.
4. The ISA Server that originally handled the request for the internal network client returns the response to the ISA Server client.

So far, so good. But let’s look at another scenario that can happen when using NLB with a single IP address bound to the external interface:

1. The internal client sends a request for an Internet resource through the ISA Server.
2. The ISA Server forwards the request to the Internet server, using the NLB VIP as the source address in the request.
3. The Internet server returns the packet to the NLB VIP that was included in the forwarded request. The NLB driver determines that the other ISA Server should handle the response. The other ISA Server is not aware of communication states (connection table entries) on the other ISA Server because neither the ISA Server software nor the NLB driver software share state information among NLB array members. Since the response is returned to an ISA Server that has no knowledge of the original communication, its drops the response. The connection attempt of the original ISA Server client fails.

Its clear that using a single VIP on the external interface of the ISA Server won’t work if you want to support outbound connections from internal network clients, as the NLB driver might assign the response to the wrong ISA Server (sometimes referred to as “asymmetric routing”). The solution is to bind at least two addresses to the external interface of each ISA Server. One address is the VIP, and the second address is the dedicated IP address. We then make the dedicated IP address the primary IP address on the external interfaces of the ISA Servers. This allows all outbound packets from the ISA Servers to use their dedicated IP address as the source IP address for outgoing communications. And because the dedicated IP address is different on each NLB array member, the responses from the Internet servers are always returned to the correct ISA Server.

The steps are depicted in the figure below:

1. The ISA Server client sends a request for Internet resources to the ISA Server that its configured to use.
2. The ISA Server forwards the request using the dedicated IP address on the external interface of the ISA Server.
3. The Internet server returns its response to the dedicated IP address of the ISA Server that send the request.
4. The ISA Server that accepted the request from the client returns the response to the client.

You’ll see the same thing happen in steps A, B C and D. The reason why these outbound requests work is that you’ve configured the ISA Servers to use the dedicated IP address as the primary IP address bound to the ISA Servers external interfaces.

At this point you might think we’ve got things fixed. Not quite. Let’s look what happens with the same ISA Server NLB configuration, but this time from the VPN client’s perspective:

1. The PPTP VPN client sends a VPN connection request to the VIP for the ISA/VPN NLB array.
2. If the PPTP VPN client is a Windows 2000 or Windows XP client (pre SP1), the PPTP VPN connection is successful and connects to resources on the internal network.
3. The Windows 2000 and Windows XP PPTP VPN client receives data from the server on the internal network via the VPN connection. The source IP address of the data returned to the PPTP VPN client is the dedicated IP address on the external interface of the ISA Server, not the VIP that the client originally connected to.
4. If the VPN client is running Win 9x, NT or Windows XP SP1, the connection request would fail because the client connects to the VIP and the response is from the dedicated IP address. These VPN clients will not allow responses to come from an IP address it did not send the connection request to.

The packet trace in the figure below shows connections attempts made to the VIP and responses returned via the external interface’s primary IP address. In this trace, the VIP is 172.16.0.1 and the primary IP address on the array member is 172.16.0.3. This is an L2TP/IPSec connection attempt, but you will see similar behavior with PPTP connections.

Things get even more interesting when the VPN client is behind a firewall (such as ISA Server). Suppose you have a PPTP VPN client behind an ISA Server. You’ve configured your ISA Server to allow outbound PPTP passthrough and the PPTP VPN client wants to connect to an ISA/VPN Server on the remote network using the VIP of the ISA/VPN array members. Here’s what happens:

1. The PPTP VPN client sends the outbound PPTP request to the ISA Server
2. The ISA Server forwards the request to the VIP on the remote ISA/VPN NLB array.
3. NLB makes a decision to not sent the frames to the ISA/VPN server with the dedicated IP address 22.22.22.3
4. NLB does decide to send the request the frames to the ISA/VPN with the dedicated IP address 22.22.22.2. The ISA/VPN Server accepts the incoming connection request
5. The remote ISA/VPN server sends its responses during the PPTP tunnel negotiation phase using the primary IP address on the external interface of the ISA/VPN Server, not the VIP. Since the ISA Server sent the original request to the VIP, the ISA Server has a state table (connection table) entry that allows it to accept responses from the VIP address of the remote ISA/VPN Server. The ISA Server has no knowledge of the dedicated IP address on the remote ISA/VPN server, and so when the response is returned with the dedicated IP address of the remote ISA/VPN Server in the source header, the ISA Server drops the request and the VPN connection fails.

As you can see, you run into some significant problems when combining the Windows 2000 NLB service with ISA Server. External clients will likely need to be directly connected to the Internet and will need to run either Windows 2000 or pre-SP1 Windows XP.

The problems with incoming VPN client connections are tied to two main issues: an issue with the Windows 2000 RRAS service that causes it to respond on an alternate IP address, and the lack of bidirectional affinity for outbound requests from internal network clients.

Both of these problems can be solved using innovative products called RainConnect and RainWall from Rainfinity. RainWall provides bidirectional affinity for inbound and outbound requests going through the ISA Server. This prevents the problem we see with the Windows 2000 NLB, where responses might be delivered to the wrong array ISA NLB array member. RainWall provides network load balancing like the Windows 2000 NLB, but it does so in a more sophisticated manner. RainWall array nodes share state table (connection table) information with one another and use this shared information to determine which node should accept incoming responses. This prevents the responses from being delivered to the wrong ISA array member.

Rainfinity also recognizes the limitations of the Windows 2000 RRAS service and how those limitations negatively affect incoming VPN connections. To help you solve this problem, they include a special limited version of RainConnect, which allows the RainWall array to accept VPN client connections. If you find the limitations mentioned in this article significantly limiting, you should definitely check out RainConnect and RainWall. They’re easy to set up and configure and you can be literally up and running with these products in less than an hour.

In this first part of our two part article on how to allow incoming VPN connections to an ISA/VPN Server NLB array, we went over some of the basic concepts of NLB and how they influence how VPN clients connect to array members. We also went over some of the limitations the Windows 2000 NLB service combined with ISA Server present to the ISA/VPN administrator and how you can deal with some of them. Finally, I went over how RainWall and RainConnect can solve these problems. In part 2 of this article, I’ll go over the details of configuring the ISA/VPN server and clients to use the NLB for incoming VPN connections.