If you would like to read the next part in this article series please go to Kicking the Tires on the TMG 2010 RC ISP Redundancy – Part 2: Enabling ISP Redundancy.
Introduction
One of the features that I have been looking most forward to in the new TMG firewall is the ISP load balancing feature. If you have been in the ISA firewall space for a while, you might know that support for multiple ISPs is a feature that has been requested since ISA 2004 came out. Good things are worth waiting for, and now we are going to have it with the upcoming TMG firewall!
Before getting into the details of the TMG multi-ISP feature, there are a few things you should know about it in advance:
-
While the term for the feature is “multi-ISP” support, it more accurately could be called “dual ISP” support, since you are limited to two ISPs
-
There must be a NAT relationship between the source and destination Networks – so if you are using a route relationship on any of the TMG firewall Protected Networks, they are not going to be able to take advantage of multiple ISPs
-
Each ISP connection needs to connect to a default gateway that is on a different network ID as the other ISP – both default gateways can not be on the same network ID (which means the external IP addresses on the TMG firewall can not be on the same network ID)
-
Your external interfaces can not use DCHP to get their addresses – if you are using a home-user type ISP connection, then multi-ISP support is not for you
-
You can host both ISP connections on one NIC, or two NICs – this article covers a two-NIC configuration where each ISP connection is represented by its own external interface
-
Network offload processing needs to be set either on or off on both NICs – if one of the NICs has it on and the other has it off, then offloading processing will be disabled on the NIC that has it on.
Those are some of the basic things you should know about multi-ISP support before we get started.
Multi-ISP support allows you to use your two ISPs in either of two ways:
-
Failover only – in this mode one ISP is used at all times until it becomes unavailable. When that happens, connections are forwarded to the secondary ISP. This is a good choice when you have one high speed link and the other is slow – but good enough to get you by until the primary link comes back up again – also, you do not have to pay for bandwidth used on a link that not being used until required
-
Failover and load balancing – in this mode both links are used. You have the option to set a weighting for each of the links, so if you do not want to use both of them equally, you don’t have to. If one of the links fails, all the connections go to the one that’s online
All that stuff sounds good on paper, but does it really work? Yes it works. But, do you really want to believe me, or do you want to set it up yourself and see that you can make it work. I suspect the latter. Thus, I am going to show you how to make it work in a virtual environment.
As you might know, my preferred dev environment for articles on ISAserver.org is VMware Workstation (I’m currently using version 6.5). Why? Because that is what I have always used, it works for me, I understand it and its networking model. There is no reason you have to use VMware Workstation to test this configuration yourself. If you like Windows Virtual PC, ESX Server, or Microsoft Hyper-V, those will work great too. The only difference is going to be in the terminology, but all the same principles apply to each product.
Let us start with the basic virtual networking scheme for this test lab. We are going to use four virtual networks or virtual switches, each of which belongs to a different physical or virtual Ethernet broadcast segment.
-
Bridged – this is the live production network in my office. The virtual NICs are bridged to this network, have valid IP addresses on the live network, and connect to the Internet on this network.
-
VMNet3 – This is a virtual switch that represents the Ethernet segment that connects the TMG firewall to the first ISP
-
VMNet4 – this is the virtual switch that represents the Ethernet Segment that connects the TMG firewall to the second ISP
-
VMNet2 – this is the virtual switch that presents that Ethernet segment connecting the internal interface of the TMG firewall to the default Internet Network
The figure below shows each of the VMNets and the devices that are connected to them:
-
RRAS1 – this is a Windows Server 2003 VM with the RRAS Service configured as a NAT server. The external interface of this VM is connected to the Bridged network and the internal interface is connected to VMNet3 – which connects the NIC on the TMG firewall dedicated to RRAS1 ISP to the RRAS Windows 2003 NAT server
-
RRAS2 – this is a Windows Server 2003 VM with the RRAS Service configured as a NAT server. The external interface is connected to the Bridged network and the internal interface is connected to VMNet4 – which connects the NIC on the TMG firewall dedicated to the RRAS2 ISP to the RRAS Windows NAT server
-
TMG Firewall – the TMG firewall has three NICs. One connected to VMNet3, which connects it to the RRAS1 ISP; one connected to VMNet4, which connects it to the RRAS2 ISP; and one connected to VMNet2, which connects it to the default Internet Network
-
DC – is a Windows Server 2003 domain controller for the msfirewall.org domain; the TMG firewall belongs to this domain and it is connected to VMNet2
Some things to note about the configuration:
-
The RRAS1 and RRAS2 nodes represent the default gateways you would use when configuring ISPs for your production configuration. So, internal IP address of RRAS1 represents the default gateway of your first ISP, and the internal IP address of RRAS2 represents the default gateway of your second ISP. Our test environment is different, in that the actual Internet connection is on the bridged network; because of this the external interfaces on RRAS1 and RRAS2 use the same default gateway
-
I am using dedicated NICs on the TMG firewall for each of the ISPs – this is not required, and in the next article, I will go over the alternate configuration where you do not have dedicated NICs for your ISP connections
-
You can achieve the same network segmentation with other virtualization solutions – Windows Virtual PC, ESX and Hyper-V all support similar methods of networks segmentation
Figure 1
Now that we have the virtual network infrastructure defined, we can take a look at the IP addressing scheme. IP addressing information used in this lab is shown in the figure below. Notice that the TMG RRAS1 ISP NIC uses the internal interface of RRAS1 as its default gateway, and that the TMG RRAS2 ISP NIC uses the internal interface of RRAS2 as its default gateway. Also, notice that the RRAS1 network segment is on network ID 10.0.1.0/24 and that the RRAS2 network segment is on network ID 10.0.2.0/24.
The TMG firewall’s default Internal Network is on network ID 10.0.0.0/24, and the DC on the default Internal Network uses the TMG firewall as its default gateway.
Note:
Notice the capitalization used for the term “network”. When “network” is spelled in lower case, it represents the term in its generic sense. However, the term “Network” with a capital “N” refers to a TMG firewall Network – which has to be defined on the TMG firewall. The TMG firewall’s default Internal Network is an example of a TMG firewall Network.
Figure 2
In this article we are going to go over the ISP load balancing feature. Because of this, I thought you might like to know how the load balancing decisions are made by the TMG firewall. Essentially what happens is that the TMG firewall looks at the source address (the client) and the destination address (the server) and creates a hash value, which is then represented as a value somewhere between 1-100. All possible hash values are equally distributed across this range. After calculating this value, the TMG firewall then looks at the weighting assigned to each ISP.
For example, suppose ISP1 was assigned 75% of the load and ISP2 was assigned 25% of the load. If the hash value based number is 30 – the connection would go to ISP1, since that hash based value is lower than value assigned to the preferred connection. If the hash based value were 80, then the connection would be forwarded to ISP2, since the value is higher than the value assigned to the preferred connection.
So, to summarize:
-
Check the load number on the preferred connection – this is presented as a percentage
-
If the hash based value is lower than that number, then it goes to the preferred connection
-
If the hash based value is higher than that number, then it goes to the secondary connection
The figure below shows you an example. ISP RRAS1 is assigned 75% of the connections and ISP RRAS2 is assigned 25% of the connections. You can see the configuration interfaces in the figure. When PC1 connects to WEB-1, a hash based value is 60 is calculated. Since this is lower than the percentage assigned to the preferred connection, the connection is forwarded through the preferred connection, which in this case is ISP RRAS1. When PC1 connects to WEB-2, the hash based value is calculated as 80, which is higher than the value for the percentage assigned the preferred connection, so it’s forwarded through the secondary (non-preferred) connection.
Of course, if you set both of the ISP connections to 50%, half the hash based values would be lower than 50 and half would be higher than 50 – so all connections would be equally distributed between the ISPs.
Figure 3
What about network interface configuration on the TMG firewall? This was something that was not very well understood before a recent posting on the TMG Firewall Team blog, and they did a great job explaining what had to be done – something that was not included in the Help File (hint!).
In the figure below you can see the network interfaces configured on the TMG firewall used in this example. The LAN interface is connected to VMNet2, WAN interface is connected to VMNet3 and WAN2 is connected to VMNet4.
Figure 4
The WAN interface was the initial NIC configured on the firewall, and therefore was given a default gateway before getting started on the TMG firewall configuration. In the figure below you can see that it has been assigned an IP address valid for ISP RRAS1 and given the default gateway that is the internal IP address of the RRAS1 virtual machine (this would be the default gateway of your actual ISP1 in a production environment).
There is one more thing you need to do though. You need to get into the Advanced TCP/IP settings and turn off the Automatic metric feature. Microsoft recommends this so that the ISP redundancy feature works correctly – maybe there is a contention between the Windows and TMG routing mechanisms – they did not mention the details. However, you do need to turn this off and set a manual metric. The value is not important from what I can tell – the only requirement is that you set your preferred interface with a low metric than the non-preferred interface. In the figure below, you can see that I set the preferred interface to use a metric of 1.
Figure 5
The figure below shows the IP addressing information for the non-preferred ISP, which is the RRAS2 ISP. I set the metric for this interface to be 2.
Figure 6
Whoa! What’s this? Windows is not happy with me assigning two default gateways on the same machine. I can not say that I blame Windows for this, since we know that we are not supposed to do that. However, the TMG firewall’s ISP redundancy feature allows us to break this rule, so you can safely click Yes because you know that in this instance, it’s going to be OK.
Figure 7
Just for fun, and before setting the ISP redundancy feature up on the TMG firewall, I wanted to see what interface would be used. I had already configured the TMG firewall and created an “all open” rule (famous for testing – but infamous and detested in production environments). Using tracert, I found that the preferred interface was used. That is good – and probably expected because it has the lower metric? Or maybe because this was the first interface configured on this computer that had a default gateway? If I have time later, I will see what happens when I switch the metrics.
Figure 8
I guess that should bring up a discussion of when to configure the interfaces and in what order. I’m afraid this is going to be a little like “shaking the skulls and bones and then pitching them onto the floor to read the signs”, since I haven’t found any information on what order to do things. However, this is what I did:
-
Created the initial virtual machine with only two NICs – internal and external
-
Assigned IP addressing information to the internal and external interfaces
-
Installed the TMG firewall software
-
Confirmed that the TMG firewall installation was successful
-
Turned off the TMG VM and installed a third virtual NIC to support the second ISP link
-
Configured the 3rd virtual NIC after restarting the VM
-
Restarted the VM after configuring the IP addressing information on the 3rd interface
I need to be clear that this is not a recommended approach, it is not a preferred approach, and it is not something I got from Microsoft or anyone else. This is what I did and it worked. In the future I will test it and see what happens with a more simple approach that does not require machine restarts.
Summary
In this, Part 1 of my series on TMG firewall ISP redundancy, we went over the virtual networking infrastructure and some details on how which ISP is selected. We then talked about interface configuration and some of the things you need to be aware of when configuring the network interfaces that connect to each ISP. In the next article, we’ll actually configure the ISP load balancing feature and see if it actually works by testing from a client and checking the TMG firewall logs and some Netmon traces. See you then! –Tom.
If you would like to read the next part in this article series please go to Kicking the Tires on the TMG 2010 RC ISP Redundancy – Part 2: Enabling ISP Redundancy.
