Exploring ISP Redundancy in Forefront Threat Management Gateway (TMG) 2010
One of the many new features in TMG that has long been requested by ISA firewall administrators is its ability to support multiple external network connections. ISP Redundancy (ISP-R) now provides this capability. With support for two unique ISPs (or more accurately, external network connections), we can now have fault tolerance and redundancy for our Internet or WAN connections. In this article we will explore the ISP-R feature in detail, look at the different operating modes, explain the load balancing algorithm, and investigate the dead link detection process. We will also discuss various deployment scenarios and considerations to be made when designing and implementing ISP-R.
ISP-R in TMG has two operating modes – Load Balancing and Failover. In Load Balancing mode, connections are balanced between two external networks evenly (by default) or unevenly (configurable by the administrator). If either external connection goes down, all communication will be sent over the remaining available connection. In Failover mode, one external network is configured as the primary connection, and the other is configured as the secondary connection. All communication is sent over the primary connection. If the primary connection goes down, all communication will be diverted to the secondary connection. Once the primary connection is available again, all communication will again be sent over the primary connection.
Preparing the Network Interfaces
ISP-R supports only two external network connections, and each connection must be on a unique subnet. For proper operation and optimum performance, both external network interfaces should be configured identically (pay special attention to your NIC driver’s offload settings). Ideally the network interface cards should be the same model.
Begin by giving each network interface a descriptive name (e.g. External_Sprint and External_Verizon). Configure the first external network interface with an IP address, subnet mask, and default gateway. If your TMG firewall is not a member of a domain and does not communicate with any internal network resources by name, you can specify your ISP’s DNS servers here. If your TMG firewall is a domain member, do not specify ISP DNS servers here (Internal DNS servers are configured on the internal network interface only). Once complete, click the Advanced… button.
Uncheck the box marked Automatic metric, and then enter 1 in the Interface metric: box.
Repeat these steps to configure the second external network interface, this time using an Interface metric: value of 2. Be sure to configure a default gateway on this second external interface. Generally this is not recommended, and Windows will complain when you attempt to do this.
In this scenario it is safe to disregard this warning and select Yes to proceed.
If your ISPs use DHCP to assign addresses, you will not be able to configure multiple default gateways. In this case you will create default persistent static routes before configuring ISP-R. In our example here, those routes would be configured as follows:
route add –p 0.0.0.0 mask 0.0.0.0 126.96.36.199
route add –p 0.0.0.0 mask 0.0.0.0 188.8.131.52
Configuring ISP Redundancy
Once the initial network interface configuration is complete, open the TMG management console and in the console tree highlight Networking, then select the ISP Redundancy tab.
In the Tasks pane, click Configure ISP Redundancy.
Choose Next, then select the ISP redundancy mode that meets your requirements. For demonstration purposes we’ll select the default option Load balancing with failover capability.
Specify the ISP connection name:, and then select a network adapter from the drop-down list.
Confirm that the gateway address and subnet mask are correct. If your TMG firewall is not a member of a domain and does not communicate with any internal network resources by name, you can specify your ISP’s DNS servers here. If your TMG firewall is a domain member, do not specify ISP DNS servers here (Internal DNS servers are configured on the internal network interface only).
In some cases there will be external servers that can only be reached via a specific external link. An example of this would be an ISP’s DNS server or mail server. If required, enter those servers here. You have the option to specify specific computers, computer sets, or address ranges.
Repeat the steps above for the second external network connection, and then select the distribution percentage by moving the slider accordingly. If both external links have the same bandwidth, you can safely leave this setting at 50%. If one link has more bandwidth than the other, configure that link to receive a greater percentage of traffic.
Choose Finish to complete the ISP-R configuration.
If you have configured DNS servers on the external network interfaces, be sure to create corresponding persistent static routes to ensure that requests for those resources are routed through the correct network interface.
In our example here, those routes would be configured as follows:
route add -p 184.108.40.206 mask 255.255.255.255 220.127.116.11
route add -p 18.104.22.168 mask 255.255.255.255 22.214.171.124
Once configured, the TMG management console will display information about each ISP connection, along with the currently configured redundancy mode.
After configuring ISP-R, to make configuration changes to a specific ISP connection you can right-click the connection and choose Properties.
Here you can change the name of the connection, alter the IP address/subnet mask information, enable or disable the connection, modify the load balancing ratio, or add, change, or remove dedicated servers.
Changing ISP-R Operating Mode
In this example we configured ISP-R for Load Balancing. If you wish to change the ISP-R operating mode, click Change ISP Redundancy Mode to Failover in the Tasks pane.
When switching from Load Balancing mode to Failover mode, be sure to edit the connection properties and select the appropriate connection role for the connection. Remember, in Failover mode all traffic will be sent over the primary external connection and the secondary connection will only be used if the primary connection is unavailable.
To view the status of each ISP connection, highlight Dashboard in the console tree.
The status for each ISP link will be displayed in the Network Status frame.
If a link becomes unavailable, the connection status will display an alert.
Additionally you will see a Connections Unavailable alert under the Alerts tab.
When the connection is back online, TMG will raise an informative alert indicating that the connection is once again available.
There are a number of ISP-R specific alerts to keep the TMG firewall administrator informed of the status and health of their external network connections.
Load Balancing and Dead Link Detection
It is important to understand that ISP-R distributes connections, not load. The manner in which ISP-R decides which external interface to distribute traffic to is determined by performing a hash of the source IP address and the destination IP address. The result is a number between 0 and 100. If the result is below the percentage configured for the first ISP connection, TMG will use this connection. If it is not, TMG will use the other external connection. This ensures session affinity – all connections for a specific source/destination address pair will be delivered through the same external network interface. The hash is computed for each outgoing connection.
To determine the availability of a particular ISP connection, TMG performs dead link detection by randomly polling one of the thirteen Internet root DNS servers on TCP port 53 (when TMG is deployed as a back firewall, make certain that TCP port 53 is open to the Internet). If the selected root DNS server responds, TMG considers the connection available. If it does not respond, TMG will poll additional root DNS servers at one minute intervals. If no replies are received after three consecutive attempts, TMG considers the connection unavailable and raises an alert. Once TMG identifies a connection as unavailable, it will wait for five minutes before attempting to poll again. Once it receives a response, TMG will continue polling at one minute intervals. When three consecutive responses have been received, TMG will consider the connection available.
The choice of ISP-R operating modes is influenced primarily by the types of Internet or WAN connections you have. For example, if you have two similar Internet connections in terms of bandwidth, Load Balancing mode is a good choice. If you have one high bandwidth connection and one low bandwidth connection, then Failover mode would be more appropriate. Although this technology is called ‘ISP’ redundancy, it is not limited to Internet-connected links. ISP-R can be used to provide load balancing and failover for WAN links between a branch office and a main office (see considerations below).
There are a few considerations to be made when designing and deploying ISP-R.
- Works with NAT only – ISP-R will only provide load balancing and failover for traffic originating from TMG protected networks and destined for the default External network, and will only work when the network relationship is configured as NAT. If the network relationship is configured as route, ISP-R will not function. This is important because traffic originating from the TMG firewall itself will not be processed by ISP-R, as the network relationship between the Local Host network and the External network is route.
- E-NAT overrides ISP-R – For traffic processed by a network rule configured with Enhanced NAT (E-NAT), E-NAT takes precedence and will override any routing decisions made by ISP-R.
- Load balancing is not perfect – The load balancing mechanism in ISP-R does not distribute traffic perfectly. Since traffic is distributed by connections, not load, the potential exists for some connections to consume more bandwidth than others, skewing the distribution percentage.
When ISP-R is configured to provide load balancing or failover for branch office WAN connections, the default dead link detection mechanism may not be appropriate. If you recall, TMG will randomly poll Internet root DNS servers to verify connectivity. If, for example, the TMG firewall is configured to NAT traffic between a branch office and a main office and the main office Internet connection is unavailable, TMG will report both of its WAN connections as being unavailable, when in fact they are.
In some cases, branch office TMG firewalls may not have direct connectivity to the Internet, which will prevent TMG from polling Internet root DNS servers. In this branch office firewall scenario it would be better to poll services located directly on the other side of the WAN connection. To change the default link detection parameters and to make changes to polling frequency, please refer to this article [http://blogs.technet.com/isablog/archive/2009/11/26/tmg-isp-redundancy-unleashed.aspx] on the Forefront TMG product team blog.
ISP Redundancy is a valuable new feature in TMG that provides fault tolerance and redundancy for external network connections; for ISP connections in the case of an edge firewall deployment, or WAN links in a branch office firewall scenario. Load Balancing and Failover operating modes provide flexible configuration options to match any external network configuration, and verbose alerting capabilities keep the TMG firewall administrator informed on the external network connection status.