Intrusion Detection and Prevention in Forefront TMG (Part 1) – Behavioral Detection

If you would like to read the next part in this article series please go to Intrusion Detection and Prevention in Forefront TMG (Part 2) – Network Inspection System.

The Forefront TMG firewall includes integrated intrusion detection and prevention capabilities designed to protect clients and itself from various kinds of network attacks. In part one of this two-part series we’ll explore TMG’s behavioral detection capabilities.

Introduction

Microsoft Forefront Threat Management Gateway (TMG) 2010 is a multi-layered perimeter defense system that includes several advanced protection technologies. In addition to integrated URL filtering, virus and malicious software scanning, and HTTPS inspection, the TMG firewall also includes intrusion detection and prevention capabilities. In part one of this two-part series we’ll examine the basic behavioral-based intrusion detection and prevention functionality. In part two we’ll take a look at the Network Inspection System (NIS), which provides enhanced vulnerability-based intrusion detection/prevention.

Common Network Attacks

Intrusion detection and prevention in Forefront TMG builds upon the behavioral-based intrusion detection and prevention capabilities that originally appeared way back in Microsoft ISA Server 2000. By performing low-level packet inspection, the TMG firewall can identify and block many common types of attacks. Intrusion detection and prevention is configured by opening the TMG management console, highlighting the Intrusion Prevention System, then selecting the Behavioral Intrusion Detection tab in the main console window and clicking the Configure Detection Settings for Common Network Attacks.


Figure 1

 

Here you’ll find that intrusion detection is enabled and configured by default to identify, block, and log the following attacks:

  • Windows out-of-band (WinNuke)
  • Land
  • Ping of death
  • IP half scan
  • UDP bomb
  • Port scan (not enabled by default)

 


Figure 2

In addition, by selecting the DNS Attacks tab you’ll find that the TMG firewall also has the ability to provide advanced application-layer protection for many common attacks against published DNS servers, such as DNS hostname overflow and DNS length overflow. The TMG firewall can even prevent DNS zone transfers from taking place (not enabled by default).


Figure 3

 

IP Options Filtering

Intrusion detection and prevention in Forefront TMG also allows for the filtering of specific IP options that may be included in an IP packet’s header. The filtering of IP options is enabled by default and can be configured by clicking the Configure IP Options Filtering link in the main console window.


Figure 4

 

Most IP options are fairly innocuous. However, there are a few IP options that, when present, are indications of potentially malicious behavior. By default, the TMG firewall is configured to filter IP options and to drop any IP packets that contain IP options 7 (Record Route), 68 (Time Stamp), 131 (Loose Source Route), and 137 (Strict Source Route). If an IP packet has specific options enabled, the TMG firewall administrator can dictate what action to take. Any packet containing an IP option can be dropped, packets with selected options can be denied, or packets with all except the selected IP options will be denied.


Figure 5

 

The TMG firewall can also prevent the forwarding of packets that contain IP fragments. This option is not enabled by default. Blocking IP fragments can have unintended side effects, so enable this feature with caution.


Figure 6

 

Flood Mitigation

Flood mitigation is an important component of the Forefront TMG firewall. This feature reduces the impact of certain types of attacks on the firewall. It mitigates potential denial of service (DoS) caused by worm flood traffic by enforcing restrictions on the amount of traffic a single host can generate. Flood mitigation is enabled by default and can be configured by clicking the Configure Flood Mitigation Settings link in the main console window.


Figure 7

 

Flood mitigation controls the following network parameters:

  • Maximum TCP connect requests per minute per IP address
  • Maximum concurrent TCP connections per IP address
  • Maximum half-open TCP connections
  • Maximum HTTP request per minute per IP address
  • Maximum new non-TCP session per second per rule
  • Maximum concurrent UDP session per IP address


Figure 8

Clicking Edit… next to any of the flood mitigation parameters allows the administrator to configure both the default connection limit and the custom limit, which applies to IP exceptions (more on this later).


Figure 9

 

When a host generates enough traffic to exceed any of these predefined limits, the TMG firewall will begin dropping packets from the offending host and will generate an alert.


Figure 10

 

Note:
It is important to remember that only communication that is allowed by firewall policy counts toward the connection quota.

If the offending host stops sending requests, or reduces the frequency of those requests to a level that is below the established limits, after one minute the TMG firewall will once again allow new connections from the host.

Most often when a host exceeds the connection limits it is due to malicious traffic (e.g. worm propagation or “phone home” traffic) or perhaps a poorly written or badly behaving application. There are instances when a particular host will generate a large amount of valid traffic, however. Some examples include DNS servers and SMTP servers. Unfortunately, some administrators will attempt to resolve these issues by raising the default connection limits, or worse, disabling flood mitigation completely. This is definitely not recommended. The proper way to deal with scenarios like this is to create an IP exception, which then enforces the custom limit threshold only to those systems included in the IP exceptions computer object. This allows more overhead for systems that are genuinely very busy and generate a lot of legitimate traffic, while still affording a high level of protection for all other systems and ensuring the stability and resilience of the firewall under busy conditions.


Figure 11

 

SIP Quotas

With the introduction of the SIP filter in TMG 2010, the firewall now has the ability to enforce quotas for SIP traffic. TMG places limits on the following SIP parameters:

  • Total maximum number of registrations
  • Maximum number of concurrent registrations per IP address
  • Total maximum number of concurrent calls
  • Maximum number of concurrent calls per IP address

Clicking Edit… next to each of the parameters allows the administrator to configure the thresholds to meet their specific requirements.


Figure 12

 

Summary

Forefront TMG’s behavioral-based intrusion detection and prevention provide a basic level of protection from common network attacks. It prevents the delivery of IP packets with IP options enabled that are indicative of suspicious and potentially malicious behavior. In addition, TMG mitigates Denial of Service attacks by enforcing connection limits, preventing malicious users or infected hosts from overwhelming the firewall with traffic. TMG firewall administrators can also configure quotas for the amount of SIP traffic generated by protected clients.

If you would like to read the next part in this article series please go to Intrusion Detection and Prevention in Forefront TMG (Part 2) – Network Inspection System.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top