Intrusion Detection and Prevention in Forefront TMG (Part 2) – Network Inspection System

If you would like to read the first part in this article series please go to rusion Detection and Prevention in Forefront TMG (Part 1) – Behavioral Detection.


Microsoft Forefront Threat Management Gateway (TMG) 2010 is a multi-layered perimeter defense system that includes several advanced protection technologies. In addition to integrated URL filtering, virus and malicious software scanning, and HTTPS inspection, the TMG firewall also includes intrusion detection and prevention capabilities. In part one of this two-part series we looked at the basic behavioral-based intrusion detection and prevention functionality. In part two we’ll examine closely the Network Inspection System (NIS), which provides both traditional exploit-based and enhanced vulnerability-based intrusion detection and prevention.

Network Inspection System

The Network Inspection System (NIS) is an all-new intrusion detection and prevention system that was first introduced with Forefront Threat Management Gateway (TMG) 2010. NIS analyzes network traffic and performs low-level protocol inspection to detect and prevent attacks on vulnerabilities in Microsoft operating systems and applications. NIS is signature based, and those signatures are developed by the Microsoft Malware Protection Center (MMPC). Signatures are made available to NIS concurrently with security updates released during the normal Microsoft update release cycle (second Tuesday of each month) or they can be released out-of-band in response to a zero-day threat if necessary.

NIS is designed to prevent known vulnerabilities in Microsoft operating systems and applications from being exploited remotely. The signature set is relatively small, but laser-focused. The result is a somewhat limited range of protection, yet the solution is very accurate and produces few false positives. The secret to the effectiveness of NIS lies with aprotocol parsing language called GAPA (Generic Application-level Protocol Analyzer). GAPA is not dissimilar from the protocol parsing functionality provided by Network Monitor. This type of inspection is much more accurate than typical byte-stream analysis. NIS analyzes each packet for protocol state, message structure, and message content. It can identify and block attacks on known vulnerabilities for which a signature is installed and enabled. It can also identify protocol anomalies and prevent protocol abuse for traffic that does not conform to RFC or implementation best practices.

NIS technology is included with many Forefront protection products, including server protection products like Forefront Protection for Exchange (FPE) and SharePoint (FPSP), as well as endpoint protection products like Forefront Endpoint Protection (FPE) and even Microsoft Security Essentials (MSE). With NIS widely deployed, Microsoft gathers extensive telemetry and feedback about the nature and types of attacks that are taking place in the wild and uses this information to improve the quality of signature updates.

With NIS enabled, once a packet has been allowed by firewall policy and has been inspected by any protocol filters required, the NIS policy engine performs low-level network protocol inspection using the currently installed signature set. If a signature match is made, NIS takes action according to the defined policy (block or detect only) and raises an alert. NIS supports network protocol inspection for DNS, HTTP, IMAP, MIME, MSRPC, POP3, SMB, and SMTP. In the future, Microsoft may add more protocols if the need arises. NIS licensing is complementary; there are no additional licenses required to enable NIS functionality.

Enabling and Configuring NIS

NIS can be enabled and configured using the Getting Started Wizard and clicking the Define Deployment Options link, then selecting Activate complementary license and enable NIS from the dropdown box.

Figure 1

Set the NIS Signature Update Settings and choose an automatic definition update action. You have the option to Check for an install definitions (recommended), only check for definitions, or no automatic action. Choose an Automatic polling frequency and specify an update alert threshold.

Figure 2

Define a New Signature Set Configuration by selecting a default response policy for new signatures. You can choose to accept the Microsoft default policy (recommended), Detect only response, or No response (disable signature).

Figure 3

Once NIS has been enabled and configured, you can access the NIS configuration by opening the TMG management console, highlighting the Intrusion Prevention System, then selecting the Network Inspection System (NIS) tab in the main console window. At the top of the main window you can view the NIS Status, the Signature Set Version, the New Signature Response:, and the Update Action:.

Figure 4

Clicking on any of the links will bring up the NIS properties page. On the General tab you can enable or disable NIS entirely.

Figure 5

On the Exceptions tab you can define any network object (networks, network sets, computers, computer sets, address ranges, subnets, or domain name sets) that should be excluded from NIS inspection. Exempting some network traffic from NIS inspection may be required in situations where, for example, trusted systems exchange a lot of information and you need to reduce the load on the TMG firewall, or perhaps an application uses network protocols that do not adhere to RFC standards.

Figure 6

On the Definition Updates tab you can review and change the settings for NIS signature set updates and alerting, as well as the default response policy for new signatures. By default, Microsoft chooses whether the signature should be enabled, and if it should be set to block or detect only. This decision is made based on the signature type and their confidence in the accuracy of the signature. By clicking the Version Control… button, the administrator can roll back to a previous signature set, if required. This may be necessary if a new signature set causes problems in your network environment. If you choose this option you will be presented with a warning indicating that “activating an older NIS signature set may expose your network to newly discovered threats”.

Figure 7

On the Protocol Anomalies Policy tab the administrator can define how NIS will respond when it identifies anomalous network traffic. As mentioned earlier, NIS performs protocol inspection and can identify when traffic does not conform to RFC or implementation best practices. By default, NIS is configured to allow anomalous traffic to avoid blocking legitimate traffic. If you elect to block protocol anomalies, this provides a higher level of security, but at the risk of blocking valid communication.

Figure 8

Exploring NIS Signatures

In the center of the main window you can view the current NIS signature set. As you can see, there are around 200 signatures currently loaded. You can group the signatures by attention required, response, policy type, business impact, category, date published, severity, fidelity, protocol, and status by selecting an option in the Group by: drop down menu. You can also sort by clicking on column headers in either ascending or descending order. You’ll see that I’ve sorted the signature set by Date Published, which allows me to quickly see the latest signatures that have been added.

Figure 9

Double-clicking a NIS signature will bring up a window that provides details about the specific signature. Here I’ve opened the properties for the vulnerability Win/MSIE.Redirect.RCE!CVE-2011-1262. You can see that the response policy for this signature is set to Microsoft default (recommended) and that the signature is enabled and set to block. The administrator has the option to override the default response policy by clicking Override. Here you can enable or disable the signature, or change the response policy as required.

Figure 10

Clicking on the Details tab reveals more information about the signature, including the affected applications, business impact, category, CVE numbers, date published, default response, default status, fidelity, protocol, related bulletins, severity, and vendor. There is also a field where an administrator can add notes about the signature, which is handy in the event a change in the default settings has been made. Clicking the More help about this NIS signature online takes you to the Microsoft NIS encyclopedia entry for this signature, where even more details about the signature are available.

Figure 11

Configuring NIS Response

The Tasks pane includes links to several configuration tasks. Two important configuration options that can be accessed here are Set All Responses to Microsoft Defaults and Set All Responses to Detect Only. If you wish to configure NIS as purely an intrusion detection system, set the default response policy to detect only. NIS will continue inspecting traffic but will alert only and not block. This might be helpful when enabling NIS for the first time on your production network. After you are confident that NIS will be non-intrusive and won’t disrupt normal communication, you can set all response back to Microsoft defaults.

Figure 12

Signature Types

There are three different types of NIS signatures.

  • Vulnerability-based – These signatures detect exploits against a specific known vulnerability. They are fundamentally different from traditional attack-based signatures in that they can identify most variants of attacks. These are high fidelity signatures that Microsoft has great confidence in and are often enabled to block by default.
  • Exploit-based – These signatures are more like traditional attack-based signatures, designed to detect a very specific exploit of a known vulnerability. These are also high fidelity signatures and are typically also enabled to block by default.
  • Policy-based – These are medium fidelity signatures designed primarily for auditing purposes. They are typically not enabled by default. If the administrator chooses to enable them, their default response policy is set to detect only. Policy-based signatures are created when either vulnerability or exploit-based signature cannot be written.
  • Other – There are several signatures created specifically for testing purposes. These signatures are enabled and set to block by default. They can be used to ensure that the TMG firewall and NIS are properly inspecting network communication and responding correctly.

Signature Updates

As a signature-based technology, NIS is most effective only when the latest signature updates are installed and enabled. NIS signatures can be downloaded from Windows Update or a local WSUS. To make sure that NIS is properly updated, highlight the Update Center node in the navigation tree. The main window will indicate the update status for the protection mechanism, and will also include details about when the last update occurred, the current signature set version number, and the license status.

Figure 13

If management console indicates that NIS signatures are out of date, you can check for and install new definitions using the corresponding links in the Tasks pane.

Figure 14


Intrusion detection and prevention systems (IDS/IPS) are an essential component of any network security architecture. Forefront Threat Management Gateway’s Network Inspection System (NIS) is a unique implementation of network IDS/IPS. Designed specifically to detect and prevent vulnerabilities in Microsoft operating systems and applications from being exploited remotely, NIS provides a valuable layer of protection for networks with Microsoft assets. With this limited scope, it is not designed to replace an existing enterprise network IDS/IPS, but rather to complement it by providing focused, dynamic threat detection and response for attacks on known Microsoft vulnerabilities. With signature updates created by the Microsoft Malware Protection Center (MMPC), NIS is very accurate and efficient, producing few false positives. NIS is included in the cost of the TMG license; there are no additional licenses required to take advantage of this functionality. Enabling NIS on your Forefront TMG 2010 firewall will significantly improve your organization’s overall security posture.

If you would like to read the first part in this article series please go to rusion Detection and Prevention in Forefront TMG (Part 1) – Behavioral Detection.

About The Author

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top