If you would like to read the next part in this article series please go to TMG Firewall Web Filtering (Part 1).
Introduction
One of the most important jobs of the TMG firewall is to make sure that users can get to the sites they need to get to and that they won’t be able to get to sites to which they don’t need access in order to do their work. In general practice, you will have worked with your security team to determine which sites fit into each of these categories. The TMG firewall enables you to do this easily, by categorizing sites for you. It’s the TMG firewall’s URL filtering feature that makes this happen. The TMG firewall works together with the Microsoft Reputation Services (MRS) to give you a flexible and real-time method for controlling site access based on policy. In this article, we’ll look at how it all works and how you can use it to help keep your network secure.
Capabilities of the TMG Firewall’s URL Filtering Feature
URL filtering enables the TMG firewall to control access to Web sites based on a URL category. Each site is assigned to a category. This is different from policies that are based on domain names or URL sets, in that URL filtering works in a dynamic fashion. Web sites that have been categorized by the MRS are posted to the Microsoft Update (MU) site and are downloaded from the Microsoft Update site by the TMG firewall. However, the MRS folks do more than just come up with their own list of categorized sites – they aggregate reputation data from multiple vendors and use telemetry obtain from various customers to improve accuracy.
Here’s how the URL filtering feature works:
- A user requests for a web site is sent through the TMG firewall.
- The TMG firewall intercepts the request and assesses whether URL categorization is required. The TMG firewall needs to determine the category to which the URL belongs in order to decide whether to allow or deny the traffic, based on the rules that are current active in the firewall policy.
- If URL categorization is required, name resolution will beperformed for the requested URL and the URL will beassigned to a category.
- When URL categorization is not required, the TMG firewall will mark the request as not categorized, but it logs the category so that it can be used if the firewall needs to send a denial to the user.
- The rule that allows the request is matched and the TMG firewall “decides” whether the rule allows or denies the category that is assigned to the requested URL.
- When categorization is assigned by a rule, the request that is marked as not categorized is blocked and the user is informed of the denied request. If the rule confirms that the category matches the settings in the rule, then the TMG firewall will allow or deny the connection based on whether the rule allows that category.
Drill Down on URL Filtering Categorization
Okay, now you know the process. But sometimes it’s easier to understand if we walk through that process with a hypothetical request, so let’s look at an example to get a better idea of how URL filtering works in practice:
A client on yourdefault Internal Network sends a request for a Web site with the URL http://www.consoso.com/Path1/Path2 to the TMG firewall. When the TMG firewall receives this request from a client that’s situated behind the firewall, the TMG firewall will assign a category to this URL, which will determine whether the connection is allowed or denied based on the firewall policy. Simple enough? But let’s delve a little deeper.
The TMG firewall parses the URL into a number of parts, which are called variantsby the MRS team at Microsoft. The variants for http://www.consoso.com/Path1/Path2 would be as follows:
- .com
- contoso.com
- www.contoso.com
- www.contoso.com/Path1
- www.contoso.com/Path1/Path2
The TMG firewall then sends these variants to the Microsoft Reputation Services site so they can be assigned to a category. In this example, MRS would provide the TMG firewall with the following information:
- .com: ”unknown”
- contoso.com “politics”
- www.contoso.com ”unknown”
- www.contoso.com/Path1 “gambling” (Not inherited)
- www.contoso.com/PathX/Path2 “computer crime”
Note that “not inherited” means that the category assigned to http://www.contoso.com/Path1is not inherited by extended paths such ashttp://www.contoso.com/Path1/Path2.
There might be times when the MRS will return a response to a variant as “inherited” instead of “not inherited”. In our current example, if the response to http://www.contoso.com/Path1 had been assigned as “inherited”, it would mean that http://www.contoso.com/Path1/Path2 would be categorized as “gambling” because it inherits the categorization from the path above it, which in this case is http://www.contoso.com/Path1.
So for our current example, based on the information provided by the MRS to the TMG firewall, the TMG firewall now knows that the following two categories apply to this URL:
- Politics
- Computer Crime
Note that Gambling cannot be assigned here because it is not inherited to http://www.contoso.com/Path1/Path2.
Now for an interesting question: How does the TMG firewall decide which of these two identified categories to assign to the request? The TMG firewall sorts the possible categories based on their influence by evaluating factors such as which category sits in the highest level of the hierarchies obtained from the returned categories.
For example, the Computer Crime category might be the most influential category, which means in this case that this is a category for which you want to block access. The URL category information sent from the MRS can then be used by the TMG firewall in several areas of analysis and decision making:
- Firewall rules, to enable allow or deny decisions
- Web Proxy Log, to enable reporting on what categories of sites have been visited by your users
- Enterprise Malware Protection (EMP) exclusion list, to determine which sites should be blocked at an enterprise level
- HTTPS exclusion list, to decide which categories of sites should be excluded from outbound SSL inspection
URL Filtering Architecture
Here’s an important thing to remember: URL filtering is not always applied, and it’s not turned on by default. The TMG firewall will filter URLs only if the following conditions are met:
- The URL filtering feature is turned on (URL filtering is disabled by default) and
- Firewall policy rules have categories assigned to them to enable allow and deny decisions based on categorization of the requested site(s)
The URL Filtering capability works as part of the Microsoft Firewall Service (wspsrv.exe). Another component that is critical to the functioning of the URL filtering feature is the MRS categorizer. The MRS categorizer obtains information from the MRS Service using the Windows Web Services API (WWSAPI) via calls to WinHTTP. The categorizer is responsible for interacting with other core TMG components, such as the firewall rule engine, the malware protection engine, the HTTPS exception feature, category query, and deny page. The changes to URL categorization made through the user interface are controlled by the categorizer.
Before you configure URL Filtering on the TMG Firewall
Now that you know how it works, you can learn how to configure it so you can start using it. Before you can get started, though, you need to work with your security and legal teams to decide how you want to implement URL filtering. In general, this isn’t a decision that you should have to make as the TMG firewall admin. There are many factors to consider here, and you should make sure the team has a common understanding of the TMG firewall’s capabilities. Then you can translate the corporate access control and compliance requirements to the configuration of the TMG firewall’s URL filtering feature.
But before we go there, let’s talk a little about how URL Categories, Bing Safe Search, and Microsoft Reputation Services work together to make the feature even better. Then in part two of this series, we’ll go into the details of how to configure the URL filtering feature on your TMG firewall.
During the beta phase of the development of the TMG firewall, Microsoft’s new Bing search engine (or “decision engine,” as the company prefers to call it) was developed. Bing provides a feature called safe search feature, which enables you to block requests for adult sites by configuring the Web proxy engine to add the query string adlt=strict to the request. This allows the Bing search service to filter content that is considered to be “adult” in nature.
The problem for the TMG firewall during that period of beta development was that adding information to the client’s original request could not be implemented by the TMG firewall at that stage of development. This problem was solved by the Bing team by creating a new domain for use by Web proxies that can’t use the Microsoft Reputation Services (MRS) or those that can’t add the required query string. The new domain is explicit.bing.net. This domain is categorized as “pornography” as shown in the figure below.
Figure 1
Summary
The TMG firewall includes a feature that wasn’t available in the prior ISA firewall – URL filtering. The URL filtering feature makes it possible for you to extend your corporate access control policy over web site connections so that users will be able to get to the sites they need to do their work, and will be blocked from accessing sites that they don’t need to get, or sites that can put your company at risk. Your job is to enable the corporate policy on the TMG firewall.
The TMG firewall makes liberal use of the information provided by the Microsoft Reputation Service, which uses information gained from its own investigations, as well as information obtained from other participants in the web security industry. The TMG then uses the information about the category to which a web site belongs in order to make allow or deny decisions. The allow or deny decision depends on the rules that are configured to support web filtering in the TMG firewall’s list of firewall rules.
The URL feature is disabled by default, and you will need to explicitly enable it and then configure rules so that URL filtering is performed. In the second part of this article, we’ll go over the steps for configuring the URL filtering feature and how to create rules that leverage it. See you then! –Deb.
If you would like to read the next part in this article series please go to TMG Firewall Web Filtering (Part 1).
