The Mystery of the HTTP Redirector and Site&Content Rules
By Stefaan Pouseele
Last Update: 17/11/2002
You thought you fully understood how ISA server applies Site&Content rules and in which order, and now you discovered that Firewall and SecureNAT clients still have access to blocked sites, despite the fact there is a proper Site&Content rule in place. How is that possible? In this article we will investigate under which conditions this can happen and what you can do about it.
To better understand this issue, I assume you have already gone through the articles:
- Configuring the HTTP Redirector by Curt Simmons
- Preventing SecureNAT and Firewall Clients from Bypassing the Web Proxy Service and How to Give Yourself a Headache with the HTTP Redirector Filter and Anonymous Access by Thomas Shinder
Also, you should keep in mind that ISA processes rules in the following order:
- Deny rules applying to any request (anonymous).
- Allow rules applying to any request (anonymous).
- Deny rules applying to client address sets or users and groups (authenticated).
- Allow rules applying to client address sets or users and groups (authenticated).
Of course, a good knowledge about the different ISA clients is a requirement. If you have any questions about them, check out first the excellent articles in Jim Harrison's section.
2. HTTP Redirector
To make the issue more easy to understand and test, we will set up a simple configuration with the following rules:
- A Protocol rule allowing access to all IP traffic for any request.
- A Site&Content rule allowing access to all destinations and all content for a client address set. The client address set contains the whole Local Address Table.
- A Site&Content rule denying access to a destination set and all content for any request. The destination set contains the entries *.playboy.com, *.penthouse.com, *.netscape.com and www.pouseele.be.
Because we have some nasty programs that don't support a Web Proxy server, we have configured the HTTP Redirector to Send to requested Web server, a very common configuration. Let us now investigate what happens for the different ISA client types.
If the client is configured as Web Proxy client, the HTTP request is sent to the Web Proxy service on ISA. As expected, the client will have access to all destinations except those listed in the destination set used by the deny Site&Content rule. Now, what happens if the web browser is not configured as a Web Proxy client? In that case the web browser is either a Firewall or a SecureNAT client and the HTTP request is sent to the Firewall service on ISA. There, the HTTP request is intercepted by the HTTP Redirector and sent to the requested destination, bypassing the Web Proxy service on ISA.
Like most of us, you probably assume now that the Firewall service will take over and apply the configured Site&Content rules to this traffic, just like for any other traffic. However, this is not true! The Firewall and SecureNAT clients will have HTTP access to any site and content, whatever Site&Content rules are defined. In the Firewall log you could clearly see that for those HTTP requests the field s-operation = Connect and that the field sc-status = 0, what means success. The field Rule#1 will be set to the protocol rule allowing the access and the field Rule#2 will be empty.
3. Site&Content Rules
When the HTTP Redirector is configured to Send to requested Web server, the request not only bypasses the Web Proxy service but also all Site&Content rules and is forwarded directly to the Web server. In my opinion, this makes the HTTP Redirector useless in such a configuration. So, let us just disable the HTTP Redirector and see what happens. Note that we still use the same Protocol and Site&Content rules and that the client is not set up as a web proxy client.
We have tested all four denied destinations and this is the result:
- Destination www.playboy.com and www.penthouse.com are always blocked.
- Destination www.netscape.com is sometimes blocked.
- Destination www.pouseele.be is never blocked.
You probably ask yourself why the Firewall service on ISA don't block all destinations listed in the destination set used by the deny Site&Content rule. This seems not logical. However, there is a good explanation for this behaviour.
You should keep in mind that the Firewall and SecureNAT client always send the requests by IP address to the Firewall service on ISA. The forward DNS lookup of the FQDN is done by the client (SecureNAT and possible Firewall client) or by the ISA server on behalf of the client (Firewall client). When the request hits the Firewall service on ISA, two things can happen. If the destination set is populated with IP addresses, the Firewall service can compare the requested destination (an IP address) with the IP addresses specified in the destination set and immediately decide what to do with that request. However, if the destination set is populated with FQDN's, the Firewall service must first do a reverse DNS lookup of the requested IP address. Once that done, it can compare the requested destination (translated to an FQDN) with the FQDN's specified in the destination set and decide what to do with that request.
The following table lists the result of a forward and reverse DNS lookup for all four denied destinations:
If you now compare the FQDN's as a result of the reverse DNS lookups with the content of the denied destination set, you can clearly see that the Firewall service will find a match for www.playboy.com and www.penthouse.com. However, for the destination www.pouseele.be the firewall service will never find a match. For the destination www.netscape.com it depends which IP address will be used. This explains exactly the behaviour we have seen.
You probably wonder why the Web Proxy service on ISA behaves differently? When a request is sent to the Web Proxy service, it has complete knowledge of the URL you want to access. So, the Web Proxy service can very easily match the URL with the destination set specified in the Site&Content rules. Normally, you would enter an FQDN as URL address. In this case, no DNS lookup is needed. However, if you enter an IP address as URL, then the behaviour of the Web Proxy service is determined by the registry key SkipNameResolutionForAccessAndRoutingRules, which can be found at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W3Proxy\Parameters\. When this registry key is not present (default configuration) or has the value 0 (DWORD), then the Web proxy service will do a reverse DNS lookup and the same logic applies as explained for the Firewall service. However, when the registry key has the value 1 (DWORD) then no reverse DNS lookup is done. In the latter case, only Site&Content rules with destination sets populated with IP addresses could possibly give a match.
When the HTTP Redirector is configured to Send to requested Web server, it really means it! The request bypasses all Site&Content rules and is forwarded directly to the Web server. If you don't like that behaviour, you should disable the HTTP Redirector. However, be aware of possible forward and reverse DNS lookup problems when creating your destination sets for Site&Content rules.
I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=3;t=002291 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks! – Stefaan.