ISA Firewall Stateful Application Layer Inspection Filters (Part 1)

If you would like to be notified when Thomas Shinder releases the next part of this article series please sign up to the ISAServer.org Real time article update newsletter.

The ISA firewall is able to perform both stateful filtering and stateful application layer inspection. Its stateful filtering feature set makes it a network layer stateful firewall in the same class as any hardware firewall that performs stateful filtering at the network and transport layers. Stateful filtering is often referred to as stateful packet inspection, which is a bit of a misnomer because packets are Layer 3 entities, and to assess connection state, Layer 4 information must be assessed. 

However, in contrast to traditional packet filter-based stateful hardware firewalls, the ISA firewall is able to perform stateful application layer inspection, which enables it to fully inspect the communication streams passed by it from one network to another. In contrast to stateful filtering where only the network and transport layer information is filtered, true stateful inspection requires that the firewall be able to analyze and make decisions on all layers of the communication, including the most important layer, the application layer.

The Web filters perform stateful application layer inspection on communications handled by the ISA firewall’s Web Proxy components. The Web Proxy handles connections for HTTP, HTTPS (SSL), and HTTP tunneled FTP connections. The Web filters take apart the HTTP communications and expose them to the ISA firewall’s application layer inspection mechanisms, examples of which include the HTTP Security filter and the OWA forms-based authentication filter.

The Application filters are responsible for performing stateful application layer inspection on non-HTTP protocols, such as SMTP, POP3, and DNS. These application layer filters also take apart the communication and expose them to deep stateful inspection at the ISA firewall.

Web and Application filters can perform two duties:

  • Protocol access
  • Protocol security

Protocol access allows access to protocols that require secondary connections. Complex protocols may require more than one connection, either inbound or outbound through the ISA firewall. SecureNAT clients require these filters to use complex protocols because the SecureNAT client does not have the power of the Firewall client. In contrast to the Firewall client that can work together with the ISA firewall to negotiate complex protocols, the SecureNAT client is a simple NAT client of the ISA firewall and requires the aid of application filters to connect using these complex protocols (such as FTP or MMS).

Protocol security protects the connections moving through the ISA firewall. Protocol security filters such as the SMTP and DNS filters inspect the communications that apply to those filters and block connections that are deemed outside of secure parameters. Some of these filters block connections that may represent buffer overflows (such as the DNS and SMTP filters), and some of them perform much deeper inspection and block connections or content based on policy (such as the SMTP Message Screener).

Application Filters

The ISA firewall includes a number of Application filters. In this section, we discuss:

  • SMTP filter
  • DNS filter
  • POP Intrusion Detection filter
  • SOCKS V4 filter
  • FTP Access filter
  • H.323 filter
  • MMS filter
  • PNM filter
  • PPTP filter
  • RPC filter
  • RTSP filter

The SMTP Filter

The ISA firewall’s SMTP filter configuration interface can be accessed by opening the Microsoft Internet Security and Acceleration Server 2006 management console, expand the server name, and then expand the Configuration node. Click the Add-ins node. In the Details Pane, double-click the SMTP Filter. Click the SMTP Commands tab. (See Figure 7.1.)

The settings on the SMTP Commands tab are mediated by the SMTP filter component. The SMTP Message Screener does not evaluate SMTP commands and does not protect against buffer overflow conditions. The commands in the list are limited to a predefined length. If an incoming SMTP connection sends a command that exceeds the length allowed, the connection is dropped. In addition, if a command that is sent over the SMTP channel is not on this list, it is dropped.


Figure 1

The DNS Filter

The ISA firewall’s DNS filter protects the DNS server published by the ISA firewall using Server Publishing Rules. You can access the configuration interface for the DNS filter’s attack prevention configuration page in the Intrusion Detection dialog box. Expand the server name and then expand the Configuration node. Click the General node.

In the Details Pane, click the Enable Intrusion Detection and DNS Attack Detection link. In the Intrusion Detection dialog box, click the DNS Attacks tab. On the DNS Attacks tab, put a checkmark in the Enable detection and filtering of DNS attacks checkbox.


Figure 2

Once detection is enabled, you can then enable prevention. You can protect yourself from three attacks:

  • DNS host name overflow
  • DNS length overflow
  • DNS zone transfer

The DNS host name overflow and DNS length overflow attacks are DNS denial-of-service (DoS) type attacks. The DNS DoS attack exploits the difference in size between a DNS query and a DNS response, in which all of the network’s bandwidth is consumed by bogus DNS queries. The attacker uses the DNS servers as “amplifiers” to multiply the DNS traffic.

The attacker begins by sending small DNS queries to each DNS server that contain the spoofed IP address of the intended victim. The responses returned to the small queries are much larger, so if a large number of responses are returned at the same time, the link will become congested and denial of service will take place.

One solution to this problem is for administrators to configure DNS servers to respond with a “refused” response, which is much smaller than a name resolution response, when they receive DNS queries from suspicious or unexpected sources.

You can find detailed information for configuring DNS servers to prevent this problem in the U.S. Department of Energy’s Computer Incident Advisory Capability information bulletin J-063, available at www.ciac.org/ciac/bulletins/j-063.shtml.

The POP Intrusion Detection Filter

The POP Intrusion Detection filter protects POP3 servers you publish via ISA firewall Server Publishing Rules from POP services buffer overflow attacks. There is no configuration interface for the POP Intrusion Detection filter.

The SOCKS V4 Filter

The SOCKS v4 filter is used to accept SOCKS version 4 connection requests from applications written to the SOCKS version 4 specification. Windows operating systems should never need to use the SOCKS filter because you can install the Firewall client on these machines to transparently authenticate to the ISA firewall and support complex protocol negotiation.

For hosts that cannot be configured as Firewall clients, such as Linux and Mac hosts, you can use the SOCKS v4 filter to support them. The SOCKS v4 filter is disabled by default. To enable the filter, open the Microsoft Internet Security and Acceleration Server 2006 management console, expand the server name, and then expand the Configuration node. Click the Add-ins node. In the Details Pane, right-click the SOCKS V4 filter and click Enable.

You will need to configure the SOCKS V4 filter to listen on the specific network(s) for which you want it to accept connections. Double-click the SOCKS V4 filter. In the SOCKS V4 Filter Properties dialog box, click the Networks tab. On the Networks tab, you can configure the Port on which the SOCKS filter listens for SOCKS client connections. Next, put a checkmark in the checkbox next to the network for which you want the SOCKS filter to accept connections. Click Apply and then click OK.


Figure 3

The SOCKS v4 filter supports SOCKS v4.3 client applications. The SOCKS filter is a generic sockets filter that supports all client applications that are designed to support the SOCKS v4.3 specification. The SOCKS filter performs duties similar to that performed by the Firewall client. However, there are some significant differences between how SOCKS and the Firewall client work:

  • The Firewall client is a generic Winsock Proxy client application. All applications designed to the Windows Sockets specification will automatically use the Firewall client.
  • The SOCKS filter supports applications written to the SOCKS v4.3 specification.
  • When the Firewall client is installed on the client machine, all Winsock applications automatically use the Firewall client, and user credentials are automatically sent to the ISA firewall. In addition, the Firewall client will work with the ISA firewall service to manage complex protocols that require secondary connections (such as FTP, MMS, and many others).
  • The SOCKS client must be configured on a per-application basis. Each application must be explicitly configured to use the ISA firewall as its SOCKS server. When the application is configured to use the ISA firewall as its SOCKS server, the SOCKS filter will manage complex protocols for the SOCKS client application.
  • The SOCKS 4.3a filter included with the ISA firewall does not support authentication. SOCKS 5 introduced the capability to authenticate the client application that attempts to access content through the SOCKS proxy.

I always recommend that you use the Firewall client because of the impressive advantages it provides by allowing you the ability to authenticate all Winsock connections made through the ISA firewall. However, SOCKS is a good “second best” when you cannot install the Firewall client.

The FTP Access Filter

The FTP Access filter is used to mediate FTP connections between Protected Network clients and FTP servers on the Internet, and from external hosts and published FTP servers. The FTP Access filter supports both PASV and PORT (passive and standard) mode FTP connections.

The FTP Access filter is required for SecureNAT clients because FTP uses secondary connections for PORT-mode FTP connections. FTP is a complex protocol that requires outbound connections from the FTP PORT-mode client and new secondary inbound connections from the FTP server. While the Firewall client does not require application filter support for secondary connections, SecureNAT clients do require application layer filter support, which is why the ISA firewall dev team included the FTP Access application filter.

If you plan to support PORT-mode FTP client connections, make sure that IP Routing is enabled on the ISA firewall (the default setting). When IP Routing is enabled, the secondary connections are handled in kernel mode rather than user mode. This kernel-mode handling of the secondary connections (which are data transfers from the FTP server to the FTP client) will significantly increase performance.

Stefaan Pouseele, an ISA Server MVP, has written an excellent article on the FTP protocol and how FTP challenges firewall security. Check out his article, How the FTP Protocol Challenges Firewall Security.

There is no configuration interface for the FTP Access filter. However, if there is an Access Rule that applies to FTP connection, the right click menu on the Access Rule will allow you to Configure FTP. The Configure FTP option allows you to control whether or not FTP uploads are allowed.

The H.323 Filter

The H.323 filter is used to support H.323 connections through the ISA firewall. To configure the H.323 filter, open the Microsoft Internet Security and Acceleration Server 2006 management console and expand the server name. Next, expand the Configuration node and click the Add-ins node. Double-click the H.323 Filter entry in the Details Pane.

In the H.323 Filter Properties dialog box, click the Call Control tab.. You have the following options:

  • Use this Gatekeeper
  • Use DNS gateway lookup and LRQs for alias resolution
  • Allow audio
  • Allow video
  • Allow T120 and application sharing

Click the Networks tab. On the Networks tab, put a checkmark in the checkbox to the left of the networks on which you want the H.323 filter to accept connections requests.

The MMS Filter

The MMS filter supports Microsoft Media Services connections through the ISA firewall for Access Rules and Server Publishing Rules. The MMS filter is an access filter that allows SecureNAT client access to the complex protocols and secondary connections required to connect to Microsoft Media Services hosted content. Firewall clients do not require the help of the MMS filter to connect to MMS servers. There is no configuration interface for the MMS filter.

The PNM Filter

The PNM filter supports connections for the Progressive Networks Media Protocol from Real Networks. The PNM filter is an access filter allowing SecureNAT client access to the complex protocols and secondary connection required to connect to Progressive Networks Media servers. There is no configuration interface for the PNM filter.

The PPTP Filter

The PPTP filter supports PPTP connections through the ISA firewall for outbound connections made through Access Rules, and inbound connections made through Server Publishing Rules. The ISA firewall’s PPTP filter differs from the ISA Server 2000 PPTP filter in that it supports both inbound and outbound PPTP connections. The ISA Server 2000 PPTP filter only supports outbound PPTP connections.

The PPTP filter is required by both SecureNAT and Firewall clients. In fact, a machine located on an ISA firewall protected network must be configured as a SecureNAT client to use the PPTP filter to connect to PPTP VPN servers through the ISA firewall. The reason for this is that the Firewall client does not mediate non-TCP/UDP protocols. The PPTP VPN protocol requires the use of the Generic Routing Encapsulation (GRE) protocol (IP Protocol 47) and TCP protocol 1723. The TCP session is used by PPTP for tunnel management.

When the outbound access to the PPTP protocol is enabled, the PPTP filter automatically intercepts the GRE and TCP connections made by the PPTP VPN client. You do not need to create an Access Rule allowing outbound access to TCP 1723 for VPN clients.

The RPC Filter

The RPC filter is used to mediate RPC connections to servers requiring Remote Procedure Calls (RPCs) for both outbound connections using Access Rules and inbound connections using Server Publishing Rules. This includes secure Exchange RPC publishing.

There is no configuration interface for the RPC filter.

The RTSP Filter

The RTSP filter supports Microsoft Real Time Streaming Protocol connections through the ISA firewall for Access Rules and Server Publishing Rules. The RTSP filter is an access filter that allows SecureNAT client access to the complex protocols and secondary connections required to connect to Microsoft Real Time Streaming Protocol hosted content (such as that on Windows Server 2003 Microsoft Media Servers). Firewall clients do not require the help of the MMS filter to connect to MMS servers.

There is no configuration interface for the RTSP filter.

Summary

In this article we went over the application layer inspection filters that come with the ISA firewall right out of the box. We saw that the application layer inspection filters can perform two primary duties: they can enable protocol access and the can secure protocol access. The ISA firewall includes application layer inspection filters that can perform both of these activities. Next week we’ll look at another application layer inspection filter, the Web Proxy filter. See you then! –Tom.

If you would like to be notified when Thomas Shinder releases the next part of this article series please sign up to the ISAServer.org Real time article update newsletter.

About The Author

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top