Understanding and Implementing ISA 2004 as an Application Firewall with the RPC Stateful Inspection Filter

 

ISA Server 2004 (ISA firewall) includes a number of technologies that provide enhanced security performance for corporate network infrastructures. The unique combination of security and functionality is highlighted by the application filters included with the ISA firewall right out of the box.

It is an important fact to realize that the RPC (Remote Procedure Call) protocol is used by many Microsoft networked applications, but that most of IT personnel, including network and firewall administrators, do not understand how the RPC protocols works. They don’t understand what potential problems are generated by the RPC protocol, and most importantly, they don’t know how to protect infrastructure servers. Typical network and firewall administrators just think that RPC is not secure and don’t even consider the fact that RPC access can be made secure.

This article discusses the RPC protocol and how the ISA firewall can provide a high level of protection for Microsoft servers requiring RPC access. Throughout this article, we will use a basic example to demonstrate a methodology for implementing RPC stateful filtering. This will enable you to create your own more robust and complex architecture later.

Get the New Book!

 

Introduction – Firewalls, Filtering and Inspection – Layer 3 vs. Application firewalls

Most organizations depend solely on layer 3 packet filtering to protect their company’s assets. Layer 3 filtering (sometimes referred to as “stateful inspection”) just opens and closes TCP or UDP ports. Due to the constant evolution of network threats and protocols, working at IP and transport (layers 3 and 4 in the OSI stack) layers is no longer sufficient to have a high level of security and dependency on stateful packet inspection cannot be considered adequate network protection. We now need to put firewall at a higher level of the OSI model, at the application layer.

If you want to filter RPC traffic and you use a stateful packet inspection-only firewall, you need to open TCP port 135 (the RPC endpoint mapper, used by RPC clients to locate the TCP port of the remote RPC service) and then, because we never know what is the port used by the RPC application will be, you also need to open the “high ports” from 1024 to 65535.

We can symbolize this configuration with the following diagram:

As you can see on this diagram, a firewall rule opens a huge hole to the RPC server on the protected network. Traditional firewall and network security administrators and engineers would consider this configuration as extremely unsecure and unacceptable. On the other hand, most of the Microsoft servers today on the production LAN totally wide open to other hosts on the production network. Traditional network and firewall administrators do not no firewall intra-LAN communications.

This is problematic situation. In contrast to the thinking of the latter 20th century where we considered the Internet the untrusted network and the corporate LAN a trusted network, the networks of the 21st century cannot be partitioned in this way. The fact is that today no network can be considered a trusted network. Worms, viruses, and other malicious mobile code, in addition to intentional and unintentional attackers on the corporate network prove that your corporate LAN should be considered as untrusted as the Internet.

Get the New Book!

Protecting the Network in a Smarter Way – Stateful Application Layer Inspection

Because we can’t implicitly trust any network, we need to protect our Microsoft servers and services in a more intelligent fashion. What we can do with the ISA firewall is protect Microsoft servers on the corporate LAN in a smarter way. To accomplish this goal we will perform application filtering or stateful application layer inspection.

Filtering or inspection at the application layer means inspecting the payload of the frame and confirming that the data transported is authorized, valid and is not a threat of the company.

When the ISA firewall performs application layer inspection, the data is more than just a “bunch” of characters moving over the wire, which is the case with a conventional stateful packet inspection-only firewall. The ISA firewall actually inspects the datastream as a network protocol and validates the protocol headers and data.

For example, an SMTP application filter is able to drop all the frames containing the RSET command, or all frames with more than X characters. SMTP filtering actually inspects, qualifies and validates the application layer protocol information contained within the communication.

When the ISA firewall performs stateful application layer inspection on the RPC protocol, it analyzes a part of the frame called the UUID (Universally Unique IDentifier), which is the identifier of the RPC service running on a server. Microsoft Exchange is a good example, as the Exchange Server services include multiple RPC service listeners.

Here is an example of the dialog between a RPC client application connecting an RPC server:

 

  1. In the above graphic, the yellow machine on the left side needs to establish an RPC connection with the RPC servers with yellow application. The yellow RPC client machine first contacts the yellow RPC server via TCP port 135, and will say “hey, what is the TCP port for UUID E1AF8308-5D1F-11C9-91A4-08002B14A0FA“, which is the unique identifier of the RPC application I need to communicate with. The ISA firewall’s stateful application inspection RPC filter authorizes the flow RPC data flow, because there is a firewall rule authorizing RPC communications using this UUID.
  2. The yellow RPC server replies with the TCP port assigned (Let’s say 2345) to the requested UUID (the yellow application).
  3. The ISA firewall’s stateful application layer inspection RPC filter dynamically authorizes the flow of data coming from yellow application on the workstation to the yellow server on this RPC application port (2345) because the filter authorizes the RPC traffic.
  4. The workstation initiates a connection to the remote server using TCP 2345, and the ISA firewall allows this connection (symbolized by a dotted blue tunnel).

So by inspecting the actual protocol instead of just “opening ports”, the ISA firewall detected that these RPC communications are authorized and valid because it understands the UUIDs used RPC protocol communications moving through the ISA firewall.

The same process is used by blue machine in the above graphic in step 3, where it connects to the blue application. The only difference here is that the target UUID is different, and so the RPC service is not listening at port 2345, but at another port, for example 5678.

Of course, if a machine tries to connect the RPC server on an RPC UUID this is not allowed by an ISA firewall rule, the connection will be impossible and the ISA firewall drops the frames.

 

With this “vision” of security, we can drop all the invalid and non-authorized RPC connections and provide an exception level of protection the internal servers from both Internet and other external exploits and also attacks that source from hosts on the corporate LAN.

Get the New Book!

Introducing the ISA Firewall’s RPC Application Layer Inspection Filter

Now that you’ve seen an example of how the RPC protocol works, let’s go a bit deeper in the ISA firewall’s architecture and networking components.

You can see on the next diagram an extract of a network FRAME from a Network Monitor packet trace. All RPC connections start with a frame containing the UUID of the requested RPC service. The ISA firewall inspects this kind of frame and validates and authorizes them.

Also in this diagram is a screenshot of all UUIDs authorized by the ISA firewall when publishing an Exchange Server using a Secure Exchange RPC Server Publishing Rule. It the UUID in the RPC connection require frame is not in the list allowed by the ISA firewall’s rule enabling access to the RPC service through the firewall, then it is dropped.

Now that we know the basics of the ISA firewall’s RPC filtering (there’s more than just UUID inspection, but this is enough information to get you started), we will use a basic scenario and see the A to Z process to follow when securely publishing RPC applications.

This is an important discussion because even if Microsoft supplies a list of UUIDs for some RPC services, such as those employed by Microsoft Exchange Server, we need the skills to create our own RPC protocol definitions when necessary. For example, you may need to create a custom RPC Protocol Definition to enable 2 domain controllers to replicate across an ISA firewall if one of them is located on a site that you not fully trust, and as we now know, you can’t trust any network.

The scenario we will use for this demonstration is to authorize using a custom RPC Protocol Definition and the RPC filter a LAN workstation to run the ISA MMC through an ISA firewall. The MMC sends RPC calls to the server for ISA firewall administration.

Get the New Book!

 

The Challenge: Discovering the RPC UUIDs used by the ISA Firewall MMC

We saw that allowing uninspected access to TCP port 135 (the RPC endpoint mapper) and the high ports is not a good idea from a network security perspective. So we will see now how to do it in a smarter way, using the ISA firewall’s RPC stateful application inspection filter.

But in order to create this rule, we need to identify the UUID(s) used by the ISA firewall MMC.

To do so, I have created a test lab. Here is the lab I used for this article:

  • A workstation with ISA MMC installed
  • An ISA 2004 firewall

Before creating our smart RPC firewall rule, we first need to know what kind of RPC communications we need to allow. Unfortunately there is no official database with all UUIDs used by all RPC services, so the best way to discover it is to use a basic example.

To analyze the data flow, I first created a firewall rule to authorize all the protocols between the administration machine and the ISA firewall’s Local Host Network.

Then run Microsoft Network Monitor to capture the traffic when connecting to the ISA firewall with the MMC. Here is a screenshot of what I have seen using Microsoft Network Monitor:

On this screenshot, you can see that the administration machine initiates an RPC communication by making a request to the RPC endpoint mapper. Here is a zoom of the screenshot where you can see the ISA UUID:

We have now the UUID used by the ISA MMC (In fact, I checked all the other frames in case the MMC uses some other UUIDs, but there is only one).

 

Creating a New RPC Protocol Definition

We know the UUID to publish (in you scenario you may have more than one !).We need now to create the firewall rule to implement it.

First of all, we need to create our own ‘RPC protocol’ by giving all the UUIDs to publish.

To do so, create a new firewall rule, but rather than selecting a protocol in the list, click New, then RPC Protocol:

 

The RPC protocol wizard starts. Give a name to your ‘RPC protocol (I advise you to use a naming convention where you can clearly identify this as a new RPC protocol).

 

On the next screen you have 2 options:

Option 1 – select interfaces: this option will list (click browse button) of RPC application running on the ISA machine itself, so you will be able to check the ones you want to publish. This option is nice, but (I don’t know why) does not list all the RPC applications loaded for our test scenario. Also, it is rarely useful because most of the time the RPC application to publish is not on the ISA firewall device itself:

 

Option 2 – add interfaces manually : This option will provide an interface where we can manually type the UUID(s) we want to publish (on that screen interface means RPC UUID) :

 

Select ‘Add interfaces manually’ and click next

Click ‘Add button’

 

When you click the Add button, you get another screen where you have to type the UUID to add in your ‘own RPC protocol”:

We have now to type the UUID detected by Netmon in the ‘Interface UUID’ text box. Give a name for this interface if you know it.

Most of the time, leave the radio button on Publish on a Dynamically Assigned Port. Select the other option only if you forced the RPC service to be executed on a ‘Forced’ port (only a few RPC services have registry keys to ‘force’ the port this service will use).

 

 TIP To make sure that you do not mistype the UUID (which is quite long). You can print all the frames captured by Network Monitor in a text file by using the ‘Text Only’ printer supplied by default with Windows. Then you can copy and paste this UUID:

 

For this article we use a basic scenario which – after a Network Monitor investigation – requests for 1 UUID need to be opened. If you need to publish a service that requests more than one, you need to do the same process for each UUID.

At the end of the process, you get that screen where you can see the configuration:

At this level we have finished the creation of our own RPC protocol. We can now use it in the network firewall.

 

Get the New Book!

 

Create the Access Rule using the Custom RPC Protocol Definition

Now that we have created our custom RPC Protocol Definition, We have previously created our RPC protocol. We need now to use it in an ISA firewall Access Rule.

Here are the details of the ISA firewall Access Rule we need to set to authorize the RPC traffic.

The rule listed as Rule number 5 is the one w will need to create.

Now let’s focus on each tab of the rule:

Name of the rule It is an ALLOW rule

 

We select the protocol we have just created

We select the machine authorized by the rule

 

 

Destination is the ISA local Host

 

Now the rule is created. Click the Apply button and you are all set.

Conclusion

Implementing stateful application layer inspection is mandatory to protect your infrastructure and cope with current threats and attacks aimed against your Microsoft infrastructure servers and services. Installing both network edge and internal perimeter firewalls like the ISA Server 2004 application layer inspection firewall, will profoundly improve the level of security for core network services and applications and significantly reduce the chance of a successful attack being launch by external and external agents. All surveys show that internal attacks generate a generous slice of the overall damage done to corporate networks, and the costs are significant..

The ISA firewall is sophisticated stateful packet inspection and application layer filtering/inspection firewall and the best firewall on the marketing today to filter the RPC protocol the “smart way”.

I hope that this document will help you creating your own RPC filters now.

Leave a Comment

Your email address will not be published.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top