TMG back to Basics - Part 3: Protocol Definitions

If you would like to read the other parts in this article series please go to:

This week we continue our TMG Back to Basics series with a simple, but very important basic TMG concept that you might not even be aware of: Protocol Definitions. The reason why you might not be aware of Protocol Definitions is that Microsoft has done a great job of including a number of Protocol Definitions in TMG right out of the box so that you don’t even need to know about them. But there are times when it is useful to be aware of this feature.

Protocol Definitions are a key feature of every firewall rule you create on the TMG firewall. The Protocol Definition defines the protocol that’s used to allow inbound or outbound access to or through the TMG firewall. For example, if you create a rule that allows outbound access to the HTTP and HTTPS protocols, that rule will include the Protocol Definitions for the HTTP and HTTPS protocols. A Protocol Definition is required for all firewall rules, which includes Access Rules and Web Publishing Rules and Server Publishing Rules.

Let’s delve a little deeper into this. To view information about your Protocol Definitions, click the Firewall Policy node in the left pane of the TMG firewall console. Click the Toolbox tab in the Tasks Tab, then click the Protocols entry on the Toolbox tab. There you will see a number of groups of Protocol Definitions, as shown in the figure below.

Figure 1

Click on the Common Protocols group. Here, as shown in Figure 2, you will see a list of Protocol Definitions for the protocols that are most commonly used when configuring firewall rules on the TMG firewall.

Figure 2

Scroll down the list of Protocol Groups. Click on the All Protocols group, as shown in Figure 3. Here you will see a list of all the Protocol Definitions that are included in the TMG firewall right out of the box. There are well over a hundred Protocol Definitions included and that selection should support just about any protocol to which you would want to allow inbound or outbound to or through the TMG firewall in typical scenarios. But there are special circumstances when you’ll need to create your own definitions. Scroll through the list of Protocol Definitions to get an idea of what’s available. Being aware of what’s available out of the box can save you time when you are considering creating your own custom Protocol Definition. After all, why go to the trouble of creating a new Protocol Definition if there’s already one there that will serve your purpose?

Figure 3

Now let’s look at a specific definition. Double click the HTTPS Protocol Definition. This brings up the HTTPS Properties dialog box that’s shown in Figure 4. On the General tab, you’ll see the name of the Protocol Definition and a short description of the protocol. You’ll also see the Associated Standard Protocol section, but it will be grayed out in most cases; we’ll talk about this feature later.

Figure 4

Now click on the Parameters tab. On the Parameters tab, shown in Figure 5, you can see the Primary Connections, the Secondary Connections and the Application Filters sections. We’ll talk about these sections in more detail later, too – but you can see here which protocols, ports and direction is associated with the protocol.

Figure 5

Speaking of directions for Protocol Definitions, check out Figure 6 below. Notice that there is an SMTP Protocol Definition and a SMTP Server Protocol Definition. You might be wondering: What is the difference?

Figure 6

It’s all about direction. If you look at the Parameters of the SMTP Protocol Definition, shown in Figure 7, you’ll see that the Direction is Outbound. When you create Access Rules, you will always use a Protocol Definition that supports outbound protocols.

Figure 7

Now if you look at the SMTP Server Parameters, shown in Figure 8, you’ll see that the direction there is Inbound. When you create Server Publishing Rules, you will always use a Protocol Definition that has the Inbound direction. In addition, notice that there is an Application Filter associated with the SMTP Server Protocol Definition. In this case, the SMTP Filter is associated with the SMTP Server Protocol Definition. This means that some level of application layer inspection will be applied to connections that match firewall rules that use the SMTP Server Protocol Definition.

Figure 8

How to Create New Protocol Definitions

Now that you’ve seen some examples of existing Protocol Definitions, let’s find out how you can create your own custom Protocol Definitions. Why would you ever want to do this? Well, you might need to create a custom definition if you have a new application that requires a protocol that isn’t included with the out of the box Protocol Definitions.

Luckily, it’s pretty simple. You start creating a new Protocol Definition by clicking the Toolbox tab in the Task Pane and then clicking the Protocols section. Then click the New menu and click Protocol, as shown in Figure 9 (the RPC Protocol is an advanced option that we won’t cover in this article).

Figure 9

This will bring up the Welcome to the New Protocol Definition Wizard page that’s shown in Figure 10. Enter a name for the new Protocol Definition in the Protocol Definition name text box. In this example, we’ll name the Protocol Definition Disambigulator Protocol and then click Next.

Figure 10

On the Primary Connection Information page, as seen in Figure 11, you enter the protocol and the port or ports required for the primary connection. The primary connection is the first connection(s) made by the system initiating the connection to another host. Click the New button.

Figure 11

This brings up the New/Edit Protocol Connection dialog box that you can see in Figure 12. Here you must select the following:

Protocol type. You select from a list that includes TCP, UDP, ICMP and IP-Level. If you select IP-Level you will need to enter the IP Protocol Number in the Protocol Number text box. If you select the ICMP option, you will need to enter an ICMP Code and an ICMP Type. For TCP and UDP protocols, you will need to enter a port or port range. For all protocols, you will need to enter a direction.
Direction. Here you enter the direction. For all protocols except UDP, you need to select either inbound or outbound. For UDP, you have more choices, as you’ll see in the next step.
Port Range. If you select TCP or UDP, you need to enter a port range here. If there is only a single port, then enter the same port number in the From and To boxes.
ICMP Properties. If you are creating a new ICMP protocol, then you will need to enter an ICMP type and code in the text boxes.

Figure 12

If you select a UDP protocol, then your choices for selecting a direction are different. As you can see in Figure 13, our choices are Receive, Receive Send, Send and Send Receive. It can be hard to know in advance which direction will work. In general, I will start with Receive or Send (depending on the direction – receive is equivalent to inbound and send is similar to outbound). If that doesn’t work, I’ll use Receive Send or Send Receive. It’s tricky in some cases, and you might need to experiment with the direction for UDP protocols.

Figure 13

In this example, our protocol is a UDP protocol and it’s used for outbound connections – so we will select UDP and Send, as shown in Figure 14. This protocol uses a single port, which is 9966, so we enter that number in the From and To boxes in the Port Range section and then click OK.

Figure 14

The Secondary Connections page, which you can see in Figure 14, is used for what we call “complex protocols”. A complex protocol is one that requires new connections after the initial connection. For example, the client might require a secondary inbound connection from the server after it establishes its connection to the server using the primary connection protocol/port. If that’s the case, you will need to configure a secondary connection. It’s important to note that if you have a protocol that requires a secondary connection, the client type you’re using is going to matter. SecureNAT clients require an Application Filter to support complex protocols. However, if you’ve deployed the Firewall Client (TMG client), then you don’t need an application filter. Application Filters are complex to develop, so it’s highly recommended that you deploy the TMG client if you need support for secondary connections.

Figure 15

Okay, we’re at the end of the wizard. Let’s review the settings on the Complete the New Protocol Definition Wizard page that’s shown in Figure 16 and click Finish.

Figure 16

You’re finished with the wizard, but you’re not done yet. Go back to the Toolbox tab and click the User-Defined group. Here you see a list of Protocol Definitions that you created. Double click on the Protocol Definition you just created, as shown in Figure 17.

Figure 17

Notice on the General tab, shown in Figure 18, that you have the option to Associate this protocol definition with this standard protocol. You would choose this option if you wanted the TMG NIS to inspect this connection with signatures written for the protocol you select here. In general, you wouldn’t want to do this, but there might be situations where you’d want to take advantage of this option. One example might be when you want to support an alternate port for HTTP or HTTPS connections that are being established outside of the web proxy filter.

Figure 18

On the Parameters tab that’s shown in Figure 19, you can also associate an Application Filter with the protocol. You need to be careful about doing this, though, since the filters are designed for specific protocols and won’t work if you assign them to Protocol Definition that doesn’t match the requirements of the filters. In general, you would not assign an application filter to a custom Protocol Definition.

Figure 19

Summary

In this article, we covered Protocol Definitions. All firewall rules require a Protocol Definition as part of the rule configuration. Protocol Definitions are generally designed to be either inbound or outbound. Outbound Protocol Definitions are used for Access Rules and inbound Protocol Definitions are used for Web and Server Publishing Rules. There are also complex Protocol Definitions, where more than one connection is required to support the application. When using complex protocols remember that SecureNAT clients require an application filter to support the complex protocol, whereas Firewall clients do not require an application filter. You likely won’t often need to create new Protocol Definitions, but there might be times when you need to create a custom Protocol Definition to support a new application or product. Let me know if you have questions about Protocol Definitions and I’ll get back to you as soon as possible. Thanks! –Deb.

If you would like to read the other parts in this article series please go to: