Enabling Secure FTP Access Through ISA 2006 Firewalls (Part 1)

If you would like to read the next part in this article series please go to Enabling Secure FTP Access Through ISA 2006 Firewalls (Part 2).

This article has been edited by Dr. Thomas Shinder.

The ISA Firewall, officially, does not support the FTPS protocol because protocol negotiations are encrypted and thus no application filter can process these negotiations. This article series provides a viable workaround. However, a side effect of the method described in these articles is that no unencrypted (FTP) connections will work. An alternative to our solution is to create a custom Server Protocol Definition for FTPS, and then use Pesach Shelnizt’s technique to prevenet conflicts with the default FTP Server Protocol Definition.


One of the big new features included in IIS 7.0 (which comes with Windows Server 2008) is secure FTP. Secure FTP uses TLS encryption to secure data moving over the FTP data channel. This is a major advance over previous versions of the IIS FTP service, which supported only unencrypted connections.

The figure below shows the IIS 7.0 interface for configuring the IIS 7.0 secure FTP service.

Figure 1

However, if you have ever tried to publish the IIS 7.0 FTP service through an ISA firewall, you might have noticed that things did not quite work out the way you wanted them to. In fact, you will encounter the dreaded error 500 Access Denied.

The figure below shows the error in the FTP client when trying to connect to the published secure FTP server.

Figure 2

Let us now examine what the problem is and see if there is a way we can solve it. In order to fully understand what is happening, you will need the following:

  • Sniffer on the FTP client
  • Monitoring on the ISA server system
  • Sniffer on the FTP server

Our test setup looks like this:

FTP client <===> ISA 2006 <===> Win2k8 (IIS7/FTP7)

The IP address of the FTP client is The IP address on the external interface of the ISA firewall is ISA and the IP address of the server is

Figure 3

Step 1:

Try to connect to the FTP server from the FTP client and sniff at both ends. When you look at the packets you can clearly see what is happening. We will first look at the packet trace which will enable us to follow the TCP stream.

1.1 packet view

Figure 4

As you can see, the FTP client goes through the TCP handshake in the first 3 packets (SYN / SYN ACK / ACK). After the TCP handshake is complete, the actual FTP communication starts. The first FTP command is the server identifying itself as an MS FTP server.

The client then responds to this by saying AUTH TLS. The client is basically saying “hello friendly FTP server, i would like to start an encrypted TLS session”. The client tries this twice but gets ACCESS IS DENIED responses back from the server.

When you look at the FTP server packets, you can see the first 3 handshake packets, then the one packet where the server identifies itself, but we never see another FTP packet coming into the FTP server with the AUTH TLS session. 

Figure 5

1.2 Follow View

If we look at the TCP follow view, we can clearly see the  the client sending the Auth SSL and the server never receiving these packets.

Figure 6

Step 2:

The above information clearly shows a response 550 Access is denied response is being sent to the FTP client. However, what might not be so obvious that that the response was not generated by the FTP server! The FTP server should be able to accept the request to negotiate a secure TLS connection. So what is the problem? Evidently, there is a device in the middle the is generating this response and sending it back to the client.

Using this information, let’s look at the ISA firewall’s log files and see if we have any deny’s at that level. The figure below shows the relevant log entries.

Figure 7

At a glance, the traffic seems to be flowing normally, we see the ISA accept the first packet and then close the connection like it would do with any normal TCP connection that ended.

Where is this Access Denied response coming from? To find out a bit more we need to add an extra column to the ISA log viewer window called result code.

Figure 8

Once we add this column, we can see the the full details of what the ISA firewall has been doing.

To understand the meaning of each result code you can check out the following link.

FWX_E_CONNECTION_KILLED => 0x80074E24 => ISA Server killed a connection.

Even though this type of end is not unusual, it seems that the ISA firewall took an active part in ending the session, and we know that the only place the 550 Access is denied packet could have come from was the ISA firewall.

At this point we need to consider what component of the ISA firewall is killing the connection. A good place to start is to look for an application filter that handles FTP connections. As you may know, the ISA firewall is a stateful packet and application layer inspection firewall, enabling it to perform application layer inspection for anuber of application layer protocols. One of these protocols is FTP. You can view this on the Add-ins node in the left pane of the ISA firewall console.

Figure 9

This means ISA understands the FTP protocol at the application layer and will recognize malformed commands with an FTP communication stream. According to the application filter dll we may or may not have configurable items.

For example, the SMTP filter has a way of adding and removing commands that are accepted by the firewall. Unfortunately, the FTP filter however does not. All you can do is enable or disable it. You can disable the filter here and completely disable the filter for all ISA firewall rules, or you can do it on each rule individually. We will show you how to disable is for individual rules later.

Figure 10

The simple fact is that the FTP application filter in ISA 2006 does not support the AUTH TLS and thus the default response from the ISA firewall to such a request is to respond with an Access denied. The ISA firewall expects the default FTP command stream, as seen in the figure below, and there is no way for us to add accepted commands.

The response does not say: filtered by ISA 2006, as this could give potential hackers information they do not need to know.

Figure 11

The only way to solve this problem in ISA 2006 is to disable the FTP application filter on the Access Rule. Open the Properties dialog box for the Access Rule for the FTP Server Publishing Rule, click the Traffic tab, and remove the checkmark from the FTP Access Filter checkbox in the Application Filters frame, as seen in the figure below.

Figure 12

If we now retry our connection and sniff the traffic on the FTP server we can see the first packet with the identification string, the second packet is the FTP client saying; “Hi, I want to do TLS authentication”. The third packet is the server responding and saying “ok let’s do TLS”, and from there on the stream goes to an encrypted mode.

Figure 13

However, every FTP story has two sides. As the authentication channel is only the first part of the connection. Once we are authenticated to an FTP server we want to list directories and transfer files, this is done over a secondary data channel. In part 2 of this article series, we will see what we need to do in order to enable data transfer over the secure channel.


In this article we demonstrated a problem with secure FTP server publishing using the ISA 2006 firewall. Using network sniffers, we say that the TLS negotation attempt by the client was denied, but it was not denied by the published FTP server. Instead, we saw that that the security negotiation attempt was denied by the ISA firewall. The ISA firewall has an FTP applciation layer filter to support FTP connections, but it is not configurable (in contrast to the SMTP filter, which does enable some level of configuration). Because the built-in FTP application layer filter on the ISA firewall does not support TLS negotiation, you need to disable the FTP application layer filter either for the entire rule set, or for specific rules. We prefer to disable the FTP filter for specific rules due to the fact that SecureNAT clients will not be able to use the FTP protocol for outbound access if the filter is disabled globally. After disabling the FTP application filter, we demosntrated that the secure FTP connection was successfully established through the ISA firewall to the published IIS 7.0 secure FTP server. However, there is one more step – while we have enabled a secure FTP connection, we have not yet enabled secure data transfer over the channel. In part 2 of this series, we will go over what you need to do enable data transfer over the secure FTP link.

If you would like to read the next part in this article series please go to Enabling Secure FTP Access Through ISA 2006 Firewalls (Part 2).

About The Author

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top