Publishing Remote Desktop Web Connection Sites with the ISA Firewall Part 3: Testing and Troubleshooting

Publishing Remote Desktop Web Connection Sites with the ISA Firewall
Part 3: Testing and Troubleshooting
By Thomas W Shinder MD, MVP

If you would like to read the other articles in this series please go here:

• Publishing Remote Desktop Web Connection Sites with the ISA Firewall Part 1: Remote Desktop Web Services Concepts

• Publishing Remote Desktop Web Connection Sites with the ISA Firewall Part 2: Creating the Web and Server Publishing Rules

Test the solution

Now we’re ready to test the solution. However, before you make your test, make sure you have configured name resolution so that external clients resolve the name used on the Web site certificate bound to the Web listener to the IP address used for the Web listener. In a production environment you will create a Host (A) record on your public DNS servers. In a test environment, you can create a HOSTS file entry so that the name resolves to the IP address on the external interface of the ISA firewall.

In this example, we’ve created a HOSTS file entry for tsweb.msfirewall.org so that it resolves to the IP address used on the Web listener.

Perform the following steps on the client machine to test the configuration:

1. Open Internet Explorer on the client machine located on an external network. Enter into the address bar https://tsweb.msfirewall.org and press ENTER.
2. Enter valid user credentials into the Connect to dialog box and click OK.

Figure 1

3. If you’re running Windows XP Service Pack 2, then you’ll find the ActiveX control is blocked the first time you hit the site. Click OK in the dialog box informing you that the ActiveX control could not be installed.
4. Click the Information Bar in Internet Explorer to install the ActiveX control. Click the Install button in the Internet Explorer – Security Warning dialog box.
5. After the ActiveX control is installed the Remote Desktop Web Connection appears. Enter the server name in the Server text box. In this example, we’ll use the name tsweb.msfirewall.org as the server name. Note that this name resolves to the IP address used on the RDP Server Publishing Rule’s listener. Also remember that this name may or may not be the actual name of the server on the corporate network. It doesn’t matter what the server’s actual name is, what matters is that the name you enter in the Server text box resolves to an IP address that will get the connection to the RDP Server Publishing Rule’s listener address. Put a checkmark in the Send logon information for this connection checkbox, and then enter the User name and Domain in the text boxes. Click Connect.

Figure 2

6. The terminal services log on dialog box appears in a full screen session. The User name and Domain are automatically entered. Enter the password and click OK.

Figure 3

7. At this point you will enter the terminal services session. Everything worked!

Troubleshooting the Solution

Even though the procedures seem straightforward, there are several common troubleshooting issues that come up time and again on the ISAserver.org message boards, mailing list, and newsgroups. Some of these issues are related to ISA firewall configuration errors and some of them have nothing to do with the ISA firewall configuration. We’ll talk about both these types of problems in this section.

Common troubleshooting issues include:

• RDP Service on the RDP Server is not Started
• External Interface of the ISA Firewall uses a Private IP Address
• Authentication with the ISA Firewall Fails
• Certificate Dialog Box Appears when Client Connects

RDP Service on the RDP Server is not Started

Imagine this scenario: you’ve gone through this article and done everything I’ve told you to do. You have a good understanding of the ISA firewall’s Web and Server Publishing models, Windows networking, and the TCP/IP protocol suite. Everything seems to be in place but when you try to connect to the remote desktop Web connection Web site, the connection attempts repeatedly fail.

You look at the log files and the results don’t make a lot of sense. It appears that the SSL connections are accepted, but you’re really not sure what to look for so you shake your head and wonder what questions you should ask yourself to come up with a solution.

The first question you ask yourself is “I wonder what would happen if I used the native Remote Desktop Connection client to connect to the RDP site, without going through the Remote Desktop Web connection?”  You fire up the RDP client software and connect to the IP address of the RDP Server Publishing Rule listener. You end up seeing what appears in the figure below.

Figure 4

When you look at the ISA firewall’s log files for entries related to this connection attempt, you’ll see something like that in the figure 5. Notice that nowhere in these log file entries is there any information regarding the connection being denied. All you see in the Action column are entries for Initiated Connection and Closed Connection.

This should lead you to two conclusions:

• There is no rule denying the connection
• There is a rule allowing the connection

If those are the two conclusions you came up with, then you’re absolutely right. There is a rule allowing the connection, the RDP Server Server Publishing Rule. There is no rule denying the connection.

Figure 5

So what’s up? What’s causing the failed connection? The key is the entry in the Result Code column. Here you see that Result Code for the Closed Connection entries is 0x80074e21. If you look up this result code you’ll see that it means that either the client or the server closed the connection with a TCP RST. OK, so what is a TCP RST?

A TCP RST is a TCP segment (notice that TCP messages are segments, NOT packets, which is a common misuse of the language among network admins) which indicates something is wrong with the connection. A RST is sent when:

• A SYN arrives for a non-existent server.
• TCP wants to abort the connection.
• A segment is received for which a connection doesn’t exist.

The problem in our case is related to the first reason. The “non-existent server” is the absence of the RDP listener being enabled on the terminal server or remote desktop connection server (like Windows XP). When you enable terminal services listener on the terminal server, or enable remote connections to a published remote desktop server (for example, Windows XP Professional or Windows Server 2003 Admin RDP) everything will work as expected.

For your future reference, Tables 1 and 2 list firewall service error codes that help you solve problems using the information in the Result Code field in the ISA firewall’s log files. This information was obtained from the MSDN Web site over at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/isasdk/isa/error_codes.asp

Table 1: Firewall Service Error Codes

Table 2: Additional run-time codes that may be returned by the Firewall service and may appear as result codes in ISA Server logs

External Interface of the ISA Firewall uses a Private IP Address

A remarkably common problem relates to deployments where there is a NAT device in front of the ISA firewall. The NAT device could be a simple NAT device like a broadband “router”, or it can be a simple stateful packet inspection firewall, such as a PIX or Sonicwall. What the broadband “router” and the simple stateful packet inspection devices have in common is that they both are responsible for the Internet connection and perform network address translation (in any of its forms) between its LAN and WAN interfaces.

When there is a NAT device in front of the ISA firewall, the ISA firewall’s external interface will have one or more private IP addresses bound to it. Connections cannot be routed over the Internet to a private IP address. That’s why all incoming connections to resources located on your internal network must be directed to the public address that is bound to the external interface of your NAT device.

Figure 6 shows a common troubleshooting situation. The NAT device has a public address assigned to its WAN interface and a private address bound to its LAN interface and performs NAT on communications moving across its interfaces. The ISA firewall has private addresses on both its internal and external interfaces. The ISA firewall also performs NAT across its internal and external interface (however, you do have the option to route between those interfaces, which is very helpful in SBS 2003 scenarios where there is a NAT/VPN device in front of the ISA firewall and a site to site VPN connection between that NAT/VPN device and another device.)

As we spoke about earlier in this article, the external client must be able to query a public DNS server to resolve the names it uses to connect to the remote desktop connection Web site and the RDP Server Publishing Rule. I’ve said several times in this article that the public DNS server must be able to resolve the names the remote desktop Web connection client uses to access the Web site and RDP Server Publishing Rule listener to the IP addresses used by the Web listener and the RDP Server Publishing Rule’s listener.

The problem I have when discussing ISA firewall scenarios is that I assume you have a public address bound to the external interface of the ISA firewall. When you have a public address bound to the external interface of the ISA firewall, then my comments about requiring that the names the external client uses to connect to the Web and RDP sites resolving to the IP addresses on the external interface of the ISA firewall is correct. However, if you have a NAT device in front of the ISA firewall, then this advice is not correct, because external client cannot connect to private IP addresses.

In order to keep things simple (mostly simple for me), I always make the assumption that the ISA firewall has a public address on its external interface or if you have a NAT device in front of the ISA firewall, I assume that you understand the implications of that configuration. However, making these assumptions are for my convenience and not necessarily reflective of the overall knowledge base of the ISA firewall community.

The figure below shows a common misconfiguration that leads to failure for both the Web connection and the RDP connection. The user has configured the DNS server with a Host (A) record that resolves the name used to connect to the Web site and the RDP site to the IP address bound to the external interface of the ISA firewall. This cannot work, since the IP address on the external interface is a private IP address and is not directly accessible through the Internet.

Figure 6

The correct configuration is shown in the figure below. The public DNS server has a Host (A) record that resolves the name used to connect to the Web and RDP sites using the public IP address bound to the external interface of the NAT device located in front of the ISA firewall. There is the only way it can work. You never used use the external address on the ISA firewall when a private address is used on the external interface.

Figure 7

Now, getting the DNS entry right is only half the battle. The other half is to configure the NAT device to forward the required connection requests to the correct IP addresses on the external interface of the ISA firewall. This is commonly referred to as port forwarding. For example, in the scenario discussed in this article you would need to forward incoming connections to TCP port 80 and TCP port 3389 to the correct IP addresses on the external interface of the ISA firewall.

In this scenario it’s fairly simple, because there is a single IP address bound to the external interface of the ISA firewall and we’re only publishing a single RDP server. It gets slightly more complex (but not terribly so) when you need to publish multiple RDP servers, because you will need multiple IP addresses bound to the WAN interface of the NAT device and multiple private addresses bound to the external interface of the ISA firewall.

One last thing related to this scenario is the default gateway setting on the external interface of the ISA firewall. The default gateway on the external interface of the ISA firewall must be the LAN IP addresses assigned to the NAT device located in front of the ISA firewall.

Authentication with the ISA Firewall Fails

If there’s one thing that I’ve been remiss in explaining in my books and articles is how the ISA firewall’s pre-authentication features requires an authentication provider. As you probably already know, the ISA firewall includes a pre-authentication feature that allows the ISA firewall to authenticate incoming connections mediated by the Web proxy filter before allowing the connection to be forwarded to the Web site on your internal network. The is a powerful security feature, since external attackers aren’t able to leverage an anonymous connection to attack your corporate Web servers.

However, in order for the ISA firewall to authenticate a user, it must have an authentication provider. For Web Publishing Rules, you can use the following authentication providers:

• The local SAM database maintained on the ISA firewall itself
• The internal network Active Directory user database
• RADIUS authentication against a RADIUS compliant user directory

Keeping the user database on the ISA firewall is not a recommended configuration, as the management overhead of synchronizing passwords can be overwhelming. Also, there is a remote, although infinitesimal possibility that that ISA firewall will be “owned” and keeping the directory on the ISA firewall potentially exposes all of them to the attacker.

The preferred configuration for an ISA firewall used to control both inbound and outbound access is Active Directory authentication. In this case, the ISA firewall must be a domain member to authenticate users against the Active Directory. This is a secure configuration and in the overall security evaluation, a domain member ISA firewall is more secure than a non-domain member ISA firewall. There are many reasons for this being true and I’ll publish a comprehensive article in the near future highlighting the security considerations for domain membership and non-domain membership and prove that domain membership for ISA firewall’s controlling inbound and outbound access is the most secure configuration.

When the ISA firewall is deployed to allow only inbound access, then RADIUS authentication is the preferred method. This brings up an ISA firewall best practice. If your budget allows for it, you should deploy ISA firewalls in a role-based configuration. When I am allowed the appropriate budget, I separate my ISA firewalls into the following roles:

• Secure Remote Access VPN server and gateway
• Inbound access firewall for Web and server publishing
• Outbound access firewall for SecureNAT, Web proxy and Firewall clients

I don’t think I’ve done an article or described in any of our books the rationale for such a role-based deployment, but I’ll put it on my list, as it’s an interesting and exceptionally powerful security and performance solution.

But back to the RADIUS authentication provider. When the ISA firewall is deployed to allow only inbound connections via Web Publishing Rules and Server Publishing Rules, then RADIUS authentication is often the best solution, as the ISA firewall is positioned well outside the security perimeter of the authentication providers. For this reason, and others, I prefer to the inbound access only ISA firewall to use RADIUS authentication and not make the ISA firewall a domain member.

BOTTOM LINE:
When configuring the ISA firewall to perform pre-authentication, think about what type of authentication provider you want to use. In single ISA firewall deployments where the ISA firewall is responsible for both inbound and outbound access control, always make the ISA firewall a domain member and use Windows Integrated (Active Directory) authentication. When the ISA firewall is providing only inbound access control, then configure the ISA firewall to use RADIUS authentication.

Certificate Dialog Box Appears when Client Connects

You have two certificate options when deploying secure SSL connections to the ISA firewall:

• Commercial certificates
• Private certificates

Commercial certificates must be purchased from a commercial certificate authority. An example of such a certificate authority is Verisign. Private certificates are those you generated yourself, or have someone else generate for you, but which are not part of a commercial certificate authority chain.

The advantage of private certificates is that you can generate them yourself. The disadvantage of private certificates is that no one trusts those certificates out of the box. If you have an Active Directory domain, you can automatically configure all the domain member computers in that domain to trust the certificates you generate. For computers that are not members of the domain, you must do some “out of band” configuration to make the those computers trust your certificates.

This points out the advantage of using commercial certificates. Most Windows computers trust most of the commercial certificate authority’s right out of the box. You don’t need to add the commercial certificate authority’s CA certificate to your machine’s Trusted Root Certification Authorities machine certificate store because Microsoft has already put those into your store and updates them on the periodic basis for you.

If you have an Active Directory domain and configure Group Policy to automatically deliver your CA certificates to all domain members and if you only allow domain member computers to connect to your published resources over the Internet, then everything will be fine. Users will never know any difference and their access to your SSL secured sites will be transparent.

However, if you have computers that are not domain members, or any computer that doesn’t have your CA’s certificate installed in its Trusted Root Certification Authorities machine certificate store, then users connecting to your SSL secured Web site will be confronted with a dialog box informing them that their machine doesn’t trust the certificate presented by the Web site. Of course, the typical user has no idea why this appears and either clicks past this dialog box or calls the Help Desk asking what to do.

The solution to the problem is to install the CA certificate into the Trusted Root Certification Authorities machine certificate store on all machines that will connect to your secure Web site. There are several ways to do this, including using the Web enrollment site, Active Directory Group Policy, and the Certificates MMC snap-in. PKI deployment and provisioning is far beyond the scope of this article, but you might want to check out the ISA Server 2000 VPN Deployment Kit at http://isaserver.org/articles/isa2000vpndeploymentkit.html for some great information on setting up different certificate authorities and how to deploy certificates to your clients.

Summary

In this article we finished up the series on how to publish remote desktop Web connection sites by testing the solution and going over some troubleshooting issues. I hope you learned something about Web and Server Publishing during this series and I’d be very happy to hear from you about things I can do to make this information even more useful to you. Send me a note at [email protected] and I’ll get back to you ASAP. Thanks! –Tom.