Configuring a Trihomed ISA Server as a VPN Server: Adventures with the DMZ Interface UPDATED 12/22/2002

Configuring a Trihomed ISA Server as a VPN Server:
Adventures with the DMZ Interface
UPDATED 12/22/2002
Thomas W Shinder M.D.

One of the great advantages of using ISA Server as your corporate firewall is that it’s very easy and cost effective to make the ISA Server a VPN server. Unlike other “black box” firewalls, you get a full featured and powerful VPN server without having to pay outrageous fees for VPN connections. ISA Server allows you to leverage the unlimited VPN connections allowed by the Windows 2000 Routing and Remote Access Service to connect your road warrior to the corporate network.

A question that’s come up a few times on the ISA Server message boards is whether you can make VPN connections to the DMZ interface on the ISA Server. This question brings up an important issue that often leads to some consternation. It’s often said that ISA Server allows only a single external interface. This is true, but we have to understand the definition of the external interface.

For ISA Server, the external interface is the interface with the default gateway. Windows 2000 supports a single default gateway. All outbound traffic from the internal network leaves the ISA Server through the external interface. The external interface uses the default gateway setting to forward Internet bound packets.

Internal network clients are defined as hosts with IP addresses contained on the LAT. All hosts with IP addresses on the LAT are trusted hosts and the ISA Server does not apply firewall policy to communications that take place between LAT hosts. All communications between internal network clients and non-internal network hosts are subject to address translation. The primary IP address bound to the external interface of the ISA Server is used as the source IP address for outbound communications.

But there’s a gray area between the external interface and the internal interface. This is the DMZ interface. These DMZ interfaces are not trusted because IP addresses in the DMZ are not contained in the LAT. DMZ interfaces are also not strictly considered the external interface because they do not have a default gateway defined on them. The DMZ interface is typically connected to a network that contains a subnet of the organization’s public address block. However, this is not a requirement and you can use any IP addressing scheme you like on the DMZ segment. The only thing required are routing table entries to support communications into and out of the DMZ segment. Note that regardless of the type of network ID you’re using on the DMZ segment that packets are translated between LAT based hosts and non-LAT hosts.

I can’t think of too many reasons why I would want to use the trihomed DMZ configuration. The problem with the trihomed DMZ setup is that you can’t really leverage the full power of ISA Server policies. ISA Server acts as a simple packet filtering router when passing packets between the external interface and the DMZ segment. The packet filtering isn’t even “stateful”. By this I mean that when an inbound request is make to a server on the DMZ segment, a packet filter is not dynamically created to allow the outbound response. Packet filters must be created to explicitly allow the inbound and outbound connection.

For example, if you create a packet filter to allow inbound connections to TCP port 80 from any port number, you must create another packet filter that allows the Web server outbound access from its TCP 80 to any port number. While creating the response filters isn’t a big deal, it’s a stark contrast to the dynamic packet filtering used when LAT hosts connect to the Internet.

Recently a question came up regarding whether you could use the DMZ interface as a second ISP connection and allow incoming VPN connections via the DMZ interface. This certainly seemed possible, as the VPN server should be able to listen on all interfaces for incoming VPN connections. Of course, what sounds good doesn’t always turn out go be in practice.

I set up a trihomed device that had an internal interface, an external interface connected to one ISP and a DMZ interface connected to a second ISP. My first test was to put a laptop on the DMZ segment between the DMZ interface the LAN interface of the router connecting the DMZ interface to the Internet. It worked perfectly. There was just one problem with this type of test: the endpoints of the tunnel were on the same network ID, so routing issues didn’t interfere with the connection. What this confirmed the VPN server was working, it didn’t confirm that I could actually connect a VPN client to the Internet and make a VPN connection to the DMZ interface.

Actually, it was strange that this worked at all. The default PPTP packet filters are automatically configured to use the primary IP address(es) on the external interface(s) of the ISA Server. Since these were the only packet filters configured to allow incoming PPTP connections, how did the host on the DMZ segment gain access to the VPN server and the internal network? This seems to argue for the possibility that for packet filters, any interface that isn’t an internal interface is an external interface.

Things got even worse when I plugged the laptop into a modem connection and connected the VPN client to the public address bound to the DMZ interface. There was no connection at all. The client continued to receive “there is no answer” errors. I decided to do a packet trace on the DMZ and external interface and it appeared that incoming PPTP packets were making it to the DMZ interface, but the were leaving the external interface.

This makes sense when you think about it. The packets arrived to the DMZ interface from a remote network ID. The ISA/VPN server tries to reply, but it can’t reply from the same interface that received the packets because it must use the external interface (which is configured with the default gateway) to send the reply. This caused a spoof attack to be detected by the ISA Server. However, as you’ll see later, disabling spoof detection didn’t help.

What’s interesting is that when I disabled the ISA Server services and pinged the DMZ interface, the ICMP Ping Request arrived at the DMZ interface and the ICMP Ping Response left the external interface and was dutifully returned to the client. But when I enabled the ISA Server services, neither the ping not the PPTP connections would work. Of course, the ping test might have failed because I didn’t create ICMP packet filters to pass from the DMZ interface to the external interface

Take a look at the packet filters properties dialog box in the figure below. For the PPTP packet filters, the local computer tab is used denote the IP address or addresses on the ISA Server receiving the packets. The Default IP address(es) on the external interface(s) and This ISA server’s external IP address apply only to the external interface. The first option is seems to be a misnomer, since you can only have a single external interface. Although at this point we’re not absolutely sure that this doesn’t include the DMZ interface (which is technically considered external, since it is not internal). The second option refers to secondary IP addresses that are bound to the external interface. Which seems to imply that there is only a single external interface, although it might be true that this includes secondary IP addresses on the DMZ NIC. (we’ll answer this question in a moment)

The This computer (on the perimeter network) option wouldn’t apply in this example because the ISA Server is the PPTP endpoint. The same goes for the These computers (on the perimeter network) option. While these options may be helpful when you’re passing PPTP packets from the Internet through an external ISA server in a public address back to back DMZ configuration, they don’t apply in this example where the trihomed DMZ interface is the endpoint of the PPTP connection.

How should the packet filter be configured to support our PPTP connection through the DMZ interface scenario? It didn’t work when the Default IP address(es) on the external interface(s) option was selected. This tends to support the hypothesis that this option only applies to the external interface. It also didn’t work when I manually set the address in the This ISA Server’s external IP address option, again supporting the external interface hypothesis. The last two options don’t apply in this scenario and weren’t tested. However, since a connection was possible when the VPN client was on a local segment to the DMZ interface, its most likely that all non-internal interface are considered external by the packet filters.

Follow Up Note:
Now for the answer to the packet filtering/external interface question. You can easily test how the packet filters are applied on a trihomed ISA Server by using the netstat -na command. For example, create a packet filter that allows the primary IP address on all external interfaces to listen on TCP 119. Now enable the NNTP service on the ISA Server (if the NNTP service is not installed, you can do the same test with the HTTP, FTP or SMTP services). Now try to telnet to TCP 119 on the external interface. The telnet will succeed and you’ll see the NNTP banner. Now telnet to the DMZ interface. The telnet will succeed again. This proves that when you configure a packet filters to apply to the Default IP address(es) on the external interface(s) option, all non-LAT interfaces are considered external, including the DMZ interface. In addition to this, you can prove that when you select the This ISA Server’s external IP address option, that you can use secondary addresses bound to either the external interface or the DMZ interface. The unifying principle here is that for packet filters any interface that is not on the LAT is considered an external interface.

So, after trying to force this square peg into a round hole for over two hours, I decided that I would try something else just for fun. I moved the former DMZ interface to the same segment as the external interface and assigned it an IP address on the same network ID as the external interface, but without a default gateway (remember you can have only a single default gateway on a Windows 2000 machine).

This configuration sort of worked, but ran into the same problem I encountered when connecting to the DMZ interface when it was configured as a DMZ interface. At first, it would not work at all because you always get spoof alerts when you put two NICs on the same external network ID. However, I was about to disable the spoofing problem by performing the following steps:

1. Click Start, point to Run, and type regedit in the Open text box. Click OK.




2. Locate the following registry key:

3. If you don’t see a Parameters key, then you’ll have to create it. To do this, point to New on the Edit menu, and click Key. Name the key Parameters.




4. Right-click Parameters the Parameters key, point to New, and click DWORD Value.




5. Name the value SpoofDetection.




6. Right-click SpoofDetection, and then click Modify.




7. In the Value data box, type 0 (zero), and then click OK.
Note that this disables IP Spoof detection, which does pose a security risk. You can reenable IP Spoof detection, by setting the SpoofDetection value to 1. This is the default value.




8. Close the Registry Editor, and then restart the ISA Server 2000 services.

However, what was quite strange about this was that I could no longer establish a VPN connection to the Primary IP address on the Primary (top listed) interface. The VPN link had to be established with the IP address used on the former DMZ interface IP address (which now has the second IP address on the same network ID as the external interface). To make matters worse, the PPTP packet filters allowing for inbound access were set for the default IP address(es) on the external interface(s). I tested it with a specific IP address on the external interface and used the IP address on the primary interface. All this accomplished was being unable to make a connection to any of the IP addresses. No VPN connection could be established on either the external interface or the former DMZ interface.

My conclusion is you can forget about establishing inbound VPN connections to the DMZ interface in a trihomed DMZ configuration. While it’s a no-brainer publishing servers on the DMZ segment when that segment represents a subnet of you public block, using the DMZ interface to allow you to connect a second ISP and allow incoming VPN connections via that ISP isn’t something that will work.

What was a little surprising was that configuring a second NIC on the external network ID didn’t work. I recall reading a description on the ISAServer.org message boards on how you can get around some problems with L2TP/IPSec by configuring a second NIC on the external network ID. While this type of configuration is well known for causing the spoofing issues, it still didn’t work even after spoof detection was disabled.

Finally, I should note that there were a some inconsistencies during the testing. Sometimes a certain configuration would work, and then when the ISA Server machine was restarted, it would not longer work, in spite of an absence of errors in the Event viewer. If you’ve tried the same thing and got similar results, or even better, if you’ve gotten it to work, please post your tips and tricks to the message boards via the link provided below.

I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=9;t=000460 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks! –Tom