Configuring Windows Server 2003-based ISA Server Firewall/VPN Server to Accept inbound NAT-T L2TP/IPSec Calls

Configuring Windows Server 2003-based ISA Server Firewall/VPN Server to accept inbound nat-t L2TP/IPSec calls



By Thomas W Shinder, M.D.


There are a lot of reasons why you would want to run your ISA Server firewall on a Windows Server 2003 machine instead of Windows 2000. Just of few of these include:



  • Windows Server 2003 appears to be significantly more secure than Windows 2000, as least right out of the box
  • Windows Server 2003 supports VPN client quarantine
  • Windows Server 2003 supports conditional DNS forwarding
  • Windows Server 2003 supports NetBIOS proxy name resolution
  • Windows Server 2003 supports NAT-T L2TP/IPSec VPN clients

  • Support for NAT-T L2TP/IPSec VPN clients is provides one of the most compelling reasons to put your ISA Server firewall/VPN server on Windows Server 2003 instead of Windows Server 2003.


    Get the Book!


    Why? Because you may want to allow external NAT-T L2TP/IPSec clients located behind a NAT device to connect to your Windows Server 2003-based ISA Server firewall/VPN server. Normally, any IPSec based protocol cannot be passed through a NAT device because NAT and IPSec are incompatible. Either the NAT device invalidates the packet, or the NAT device cannot read the packet headers required for address translation. The only other option you have is PPTP. While some NAT devices handle multiple outgoing PPTP connection intelligently, more often than not your outbound PPTP through a hotel conference center will get “bumped” after a certain number of other outbound PPTP connections are established



    Note:
    For an excellent review of the issues involved with passing IPSec based protocols through a NAT device, please refer to Stefaan Pouseele’s article How to pass IPSec traffic through ISA Server


    The figure below shows the typical remote access VPN scenario. A user is located at a hotel or home office and needs to create a secure L2TP/IPSec connection to the corporate network. This VPN user as two choices: PPTP or NAT-T L2TP/IPSec. While normal IPSec packets are stopped by NAT devices (such as NAT routers and “Internet gateways”), the NAT-T L2TP/IPSec packets are wrapped or “encapsulated” by UDP headers. These UDP headers protect the IPSec protected portion of the packet and allow the VPN connection to pass through the NAT device without harm. Note that in the figure below that the UDP 1701 header is encapsulated in the UDP 4500 header. The NAT device only needs to be able to pass UDP 500 and UDP 4500.



    The advantage of using the Windows VPN client software to connect to the Windows Server 2003-based ISA Server firewall/VPN server is that both the client and server are RFC compliant. Unlike other major VPN server vendors that use non-RFC, proprietary and incompatible methods of NAT Traversal, the Microsoft NAT-T solution is compliant with IETF Internet draft standards.



    Note:
    For comprehensive information on how to install the Microsoft NAT-T L2TP/IPSec client, please refer the ISA Server 2000 VPN Deployment Kit document that applies to your Windows client operating system at Complete List of ISA Server 2000 VPN Deployment Kit Documents. For more information on the details of the Windows NT/9x NAT-T L2TP/IPSec client, check out Description of the Microsoft L2TP/IPSec Virtual Private Networking Client for Earlier Clients. For more information on the details of the Windows 2000/Windows XP NAT-T L2TP/IPSec client, check out L2TP/IPSec NAT-T Update for Windows XP and Windows 2000.


    Packet Filters Required to Allow Inbound NAT-T VPN Calls


    You need to do the following on the ISA Server firewall/VPN server to support inbound VPN calls from NAT-T RFC compliant L2TP/IPSec clients that are situated behind a NAT device:



  • Create a packet filter for inbound UDP 500 (receive/send)
  • Create a packet filter for inbound UDP 4500 (receive/send)
  • Create a packet filter for inbound UDP 1701 (receive/send)

  • The UDP 500 receive/send packet filter allows for Internet Key Exchange Protocol (IKE) packets to be received by the ISA Server firewall/VPN server. This packet filter is required for both NAT-T VPN clients and non-NAT-T VPN clients.


    The UDP 4500 receive/send packet filter is specific for NAT-T VPN clients. The IPSec ESP header is encapsulated in the UDP port 4500 header. When the Windows Server 2003 ISA Server/VPN server receives the packet, it removes the UDP header and exposes the ESP header. This is how the server determines that the VPN client is a NAT-T client.


    The UDP 1701 receive/send packet filter allows the L2TP control channel to be established and maintained. The are a number of different control messages that are sent through the L2TP control channel. The purpose of the control messages is to establish the VPN tunnel, maintain the VPN tunnel, and tear down (close) the tunnel in an orderly fashion when the connection is no longer needed.


    The figure below shows the structure of an L2TP/IPSec packet. Notice that the IPSec ESP header is located in front of the L2TP UDP header. The IPSec ESP header does not require an open port. However, it does require that the firewall listen and accept incoming connections to IP Protocol 50. Only the tunnel IP header containing the tunnel endpoint information and the datalink layer header encapsulate the IPSec ESP header.



    Note:
    You do not need to create a packet filter to allow incoming IP Protocol 50. The reason for this is unknown.




    Create the three packet filters at the ISA Server firewall/VPN server accepting the L2TP/IPSec connections from L2TP/IPSec clients located behind a NAT device. If you do not want to support NAT-T L2TP/IPSec clients, then you can use the ISA Server VPN Wizard and all the required packet filters are created for you.


    Creating the Packet Filter for UDP Port 500


    Perform the following steps to create the packet filter for UDP Port 500:



    1. In the ISA Management console, expand the Server and Arrays node, then expand your server name. Expand the Access Policy node. Right click the Packet Filters node, point to New and click Filter.



    1. Type a name for the packet filter in the IP packet filter name text box on the Welcome to the New IP Packet Filter Wizard page. I recommend you name it UDP 500 (receive/send). Click Next.



    1. Select the Allow packet transmission option on the Filter Mode page. Click Next.



    1. Select the Custom option on the Filter Type page. Click Next.



    1. Configure the details of the packet filter on the Filter Settings page. Select the UDP option from the IP protocol drop down list box. Select the Receive send option in the Direction drop down list box. Select the Fixed port option in the Local Port drop down list box. Set the local Port number to 500. Select the All ports option in the Remote port drop down list box. Click Next.



    1. Select the Default IP addresses for each external interface on the ISA Server computer option on the Local Computer page. The default IP address is the primary IP address bound to the interface. The primary address is the IP address at the top of the list in the Advanced TCP/IP Properties dialog box. Click Next.



    1. Select the All remote computers option on the Remote Computers page. Click Next.



    1. Review the settings on the Completing the New IP Packet Filter Wizard page, then click Finish.


    Get the New Book!


    Creating the Packet Filter for UDP 4500


    Perform the following steps to create the packet filter for UDP 4500:



    1. In the ISA Management console, expand the Server and Arrays node, then expand your server name. Expand the Access Policy node. Right click the Packet Filters node, point to New and click Filter.
    2. Type a name for the packet filter in the IP packet filter name text box on the Welcome to the New IP Packet Filter Wizard page. I recommend you name it UDP 4500 (receive/send). Click Next.
    3. Select the Allow packet transmission option on the Filter Mode page. Click Next.
    4. Select Custom on the Filter Type page. Click Next.
    5. Configure the details of the packet filter on the Filter Settings page. Select the UDP option from the IP protocol drop down list box. Select the Receive send option in the Direction drop down list box. Select the Fixed port option in the Local Port drop down list box. Set the local Port number to 4500. Select the All ports option in the Remote port drop down list box. Click Next.



    1. Select the Default IP addresses for each external interface on the ISA Server computer option on the Local Computer page. The default IP address is the primary IP address bound to the interface. The primary address is the IP address at the top of the list in the Advanced TCP/IP Properties dialog box. Click Next.
    2. Select the All remote computers option on the Remote Computers page. Click Next.
    3. Review the settings on the Completing the New IP Packet Filter Wizard page, then click Finish.

    Neither the Windows 2000/Windows Server 2003 server, nor the ISA Server services, need to be restarted. The packet filters will start working automatically. If you have a very busy machine and you need the packet filters to start working immediately, you should restart the Firewall service.



    Note:
    You can restart the firewall service by navigating to the Servers and Arrays/Server Name/Monitoring/Services node in the ISA Management console. Then right click on the Firewall service entry in the right pane. Click the Stop command. After the service is stopped, right click the Firewall service entry again and click the Start command. You can also stop the Firewall service from the command prompt. Open a command prompt and type “net stop Microsoft firewall” (without the quotes). After the Firewall service stops, restart the Firewall service by typing “net start Microsoft firewall” (without the quotes).


    Get the New Book!


    Creating the Packet Filter for UDP 1701


    Perform the following steps to create the packet filter for UDP 1701:



    1. In the ISA Management console, expand the Server and Arrays node, then expand your server name. Expand the Access Policy node. Right click the Packet Filters node, point to New and click Filter.
    2. Type a name for the packet filter in the IP packet filter name text box on the Welcome to the New IP Packet Filter Wizard page. I recommend you name it UDP 1701 (receive/send). Click Next.



    1. Select the Allow packet transmission option on the Filter Mode page. Click Next.
    2. Select the Custom option on the Filter Type page. Click Next.
    3. Configure the details of the packet filter on the Filter Settings page. Select the UDP option from the IP protocol drop down list box. Select the Receive send option in the Direction drop down list box. Select the Fixed port option in the Local Port drop down list box. Set the local Port number to 1701. Select the All ports option in the Remote port drop down list box. Click Next.



    1. Select the Default IP addresses for each external interface on the ISA Server computer option on the Local Computer page. The default IP address is the primary IP address bound to the interface. The primary address is the IP address at the top of the list in the Advanced TCP/IP Properties dialog box. Click Next.
    2. On the Remote Computers page, select the All remote computers option and click Next.
    3. Review the settings on the Completing the New IP Packet Filter Wizard page and click Finish.

    The L2TP/IPSec NAT-T VPN clients are able to connect after you create all three packet filters. Note that while the ISA Server VPN Wizard creates L2TP/IPSec packet filters, you should recreate the packet filters as noted in this article. These NAT-T L2TP/IPSec filters differ slightly from those created by the Wizard.


    Get the Book!


    Summary


    In this article we discussed the issue of passing IPSec based protocols through a NAT device. NAT-T (NAT Traversal) protocols allow VPN clients to pass IPSec protected packets through a NAT device. The Windows L2TP/IPSec NAT-T VPN clients software works together with the Windows Server 2003-based ISA Server firewall/VPN server to allow VPN clients located behind a NAT device to pass IPSec protected through the NAT. We also went through detailed step by step procedures required to create the packet filters on the ISA Server firewall/VPN server that allow it to accept the inbound ISA Server firewall/VPN server calls.


    I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=13;t=001725 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks! –Tom

    1 thought on “Configuring Windows Server 2003-based ISA Server Firewall/VPN Server to Accept inbound NAT-T L2TP/IPSec Calls”

    1. Hi, I have found your article (well written) but i still have a problem with this configuration:

      Client – Windows XP connecting with L2TP with pre-shared key
      Firewall – ISA 2006
      Server – Windows Server 2003

      All works!

      Problem is when i try to connect with Windows 10 … server asks for credentials and then, after some seconds, it blocks me saying “The security level encountered a processing error during initial negotiations with the remote computer”.

      Can you help me in any way?

    Leave a Comment

    Your email address will not be published.

    This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

    Scroll to Top