How to enable ESP Encryption on ISA 2004 in a site-to-site VPN scenario

Introduction

In many documents on http://www.microsoft.com/isaserver/techinfo/guidance/2004/vpn.mspx the author includes a section that briefly describes how IPSec works in tunnel mode. When using Encapsulating Security Payload (ESP), traffic is typically encrypted using Data Encryption Standard (DES) or Triple DES (3DES) and authenticated with SHA1 or MD5. However, you can specify to use Null Encryption, that is no encryption at all, so that the packet structure with ESP can be seen as it traverses the network. It should be obvious that this could be a very valuable feature, especially in a lab environment to learn more about IPSec and probably also to debug some typical scenarios, particularly with third-party IPSec devices.

Unfortunately, none of those documents explain how to enable ESP Null Encryption on ISA 2004 in a site-to-site VPN scenario. Therefore, I used the following lab environment to find it out myself:


Figure 1

As you can see in the above figure, the setup is rather simple. We have 3 networks: #22 (192.168.22.0/24), #1 (192.168.1.0/24) and #44 (192.168.44.0/24). Between the ISA 2004 server and the IPSec Gateway we create a site-to-site IPSec Tunnel Mode VPN which enables the networks #22 and #44 to talk to each other in a secure way. The ISA 2004 server runs on Windows 2003 SP1. The IPSec Gateway is implemented on a Windows 2003 SP1 server used as a router with IPSec capabilities. It is not within the scope of this article to explain how to set up such a site-to-site IPSec Tunnel Mode VPN. However, on the ISA 2004 server you should use the Add Remote Site Network wizard and choose IPSec tunnel mode as VPN protocol (check out Creating IPSec Tunnel Mode Site to Site VPNs with ISA Server 2004 Firewalls for more info), and for the IPSec Gateway you should follow the instructions outlined in the Microsoft knowledgebase article How To Configure IPSec Tunneling in Windows Server 2003.

Configuration

On the ISA 2004 server we created a new remote site network called RemoteSite#44. If you look at the properties of this network, more specifically the IPSec Configuration, you will see that for the Phase II parameters you can only choose between DES and 3DES:


Figure 2

On the IPSec Gateway, remember this is just a Windows 2003 server, we created a new IPSec policy called RemoteSite#22. If you look at the properties of this IPSec policy, more specifically the Custom Security Method Settings, you will see that you have the choice between <None>, 3DES and DES for the ESP Encryption algorithm as shown in the following figure:


Figure 3

The obvious conclusion is of course that you can not configure ESP <None> or Null encryption through the ISA MMC, although ISA server does use the IPSec capabilities of Windows 2003 in this scenario. Nevertheless, it must be clear that the ISA MMC simplifies a lot of the configuration of a site-to-site VPN scenario. So, maybe if we look up the properties of the Windows 2003 IPSec policy created by the ISA MMC, we can configure ESP <None> or Null encryption. The following figure shows the IPSec Policies MMC on the ISA 2004 server:


Figure 4

As you can clearly see, no policy is shown that can be related to the remote site network RemoteSite#44 we created. After some further investigation we learn that the IPSec Policies MMC only shows the static configured IPSec policies. If you want to configure and view all the static or dynamic IPSec main mode settings (Phase I), quick mode settings (Phase II), rules and configuration parameters, then you have to use the Netsh commands for IPSec. Detailed information about the Netsh commands for IPSec can be found in the Windows Server 2003 Technical Libary on Microsoft TechNet. In the context of this article we will only use a few of those commands.

The goal of this article is to figure out if we can configure ESP Null Encryption on ISA 2004. We know already that we should not use the Netsh IPSec static mode commands because they give you the same management and monitoring tasks that you can perform by using the IP Security Policy Management and IP Security Monitor consoles. In addition, the ESP Encryption is a Phase II or quick mode configuration setting. Therefore, if it is possible to configure ESP Null Encryption then it should be possible with the netsh ipsec dynamic set qmpolicy command. According to the documentation we have indeed the choice between None, DES and 3DES as encryption algorithm. That’s what we expected, but first we need to find the name of the IPSec quick mode policy that the ISA MMC created when we configured the network RemoteSite#44. To find that out, use the command netsh ipsec dynamic show qmpolicy all. Here is the result in my lab environment:

C:\>netsh ipsec dynamic show qmpolicy all

QM Negotiation Policy Name : ISA Server RemoteSite#44 QM Policy

Security Methods Lifetime (Kb:secs) PFS DH Group
————————- ——————— ————
ESP[3DES,SHA1] 0:3600 Medium (2)

Now that we know the name of the quick mode policy we can try to change the encryption algorithm to None and see if it works. These are the Netsh commands for IPSec we ran on the ISA 2004 server:

  • To change the encryption algorithm to Null:

C:\>netsh ipsec dynamic set qmpolicy name="ISA Server RemoteSite#44 QM Policy" qmsecmethods=ESP[None,SHA1]:3600s

  • To verify the change:

C:\>netsh ipsec dynamic show qmpolicy name="ISA Server RemoteSite#44 QM Policy"

QM Negotiation Policy Name : ISA Server RemoteSite#44 QM Policy

Security Methods Lifetime (Kb:secs) PFS DH Group
————————- ——————— ————
ESP[NONE,SHA1] 0:3600 Medium (2)

  • To verify that a quick mode security assocation can be established:

C:\>netsh ipsec dynamic show qmsas all

Quick Mode SAs
————–

Tunnel Filter

Policy Name : ISA Server RemoteSite#44 QM Policy
Tunnel Source : 192.168.1.10
Tunnel Destination : 192.168.1.30
Source Address : 192.168.22.0
Destination Address : 192.168.44.0
Protocol : ANY
Source Port : 0
Destination Port : 0
Direction : Outbound

Offer Used

AH(b/r) ESP Con(b/r) ESP Int PFS DH Group
———- ————- ——- ————
None None SHA1 Medium (2)

Take note that the ESP encryption algorithm is shown under the heading ESP Con(b/r). Also we ran the ping command from the client (192.168.44.20) to the server (192.168.22.2) to generate meaningful traffic accross the site-to-site VPN.

Be aware that the changes we made to the IPSec policy are not persistent. The Netsh IPSec dynamic mode commands directly configure the security policy database (SPD). Changes that you make to an IPSec policy while using these commands take effect only while the IPSec service is running. If the IPSec service is stopped, the dynamic policy settings are discarded. Likewise, if you push a new ISA Firewall policy, the changes you made with the Netsh IPSec configuration commands might be overwritten by the new ISA Firewall policy. Therefore, you should only use those commands in a lab environment and maybe also to debug some typical scenarios.

Network Capture

To see what is really happening on the wire, we have taken some network traces with my favorite tool Ethereal when the client (192.168.44.20) pings the server (192.168.22.2) across the site-to-site IPSec Tunnel Mode VPN. The first capture was taken at the IPSec Gateway on the interface ‘192.168.44.1’. That means we should see the original packets as sent by the client just before the IPSec protection. Take note that you can read the ping data, abcdefghijklmnop…, as shown in the figure below.


Figure 5

The second capture was taken at the IPSec Gateway on the interface ‘192.168.1.30’. That means we should now see the client ping packet protected by the site-to-site IPSec Tunnel Mode VPN. In this case the ESP 3DES encryption algorithm was configured in the IPSec quick mode policy. As shown in the figure below, you see a completely new IP packet with, as source and destination IP addresses, the VPN tunnel endpoints (the IPSec Gateway interface ‘192.168.1.30’ and the ISA 2004 server interface ‘192.168.1.10’) and an ESP packet as data payload. The original client packet is completely buried in the data payload of the ESP packet. In other words, by looking at the packet alone we cannot even determine if the packet shown in the figure below is indeed the ESP packet carrying the original client’s ping packet.


Figure 6

The third capture was also taken at the IPSec Gateway on the interface ‘192.168.1.30’. However, in this case the ESP Null encryption algorithm was configured in the IPSec quick mode policy. That means that we should now be able to completely decode the packet structure. As shown in the figure below, you can clearly see that the original client IP packet is the data payload of an ESP packet. That ESP packet is in turn the data payload of a new IP packet with, as source and destination IP addresses, the VPN tunnel endpoints (the IPSec Gateway interface ‘192.168.1.30’ and the ISA 2004 server interface ‘192.168.1.10’). Because the ESP data payload is not encrypted, although it is still authenticated, you can read the ping data, abcdefghijklmnop…, as shown in the figure below.


Figure 7

Note:
If Ethereal does not show the complete packet structure, check if you have enabled the flag Attempt to detect/decode NULL encrypted ESP payloads in the ESP protocol preferences.

Conclusion

In this article we learned that ISA Server 2004 leverages the IPSec capabilities of Windows 2003 by creating dynamic IPSec policies to build a site-to-site IPSec Tunnel Mode VPN. With the help of the powerful Netsh IPSec commands we can enable ESP Null Encryption on ISA 2004 for this VPN scenario, although it should only be used in a lab environment and for debugging purposes.

I hope you enjoyed this article and found something in it that you can apply to your own environment. If you have any questions on anything I discussed in this article, head on over to http://forums.isaserver.org/m_2002012438/tm.htm and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks!  – Stefaan.

About The Author

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top