Troubleshooting IPsec based VPN’s can be very challenging. The reason for this is that you need to have a thorough knowledge of the IPsec protocol. Moreover, how to enable and interpret the more advanced IPsec diagnostic logging, more precisely the Oakley log, is not very well documented. In this short article we will summarise some troubleshooting steps you can apply to the IPsec part of the VPN. However, it is outside the scope of this article to explain how to read the Oakley log.
Before diving into the details, it’s necessary to have a good understanding of how the IPsec protocol works. Of course the IETF RFC documents are the authorative resource but they are probably too boring to read for most people. Instead, we can recommend the excellent Technet article series IPSec Technical Reference. This is definitely a must read! When you only want to refresh your knowledge on how the IKE Negotiation happens, check out the Cable Guy articles IKE Negotiation for IPsec Security Associations and IPsec NAT Traversal Overview, as well as section 2. IPSec NAT Traversal in my article How to pass IPSec traffic through ISA Server.
1. Enable IKE Auditing
The Windows Security Log is the recommended starting point when trying to determine the reason for an IKE negotiation failure. To view these IKE events, enable success and failure auditing for the Audit logon events audit policy for your domain or local computer. The IKE event category is also used for auditing user logon events in services other than IPsec.
When you enable success or failure auditing for the Audit logon events audit policy, IPsec records the success or failure of each Main mode and Quick mode negotiation and the establishment and termination of each negotiation as separate events. Keep in mind that enabling this type of auditing can cause the security log to fill with IKE events.
In Windows XP SP2 and Windows Server 2003, all IKE audits can be disabled with a DisableIKEAudits registry key. Be sure to check this value on computers that are being investigated. To disable auditing of IKE events in the security log, even if the Audit logon events audit policy is enabled, do the following:
set the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Audit\DisableIKEAudits registry key to a value of 1.
Deleting the DisableIKEAudits registry key or setting the value to 0 restores the default setting. After making a change to this registry key, you must either restart the computer or stop and then restart the IPSec service by running the net stop policyagent and net start policyagent commands at the command prompt. Note that stopping and restarting the service may disconnect all computers using IPsec from the computer on which the service is stopped.
In Windows Vista, the IKE audits can granulary be enabled or disabled with the auditpol.exe commandline tool. Moreover, no IPsec service must be restarted before the new settings become active. To see all possible Audit logon events subcategories, use the following command at the command prompt:
auditpol /get /category:”Logon/Logoff”
To change the audit settings for one of the IKE subcategories, use the following command at the command prompt:
auditpol /set /subcategory:<name> /success:<enable>|<disable> /failure:<enable>|<disable>
Now that we know how to enable and fine tune the IKE auditing, how does a normal IKE audit looks like in Windows XP SP2, Windows Server 2003 and Windows Vista?
In Windows XP SP2 and Windows Server 2003, an IKE audit for a successful L2TP/IPsec VPN connection shows the following sequence of events:
- ID 541: IKE security association established, Mode: Key Exchange (Main mode).
- ID 541: IKE security association established, Mode: Data Protection (Quick mode).
- ID 542: IKE security association ended, Mode: Data Protection (Quick mode).
- ID 543: IKE security association ended, Mode: Key Exchange (Main mode).
When something goes wrong during the IKE negotiation, one or more of the above event ID’s will be missing. Instead you will see some IKE failure audit events such as:
- ID 544: IKE security association establishment failed because peer could not authenticate. The certificate trust could not be established.
- ID 545: IKE peer authentication failed.
- ID 546: IKE security association establishment failed because peer sent invalid proposal.
- ID 547: IKE security association negotiation failed.
In Windows Vista, an IKE audit for a successful L2TP/IPsec VPN connection shows the following sequence of events:
- ID 4650: An IPsec Main Mode security association was established. Extended Mode was not enabled. Certificate authentication was not used.
ID 4651: An IPsec Main Mode security association was established. Extended Mode was not enabled. A certificate was used for authentication.
- ID 5451: An IPsec Quick Mode security association was established.
- ID 5452: An IPsec Quick Mode security association ended.
- ID 4655: An IPsec Main Mode security association ended.
When something goes wrong during the IKE negotiation, one or more of the above event ID’s will be missing. Instead you will see some IKE failure audit events. However, we didn’t find a list of all possible IKE failure audit events. Therefore we can only list those we have seen during our limited testing with Vista L2TP/IPsec VPN connections:
- ID 4652: An IPsec Main Mode negotiation failed. Failure Reason: IKE authentication credentials are unacceptable.
- ID 4653: An IPsec Main Mode negotiation failed. Failure Reason: New policy invalidated SAs formed with old policy.
As you can see, Windows Vista include new IPsec audit-specific events and the text of existing events has been updated with more useful information. One of the goals of these improvements is to help you troubleshoot failed IPsec negotiations without having to enable the advanced Oakley logging capability.
2. View IKE/IPsec Security Associations and Statistics
In Windows XP SP2, Windows Server 2003 and Windows Vista, IP Security Monitor is implemented as a Microsoft Management Console (MMC) snap-in. IP Security Monitor allows you to view details about an active IPsec policy that is applied by the domain or locally, and to view quick mode and main mode statistics, as well as IPsec security associations (SAs). IP Security Monitor also enables you to search for specific main mode or quick mode filters.
Another way of viewing the same information is by using the commandline tools IPseccmd (Windows XP SP2) or Netsh (Windows Server 2003 and Windows Vista).
IPseccmd is a command-line alternative to the IP Security Policy MMC snap-in for Windows XP. In Windows XP SP2 the tool provides additional functionality and must be installed from the Support Tools folder on the Windows XP SP2 CD. Some useful commands are:
ipseccmd show sas
ipseccmd show stats
Netsh is a command-line scripting utility that allows you to display or modify the network configuration. In Windows Server 2003 and Windows Vista the tool is enhanced to provide IPSec diagnostic and management functionality and replaces therefore the IPseccmd in Windows XP. Some useful commands are:
netsh ipsec dynamic show mmsas all
netsh ipsec dynamic show qmsas all
netsh ipsec dynamic show stats
3. Use Network Monitor and IKE Tracing
When you really want to know how the IPsec stuff works on the wire or when you have to analyse IKE negotiation failures, you should be handy with a network monitor tool and learn to explore the Oakley log. Both skills are required to be able to diagnose the hard to find problems. The most popular and free network monitor tools are Wireshark and Microsoft Network Monitor 3. Which one you use isn’t that important. Just make sure you know how to use the features available in the tool.
As an example, an excerpt of a ‘Network Monitor 3’ frame summary for a successful L2TP/IPsec VPN connection:
In the above figure you see clearly the important phases in the startup of an IPsec session. First you have the main mode negotiation shown in the frames 1 – 8. Second you find the quick mode negotiation in the frames 9 – 12. At last you have the data transfer (ESP packets) starting from frame 13 onwards. Take note that in the case of a L2TP/IPsec VPN connection the L2TP negotiation starts at the first ESP packet (frame 13 in this example).
Additional you can see that message 5 and 6 in the main mode negotiation are fragmented in two packets (frames 5-6 and 7-8). That’s normal behavior because the identification is generally based on certificates and they are usual too big to fit in one packet. Therefore, IP fragments must be supported along the path. Also, you can observe that from frame 5 onwards the flag “E” is set. This means that the payload is encrypted. In other words, you can’t inspect the content of those payloads. This shows clearly a major limitation in using a network monitor tool for debugging IPsec traffic. Nevertheless, it’s an excellent tool to check out the general packet flow up to and including layer-4 (e.g. NAT-T Encapsulation) as shown in the figure below:
When troubleshooting IPsec and taking network traces between hosts, especially in a lab environment, it is considered best practice to lower the ESP encryption level (if it’s currently at DES/3DES) to ESP-Null. This will help in reading and understanding the network trace much better than going through the encrypted traffic capture. One method to accomplish that is described in my article How to enable ESP Null Encryption on ISA 2004 in a site-to-site VPN scenario.
As a last resort, you can enable IKE tracing or Oakley logging to analyse IKE negotiation failures. The Oakley log records all IKE (ISAKMP) main mode and quick mode negotiations in great detail. In Windows XP SP2 and Windows Server 2003, the Oakley log is stored in the systemroot\Debug folder. A new Oakley.log file is created each time the IPsec policy agent is started and the previous version of the Oakley.log file is saved as Oakley.log.sav.
To enable Oakley logging in Windows XP SP2, use the following command:
ipseccmd set logike|dontlogike
To enable Oakley logging in Windows Server 2003, use the following command:
netsh ipsec dynamic set config ikelogging 1|0
As already mentioned, one of the goals of the IKE audit improvements in Windows Vista is to help you troubleshoot failed IPsec negotiations without having to enable the advanced Oakley logging capability. Apparently Microsoft decided to push that idea to such an extreme they removed the possibility to enable or disable the Oakley logging in the netsh command, though it is still listed in the usage of the netsh command.
To enable Oakley logging in Windows Vista, do the following (of course you need elevated privileges):
- Create a local \Tools directory and copy in TRACEFMT.exe and TRACEPRT.dll from the Windows XP SP2 Support Tools as a free download.
- Set the registry key HKLM\SYSTEM\CurrentControlSet\Services\Ikeext\Parameters\EnableLogging = DWORD 0x1.
- Stop/Start the IKEEXT service. Do not use “restart” because that does not have the same effect. Alternatively, run “NET STOP IKEEXT” and “NET START IKEEXT” at a command prompt.
- Repro the problem by initiating the IPsec based VPN connection that you wish to troubleshoot.
- Verify the log file was created in the directory %windir%\system32\ikeext.etl and has flushed the memory buffers into the file (e.g. make sure the file is not a 0 byte value).
- Run the following command from a command prompt with the working directory set to the \Tools directory:
tracefmt.exe %systemroot%\system32\ikeext.etl -tmf %systemroot%\System32\wfp.tmf -o %TEMP%\WFPDiag.txt
- Review the WFPDiag.txt file for Oakley-level data. Given the addition of AuthIP for transport mode IPsec policies from the Windows Firewall with Advanced Security, the file is a little busier than it used to be.
- Set the registry key HKLM\SYSTEM\CurrentControlSet\Services\Ikeext\Parameters\EnableLogging = DWORD 0x0.
- Stop/Start again the IKEEXT service. Do not use “restart” because that does not have the same effect. Alternatively, run “NET STOP IKEEXT” and “NET START IKEEXT” at a command prompt.
- Optional: Delete the ‘%systemroot%\system32\ikeext.etl’ file.
To prove the above procedure really works, here is a small excerpt of an Oakley logging on a Windows Vista machine:
For further troubleshooting information, check out the following articles:
- IPSec Tools and Settings
- IPSec troubleshooting tools
- Troubleshooting VPN over IPsec
- Server and Domain Isolation Using IPsec and Group Policy – Chapter 7: Troubleshooting IPsec