Troubleshooting IPSec Tunnel Mode Scenarios

Symptom


Receive “Negotiating IP Security” when testing connectivity from ISA Server across a Remote Site connection


Consider the following scenario : ISA-A has a Remote Site connection to ISA-B (or a 3rd party IPSec gateway) using IPSec Tunnel Mode.




Figure 1


After creating the Remote Site and creating the Firewall Rule to allow the Local Host Network (at the ISA firewall) access to the Remote Site, you are unable to establish a connection with any protocol. If you test connectivity with PING, you receive the Negotiating IP Security response indefinitely.


Solution


On each ISA Server (or 3rd party VPN gateway), add the external IP address of the opposing ISA firewall into the Addresses tab of the connection.


Description


If the ISA firewall is installed on Windows 2003, you can use netsh ipsec dynamic show qmfilters all command to see the filters referenced below.



SubnetA — ISA-A — Internet — ISA-B — SubnetB


ISA-A has a Remote Site for SubnetB containing the addresses of that subnet.


This results in ISA-A having an IPSec Filter List of:












A1   SubnetA > SubnetB


A3   ISA-A > SubnetB


A2   SubnetA < SubnetB


A4   ISA-A < SubnetB


ISA-B has a Remote Site for SubnetA containing the addresses of that subnet.












B1   SubnetB > SubnetA


B3   ISA-B > SubnetA


B2   SubnetB < SubnetA


B4   ISA-B < SubnetA


When you PING from ISA-A to SubnetB, the traffic sources from ISA-A’s external IP address. Because of this, ISA-A has a matching filter for the traffic (A3 above) but ISA-B doesn’t have a matching filter for this (B1 through B4 don’t match the traffic). As a result, ISA-A continues trying to negotiate IP Security with ISA-B but this will never complete as there is not a match for the traffic on ISA-B.


To fix this, on ISA-A, you’ll need to add ISA-B’s external IP address into the Addresses tab of the Remote Site. On ISA-B, you’ll need to add ISA-A’s external IP address.


What happens is now ISA will now have the following filters…


ISA-A














A1 SubnetA > SubnetB


A3 ISA-A > SubnetB


A5 ISA-B > SubnetA


A2 SubnetA < SubnetB


A4 ISA-A < SubnetB


A6 ISA-B < SubnetA


ISA-B














B1 SubnetB > SubnetA


B3 ISA-B > SubnetA


B5 ISA-A > SubnetB


B2 SubnetB < SubnetA


B4 ISA-B < SubnetA


B6 ISA-A < SubnetB


With this setup, when ISA-A tries to communicate with SubnetB, A3 now matches B5 and A4 matches B6 and the Security Associations can come online.

Leave a Comment

Your email address will not be published.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top