Troubleshooting IPSec Tunnel Mode Scenarios


Receive “Negotiating IP Security” when testing connectivity from ISA Server across a Remote Site connection

Consider the following scenario : ISA-A has a Remote Site connection to ISA-B (or a 3rd party IPSec gateway) using IPSec Tunnel Mode.

Figure 1

After creating the Remote Site and creating the Firewall Rule to allow the Local Host Network (at the ISA firewall) access to the Remote Site, you are unable to establish a connection with any protocol. If you test connectivity with PING, you receive the Negotiating IP Security response indefinitely.


On each ISA Server (or 3rd party VPN gateway), add the external IP address of the opposing ISA firewall into the Addresses tab of the connection.


If the ISA firewall is installed on Windows 2003, you can use netsh ipsec dynamic show qmfilters all command to see the filters referenced below.

SubnetA — ISA-A — Internet — ISA-B — SubnetB

ISA-A has a Remote Site for SubnetB containing the addresses of that subnet.

This results in ISA-A having an IPSec Filter List of:

A1   SubnetA > SubnetB

A3   ISA-A > SubnetB

A2   SubnetA < SubnetB

A4   ISA-A < SubnetB

ISA-B has a Remote Site for SubnetA containing the addresses of that subnet.

B1   SubnetB > SubnetA

B3   ISA-B > SubnetA

B2   SubnetB < SubnetA

B4   ISA-B < SubnetA

When you PING from ISA-A to SubnetB, the traffic sources from ISA-A’s external IP address. Because of this, ISA-A has a matching filter for the traffic (A3 above) but ISA-B doesn’t have a matching filter for this (B1 through B4 don’t match the traffic). As a result, ISA-A continues trying to negotiate IP Security with ISA-B but this will never complete as there is not a match for the traffic on ISA-B.

To fix this, on ISA-A, you’ll need to add ISA-B’s external IP address into the Addresses tab of the Remote Site. On ISA-B, you’ll need to add ISA-A’s external IP address.

What happens is now ISA will now have the following filters…


A1 SubnetA > SubnetB

A3 ISA-A > SubnetB

A5 ISA-B > SubnetA

A2 SubnetA < SubnetB

A4 ISA-A < SubnetB

A6 ISA-B < SubnetA


B1 SubnetB > SubnetA

B3 ISA-B > SubnetA

B5 ISA-A > SubnetB

B2 SubnetB < SubnetA

B4 ISA-B < SubnetA

B6 ISA-A < SubnetB

With this setup, when ISA-A tries to communicate with SubnetB, A3 now matches B5 and A4 matches B6 and the Security Associations can come online.

About The Author

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top