I observed that if an IPSec tunnel mode site-to-site VPN is used between two ISA 2004 servers, or between two Windows 2003 RRAS servers, or between an ISA 2004 server and a Windows 2003 RRAS server, then every 5 minutes the Quick Mode Security Association is deleted (Event ID 542) and a complete new Quick Mode Security Association is renegotiated (Event ID 541), although the Security Association (SA) lifetime is 3600 seconds (default value) and despite there is traffic all the time within the tunnel (i.e. ping -t).
Value name: SAIdleTime
Data Type: REG_DWORD
Value data: 300 – 3600 (default=300)
Back in Januari 2006, I logged a case with Microsoft PSS to fix that problem. Microsoft PSS confirmed that on a Windows 2003 SP1 based server the Quick Mode SA idle timeout function was indeed broken. In fact, the Quick Mode SA idle timeout function no longer resets the timer if there is actual traffic passing for that Quick Mode SA. In other words, by default every 300 seconds (SAIdleTime value) the timeout function will trigger a drop of the Quick Mode SA regardless if there is traffic or not. When new traffic is seen, a new Quick Mode SA will be negotiated. To workaround the problem Microsoft PSS advised to change the SAIdleTime value to 3600 seconds.
Shortly thereafter, I got a private fix for the Windows 2003 SP1 ipsec.sys driver that solves the problem. I was told at that time that this fix would be released with Windows 2003 SP2. However, I noticed that on August 22, 2006 Microsoft released the Knowledge Base article KB923339 FIX: The client connections are dropped frequently when you use the IPsec tunnel through a NAT device on a Windows Server 2003 Service Pack 1-based server. Because the mentioned hotfix contains an update of the ipsec.sys driver and has the keyword “kbwinserv2003presp2fix”, it sounds worth the effort to find out if this hotfix would solve de “Quick Mode SA idle timeout” problem too. So, I have requested that hotfix, installed and tested it in my lab and can confirm it effectively solve the “Quick Mode SA idle timeout” problem. Because this was a Windows 2003 SP1 issue, I think this fix should apply to all ISA Server versions, though I tested it with ISA 2004 SP2 only.