AuthIP – Enhancing IPsec Capabilities and Functionality


I remember when Windows 2000 Server was released. One of the things that I was looking most forward to was IPsec. With IPsec support baked right into the operating system, I would be able to secure the connections between all the computers on my network. No longer would I have to worry about someone with a network sniffer stealing passwords, users’ names and application information as it moved across the wire. While IPsec could have made that possible, it turned out to be too complex, too limited, and frankly, too much of a pain in the neck to get it to work.

With Windows Server 2003, things got to be a bit easier. While Windows 2000 Server required you to run the IPsec wizard on each of the machines on which you wanted to enabled IPsec policy, Windows Server 2003 let you use Group Policy, in the form of anIPsec Group Policy snap-in, to configure IPsec policy for Windows Server 2003 and Windows XP on the network. With Active Directory enabled Group Policy, you could scale IPsec policies. However, the same complexity in the IPsec Configuration Wizard that we had with Windows Server 2000 IPsec configuration continued on with Windows Server 2003 and Windows XP.

IPsec is part of the Windows TCP/IP stack, with the IPsec driver being a very low level driver. IPsec policy, as the name implies, is imposed at the network layer, before any of the application layer data is received by a system. When you configure an IPsec policy, you actually configure IPsec rules that determine:

  • what traffic should have IPsec protection enabled on it,
  • whether the traffic should be allowed or blocked,
  • if the traffic is allowed, whether it should require authentication or encryption, and
  • if authentication and/or encryption is required, what type of authentication or encryption should be used.

There are other settings too, such as one that defines the source and destination address or network for which the policy should be applied.

While I thought IPsec was really cool and one of the reasons to get excited about Windows 2000 Server, that excitement didn’t generalize to the general community, most likely for the reasons that I noted above. However, there seemed to be a bump in the IPsec road that shook things up a bit around 2005, when Microsoft started publishing information about Server and Domain Isolation (SDI). The vision around SDI was that you could logically segment your network to protect your domain assets from hosts of lower trust. The promise of SDI was that you could logically segment high value assets from lower value assets (or high trust assets from lower trust assets) instead of physically isolating these security zones from one another. The ability to logically segment using IPsec as the security boundary seemed very attractive and potentially a huge cost saver.

SDI sounded extremely attractive on paper. However, as they say, the devil is in the details and it turned out that the devil was alive and well. In order to create an effective SDI solution, you had to create a large number of rules, which resulted in a very complex deployment scenario.  It was complex not only in its design, but also in its ongoing management. You had to create a large number of rules, and then you had to create even more rules that created exceptions to the rules you had already created. The potential for a “tangle of rules” similar to a poorly managed datacenter’s tangle of cables was the norm and administrators lost interest in SDI over time.

Everyone knows that it sometimes takes Microsoft a few tries to get things right. They didn’t want to give up on the dream of what IPsec could deliver to IT admins, so they took a look at the problems with how IPsec was implemented and decided to fix some of those problems. This led to the development of AuthIP – an implementation of IPsec that features improvements in the Internet Key Exchange (IKE) protocol that fixed some of the most important limitations in previous versions of IPsec. AuthIP was first introduced with Vista and Windows Server 2008, and is also included in Windows 7 and Windows Server 2008 R2.

AuthIP Enhancements to IPsec

AuthIP is actually a set of extensions to the IKE protocol that includes new flags and negotiation methods to increase the utility of IPsec. AuthIP can be used with computers running Vista, Windows 7, Windows Server 2008 and Windows Server 2008 R2. And if these computers need to use IPsec to communicate with down-level computers that don’t support AuthIP, they can fail back to use IKE.

AuthIP enables the following features which were not available with previous versions of IKE:

  • Authenticates users in new ways
  • Authenticates using a variety of protocols or methods
  • Uses more efficient protocol negotiation
  • Uses multiple sets of credentials to authenticate

Authenticate Users in New Ways

If you remember how authentication worked with previous versions of IPsec, you know that authentication was based on computer credentials. You had two choices for computer authentication: computer certificate authentication or computer account authentication using Kerberos. Oh, you could also use a pre-shared key, but that’s pretty low in the food chain when it comes to scalability and security. If you wanted to control which users had access to a server application, you had to deal with user authentication at the application level – which meant untrusted users from trusted machines could potentially leverage an attack at the network level against the server hosting the application of interest.

With AuthIP, you can control access by computer account, user account or both. This forces authentication to a lower level in the TCP/IP stack so that now you don’t have to depend on the application to authenticate the user. With AuthIP, you can authenticate the user at the network layer and protect the application from network layer attacks that happen before application layer authentication takes place.

You can use the following methods of user authentication with AuthIP:

  • Computer health certificate, which is used by Network Access Protection (NAP)
  • User certificate (client certificate in the user certificate store)
  • NTLMv2 for the logged on user account
  • Kerberos for the logged on user account

Authenticate using Different Methods

AuthIP opens up deployment scenarios not available with previous versions of IPsec. For example, there are scenarios where you need to support different authentication methods on each side of the negotiation. To put forward an example, maybe one member of the IPsec peer negotiation supports certificate authentication while the other side supports computer account authentication. This scenario would fail in previous versions but is enabled by AuthIP.

For example, consider a situation where you want to use IPsec to secure communications between a server in the DMZ and a server located in the datacenter. The server in the DMZ is in a forest that has a one-way trust with the datacenter server’s forest so that the DMZ forest trusts the datacenter forest, but the datacenter forest does not trust the DMZ forest. This allows users in the datacenter forest to connect to the DMZ server using user or computer credentials, but does not allow users in the DMZ forest to connect to datacenter resources using computer or user credentials.

However, you have deployed certificates on both of the servers and each server trusts the other server’s certificate. With AuthIP you can now support scenarios where authenticating peers use different methods of authentication. In this scenario, when the datacenter server authenticates its IPsec connection to the DMZ server, it will use user or computer account Kerberos authentication. When the DMZ server authenticates to the datacenter server, it will use computer or user certificate authentication. Even though each of the IPsec peers is using a different authentication method, they can still establish an IPsec connection because the goal is successful authentication, not just successful authentication using the same authentication protocol.

Negotiate More Effectively

As I noted above, with previous versions of IPsec (IKE) you could authenticate based on computer account using either computer certificates or computer account using Kerberos. However, if the preferred method of authentication failed, then the entire IPsec negotiation failed.

Suppose you have two computers that you want to use IPsec to protect communications between them. These two computers belong to different forests and there is no trust between the forests. However, you have deployed computer certificates to both of these hosts, and populated the Trusted Root Certification Authorities’ certificate stores so that each machine trusts the other machine’s certificate. However, with previous versions of IKE, IPsec negotiations in this scenario will fail because after failing the Kerberos authentication, the potential IPsec partners will not try to use computer certificate authentication.

AuthIP fixes this problem by enabling negotiation of authentication methods. If one method fails, you can configure IPsec to use other methods, and the IPsec peers will continue to negotiate authentication methods until they find one in common. This significantly opens up the number of scenarios in which you can deploy IPsec and makes the entire solution more manageable and easier to configure.

Use More than a Single Set of Credentials to Authenticate

As you might have surmised from reading the various new scenarios supported by AuthIP, you can use multiple credentials to authenticate IPsec peers. You can authenticate IPsec peers using computer credentials, user credentials, or both user and computer credentials. If you know about DirectAccess, the new remote access technology available with Windows 7 Enterprise/Ultimate edition and Windows Server 2008 R2, you might know that the infrastructure tunnel uses computer certificate and computer account NTLMv2 authentication, while the intranet tunnel uses computer certificate and user account Kerberos authentication. By adding user credentials to the authentication mix, you can protect servers from malicious users who might be using a trusted machine.

Another scenario enabled by multiple credentials support is NAP. With NAP, a computer health certificate is used to authenticate a computer to prove that it’s a domain member and is in compliance with network health policies. These credentials can be stacked on top of computer and user account authentication if you like; AuthIP makes this possible.

Domain Controller Support Now Makes Server and Domain Isolation Possible for the Rest of Us

One of the reasons IPsec was not used much in the past was the problem of domain controllers. If you wanted to use IPsec on your network with previous versions of IPsec, you had to exempt domain controllers from your IPsec policies. The reason for this was that in order to authenticate a computer, you had to have access to the DC, and if you can’t access the DC, you can’t authenticate – so you ran into the classic “chicken and egg” paradox.

Windows Server 2008 and above and Vista and above solve this problem by enabling the following:

  • You no longer need to create exemptions for domain controllers, as you can configure IPsec to decide when to use IPsecin communicating with domain controllers.
  • Computers running Vista and above and Windows Server 2008 and above can be prompted for credentials when connecting to a domain controller – this solves the old “domain join” problem whereby you needed to exempt a computer from IPsec protection if you wanted to join a computer to the domain. By enabling prompting for credentials, non-domain computers can prompt for credentials to establish the IPsec session.
  • For down-level clients, you can easily create policies (without needing to create exemption rules) that request IPsec protection but do not require it.

These improvements in intradomainIPsec negotiations go a long way toward making IPsecdeployment actually feasible on networks today.


IPsec was first introduced with Windows 2000 Server. While it held a lot of promise, there were a number of reasons it didn’t take hold on enterprise networks. With the introduction of Vista and Windows Server 2008 (and then Windows 7 and Windows Server 2008 R2), you now have the services of AuthIP, which significantly improves the flexibility and manageability of IPsec. Now you have powerful authentication mechanisms that are easy to configure using Group Policy based management tools that allow you to create robust and powerful IPsec policies without the complexity that discouraged use of previous versions of IPsec. The combination of new user authentication options, the ability to use multiple credentials, more flexible authentication negotiation and the ability for each side to use different authentication methods makes AuthIP the salve that IPsec needed to break through and find its full potential.

About The Author

Leave a Comment

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top