Troubleshooting Kerberos in a Sharepoint Environment (part 2)

If you like to read the other parts in this article series please go to:

• Troubleshooting Kerberos in a Sharepoint Environment (part 1)
• Troubleshooting Kerberos in a Sharepoint Environment (part 3)

Continuing the investigation

In the first part we took a look on troubleshooting time and date issues, application pool accounts and the basic Service Principal Name (SPN) configuration. In this part will take a look at the following topics:

• SPN Configuration – note for IIS 7
• Duplicate Service Principal Names
• DNS Configuration mismatch

SPN Configuration – note for IIS 7

It is always nice to get feedback on the articles and a couple mentioned that they could not get websites to work with Kerberos if they used Internet Information Server 7 on Windows Server 2008. The application pool account and SPN registration was configured correct, but this good old error message still popped up in the Windows System Event Log: KRB_AP_ERR_MODIFIED.

This problem occurs because Kernel-mode authentication is enabled by default, causing the Kerberos tickets to be decrypted with the SharePoint web frontend machine account. If you are running a single server SharePoint installation and register your SPN to the servers NETBIOS name you are fine. But, in a web farm you need to either disable the Kernel-mode authentication or configure the web application to use the credentials from the application pool.

Let us take a look at our environment from part 1 of this article that I will use in the example.

Information about IP addresses used:

172.16.189.11 is the domain controller (and the KDC) DC1
172.16.189.15 is the SQL Server (and the KDC) SQL1
172.16.189.20 is the website http://intranet.domain.local
172.16.189.21 is the SharePoint Server WSS1
172.16.189.22 is the SharePoint Server WSS2
172.16.189.101 is the computer PC1 accessing the website

I disabled the Kernel-mode authentication in my environment and for this test, I simply enabled it again (as it is the default mode) in Internet Information Manager 7:

1. In IIS 7 Manager select the “Sites/<name of your SharePoint website>” and choose “Authentication”.

Figure 7

2. Choose “Windows Authentication” and click on the “Advanced Settings”

Figure 8

3. I ticked the “Enable Kernel-mode authentication” which was the default setting

Figure 9

4. I then reset my IIS with the command: IISRESET /NOFORCE

We want to see some more packet details, so, before trying to access the website I start up Wireshark, a network protocol analyzer, on DC1 and PC1 so we can investigate the Kerberos errors on the wire. I recommend setting a filter to display only the Kerberos packets:

Figure 10

Now I will access the website: http://intranet.domain.local – here, I will get a logon box because of a failed login attempt. First step is to check the event log on the computer accessing the website. This is the event from the Windows System Event Log on the computer accessing the website checked on PC1:

Figure 11

If we take a look at the packets and error captured on PC1 with Wireshark we see the same error:

Figure 12

What we can see, in both the Windows Event Log and the packets captured, is that we get a KRB_AP_ERR_MODIFIED and the account replying is wss1$. We know that the server WSS1 reports this error when we access the website. We can also see this from the IP address if we look at the source/destination IP address information in Wireshark. Let us take a look on this server. The KRB_AP_ERR_MODIFIED means that the computer thinks that the Client/Service exchange packet seems to be changed and the parameters that are checked are the date/time, IP addresses, hostnames and if the decryption key works. We quickly check the date/time, IP addresses and hostnames (see DNS configuration section) and these are (supposedly) correct. The encryption/decryption key is determined by the SPN – account mapping. This account must be the account the IIS website on the server WSS1 is using.

We check this out with the following command:

Figure 13

The Service Principal Name maps to the domain account SPContentPoolAcct and lets us check the IIS Application Pool that the website uses. In the IIS manager locate the Application Pool used on the IIS Website by. Then check the Advanced Settings for this – in figure 14 this shows that the account is configured correctly.

Figure 14

But because of the new structure in IIS 7 on Windows Server 2008 this is only used if Kernel mode authentication is deactivated or the host configuration is altered.

We check and correct the configuration file using this method:

1. First we open the configuration file on every SharePoint web frontend server: %WinDir%\System32\inetsrv\config\ApplicationHost.config
2. Then we check that the useAppPoolCredentials setting is present. If not we add this attribute like in the example highlighted in green below (REMEMBER to type exactly like the text and only the A, P and C are in capital letters):
   <system.webServer>
   <security>
   <authentication>
   <windowsAuthentication enabled=”true” useKernelMode=”true” useAppPoolCredentials=”true” />
   </authentication>
   </security>
   </system.webServer>
3. Lastly, we reset the Internet Information Server from a command prompt:
   IISRESET /NOFORCE

We are now able to access the website again with no login prompts or Event log errors. For the record I just include the correct Wireshark protocol details captured from the client computer PC1:

Figure 15

Checking for duplicate Service Principal Names

It is easy to create duplicate SPNs in Active Directory unless you use the setspn.exe command with the “-S” or “-F” switch (only the setspn.exe from Windows Server 2008 and higher). For a better understanding of how the Service Principal Names are stored and how the KDC look up the account based on the SPN I made the diagram in figure 16. An example of a duplicate SPN registered on SPWrongAcct is highlighted in red.

Figure 16

Having duplicate SPNs can give intermittent error messages saying KRB_AP_ERR_MODIFIED, as the KDC could encrypt the service ticket with the public key of the account the SPN was found on – which could be another account that the application pool uses to decrypt the packet with. Checking for duplicate SPNs is highly recommended in any environment and this can be done in several ways.

• ldifde – extract SPNs directly and filtered from Active Directory
  Getting accounts where the HTTP SPNs is defined:
  ldifde -d “dc=domain,dc=local” -r “servicePrincipalName=http*” -p subtree -l “dn,servicePrincipalName” -f output.txt
  Getting accounts where the MSSQLSvc SPNs is defined:
  ldifde -d “dc=domain,dc=local” -r “servicePrincipalName=mssqlsvc*” -p subtree -l “dn,servicePrincipalName” -f output.txt
• setspn.exe – on Windows Server 2008 it is possible to check for duplicates
  Getting the account where a HTTP SPN is defined:
  setspn.exe -Q http/intranet.domain.local
  Checking for a HTTP SPN registered on multiple accounts (duplicate SPNs):
  setspn.exe -X http/intranet.domain.local
• ADSIEdit – manually go through the user and computer accounts one by one and check that the value does not exist on multiple accounts. This approach is hard work so I would choose one of the other methods.

Figure 17

The thing to look for in the output of any of the above procedures is that the SPN (e.g. HTTP/intranet.domain.local) is not listed on more than one account. The account itself can have multiple SPNs listed without any problems (see Figure 17).

I recommend using the following commands if you run Windows Server 2008:

• “setspn.exe -S” for registering a SPN on an account and check for duplicates in the domain
• “setspn.exe -F -S” for registering a SPN on an account and check for duplicates forest-wide

I made a screenshot of checking for duplicates and configure a Service Principal Name (Figure 18) to the DOMAIN\SPContentPoolAcct account

Figure 18

The next screenshot is showing an attempt to create a duplicate SPN. The setspn.exe finds this duplicate is shown in Figure 19 and even tells you which Active Directory object this conflicts with.

Figure 19

DNS configuration

A good solid DNS configuration must be in place in every domain today and when using Kerberos this is no exception. All hostnames must be created in the forward and reverse lookup zones and no duplicates on either the hostname or IP address is permitted. In the forward lookup zone the records should be created as A-records – also on host headers that point to servers. If CNAME is used sometimes Kerberos will construct tickets using the wrong hostname – so basically a rule against the best practices in DNS configuration to make it work.

The information send and received will be checked both with forward and reverse lookup. I will test my configuration first to see if the DNS responds correctly to my website hostname.

Figure 20

Both commands must return correct host-name and IP address – but a good idea is to check the DNS configuration manually. You need to be sure that all hostnames in your setup (servers DC1, SQL1, WSS1 and computer PC1) and used host-headers (intranet.domain.local) are created and are unique. Below are screenshots from our setup’s DNS configuration shown in Figure 21 and Figure 22.

Forward Lookup Zone for domain.local:

Figure 21

Reverse Lookup Zone for domain.local:

Figure 22

Conclusion

If you like to read the other parts in this article series please go to:

• Troubleshooting Kerberos in a Sharepoint Environment (part 1)
• Troubleshooting Kerberos in a Sharepoint Environment (part 3)