Troubleshooting Kerberos in a Sharepoint Environment (part 3)

If you would like to read the other parts in this article series please go to:

• Troubleshooting Kerberos in a Sharepoint Environment (part 1)
• Troubleshooting Kerberos in a Sharepoint Environment (part 2)

Introduction

I would like to start this section with explaining what Kerberos Delegation is and when we need to configure it. In past articles I included the configuration steps but delegation is not always needed because it depends on your scenario, application design and security requirements.

Kerberos Delegation makes it possible to pass on an authenticated account through multiple servers by use of impersonation. This problem is often referred to as “double-hop issue” and you will experience this when using, for example, Excel Calculation Services (ECS). When we configure the trust for delegation and use an impersonation of the user accounts credentials, we ensure that the correct level of access is maintained throughout the system. Another example could be seen when a user requests data from a web application. The web application makes a query to a SQL database on another physical server and therefore uses impersonation. If delegation and impersonation is used, the SQL server only returns the data the user has access to.

Figure 22: The two physical hops leading to the double-hop issue

Let us set up a test environment for this and see what kind of errors we will see if delegation is not configured correctly. I have a working aspx-page that displays data from a custom database on the SQL server.

Figure 23: Working query to the SQL server

Kerberos Delegation is already configured correctly in our demo environment, but let us see what happens if we remove the trust for delegation rights for the Application Pool account; SPContentPoolAcct. Figure 24 shows the current configuration for this account.

Figure 24: Correct setup of constrained delegation

I will now try to remove the trust rights for the MSSQLSvc service highlighted in yellow in the figure and perform a IISRESET /NOFORCE on the SharePoint server WSS1. With these rights missing the SPContentPoolAcct account can no longer impersonate the user credentials and present these to the SQL server. The user now gets the Microsoft SharePoint standard error shown in Figure 25.

Figure 25: SharePoint page returning standard custom error message

Okay, this is a troubleshooting article so we better start doing that. First of all, we can not do much with this standard error message. As an administrator we can disable this in the web.config file found in the file system of the Microsoft SharePoint Web FrontEnd server (WFE).

The test environment this is located here: C:\inetpub\wwwroot\wss\VirtualDirectories\intranet.domain.local80

Disable Custom Error messages in Microsoft SharePoint

Note:
This is done on every Microsoft SharePoint WFE server you have

1. Make a copy of the web.config file before editing it (just to be sure nothing breaks too much)
2. Open it with our favorite text-editor Notepad
3. Search for <customErrors mode=”On” /> and change this to <customErrors mode=”Off” />
4. Search for CallStack=”true” and change this to CallStack=”false”
5. Restart your Internet Information Server (IIS) with the command line: IISRESET /NORFORCE

Now let us take a look on the page again:

Figure 26: SharePoint page returning detailed error message

This gives us a bit more information about the problem and the page clearly indicates that it is a SQL client login error with the message: System.Data.SqlClient.SqlException: Login failed for user ‘NT AUTHORITY\ANONYMOUS LOGON’

An anonymous logon was attempted by the .NET code and therefore our user account (DOMAIN\Administrator was logged on to the workstation attempting to load the page) was not used in this process. For more information on this issue we can take a look at the Application event log at the SharePoint server (we have only one in this scenario, but if you have multiple WFE servers in a Network Load balanced Network (NLB) you need to check every single one to find your error message).

In the Event Viewer on WSS1 we find a warning event:

Warning, ASP.NET 2.0.50727.0, Event ID: 1039, Category:Web Event
Event code: 3005
Event message: An unhandled exception has occurred
…
…
User: DOMAIN\Administrator
Is authenticated: True
Authentication Type: Negotiate
Thread account name: DOMAIN\spcontentpoolacct
Thread information:
Thread ID: 4
Thread account name: DOMAIN\spcontentpoolacct
Is impersonating: False
…
…

Figure 27: Warning from event log on WSS1 SharePoint WFE

The information in the event viewer shows us that the .NET throwed a web warning event. The user who activated this was the authenticated user DOMAIN\Administrator and the thread account is DOMAIN\spcontentpoolacct (our web application pool user). It also shows us that the thread user is not impersonating the actual user account. So, impersonation is not used for the .NET code and this is the problem (Is impersonating: False).

Check the Kerberos delegation configuration which includes:

• Service Principal Names (SPN) in Active Directory
  – Service type (MSSQLSvc)
  – Server name (SQL and SQL.DOMAIN.LOCAL)
  – Port, if used (1433)
  – Account that it is registered to (DOMAIN\SQLSvcAcct because this is configured to run the SQL Server service)
• Delegation configuration for the application pool account (DOMAIN\SPContentPoolAcct so this is allowed to pass the credentials to the SPN service, port and account)
• .NET code (authentication must be set to trusted and use impersonation)

Note:
Notice that I left out the delegation configuration on the DOMAIN\SPContentPoolAcct. Simply add the information and restart the IIS services on the SharePoint WFE server.

Kerberos for the Shared Service Provider (SSP)

Before the Infrastructure Update (July 2008) for Microsoft Office Servers was released Kerberos authentication was not fully supported for the Shared Service Provider. This gave some issues with e.g. multiple SSPs and search functionality. Therefore I recommend you to install this update on your systems if you have not done this already. Microsoft has published some cumulative updates later that includes this infrastructure update, so please check the Microsoft Team Blog website for the lastest recommended upgrade path. You can find a link to this blog links section last in this article.

In this Infrastructure Update Microsoft added a special Service Principal Name (SPN) format: MSSP/<host:port>/<SSP name>.

For the configuration details for the SSP using this new SSP format please visit the TechNet website mentioned in the links section of this article. I include a quick summarization of the Kerberos configuration steps mentioned in the TechNet article for the Shared Services Provider.

Shared Services Provider (SSP)

• Register the Shared Service Provider SPNs  in Active Directory

• Change the SSP to use Kerberos authentication using the STSADM command line tool

• Make a change in the registry on all MOSS 2007 servers to use the new SPN format for the SSP

• Confirm that the Kerberos authentication works for the root and virtual-directory shared web services access

Figure 28: Service Principal Name registration for the SSP

Miscellaneous issues and notes

User and SharePoint services accounts

Check if the user or SharePoint accounts are not locked in the domain or have an expired password. A company policy could be implemented in your environment if this is the case.

Computer account secure connection

Some clients/servers fail to set up a correct secure connection with the domain and Kerberos is built on this type of secure infrastructure. If this happens you need to reset and rebuild this relationship. Microsoft Knowledgebase article KB216393 (see links section at the end of the article) has the instructions for this procedure.

If your server/client has been cloned you need to generate a new security ID (SID) and the recommended way to do this is to run the Microsoft sysprep-utility. Another way is to use the former Sysinternals, now Microsoft, utility NewSID.

Issues with the MTU Size

The network packets that are sent through the wires have a certain length. If an account is member of a large number of groups this can been seen. Another way to deal with the MTU-problem is to force Kerberos to use TCP. You can find information about this in Microsoft knowledgebase article KB244474 (see links section at the end of the article).

Multiple Web applications with different authentication methods

Disable negotiation on web applications if they do not need or use Kerberos authentication. If you have multiple web applications on the same server and uses Kerberos on some web applications and NTLM you can experience issues. Especially if the web applications uses the same server-name and only different port-numbers – and using different application pool accounts with these usually starts a few hours of work troubleshooting the systems. I have only seen these types of setups in small deployments though.

I would like to recommend some guidelines for web applications

• Use unique host-headers
  e.g. intranet.domain.local instead of the servername of the SharePoint server
  
  

• Use unique IP addresses for host headers
  – this makes it easier to load balance later
  
  

• Use different Application Pool accounts for different web applications
  – For security purposes please create your own. If you use the NETWORK SERVICE account the computer-account is the account used by the Web Application to identify the SPN.
  
  

• Use Network Load Balancing (NLB) for your WFE servers
  – you get fault tolerance and better performance
  
  

• Enable negotiation when using Kerberos. Disable it when using NTLM
  – Check your web application using the command line tool adsutil.vbs
  

Check if Kerberos is configured for your web application

Start a Command Prompt.

cd c:\inetpub\adminscripts
cscript adsutil.vbs get w3svc/<id of your website>/root/NTAuthenticationProviders

(you will find the I.D of your website through IIS Management under identifier)
For Kerberos the result is: NTAuthenticationProviders : (STRING) “Negotiate,NTLM”
For NTLM the result is: NTAuthenticationProviders : (STRING) “NTLM”

Configuring the IIS website with another method (in this example to Kerberos) use the following command:

cd c:\inetpub\adminscripts
cscript adsutil.vbs set w3svc/<id of your website>/root/NTAuthenticationProviders “Negotiate,NTLM”

This is needed on every SharePoint web frontend server.

SQL Server 2005 script to check users auth. method

I made this SQL script to check what kind of authentication methods are used for the databases. This is very useful when you need to make sure everything is working as you expect it to.

SELECT DB_NAME(dbid) AS DatabaseName, loginame AS LoginName, sys.dm_exec_connections.auth_scheme as AuthMethod
FROM sys.sysprocesses
JOIN sys.dm_exec_connections
ON sys.sysprocesses.spid=sys.dm_exec_connections.session_id
WHERE dbid > 0
GROUP BY dbid, loginame, spid,sys.dm_exec_connections.auth_scheme

The output of this in my test environment (including the CompanyDatabase example above) is shown in Figure 29 below.

Figure 29: Output from the script in the SQL Management Studio

Conclusion

In this article I discussed Kerberos delegation, impersonation and Shared Service Provider (SSP) configuration. I shared my “notes from the field” and scripts to check the authentication method used – this can be used with SharePoint and other configurations as well.

Kerberos configuration and especially troubleshooting various issues can be a lot of work. My task was to give you some more insight in the techniques to pinpoint the reasons for the issues and solve them. I hope you see this task completed and also will give me feedback.

Links

• MSDN: Microsoft SharePoint Team Blog
• Microsoft Technet: Configuring Kerberos authentication (Microsoft Office Servers)
• Microsoft Knowledgebase: How to force Kerberos to use TCP instead of UDP in Windows
• Microsoft Knowledgebase: Resetting computer accounts in Windows
• Joel Olsons’ Blog: Protocols, ports and firewall rules

If you would like to read the other parts in this article series please go to:

• Troubleshooting Kerberos in a Sharepoint Environment (part 1)
• Troubleshooting Kerberos in a Sharepoint Environment (part 2)