Active Directory Insights (Part 12) – MaxTokenSize

If you would like to read the other parts in this article series please go to:

• Active Directory Insights (Part 1): Configuring DNS on domain controllers
• Active Directory Insights (Part 2): Digging into Trusts
• Active Directory Insights (Part 3): Re-examining read-only domain controllers
• Active Directory Insights (Part 4): More on read-only domain controllers
• Active Directory Insights (Part 5) – Domain controller hardware sizing
• Active Directory Insights (Part 6) – Domain controllers and NIC teaming
• Active Directory Insights (Part 7) – More on using virtual domain controllers
• Active Directory Insights (Part 8) – Virtual domain controllers and disaster recovery
• Active Directory Insights (Part 9) – Automating user account provisioning
• Active Directory Insights (Part 10) – DHCP and domain controllers
• Active Directory Insights (Part 11) – sIDHistory
• Active Directory Insights (Part 13) – Digging into the Global Catalog
• Active Directory Insights (Part 14) – More about the Global Catalog

Introduction

In the previous article of this series our guest expert Justin Harris explained the basic concepts of SID and sIDHistory as they apply to Active Directory environments and how certain problems can arise associated with the MaxTokenSize parameter. In this present article Justin continues along these lines by explaining MaxTokenSize in detail and discussing various remediation steps that Active Directory administrators can perform to ensure the resiliency of their environments. Justin is a Microsoft Most Valuable Professional (MVP) and Microsoft Exchange Certified Master who has worked with a global customer base over the last 18 years to deliver enterprise-level messaging, unified communications, and cloud-enabled virtualization solutions. He is one of a handful of professionals globally who holds the MCM/MCSM and MVP designations with Microsoft for Exchange. Justin can be found on Twitter at @ntexcellence and on the web at http://www.ntexcellence.com.

What is MaxTokenSize?

The traditional user authentication protocol within a Microsoft infrastructure since the Windows 2000 time frame has been Kerberos. The Kerberos token leverages a predefined buffer to house authorization requests. This predefined Kerberos buffer size is set by the MaxTokenSize setting found in the registry here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters

Essentially, Kerberos uses this authorization buffer to allow protocols like HTTP to set memory allocation for authentication duties. So the MaxTokenSize setting will instruct Windows how large an authentication request using a protocol like HTTP, for instance, can be before the request fails.

Through the years, different types of Windows operating systems have had different default MaxTokenSize buffer sizes. The buffer setting has changed even for different patch revisions of the same operating system as you will see in the table below. The default MaxTokenSize buffer size since the Windows 2000 time frame up to Windows 7/2008 R2 was 12,000 bytes. The release of Windows 8/2012 bumped the default MaxTokenSize buffer up to 48,000 bytes.

Table 1

So what happens when a user tries to present an access token that is larger than the MaxTokenSize buffer defined on a remote Windows server? The user’s access token (key ring) will be sized down automatically to fit the configured buffer size. This means that SIDs (keys) will be stripped off the access token (key ring) during the request. The SID required to access a resource may be filtered out and result in an access denied message to the user even though they really have appropriate rights on the access control list.

Another common problem that results in MaxTokenSize buffer issues is when users are added to a large number of groups. Most companies will continue to grow through the years and in my experience it is not uncommon to see MaxTokenSize issues after a user has been migrated several times and group membership cleanup is left unchecked. Once the user’s Kerberos token grows and surpasses the MaxTokenSize buffer they will begin to see error messages from applications that leverage Kerberos for authentication.

In my opinion the default MaxTokenSize up to the Windows 7/2008 R2 timeframe was way too low and resulted in many sleepless nights and trouble tickets trying to track down user issues in larger enterprises. For us Exchange administrators we became intimately aware of the MaxTokenSize issue when small groups of users were not able to successfully make Outlook Anywhere connections (RPC/HTTP) to Client Access servers but the majority of users could without any issue. These issues are always sneaky in nature because they are hard to troubleshoot and can be seemingly sporadic in nature which serves to waste a lot of troubleshooting time to resolve.

Remediation Steps

So you have found yourself in the middle of a Kerberos issue due to users being a member of too many groups or because of sIDHistory and token bloat. Now what? Thankfully there are numerous scripts, registry hacks, utilities and built-in mechanisms to help!

First off though, how does one figure out how large the MaxTokenSize should be within a given environment? Microsoft provides a formula (TokenSize = 1200 + 40d + 8s) that helps us here by adding up the number of domain local groups a user is a member of along with any universal groups outside the domain and the number of groups found in sIDHistory and then multiplying that by 40 and adding 1,200 for Kerberos overhead. We also need to add the total number of global groups and universal groups within the domain that the user is member of and multiply it by 8. The resulting number will provide us the token size for a given user.

If you are thinking that is a lot of work for just one user, you are not alone! Fortunately, there are scripts available in the TechNet Gallery that can be particularly useful. One of my favorite scripts can check a group of users or a single user to apply the above formula and provide token size. While the script may take a bit of time to run in a large environment or against a large subset of users, the output is fantastic. After script completion you will have an understanding of how many groups a user is a member of, how many of these are local, global or universal groups, and how many SIDs reside in the sIDHistory field for a user. That is certainly handy!

While scripts provided by the community certainly help to make life a bit easier, the introduction of Windows 2012 server made the activity of identifying token bloat issues and remediating them even easier!

One of the most straight forward and repeatable methods to remediate token bloat issues is to create a new GPO per Microsoft’s guidance as shown in Figure 1. This avenue will allow you to change the MaxTokenSize on all your servers or you can even use a WMI filter to further refine what machines are within scope for this setting. Options that allow granularity are nice.

Figure 1: The Set Maximum Kerberos SSPI Context Token Buffer Size policy setting in Group Policy.

Using a Windows 2012 or above domain controller you have the ability to define a preset GPO setting to set the maximum buffer size. For operating systems below 2012 you will need to configure a registry entry to push within the GPO as outlined in Microsoft’s KB article.

So obtaining the ability to create a GPO within an organization to extend the MaxTokenSize is great but that does not provide you the ability to simply monitor your environment for any MaxTokenSize violations. It would be great to have the ability to proactively monitor your environment for these violations so you can take proactive measures to mitigate the issue before end users can log support tickets.

Here comes something that made me extremely excited when Windows 2012 server came out. You can now configure a GPO setting, as shown in Figure 2, to write a warning to the event log (Kerberos-Key-Distribution-Center) as event ID 31 whenever a Kerberos ticket reaches the predefined size set. Common sense will tell you that this setting should not be configured to a threshold that is higher than what you have set the MaxTokenSize to within your environment. This setting will allow you to quickly see any authentication errors within the domain due to the Kerberos token being too large all without the need to fuss with third-party scripts! At this point you can configure your monitoring system to key in on event ID 31 and alert you as necessary. Easy, right?

Figure 2: The Warning For Large Kerberos Tickets policy setting in Group Policy

Caveats

A quick internet search will quickly turn up recommendations that the MaxTokenSize should be bumped up to 65,535 bytes. While this was the case several years ago, the guidance from Microsoft has changed even though the maximum value that can be used for MaxTokenSize is still 65,535 bytes. What has changed, however, as Microsoft points out in this KB article, the base64 encoding of HTTP authentication tokens means that 48,000 bytes is the largest value recommended to meet best practice.

Another item to keep in mind is that once the MaxTokenSize buffer has been increased, IIS (http.sys) can start exhibiting negative symptoms. A typical problem would be that a user goes to an IIS page and receives an HTTP 400 Bad Request error and the page is not presented correctly. The issue is that increasing the MaxTokenSize also increases the size of the authentication header that is encapsulated in the HTTP request which can violate the configured size limits within IIS. While the simple fix is to remove the user from any groups that are no longer required, that is rarely an acceptable fix in the real world.

The suggested fix from Microsoft is to increase the MaxFieldLength and MaxRequestBytes values in the registry to a value large enough to ensure authentication headers fall within the acceptable configured limits. This means that you may need to take the extra step and calculate the size of your users’ Kerberos token as outlined in the remediation steps above.

Conclusion

Identifying and remediating MaxTokenSize issues within your environment can be a complex task and normally is not something that is remediated quickly. Normally, if left unchecked these types of problems linger in the enterprise and cause numerous service tickets. It is important to understand though that problems with Kerberos buffer sizes is not something that should be blindly resolved by increasing the MaxTokenSize to the maximum value as that behavior can lead to additional IIS issues.

Following a steady and cautious approach as outlined in these articles will ensure a successful risk mitigation strategy within your environment. Turning on warnings for large Kerberos ticket sizes and monitoring violations will help you to determine who is running into token bloat issues. Additionally, these warnings will serve to provide general guidance on what size the MaxTokenSize value should be configured at within your environment. Thankfully, I learned the hard way 16 years ago while rebooting servers that had not been backed up in several weeks that risk avoidance and preparation is paramount to a successful career and sleep.

Still got questions about Active Directory?

If you have any questions about domain controller hardware planning, the best place to ask them is the Active Directory Domain Services forum on TechNet. If you don’t get help that you need there, you can try sending your question to [email protected] so we can publish it in the Ask Our Readers section of our newsletter and see whether any of the almost 100,000 IT pro subscribers of our newsletter have any suggestions concerning your problem.

If you would like to read the other parts in this article series please go to:

• Active Directory Insights (Part 1): Configuring DNS on domain controllers
• Active Directory Insights (Part 2): Digging into Trusts
• Active Directory Insights (Part 3): Re-examining read-only domain controllers
• Active Directory Insights (Part 4): More on read-only domain controllers
• Active Directory Insights (Part 5) – Domain controller hardware sizing
• Active Directory Insights (Part 6) – Domain controllers and NIC teaming
• Active Directory Insights (Part 7) – More on using virtual domain controllers
• Active Directory Insights (Part 8) – Virtual domain controllers and disaster recovery
• Active Directory Insights (Part 9) – Automating user account provisioning
• Active Directory Insights (Part 10) – DHCP and domain controllers
• Active Directory Insights (Part 11) – sIDHistory
• Active Directory Insights (Part 13) – Digging into the Global Catalog
• Active Directory Insights (Part 14) – More about the Global Catalog