If you missed the previous article in this series please read Fileserving in Terminal Server Environments (Part 1).
What’s your problem?
Oddly enough, it can be quite difficult to determine of you have a performance problem. I mean, usually it’s quite obvious that there is a performance problem but it’s much harder to pinpoint these problems as being related to Fileserving. There are quite a few programs out there that can pinpoint performance problems for you but if you’re on a budget, there’s a lot you can do yourself to diagnose these performance problems.
If you have performance issues relating to Fileserving, you could experience any of the following symptoms (by no means a complete list):
- The good ol’ hourglass
You catch yourself staring at the hourglass for sustained periods of time when browsing files or folders. Remember that this can be very obvious, as in browsing in your home directory, or less obvious, such a Word performing an auto-save over the network. - Logging on or off seems to take forever
Even if you have a small profile, logging on or off takes a long time. The “Loading your settings” and “saving you settings” captures on the logon and logoff dialog boxes are key indicators of this. - Lost Delayed-Write Data
This is an entry you’ll find in the system event log if you have problems writing to a network share. Sometimes the error also pops up on the (Terminal) server console, a clear indicator that an error occurred while the terminal server was trying to write to the file server. These errors usually occur when the file server does not accept the write request in a reasonable amount of time and the requests times out. These errors can also occur when the file server experiences a network outage. - Random Application Crashes
This can happen when application (configuration) data, for example the Application Data folder or an application .ini configuration file, is redirected to the file server. These application crashes seem quite random because the application itself doesn’t always mention an error in writing to the network. For example if an application tries to write to a redirected configuration file and the file server is under too much load to service that request, the application usually goes into a stall. The infamous “Application is not responding” dialog box is a good indicator of this.
Measuring
So if I have file server related problems, what settings do I change? Unfortunately it’s not that simple. The only way you can try to determine the cause of your Fileserving performance issues is to get down and dirty and start measuring. Good old perfmon is a great place to start. It’s beyond the scope of this article to discuss every single counter that could be of use so I’ll just mention the key counters.
Assuming you suspect the client side (Terminal Server) to be the bottleneck, fire up perfmon.msc and locate the Redirector object. All the counters in the Redirector object provide valuable information but also create a flood of information. Key counters to watch are:
- Current Commands
Current Commands counts the number of requests to the Redirector that are currently queued for service. According to Microsoft PSS, if this value is larger than 20, then the network(s) and/or the server(s) being accessed are seriously bottlenecked. - Server Sessions Hung
Server Sessions Hung counts the number of active sessions that are timed out and unable to proceed due to a lack of response from the remote server. - Network Errors
Network Errors/sec is the rate at which serious unexpected errors are occurring. Such errors generally indicate that the Redirector and one or more Servers are having serious communication difficulties. For example an SMB (Server Manager Block) protocol error is a Network Error. An entry is written to the System Event Log and provides details.
Assuming you suspect the File server itself to be bottlenecked, monitoring the following counters could point you in the right direction:
- Physical Disk\Average Queue Length
Avg. Disk Queue Length is the average number of both read and write requests that were queued for the selected disk during the sample interval. This is the best way to check if your Physical disks are potent enough to handle all the read and write requests in a timely fashion. If the Avg. Disk Queue Length exceeds 2 per physical disk (in for example a RAID 1 or RAID 5 configuration) then you probably have a disk bottleneck. - Server\Server Sessions
The number of sessions currently active in the server. Indicates current server activity. - Server\Blocking Requests Rejected
The number of times the server has rejected blocking SMBs due to insufficient count of free work items. Indicates whether the MaxWorkItem or MinFreeWorkItems server parameters might need to be adjusted. - Server\Work Item Shortages
The number of times STATUS_DATA_NOT_ACCEPTED was returned at receive indication time. This occurs when no work item is available or can be allocated to service the incoming request. Indicates whether the InitWorkItems or MaxWorkItems parameters might need to be adjusted.
Best Practices And Solutions
It’s impossible to discuss every possible step you can take to solve your Fileserving related performance problems. I will however suggest some best practises that, if implemented, decrease the likelihood that the Fileserving component of your environment can become a performance bottleneck.
- Best of the Best I/O
Your file server is extremely dependent on Disk I/O. Use the best hardware you can find. Use SCSI disks (preferably 15K Disks). Use hardware RAID solutions, avoid using software-based RAID (5) solutions. Consult your hardware provider best practises for achieving optimum I/O. Consider using SAN / NAS storage solutions. - Maximize Network Bandwidth
Use proven technologies providing high network bandwidth. Gigabit Ethernet is almost a commodity these days. If you haven’t done so, upgrade your NICs. Copying files across a network (like loading profile) sucks up quite a bit of network bandwidth. - Hardcode Duplex and Speed Settings
Very much related to the previously discussed network settings, hardcode those duplex and speed settings! Configure your NICs to operate on “maximum” speed and duplex levels and hardcode the same on the corresponding ports of the network access switches. Do not rely on auto-detect settings. Messing this up could cause you a lot of headaches. - Limit Redirecting Files
Generally it’s a good thing to redirect “My Documents” to your file server for data protection goals. Outside of “My Documents”, consider redirecting nothing else. Leave / put everything into the profile, including .ini files. This way your profile size may increase slightly but it is no match for the performance gain you’ll have. This way you’ll access all the data you need on local storage which is by far the fastest. As an added bonus you will also decrease the load on your file server. - Tuning the Server Service
Windows does allow for you to (partially) tune the Windows Server service. Windows should automatically select the appropriate setting, but you can take a look for yourself. When you select the properties of a connection and then select the properties of “file and printer sharing for Microsoft networks”, you should end up with a windows like this:
The setting “maximize data throughput for file sharing” is the preferred one for file servers. Selecting one of these radio buttons actually changes some Fileserving related settings in the registry. These settings change some of the registry settings you could change to gain optimal file server performance.
If you really want to tune the registry of both the client and the server for optimal Fileserving performance, you should read this article published on Thincomputing.net. It’s not for the faint of heart but it does explain exactly why registry settings are important and how you can push your Fileserving environment to its limits if you need to.
- Limit Profile Size
In addition to my recommendation to “put everything in the profile”, it’s very important to monitor and limit the size of profiles. Remember this includes Terminal Server profiles as well as “normal workstation” profiles. It’s not uncommon to have profiles of several hundred megabytes traversing the network if you don’t keep this in check. Use a group policy to limit growth or third party software. Alternatively you could use mandatory profiles or flex profile to cut down on profile size. - Disable SMB Signing
This is a typical case where increased security can lead to decreased performance. In a nutshell, Server Message Blocks (SMB) is what traffic between a file server and a client is made of.
SMB signing is a process that is used to digitally sign Server Message Blocks (SMB) to make sure that they are not altered in transit. This process can be rather processor intensive, depending on the amount of traffic. It’s up to you to decide whether the decrease in security is worth the increase in performance.
Conclusion
Unfortunately Fileserving in Terminal Server environments is a common performance bottleneck. In this two part series we took a look at what comprises Fileserving in Terminal Server environments, how you can determine if you have a Fileserving bottleneck and what you can do to prevent performance issues from arising. Be sure to read my other article on Thincomputing.net if you need to push your Fileserving environment to its limits.
If you missed the previous article in this series please read Fileserving in Terminal Server Environments (Part 1).
