Optimising and Troubleshooting Logon Time to the Server
When asking administrators around the world for their SBC issue list, it’s usually number 1 on printing, and number 2 on profiles (ie logon/logoff issues). It’s not always easy as an administrator to fully understand the process of how an application appears on the screen, and what the server is doing in the background, before the application is fully loaded. Understanding this process gives you an edge when troubleshooting the processes of user logon and profile issues and will get the servers stable before users even get a chance to get a complaint in.
First contact at logon
When a Citrix client makes the connection with the server, it will start with a small handshake for identification between the two. The client then tells the server of all the capabilities it is allowed to do, set on the client side. (If nothing has been disabled on the Citrix client, all capabilities will be sent.)
The next step, is the invoking of the licensing process. The server gets a signal from the client saying what OS it resides on, because when the client is 2k or XP pro, and the server is Windows 2000, it’s eligible for a free TS cal. If not, the server will try to get one, and if it fails, the session just disappears without any clue, which is very confusing for the administrators.
Keep in mind, that this same thing can happen if the server tries to serve a valid license, but the client does not have write access to “HKEY_LOCAL_MACHINE\Software\Microsoft\MSLicensing\Store” which happened quite a few times with the older Citrix ica32t.exe web clients.
If the licensing process is ok, the user gets the logon screen presented via the gina. (Assuming WI is not used, where validation already took place). Then the user’s profile is loaded. The gina calls the userinit process, which polls the following registry key to start: “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\appsetup”
In this appsetup key, you will find entries for:
- UsrLogon.Cmd (Terminal Server only. Legacy way of starting compatibility scripts, but still exists)
- cmstart.exe (Metaframe XP and higher)
- CtxHide (Metaframe 3.0 and higher)
Then the userinit process starts the user’s shell “explorer.exe”. If a desktop is started, the registry HKLM and HKCU run key is polled, and the startup folder shortcuts are fired off.
Tip: Double-check that no apps got installed in the run key by accident, as they will be started by every desktop user, resulting in unneeded resource waste, and/or conflicting instances of that application.
So what does that exe trigger? It launches the seamless engine wfshell.exe and it launches the cltmgr.exe (Citrix client manager) to check for an update of the Citrix client in the client update system. So simply said; without a cmstart.exe entry, you will lose seamless functionality immediately.
The crashing of the wfshell.exe process is a known issue among Citrix administrators running Citrix XP. That is because the wfshell process is also responsible for the auto creation process of client printers. So if you installed a driver that is not 100% TS aware, the auto creation process will hit a limitation of that driver, resulting in a crashing wfshell for that user. Once the user gets the wfshell error message, the process is interrupted, and no printers will be auto created at all. So keep in mind to always use the printer drivers from the OS cdrom where possible. Make sure to check my previous article on this subject.
Now you might think that this only hurts the user logging on, but it can hurt the other users as well. The hanging wfshell process can cause high CPU spikes because it tries to recover, and that affects everybody. The user will also take a much longer time to logon, as the wfshell process will try to auto create printers for some time, while hanging. So the next time a user reports the wfshell error on screen, do not dismiss it that easily and investigate the matter.
The client update process cltmgr.exe polls the client side Citrix version from the version.dat file, and starts the check on the system to see whether auto update has been enabled. If you are not using this update system, it’s vital to disable the use of the client update database system. On the server just start cudutil.exe and go to database, properties and tick off “enable”. That small tick saves quite some logon time, even on a healthy server.
Tip: If you’re using the new msi clients from Citrix, the client update system can be disabled safely, as that mechanism is not capable of deploying/updating msi clients.
Users have a habit of creating large profiles, if you leave all default. If you’re running desktop sessions including a full office suite, internet surfing etc, the profile can get huge. You can either redirect important folders from that profile to a fileserver using GPO’s or scripting, but a killer extra is a free product called the “Flex Profile Kit” from Login Consultants. Get your copy here. That profile stripper will get your profiles back to a few hundred kb, and it will load like greased lightning, I promise you.
The Flex Profile Kit is a free initiative. If your company does not like free, thus unsupported tools on the server, there are plenty of equivalents that do the same thing; Jumping Profiles from messerknecht, Managed Profiles from Managed Profile, Simplify Profile from Tricerat, and FullControl from 2X. (And then some, I probably forgot).
If a certain functionality is not needed, disable the virtual channel for it. I am talking about com port use, lpt port use, clipboard use, the need for more than just the default printer. All these virtual channels are just ticks in the configuration for your terminal server or Citrix configuration. Together they can cause a very big speed difference. Just for the fun of it, test your logon process time with everything ticked on, and everything ticked off. (Even if it’s not used). You should see quite a difference.
If you’re using Citrix Webinterface solution, and would like to disable some virtual channels for outside use, but no the inside users (assuming they do not use the portal), you could also disable virtual channel functionality in the template.ica.
For example the entry to disable the use of local drives would be:
You can find a full list of these settings in the Citrix ini file reference.
During logoff, it can happen that users’ registry strings are not released by an application, and it looks like the logoff takes forever while the profile is unloading. This is solved by installing the “UPHClean service on the server. Get your copy here. If you’re getting access denied errors and event viewer 1000 entries, this is your killer tool.
It can also happen that you have a badly behaving application that seems to be hanging during the logoff process. Open the TS console to see the processes a user that is logging off has, and note down everything but winlogon.exe, csrss.exe and explorer.exe to be the suspect. (ctfmon is a known bad process.)
Then kill the process, and see if the logoff continues successfully. If it does, you have found the culprit.
I hope you’re on Citrix, because then you can set the bad exe to be killed at logoff via:
LogoffCheckSysModules REG_SZ = ctfmon.exe
To find the process, you could also use process explorer from sysinternals.com
If you still have the feeling that you have a profile issue, make sure to check that you have the latest hotfixes loaded for the OS and, if needed, for your Citrix version. If that does not help, you can activate user debugging on the server, to track down the profile issue you’re having, The how-to of the activation can be found here.
In this article I tried to explain the logon process and how to find the culprits in the logon/logoff process which can cause major headaches for an SBC administrator. Once tuned, this process can be brought back to a few seconds, making the acceptance of this type of application easier.
Hopefully these guidelines and tips will get you back in control of the situation, giving you time for other work besides end user troubleshooting.