Introduction
I know the feeling of reviewing logs on servers and domain controllers, even the most impressive network administrator beg for a job being a greeter at the nearest convenience store. However, there are some that must review the logs, others that review logs as a job, yet others that should review logs in order to determine what is occurring on their servers. Events in logs can be a bit confusing, even in Windows Server 2008/2008 R2 with the updated interface and details, but knowing how to get to the bottom of the issue that is causing the event is essential. In this article we will review some of the tips, concepts, tools, and overall approaches in determining the root cause of some of your log events.
Windows Server 2003 vs. Windows Server 2008/2008 R2 Event Viewer and Logs
There have been some radical changes in the new Windows Server 2008 and around Event Viewer. There is no question that the Windows 2000/2003 Event Viewer has issues. Most administrators try not to rely on the Event Viewer anymore due to these significant issues.
Historically, the Windows 2000/2003 Event Viewer had limitations such as:
- Cryptic messages that did not lead or indicate the root issue
- Clumsy maneuvering of the events (up is down for example)
- Limited, if not abysmal help files
- Limited controls to filter and sort through the logs
- No centralized logging capability
- No triggering of events, such as email or messages
The new Windows 2008/2008 R2 Event Viewer fixes much of these limitations, adding some amazing new features, which all will help in troubleshooting logs and events. Features for 2008/2008 R2 include:
- Verbose messages which can be displayed in standard, verbose, and XML views
- Easy maneuvering of log to log and event to event
- Help links to the Internet, which are usually very helpful
- Filtering, custom views, and filtering of nearly any log combination
- Centralized logging (backwards compatible with Windows XP and 2003)
- Event triggers, through scheduled tasks, which allow for emails, message boxes, or running of applications (See Figure 1)
Figure 1: Events can be associated with event IDs and logs.
Common Windows Event Issues
Most of you are in charge of a Windows Active Directory enterprise. With that said, I am going to focus on issues that can cause events to arise, which don’t seem to make much sense. What I mean is that an application error could be caused by the service account not being able to authenticate. A memory issue can be due to a process that is failing due to a failed communication to DNS. There are some strange issues that can come up and knowing what the common faulty issues typically are can help. Here are some common issues that can cause random (or what appear to be random) events to occur on your servers:
- DNS can’t be reached, so Kerberos is not being used
- DNS can’t be reached, so Group Policy is not applying correctly
- Cached credentials are being used
- The secure session to the domain controller is lost
- Communication with a domain controller was interrupted briefly, causing errors
- Group Policy settings updated, causing events to occur
- Group Policy updates changed local settings such as group membership, user rights, IE settings, etc.
- Service account can’t not authenticate (locked out, expired password, wrong auth. protocol, update to system caused change to authentication, etc.)
- DLL was updated by update or other product installation
- File permissions were altered, causing denial of access to data
For your domain controllers, a totally different set of issues can arise:
- Replication issues due to permissions, name changes, IP address issues, upgrades, etc.
- Incorrect delegation within Active Directory
- Incorrect delegation within Group Policy Management
- Errant Group Policy settings to user rights, group membership, etc.
- Errant permissions causing failed writes, synchronization, and access to Active Directory, logon files, etc.
Using New Event Viewer to Help Troubleshoot Events
As you can see and most likely already knew, there are many reasons that a single, or multiple, events can be triggered. Knowing how to read the events, combine the different logs and those events, into one list can be very helpful. The key is to use a combination of the new features, which will allow you to get a very clear view of what is happening on a stream of events, not just System, Security, or Application specific.
First, make sure you set up event log forwarding and subscriptions. This is free and will allow you to get events from many different computers into a single log. To get help with setting up event log forwarding go here. Be sure to only forward event IDs that you need in your logs. Taking the entire security or system log from every computer will not do you much good. However, taking events that all relate to authentication, group changes, etc. will be very useful.
Second, be sure to combine events from multiple logs including the forwarded log from step 1, into a custom view. Custom Views as shown in Figure 2, allow you to view just what you want from many logs in a continuous stream.
Figure 2: Custom Views allow you to choose event IDs from different logs.
Third, set scheduled tasks up for when key event IDs register in the log. This will give you continuous control over the logs and events that occur. So, when you have a failed authentication you will receive an email letting you determine if the failure is due to the server, network, or domain controller.
Fourth, look for trends. Most applications, services, authentications occur on a regular basis. Every 10, 12, 15, 20, 30, etc. minutes is a common pattern for authentication for service accounts. Be sure to see if the event is isolated or recurring. If recurring, look for events in the System and Application log to see if the authentication failures coincide with other issues that are being tracked.
Summary
Event logs and event IDs can be hard to track and troubleshoot. Make sure you use the new Windows Server 2008 and Windows Server 2008 R2 Event Viewer. The new features make troubleshooting servers much easier, as you are able to view the multiple logs and events in a continuous stream. This will allow you to clearly see trends, combinations, and otherwise unrelated events together. The addition of the event log forwarding, custom views, and scheduled tasks for event IDs has made the new Event Viewer a powerful and inexpensive way to troubleshoot events.
