Of the many improvements in the logging capabilities included with Service Pack 3 for the 2004 ISA Firewall, perhaps the most impressive is the new diagnostic logging feature. The new diagnostic logging feature adds over 200 new events that can be logged when diagnostic logging is enabled. These events appear in a new section added to the Windows Event Viewer, which is aptly named ISA Server Diagnostics.
There is one problem with diagnostic logging: it has the potential to log thousands of entries in a very short period of time. Even if you are very restrictive in your logging parameters, you can still end up with hundreds of entries, which might cause the most interesting information to be lost. To help solve this problem, the ISA Firewall team has made available the Microsoft ISA Server Diagnostic Logging Viewer, which you can use to help weed out some of the less interesting entries that appear in the ISA Server Diagnostics log.
The ISA Server Diagnostic Logging Viewer is a command-line tool for viewing and querying log entries generated by the diagnostic logging feature. The tool builds queries for filtering diagnostic logging events based on the command-line options that you select, and then uses Log Parser 2.2 to perform the filtering.
The output can be displayed in the Command Prompt window or in a graphical window opened by Log Parser, or it can be saved as an HTML file that can be opened in a browser application, such as Internet Explorer. Output that is displayed in a browser application can be filtered further to obtain the log entries related to a specific logging context.
You probably know by now that I’m no fan of using command line tools when a graphical interface would better serve the admin. However, the Diagnostic Logging Viewer is pretty easy to use and doesn’t require you to memorize 600 character command line strings like those you need to know to do simple Exchange 2007 Server management.
In this article we’ll do the following to see how you can use the Diagnostic Logging Viewer to help troubleshoot ISA Firewall issues:
- Download and install Log Parser 2.2
- Download and install the ISA Firewall Diagnostic Logging Viewer
- Enable Diagnostic Logging
- Do some stuff
- Run the Diagnostic Logging Viewer to Troubleshoot
Download and install Log Parser 2.2
The diagnostic logging tool uses Log Parser 2.2 to perform the filtering of log entries. The first step then is to download and install Log Parser 2.2.
Double click on the LogParser.msi file and follow the installation wizard, using the Complete installation option. You won’t have to restart the machine after the installation is complete.
Download and install the ISA Firewall Diagnostic Logging Viewer
The next step is to install the ISA Server Diagnostic Logging Viewer. Now don’t try to find this file on your own, because if you do a search of the www.microsoft.com Web site for this file, you won’t find it!
Double click on the dlviewer.exe file and enter a location where you want the viewer to be installed. Since we’re going to have to use the command line here, I like to use a simple directory name so that I don’t have to suffer the innumerable typos that happen when you’re trying to get to a complex directory name. In this example I’ll use C:\ISALOG. The installer doesn’t make any changes to the operating system or install any system files – the only thing it does is unpack the files into the directory you specified.
Enable Diagnostic Logging
Now that we’re set up to do filtering on our diagnostic logging, we’re ready to enable it. After installing ISA 2004 SP3, you’ll notice a new node in the left pane of the ISA Firewall console, which is named Troubleshooting, as seen in the figure below.
In the middle pane of the ISA Firewall console you’ll see five links. These are:
- Use the ISA Server Best Practice Analyzer
- View ISA Server Alerts
- View ISA Server Logging
- Configure Diagnostic Logging
- Read ISA Server Documents and Troubleshooting Guides
You can see these links in the figure below.
Click on the Configure Diagnostic Logging link in the middle pane of the ISA Firewall console and you’ll see the Diagnostic Logging dialog box like the one in the figure below. You have two options for the type of events you want to log:
- Firewall policy This enables logging information about firewall policy rules, which includes Web proxy traffic.
- Authentication This enables logging information about firewall policy rule authentication
There are also three buttons:
- Start Logging Click this button to start logging
- Clear Log Data Click the button to clear the log data (this empties out the ISA Server Diagnostics section in the Event Viewer)
- View Log Data Click this button to open the Windows Event Viewer and view the entries in the ISA Server Diagnostic section
When you click the Start Logging button, very detailed events will be written to the Event Log. This can generate significant processor utilization, so only run the diagnostic logging as long as you need it. After starting the logging, click the Close button. You’ll see a warning dialog box like the one below. Click No to indicate that you don’t want to stop logging.
The ISA Firewall console lets you know that diagnostic logging is taking place so that you don’t have to wonder why performance is lagging during the logging process. The figure below shows this information in the middle pane of the ISA Firewall console.
Do Some Stuff
Now let’s generate some traffic that we need to troubleshoot. In this example, I’ve created three rules, one to allow outbound SMTP traffic, one to allow outbound NNTP traffic and one SMTP Server Publishing Rule.
In the figure below you can see that I tried to telnet to an external SMTP server and the connection attempt fails. Why? Maybe we’ll be able to use the diagnostic logging to figure this out later.
In the figure below we can see that I have to telnet to the published SMTP server from an external client. The connection attempt fails, but all we have is a blank screen. Not very helpful for troubleshooting. Maybe the diagnostic logging will help us solve this problem?
In the figure below, you can see a couple of failed outbound connection attempts to NNTP servers. Can the diagnostic log viewer help us figure out why?
Run the Diagnostic Logging Viewer to Troubleshoot
There are four ways you can view data when using the diagnostic log viewer:
- Directly from the Windows Event Viewer
- As a simple printout at the command prompt, which can be piped to a text file
- In a log parser “grid” format
- As an interactive Web page
The figure below shows how to view the data in the Windows Event Viewer. We don’t need to use the diagnostic logging viewer to access this data. However this data is not filtered. It would be very difficult to find the information you’re looking for if you had only the Event Viewer to view the diagnostic events. You can see that almost 6000 events have been logged, and the logging had been going on for maybe 10 minutes at the most, on a very quiet lab network.
The ISA Firewall diagnostic log viewer allows you to drill down on the data. Since it’s based on Log Parser 2.2, you can event use SQL queries, if you happen to be a DBA kind of guy. But even if you’re not a DBA, you can still search on simple strings and find what you’re looking for.
The instructions on how to use the diagnostic log viewer are included in a text file that’s placed in the directory in which you placed the log viewer files. The figure below shows the contents of that file.
Let’s take a look at some examples. In the figure below we entered the command:
Dlviewer “nntp outbound”
Where nntp outbound is the name of an Access Rule that allows outbound NNTP connections.
The result of this command can be seen in the figure below. We are presented with log entries that match the NNTP connections made outbound through the ISA Firewall. This information isn’t very interesting and really doesn’t provide any insight other than the ISA Firewall received the connection request.
Another way to view diagnostic log information is to see it in a “grid”. In the figure below, I entered the command:
dlviewer –ogrid “smtp outbound”
Where “smtp outbound” is the name of an Access Rule that allows outbound SMTP connections.
The result of this command is a Log Parser grid that gives up some detailed information about outbound SMTP connections matching the name of the Access Rule. The Context column includes a number that identifies each connection attempt that matches the SMTP Access Rule. In this case, we can easily determine why the outbound connection attempt failed – The rule SMTP outbound requires user authentication. Also notice the Log Source column. This column provides information regarding whether the Firewall Engine or the Firewall Service is evaluating the connection. If the packet filter Firewall Engine can’t make a definitive decision, then it passes the connection request up to the Firewall Service.
In the figure below I’ve entered the command:
dlviewer –odir tst1 “smtp outbound”
Where “smtp outbound” is the name of an Access Rule allowing outbound SMTP connections.
After the command finishes, you can go to the directory that you included in the command as to where to place the HTML files. In this case, I told it to place the files in the tst1 folder, which will be placed under the folder where the Diagnostic Logging Viewer files are placed, as you can see in the address bar in the figure below.
The Web page gives you detailed information about the connection attempts matching the rule included in the query. Notice the Context column. You can click on the context number to get very fine tuned information about how the ISA Firewall evaluated the connection matching the connection attempt.
The figure below shows a partial view of what the ISA Firewall logged for a single outbound SMTP connection attempt. If you read this information closely, you can see the order in which the ISA Firewall evaluates rules. Some of the information might be confusing, as it might not seem to be related to the connection of interest. This “extraneous” information is mostly related to how the ISA Firewall evaluates connections outside of what is specified in the Firewall Policy for that ISA Firewall, or to the fact that many of the log file entries aren’t defined in public (yet?).
In this example, you’ll see at the end that the SMTP connection failed because authentication is required and the client was not able to authenticate with the ISA Firewall.
If you click on one of the context entries, you’ll see even more detailed descriptions of what happened related to that connection.
In the figure below I entered the following command:
dlviewer –odir tst5 “nntp outbound”
Where “nntp outbound” is the name of an Access Rule that allows outbound NNTP connections.
The figure below shows the result of this query. There’s nothing there that looks like it’s going to help solve this problem. Maybe if we click on the Context number related to a specific connection attempt.
There might be a hint here. In several places the log notes that source does not match the packet. What does this mean? Your guess is as good as mine. But it might mean that the source IP address does not match the requirements in the Access Rule. If so, then the diagnostic logging has helped us, as the reason for the failed connection attempt is that the Access Rule limited outbound NNTP connections to a specific source IP address.
In the figure below, I entered the following command:
dlviewer –odir tst “smtp server”
Where “smtp server” is the name of a Server Publishing Rule publishing an SMTP server on the internal network behind the ISA Firewall.
The figure below shows the results of the query. It shows that the SMTP Server Publishing Rule accepted the packet. OK, so if the SMTP Server Publishing Rule accepted the packet, then everything must be right with the ISA Firewall configuration. So what was the problem? The SMTP service was disabled on the SMTP server!
Overall, the new diagnostic logging and diagnostic logging viewer is a great new addition included with Service Pack 3 for the ISA 2004 firewall. However, the current documentation (which includes a very short .txt file) is woefully lacking in information that would help us get the most out of this tool. There are far too many entries in the log files that do not make sense. What does it mean when the “source doesn’t match the packet” and “the source port doesn’t match the rule”? With full documentation of each of the 200+ diagnostic logging events, we’ll be able to get the most out of this new feature.