Understanding TMG Logging (Part 4)

If you would like to read the other parts in this article series please go to:

• Understanding TMG Logging (Part 1)
• Understanding TMG Logging (Part 2)
• Understanding TMG Logging (Part 3)

We’re now at the end of our series on TMG logging with this article, which is part 4 of the series. In this last installment, we’ll finish up the discussion of TMG logging by talking about the Log Queue, options for logging and logging best practices. Let’s begin the discussion by talking about TMG firewall Log Queues.

Purpose and function of the log queue

The TMG log queue was created to solve some of the problems that were caused when network and SQL server errors resulted in log failure in the old ISA firewall. This mechanism is designed to buffer logging information when the log destination is unavailable. The queue saves each collection of log entries into a binary file until the logging destination becomes available again.

Log queue configuration is a simple matter. It’s limited to a single decision: where to place the log queue. By default, the log queue is stored in the same place that the TMG logs are stored, which is in %ProgramFiles%\Microsoft ISA Server\ISALogs. As shown in the figure below, you can change this and as you’ll see later, you absolutely should change this location.

Figure 1

Logging options

You can select between logging to a local SQL Server Express database or a remote SQL Server computer or you can log to a text file. In general, you need to consider the following factors when you’re deciding which logging option to choose:

• Local-host SQL logging is more compute intensive than local-host text logging or either of the remote logging options.
• Remote logging requires additional network traffic in direct proportion to the amount of traffic being processed by the TMG firewall. The more traffic TMG processes, the more log data it will generate and thus, a larger number of log entries will be created.
• Regardless of the logging destination and format, the resources that are required to satisfy the logging will be proportional to the log fields that you have selected for each log. The more fields you choose to log, the more information needs to be maintained in the log file.

Each logging method that’s available on the TMG firewall has its advantages and disadvantages. The method you choose needs to be based on two core factors: your specific environmental status and your performance requirements.

Logging is an important subject because it allows you to understand your traffic patterns and also to perform analysis when troubleshooting issues, such as failed to access Web sites. To configure TMG firewall logging to best fit the needs of your environment, you should collect some information and answer the following questions as the first step in the decision-making process:

• Do you need to use historical or offline log viewer using the TMG Console?
• Do you have (or plan to have) a SQL Server computer that’s dedicated for logging?
• What connectivity does the TMG firewall have with SQL Server?
• Is your SQL Server computer configured for high availability?
• Does your TMG firewall have enough disk space for local logging?
• Does your TMG firewall have a separate disk just for logging?

The answers to these questions will help you to decide whether local logging or remote logging is the best choice in your environment. Let’s look at how the thought process would work. We’ll use the EndToEdge.com Company as an example. Suppose that the organization answered those questions with: yes, yes, 1Gbps, yes, no and no. These answers would clearly lead the TMG firewall administrator to choose SQL Server as the best logging method.

Comparing the logging options

After your initial assessment of your environment, the next step is to understand the differences between the logging options and what each one can offer to best address your needs. The table below compares key elements for each logging option.

Table 1

Note that the TMG firewall uses an SSL-encrypted connection to log information to the SQL Server database. For more information about SQL encrypted connection, see this link.

Using log files to troublshoot

Let’s look at an example of how you can use TMG logs to troubleshoot a performance issue. As we all know, when the web browsing is affected and all users start to complain about slow access, the TMG firewall is the “usual suspect.” In our scenario, the problem was happening for all users and for all Web sites. The interesting part is that the TMG firewall didn’t have any apparent bottleneck. All the core subsystems (disk, memory, network, and CPU) were working as expected.

Troubleshooting browsing problems can be difficult because so many different elements are involved. In this example, the issue began right after a service pack was applied, so there was a natural concern about the networking changes that the service pack might have caused.

After intensive troubleshooting that covered elements such as name resolution, local disk performance, and networking, it was decided to get a user mode dump of the process wspsrv.exe while the user was trying to browse the Internet. The result indicated that the firewall was waiting for the remote SQL Server to answer logging requests, and therefore causing browsing delays. As a workaround, the logging method was changed to TXT while the SQL administrator worked to fix the performance issue that the SQL Server was having.

This is an example of how an external component can cause delays for users. The SQL Server performance issues directly affected TMG because of the logging options.

For more information on this scenario, take a look at this link.

Optimizing logging

At this point, you should have a good idea of which logging method will best fit your environment. The next step is to follow some general guidelines for best logging experience.

RAID levels

Regardless of whether you use local or remote logging, the disk configuration you choose for logging should be planned and some guidelines should be considered, including the following recommendations:

• TMG log files should be located on a dedicated disk, preferably one that is using RAID, for best performance and fault tolerance. See the table below for a comparison between the RAID levels and the impact on logging of each.
• You should format the disk in the NTFS file system.
• You need to ensure that you have enough space for logging. Microsoft recommends that you have at least 8 GB for logging, but this value might be larger, based on your network usage. Remember that more traffic means more data is being logged, which means more disk space may be required. You also need to take into consideration your log retention policy. For example, if you want to keep the logs on the box for the last three months and then move to tape, 8 GB might not be enough.

Table 2

Although these four levels of RAID are the ones used most often for TMG firewall logging, other types of RAID are also available and you can use them for TMG firewall logging if you wish. For more information on other RAID levels, take a look at this link.

Connectivity issues

Connectivity issues are important when you choose to use remote logging. If you use a remote SQL Server as your logging option, you will need a high performance and high availability connection between the TMG firewall and the SQL Server. You should not place the SQL Server in a remote location that can only be accessed from over a WAN link, because this can cause a delay that will affect the end users’ experience. In a situation where the TMG firewall needs to log all access, a database that is located across a WAN can cause a bottleneck. Connectivity issues with the SQL Server database can be caused by many different situations. A great resource for narrowing down SQL connectivity issues is available here.

Additional issues

By default, the Log Queue is located in the ISALogs folder (in the same folder hierarchy in which the TMG firewall is installed). It’s best practice for you to store the log files on a separate disk to improve performance and availability, especially in the event that the disk where the TMG firewall is installed gets full.

Another issue that you need to be aware of is that if you have a file-level antivirus installed on the firewall, you should exclude the LLQ folder from the real time scanning and any other scan job that you have the antivirus software configured to run. Antivirus should generally not be required on the firewall if you use firewall configuration and management best practices, but some admins like to follow the “better safe than sorry” principle.

If you already have a general log retention policy, the TMG firewall should adhere to it; if your company doesn’t have a log retention policy, now is a great time to think about putting one together. Log retention policies might vary with the logging method you choose. For the logs that are stored locally, TMG allows you to control how large the log file can get and which files will be deleted (by default files older than seven days will be deleted) as showed in the options in the figure below.

Figure 2

If you want to put the logs on the local TMG firewall then you will need to plan disk sizes according to the volume that you store per day. Consider calculating the future growth of your logging requirements by creating a baseline for at least a month and then verify how much space appears to be used on a daily basis. To reduce contention on the system drive where you are installing, it is highly recommended that you have another disk for logging, as mentioned earlier.

Finally, if you use the remote SQL Server database logging option, you will need to consider whether this SQL Server computer is dedicated to TMG logging or whether there are other databases on it that are sued by other applications. The amount of disk space required for log retention will vary according to the scenario where SQL Server is being used. For SQL Server storage best practices, see this link

Summary

In Part 4 of this article, we finished up our series on logging by discussing the log queue and some best practices for configuring and managing TMG log files. I hope you enjoyed this series! Thanks! –Deb.

If you would like to read the other parts in this article series please go to:

• Understanding TMG Logging (Part 1)
• Understanding TMG Logging (Part 2)
• Understanding TMG Logging (Part 3)