Disk Performance Testing with JetStress Tool for Exchange Server 2010
Once again, Microsoft has significantly dropped down the I/O throughput requirements for Exchange Server 2010 when comparing with previous versions. But then again, the storage subsystem is still a crucial piece of the overall Exchange performance, that's why tools like JetStress are so important. JetStress 2010 simulates disk I/O loads produced by a specific number of users, thus helping verifying the performance and stability of the storage subsystem prior to putting a Microsoft Exchange Server into production.
The newest version of JetStress supports Exchange 2003, 2007 and 2010, but for legacy Exchange (2003 and 2007) it is recommended that you use Jetstress 2007.
There are two categories of test scenarios: disk subsystem throughput and mailbox profile.
- In the disk subsystem throughput test scenario, you can do the following types of tests:the Exchange
- Performance of database transactions (the performance test becomes a stress test when its duration is longer than six hours)
- Streaming backup
- Soft recovery
- In the Exchange mailbox profile test scenario, you can specify the number of mailbox users and I/O per seconds per mailbox to simulate the profiled Exchange mailbox load.
To further complement the results provided by JetStress, it is recommended that you also use System Monitor and Event Viewer to validate that the performance criteria is met or exceeded and that no errors are logged.
The previous version of JetStress introduced a help file (Microsoft Exchange Server Jetstress.chm) that replaced the old Word document with instructions. Well, for this new version (14.01.0180.003), although the .CHM file still installs with the tool, its content is basically a link to the online documentation.
Figure 1: Help File (.CHM)
But of course there are some other improvements and minor bug fixes:
- Exchange 2010 mailbox I/O profile is updated, including 32-kilobyte (KB) page size, average database read size, average database write size, and average log write size.
- Database duplication is now multicast, which reduces the time to prepare databases for Jetstress testing.
- Instead of using Logical Disk performance counters, MSExchange Database I/O counters are used for I/O measurement. Databases and logs can be placed on the same volume. Jetstress now works correctly when there's a Logical Disk performance counter problem.
- Calculations for log checksums are now multithreaded. (Log checksums from different ESE instances are calculated in parallel)
- Maximum number of databases per test is increased to 100 for Exchange 2010 ESE binaries.
- Log replication I/O is now simulated based on Exchange 2010 high availability architecture. You can simulate a maximum number of 16 database copies to drive the log replication I/O (log read I/O) via XML configuration or UI in JetstressWin.exe. The I/O generated by simulating log shipping as well as the configuration is provided in the HTML report.
- Background database maintenance (checksum) is now simulated for Exchange 2010 test configurations. This is the default behavior. The background database maintenance work performed during the test is reported separately from the normal transactional I/O in the HTML report. This can be disabled via the XML configuration or via a check box in JetstressWin.exe.
- Traces now contain date and time information for every entry.
- Improved informative messages and warnings are provided to prevent administrators from using mismatched ESE/ESE performance counters.
- Only the minimum necessary number of databases is created or replicated, rather than five databases per ESE instance.
- ESE performance counters are now unregistered or registered again if it's detected that the current registered version is inconsistent with the version Jetstress will use. Restarting Jetstress is still required.
- JetstressCmd.exe and JetstressWin.exe are now consistent in overriding the test type from Performance to Stress if the duration is longer than six hours.
- System and application events are now saved for each Jetstress run for the time of the run, along with HTML reports and performance monitor logs.
- Simplified configuration of JetstressWin.exe has exposed many parameters on configuration pages to help the beginner to intermediate user focus on the important parameters of a test scenario. Other parameters in the XML configuration file are hidden. The XML configuration file can be used by the advanced user to fine-tune a test run.
- Test pass/fail criteria in Jetstress still reports memory system information with performance counters such as Memory/Available/Available MBytes, Memory\Free System Page Table Entries, Memory\Pool Nonpaged Bytes, and Memory\Pool Paged Bytes. It no longer uses the test/fail criteria on 64-bit editions of the Windows Server operating systems.
- Throttled load generation, also known as fine-grained disk throughput, is now available. Jetstress artificially throttles the amount of disk I/O that a thread generates to support the fine-grained control on the overall I/O throughput.
- If you select the time range of a performance log, Jetstress saves a performance log for the whole time range of a test run. But the test report accounts for a valid time range of performance samples from a point where it has reached 90 percent of the maximum cache size.
Preferably, Jetstress testing should be performed before you install Exchange on the server. There are some well-known risks associated with running Jetstress on a machine with Exchange installed. If you use a version of Jet (ESE.DLL and ESEPERF.*) different than the version installed with Exchange, the registration of the Jet database counters in the Jetstress install directory will broke the database counters for Exchange after Jetstress is removed. You may have to uninstall the existing Exchange Database Engine counters before you run Jetstress, by running unlodctr ESE at a command prompt. However, you will still have to uninstall and reinstall the Exchange version of the Exchange Database Engine counters when testing is completed
To install Jetstress follow these steps:
- Ensure that installation requirements are met:
- Jetstress 2010 requires that the server that is running it have .NET Framework Version 2.0 or higher installed.
- Jetstress is supported only on Windows 2000 Server SP4, Windows Server 2003 (32 or 64 bit), Windows Server 2008 and Windows Server 2008 R2.
- Configure the storage subsystem:
- Although not really a pre-requisite, consider following these recommendations: use Windows Hardware Quality Labs (WHQL) certified hardware. If the server is a cluster, the whole server/storage configuration must be Cluster Certified.
- Verify that all the storage components have the latest firmware and the drivers are current.
- Format the LUNs within Windows with NTFS file system (64KB allocation unit size).
- Verify that the HBA/SAN specific configuration is set correctly. Many HBAs use registry keys to customize the configuration to a specific SAN platform (for example, Queue Depth).
- Configure the storage logical unit numbers (LUNs) (consider Exchange log devices and database devices).
- Download Jetstress and start the installer by double-clicking the Jetstress.msi file. Follow the prompts to install the tool on your computer.
- Copy the following ESE database modules to the directory where you installed Jetstress.
ESE.DLL can be found at C:\Program Files\Microsoft\Exchange Server\V14\Bin\, in a default Exchange Server 2010 installation. The rest of the files reside in C:\Program Files\Microsoft\Exchange Server\V14\Bin\perf\AMD64\
If you forget to copy these files, you'll get the following warning when you try to run the tool:
Figure 2: Validating required files
- Running the tool will trigger some validation tests, such as the one mentioned on the previous step. Checking if the LogicalDisk counters are enabled is another test that is run and that will be automatically fixed.
Figure 3: Validating performance counters
- Every time the tool fixes something, you'll need to restart JetStress. Eventually, when all the requisites are met, you'll be able to continue.
Figure 4: Checking test system
Running the Test
There is a command line version of the tool, and a GUI of Jetstress. The steps described in this article relate to the GUI version.
- To start Jetstress simply click the shortcut available at the Start Menu or start JetstressWin.exe. On the Open Configuration page, you can either create a new test configuration or open an existing configuration file.
Figure 5: Test configuration
- There are two categories of test scenarios, listed here, that you can select from the Define Test Scenario page:
- Test a disk subsystem throughput
- Test an Exchange mailbox profile
In this example we'll use the first option, test a disk subsystem throughput.
Figure 6: Define test scenario
- Since we're performing a disk subsystem throughput test, the Select Capacity and Throughput page is available. There are options to size the test databases using the percentage of the maximum storage capacity, and target I/O throughput (IOPS) by the percentage of the maximum throughput capacity of the disk subsystem. Jetstress reserves 25% of the initial database file size for its future growth during test runs. You also get the option to suppress tuning and use thread count, if the automatic tuning frequently fails. The following table provides guidelines for the number of threads for the log disk writes per second and the database disk transfers per second. These threads are applied on a per database basis.
Log disk writes/sec
Database disk transfers/sec
Table 1: Thread guidelines
Figure 7: Capacity and throughput
- The next screen it's where you have the option to select the kind of test you want to perform:
- Performance - Generates the Exchange type of I/O by accessing a database that has Jet transactions.
- Streaming backup - Measures the performance of a backup solution. Be aware that Jetstress can perform a streaming backup only to a device that can be mounted with a drive letter.
- Soft recovery - Measures the log replay rate.
- Multi-Host Test - Select this check box to pause the Jetstress tests (both performance and soft recovery) before the database checksums are run. By pausing, you can coordinate multiple hosts running Jetstress in parallel, and prevent the checksum of one host from interfering with the performance test on another host. Multiple host tests should only be used when testing multiple hosts against a common storage area network (SAN).
Figure 8: Test type
- The Define Test Run page has the following options:
- Output path for test results enables you to specify the directory where Jetstress will save the performance logs, and test reports.
- Test duration (hours) enables you to specify the period for performance sample gathering. The performance test becomes a stress test when the duration is longer than six hours so that it uses a lenient set of the disk I/O latency thresholds.
Figure 9: Test duration
- Jetstress limits the number of databases based on the physical memory of the system. For Exchange 2010, there's a requirement of 256MB per database.
Figure 10: Define Database Configuration
- The Select Database Source page has three options:
- Create new databases
- Restore backup databases
- Attach existing databases
Since we are building this test from scratch we'll select Create new databases. If we had previously prepared this test, we could just attach existing databases, saving some time.
Figure 11: Database source
- The Review & Execute Test page gives you a summary of the test scenario that Jetstress will run. You have the following options available:
- Prepare test - Creates the test database(s).
- Execute test - Proceeds to prepare test databases, performs automatic tuning, and runs the configured test.
- Save test - Saves the settings that you have configured to a new or an existing configuration file.
Figure 12: Review & Execute Test
- When you hit Execute test, you get the Test in Progress page, where you have the option to cancel the test and exit from the application.
Figure 13: Test in progress
After the test is completed, the performance data is analyzed and reported in a summary report. Result will be saved to Performance_(DateTime).html file. All the performance counters collected will be gathered in a counter log file named Performance_(DateTime).blg that you can use for some more advanced analysis.
Consider the following guidelines when examining the data collected.
Performance counter instance
Guidelines for performance test
Guidelines for stress test
I/O Database Reads Average Latency (msec)
The average value should be less than 20 milliseconds (msec) (.20) and the maximum values should be less than 50 msec.
The maximum value should be less than 100 msec.
I/O Log Writes Average Latency (msec)
Log disk writes are sequential, so average write latencies should be less than 10 msec, with a maximum of no more than 50 msec.
The maximum value should not exceed 100 msec.
Average should be less than 80% and the maximum should be less than 90%.
Same as for performance test.
Available Mbytes (32-bit Windows Server only)
Minimum should be no less than 50 megabytes (MB).
Same as for performance test.
Free System Page Table Entries (32-bit Windows Server only)
Minimum should be no less than 5,000.
Same as for performance test.
Transition Pages Repurposed/sec (Windows Server 2003, Windows Server 2008, Windows Server 2008 R2)
Average should be less than 100.
Same as for performance test.
Table 2: Guidelines for examining Jetstress 2010 analysis reports
If you open Performance_(DateTime).html you'll see a nice HTML report with some tables with performance analysis, like the following example:
Figure 14: Performance Test Result Report
How to interpret the summary report:
- The Test Summary legend summarizes the test run.
- The Test Issues legend indicates whether the test passed or failed. It also indicates any performance samples averages and spikes that have exceeded thresholds.
- The Database Sizing and Throughput legend explains your planned Exchange database sizing in addition to the input/output (I/O) throughput target.
- The Jetstress System Parameters legend explains the parameters that relate to the Exchange database engine.
- The Database Configuration legend lists the database files and log paths, grouped by Jetstress 2010 instance.
- The Transactional I/0 Performance legend shows the throughput, average I/O size and latency for read and write disk operations that are related to user activity, and log replication. The numbers are grouped by Jetstress 2010 instance for both database and log operations.
- The Background Database Maintenance I/0 Performance legend shows the database read throughput and average size for operations triggered by background database maintenance.
- The Log Replication I/0 Performance legend explains log read throughput and average size for operations triggered by log replication.
- The Total I/0 Performance legend explains throughput, average I/O size and latency for read and write disk operations, taking into account all the operations triggered by Jetstress 2010.
- The Host System Performance legend summarizes other non-I/O performance metrics. Be aware that these figures could be indirectly affected by storage performance.
Since I made this test using a virtual machine on my laptop, the test just failed. That was expected, since I don't have enough performance to run Exchange in a production environment.
The storage subsystem is key for a successful Exchange deployment. Planning it and testing it should precede any Exchange deployment. The Jetstress tool is the right tool for the job, if used correctly and in conjunction with Performance Monitor and Event Viewer will probably prevent any future surprises regarding overall system performance.
- JetStress Download (64-bit); JetStress Download (32-bit)
- Tools for Performance and Scalability Evaluation
- Mailbox Storage Design Process
- Exchange 2010 Mailbox Server Role Requirements Calculator