You can’t call yourself a true Exchange Administrator until you have run the JetStress Tool at least once. This nice piece of software addresses one of the most critical areas of Exchange server, the storage subsystem. You can use it to verify the performance and stability of the disk subsystem prior to putting an Exchange server into a production environment or as a troubleshooting tool when in the presence of a bottleneck.
Introduction
Jetstress is a tool that helps verify the performance and stability of the disk subsystem by simulating some heavy loads to the disks, as if the server was being used by a large number of users. You then use System Monitor, Event Viewer and the Eseutil tool together with Jetstress to verify that your disk subsystem meets the established performance criteria.
Jetstress enables you to do three types of tests:
- Disk Performance Test runs for 2 to 24 hours and enables you to verify the performance and sizing of your storage solution.
- Disk Subsystem Stress Test runs for 24 hours and enables you to test the server storage reliability over a longer time.
- Streaming Backup Test measures the database throughput and end-to-end streaming backup (both read and write) performance.
The best way to verify the integrity and performance of your disk subsystem is to run the first two tests mentioned above.
In the latest download package for Jetstress which I used (6.5.7795.0) there are two separate applications:
- JetstressUI.exe (the graphical user interface (GUI) version of Jetstress named Jetstress 2004)
- Jetstress.exe (the original command-line version of Jetstress)
Both versions can be used to successfully test the performance of an Exchange disk subsystem, but using the command-line requires expertise in specifying the parameters and analyzing performance results. Jetstress 2004 and its graphical user interface reduces the complexity of configuring the test. Additionally, it facilitates the analysis of results by producing a performance analysis report. There are also some unique features available in the graphical tool. For more information please read the manual that comes with the tool.
Although it is recommended that you run the tests in a non-production environment, in the real world sometimes it’s not possible to have a lab that mirrors the actual systems. So, my advice here is that you use the production server just before going live (even before installing Exchange), but remember to format the used storage volumes at the end of the tests.
Installing JetStress
Preferably, Jetstress testing should be performed before you install Exchange on the server. There are some well-known risks associated with running Jetstress on a machine with Exchange installed. The first is that Jetstress could potentially delete some existing logfiles if it’s configured to use the same log drives that Exchange is using. The second is that if you use a version of Jet (ESE.DLL and ESEPERF.*) different than the version installed with Exchange, the registration of the Jet database counters in the Jetstress install directory will break the database counters for Exchange after Jetstress is removed.
To install Jetstress follow these steps:
- Ensure that installation requirements are met:
- Jetstress 2004 requires that the server that is running has .NET Framework Version 1.1 installed.
- Jetstress is supported only on Windows 2000 Server and Windows Server 2003. Microsoft Windows NT 4.0 and older versions are not supported (who uses them these days?).
- Although not really a pre-requisite, consider following these recommendations: use Windows Hardware Quality Labs (WHQL) certified hardware, verify that all the storage components have the latest firmware and, finally, consider aligning the disks with diskpart (for information about how to run diskpart, read Disk Geometry).
- Extract the files of the Jetstress tool download package to a local directory, for example C:\Jetstress, and start the installer by double-clicking the Jetstress.msi file. Follow the prompts to install the tool on your computer. The setup will install the tool into Program Files, but you can move the files to any folder you want.
- Copy the following ESE database modules to the directory where you installed Jetstress.
- ESE.DLL
- ESEPERF.DLL
- ESEPERF.INI
- ESEPERF.HXX
If you forget to copy these files, you’ll get the following warning when you try to run the tool:
Figure 1: Validating required files
- Double click the JetstressUI.exe file. Running the tool will trigger some validation tests, such as the one mentioned in the previous step. Checking if the LogicalDisk counters are enabled is another test that is run and that you have the option to fix.
Figure 2: Validating performance counters
Running the Tests
Before running the test, there are some settings you must first configure and that will impact your system’s performance:
- Number of storage groups
- Number of disk drives per storage group
- Number of mailboxes
- Mailbox size limit
- User profile (estimated IOPS per mailbox)
The way that each of these factors influences performance is not the scope for this article, but if you want to know more about this subject, you’ll find, at the end of this article, some links to additional reading with detailed information about storage performance on an Exchange system.
So, let’s start configuring a Jetstress performance test:
- Start by configuring storage info. Insert the number of storage groups and disk drives. Jetstress won’t accept using the same drive for both database and logs unless you tick the checkmark on “Use storage volumes on NAS”. Selecting this option also causes Jetstress to run the performance analysis using database counters instead of LogicalDisk counters. For the purpose of this article we’ll do a simple test using just one storage group and one disk drive.
Figure 3: Storage Info
- Select the Test Run Info tab to configure test settings. Here is where you have the option to select the kind of test you want to perform: Performance, Stress or Streaming Backup. Additionally you have some tuning options that I advise you to leave on the default settings. Microsoft doesn’t recommend that you turn off automatic tuning unless you have a specific testing purpose.
Figure 4: Test Run Info
- Finally, the Database Info tab is where you’ll find the options to configure the test database. It is very important to create Jetstress databases that are matched appropriately to the expected database sizes in the production environment. Generally, the larger the Exchange database, the slower the disks respond to read and write requests. The database size will be roughly given by the ‘number of mailboxes’ and ‘mailbox size limit’. You must also configure the user profile by specifying the expected IOPS per mailbox. There are 3 well accepted usage patterns that you can use: light profile (0.2), average profile (0.5) or heavy profile (1.0).
Figure 5: Database Info
- Before running the test, you may want to save your configuration for future use by selecting “Save Configuration” on the File menu. Although you can immediately hit the “Start Test” button I like to populate the database first by clicking “Prepare Database”. Jetstress will first create a single .edb file and then copy it 5 times in order to reduce the time required to create the database. Jetstress will always validate the input parameters before starting any operation.
Figure 6: Database creation process
- When you hit “Start Test”, the automatic tuning process starts with a set of default parameters if the server has not run Jetstress before. The tuning process will try to adjust so that the following ratios are met:
- Database disk read/sec to disk write/sec is 3:2 to 2:1.
- Database disk transfer I/O to Log disk transfer I/O is 8:1 to 12:1.
- Log disk bytes /write should be is within 6 to 8 K.
- After tuning ends, Jetstress starts the actual test, collecting performance counters during the time specified. In our case, since we’re doing a regular performance test, this period will be 2 hours. As I said before, it is recommended that you run Jetstress before actually installing Exchange, but if it is already installed, make sure the following services are stopped: System Attendant, Information Store and IIS Admin.
Analyzing Results
After the test is completed, the performance data is analyzed and reported in a summary report. Results will be saved to Performance_(DateTime).html or Stress_(DateTime).html file. The status pane of Jetstress 2004 will provide a link to the summary report. All the performance counters collected will be gathered in a counter log file named Performance_(DateTime).blg that you can use for some more advanced analysis.
Consider the following guidelines when examining the data collected.
Performance Counter Instance
Guidelines for Performance Test
Guidelines for Stress Test
Database Avg. Disk sec/Read
The average value should be less than 20 ms (.020), and the maximum value should be less than 50 ms.
The maximum value should be less than 100 ms.
Database Avg. Disk sec/Write
This counter is not evaluated to determine whether a test passed or failed, but in an environment where storage replication is not being used, the average value should be less than 20 ms (.020).
Log Avg. Disk sec/Read
The average value should be less than 20 ms, and the maximum value should be less than 50 ms.
The maximum value should be less than 50 ms.
Log Ave. Disk sec/Write
Log disk writes are sequential, so average write latencies should be less than 10 ms, with a maximum of no more than 50 ms.
The maximum value should be no more than 100 ms.
Database Disk Reads/Sec
Database Disk Writes/Sec
Log Disk Writes/Sec
The sum of the averages for these values gives you the total disk transfer I/O. The ratio between read and write should be approximately 3:2.
Log Avg. Disk Bytes/write
This value should be between 6 to 8 K.
%Processor Time
Average should be less than 80 percent and the maximum should be less than 90 percent.
Available Mbytes
Minimum should be more than 50 MB.
Free System Page Table Entries
Minimum should be more than 5000.
Pages/sec
Average should be less than 100, and the maximum should be less than 1000.
Pool Nonpaged Bytes
Maximum should be less than 75 MB.
Pool Paged Bytes
Maximum should be less than 180 MB.
Database Page Fault Stalls/sec.
Should never go above 0.
Table 1: Guidelines for examining Jetstress 2004 analysis reports
If I had to choose just one performance counter to determine the performance of an Exchange system, that would be Database Avg. Disk sec/Write. But, if you noticed in the previous table, from version 6.5.7720 or later of JetstressUI, the database disk write latency criteria has been removed. Why? Because Jetstress now supports complex disaster recovery scenarios, where data replication technologies are used (sometimes based on geographically dispersed clusters). Although Jetstress doesn’t take this counter into account, if you’re not testing an infrastructure with data replication, my advice is that you still use it but make sure it stays below 20ms for 95% of the time.
The tool will produce a nice HTML report, the Jetstress Test Result, a file named Performance_(DateTime).html. If you open it you’ll see some tables with performance analysis, like the following ones:
Total test database size
Production data size
Total number of databases
Expected I/O
Achieved I/O
48.72 GB (based on the attached database)
(n/a)
6 (1 storage(s) * 6 database(s))
250.00 (500 mailboxes of 0.50 IOPS)
343,45
Table 2: Planned disk subsystem profile
Volume
Avg. Disk sec/Read
Avg. Disk sec/Write
Disk Reads/sec
Disk Writes/sec
Avg. Disk Bytes/Write
Data (C:)
0,009
0,005
81,38
62,07
(n/a)
Log (D:)
0,005
0,002
85,36
41,97
10303,92
Table 3: Disk subsystem performance summary
Alternatively you can build your own performance graphics, using the counter log file. See Figure 7 for an example of some disk performance counters.
Figure 7: Disk performance counters
Conclusion
Jetstress is really a neat tool. Making sure your storage is correctly sized will avoid future headaches due to performance bottlenecks. In this article I gave some basic guidelines on how to operate the graphical user interface of Jetstress. If you are one of those command-line addicted guys, give it a try instead of the command-line version. Also don’t forget to carefully read the manual (JetStress.doc) that comes with the tool where you’ll find much more in-depth technical stuff.
Additional Reading
JetStress Download Package
http://go.microsoft.com/fwlink/?LinkId=27883
Optimizing Storage for Exchange Server 2003
http://go.microsoft.com/fwlink/?linkid=49324
Exchange Server 2003 Performance and Scalability
http://go.microsoft.com/fwlink/?linkid=47576
Planning an Exchange Server 2003 Messaging System
http://go.microsoft.com/fwlink/?linkid=47584
Exchange Server 2003 Deployment Guide
http://go.microsoft.com/fwlink/?linkid=47569
Exchange Server 2003 Administration Guide
http://go.microsoft.com/fwlink/?linkid=47617
Troubleshooting Exchange Server 2003 Performance
http://go.microsoft.com/fwlink/?LinkId=22811
Blog about Exchange Pre-Deployment Testing and Sizing
https://blogs.msdn.com/hmlee
