Categories Exchange 2003Tools

JetStress Is Your Friend

You can’t call yourself a true Exchange Administrator until you have run the JetStress Tool at least once. This nice piece of software addresses one of the most critical areas of Exchange server, the storage subsystem. You can use it to verify the performance and stability of the disk subsystem prior to putting an Exchange server into a production environment or as a troubleshooting tool when in the presence of a bottleneck.

Introduction

Jetstress is a tool that helps verify the performance and stability of the disk subsystem by simulating some heavy loads to the disks, as if the server was being used by a large number of users. You then use System Monitor, Event Viewer and the Eseutil tool together with Jetstress to verify that your disk subsystem meets the established performance criteria.

Jetstress enables you to do three types of tests:

  • Disk Performance Test runs for 2 to 24 hours and enables you to verify the performance and sizing of your storage solution.
  • Disk Subsystem Stress Test runs for 24 hours and enables you to test the server storage reliability over a longer time.
  • Streaming Backup Test measures the database throughput and end-to-end streaming backup (both read and write) performance.

The best way to verify the integrity and performance of your disk subsystem is to run the first two tests mentioned above.

In the latest download package for Jetstress which I used (6.5.7795.0) there are two separate applications:

  • JetstressUI.exe (the graphical user interface (GUI) version of Jetstress named Jetstress 2004)
  • Jetstress.exe (the original command-line version of Jetstress)

Both versions can be used to successfully test the performance of an Exchange disk subsystem, but using the command-line requires expertise in specifying the parameters and analyzing performance results. Jetstress 2004 and its graphical user interface reduces the complexity of configuring the test. Additionally, it facilitates the analysis of results by producing a performance analysis report. There are also some unique features available in the graphical tool. For more information please read the manual that comes with the tool.

Although it is recommended that you run the tests in a non-production environment, in the real world sometimes it’s not possible to have a lab that mirrors the actual systems. So, my advice here is that you use the production server just before going live (even before installing Exchange), but remember to format the used storage volumes at the end of the tests.

Installing JetStress

Preferably, Jetstress testing should be performed before you install Exchange on the server. There are some well-known risks associated with running Jetstress on a machine with Exchange installed. The first is that Jetstress could potentially delete some existing logfiles if it's configured to use the same log drives that Exchange is using. The second is that if you use a version of Jet (ESE.DLL and ESEPERF.*) different than the version installed with Exchange, the registration of the Jet database counters in the Jetstress install directory will break the database counters for Exchange after Jetstress is removed.

To install Jetstress follow these steps:

  1. Ensure that installation requirements are met:
    • Jetstress 2004 requires that the server that is running has .NET Framework Version 1.1 installed.
    • Jetstress is supported only on Windows 2000 Server and Windows Server 2003. Microsoft Windows NT 4.0 and older versions are not supported (who uses them these days?).
    • Although not really a pre-requisite, consider following these recommendations: use Windows Hardware Quality Labs (WHQL) certified hardware, verify that all the storage components have the latest firmware and, finally, consider aligning the disks with diskpart (for information about how to run diskpart, read Disk Geometry).
  2. Extract the files of the Jetstress tool download package to a local directory, for example C:\Jetstress, and start the installer by double-clicking the Jetstress.msi file. Follow the prompts to install the tool on your computer. The setup will install the tool into Program Files, but you can move the files to any folder you want.
  3. Copy the following ESE database modules to the directory where you installed Jetstress.
    • ESE.DLL
    • ESEPERF.DLL
    • ESEPERF.INI
    • ESEPERF.HXX

If you forget to copy these files, you’ll get the following warning when you try to run the tool:


Figure 1: Validating required files

  1. Double click the JetstressUI.exe file. Running the tool will trigger some validation tests, such as the one mentioned in the previous step. Checking if the LogicalDisk counters are enabled is another test that is run and that you have the option to fix.


Figure 2: Validating performance counters

Running the Tests

Before running the test, there are some settings you must first configure and that will impact your system’s performance:

  • Number of storage groups
  • Number of disk drives per storage group
  • Number of mailboxes
  • Mailbox size limit
  • User profile (estimated IOPS per mailbox)

The way that each of these factors influences performance is not the scope for this article, but if you want to know more about this subject, you’ll find, at the end of this article, some links to additional reading with detailed information about storage performance on an Exchange system.

So, let’s start configuring a Jetstress performance test:

  1. Start by configuring storage info. Insert the number of storage groups and disk drives. Jetstress won’t accept using the same drive for both database and logs unless you tick the checkmark on “Use storage volumes on NAS”. Selecting this option also causes Jetstress to run the performance analysis using database counters instead of LogicalDisk counters. For the purpose of this article we’ll do a simple test using just one storage group and one disk drive.


Figure 3: Storage Info

  1. Select the Test Run Info tab to configure test settings. Here is where you have the option to select the kind of test you want to perform: Performance, Stress or Streaming Backup. Additionally you have some tuning options that I advise you to leave on the default settings. Microsoft doesn’t recommend that you turn off automatic tuning unless you have a specific testing purpose.


Figure 4: Test Run Info

  1. Finally, the Database Info tab is where you’ll find the options to configure the test database. It is very important to create Jetstress databases that are matched appropriately to the expected database sizes in the production environment. Generally, the larger the Exchange database, the slower the disks respond to read and write requests. The database size will be roughly given by the 'number of mailboxes' and 'mailbox size limit'. You must also configure the user profile by specifying the expected IOPS per mailbox. There are 3 well accepted usage patterns that you can use: light profile (0.2), average profile (0.5) or heavy profile (1.0).


Figure 5: Database Info

  1. Before running the test, you may want to save your configuration for future use by selecting “Save Configuration” on the File menu. Although you can immediately hit the “Start Test” button I like to populate the database first by clicking “Prepare Database”. Jetstress will first create a single .edb file and then copy it 5 times in order to reduce the time required to create the database. Jetstress will always validate the input parameters before starting any operation.


Figure 6: Database creation process

  1. When you hit “Start Test”, the automatic tuning process starts with a set of default parameters if the server has not run Jetstress before. The tuning process will try to adjust so that the following ratios are met:
    • Database disk read/sec to disk write/sec is 3:2 to 2:1.
    • Database disk transfer I/O to Log disk transfer I/O is 8:1 to 12:1.
    • Log disk bytes /write should be is within 6 to 8 K.
  2. After tuning ends, Jetstress starts the actual test, collecting performance counters during the time specified. In our case, since we’re doing a regular performance test, this period will be 2 hours. As I said before, it is recommended that you run Jetstress before actually installing Exchange, but if it is already installed, make sure the following services are stopped: System Attendant, Information Store and IIS Admin.

Analyzing Results

After the test is completed, the performance data is analyzed and reported in a summary report. Results will be saved to Performance_(DateTime).html or Stress_(DateTime).html file. The status pane of Jetstress 2004 will provide a link to the summary report. All the performance counters collected will be gathered in a counter log file named Performance_(DateTime).blg that you can use for some more advanced analysis.

Consider the following guidelines when examining the data collected.

Performance Counter Instance

Guidelines for Performance Test

Guidelines for Stress Test

Database Avg. Disk sec/Read

The average value should be less than 20 ms (.020), and the maximum value should be less than 50 ms.

The maximum value should be less than 100 ms.

Database Avg. Disk sec/Write

This counter is not evaluated to determine whether a test passed or failed, but in an environment where storage replication is not being used, the average value should be less than 20 ms (.020).

Log Avg. Disk sec/Read

The average value should be less than 20 ms, and the maximum value should be less than 50 ms.

The maximum value should be less than 50 ms.

Log Ave. Disk sec/Write

Log disk writes are sequential, so average write latencies should be less than 10 ms, with a maximum of no more than 50 ms.

The maximum value should be no more than 100 ms.

Database Disk Reads/Sec

Database Disk Writes/Sec

Log Disk Writes/Sec

The sum of the averages for these values gives you the total disk transfer I/O. The ratio between read and write should be approximately 3:2.

Log Avg. Disk Bytes/write

This value should be between 6 to 8 K.

%Processor Time

Average should be less than 80 percent and the maximum should be less than 90 percent.

Available Mbytes

Minimum should be more than 50 MB.

Free System Page Table Entries

Minimum should be more than 5000.

Pages/sec

Average should be less than 100, and the maximum should be less than 1000.

Pool Nonpaged Bytes

Maximum should be less than 75 MB.

Pool Paged Bytes

Maximum should be less than 180 MB.

Database Page Fault Stalls/sec.

Should never go above 0.

Table 1: Guidelines for examining Jetstress 2004 analysis reports

If I had to choose just one performance counter to determine the performance of an Exchange system, that would be Database Avg. Disk sec/Write. But, if you noticed in the previous table, from version 6.5.7720 or later of JetstressUI, the database disk write latency criteria has been removed. Why? Because Jetstress now supports complex disaster recovery scenarios, where data replication technologies are used (sometimes based on geographically dispersed clusters). Although Jetstress doesn’t take this counter into account, if you’re not testing an infrastructure with data replication, my advice is that you still use it but make sure it stays below 20ms for 95% of the time.

The tool will produce a nice HTML report, the Jetstress Test Result, a file named Performance_(DateTime).html. If you open it you’ll see some tables with performance analysis, like the following ones:

Total test database size

Production data size

Total number of databases

Expected I/O

Achieved I/O

48.72 GB (based on the attached database)

(n/a)

6 (1 storage(s) * 6 database(s))

250.00 (500 mailboxes of 0.50 IOPS)

343,45

Table 2: Planned disk subsystem profile

Volume

Avg. Disk sec/Read

Avg. Disk sec/Write

Disk Reads/sec

Disk Writes/sec

Avg. Disk Bytes/Write

Data (C:)

0,009

0,005

81,38

62,07

(n/a)

Log (D:)

0,005

0,002

85,36

41,97

10303,92

Table 3: Disk subsystem performance summary

Alternatively you can build your own performance graphics, using the counter log file. See Figure 7 for an example of some disk performance counters.


Figure 7: Disk performance counters

Conclusion

Jetstress is really a neat tool. Making sure your storage is correctly sized will avoid future headaches due to performance bottlenecks. In this article I gave some basic guidelines on how to operate the graphical user interface of Jetstress. If you are one of those command-line addicted guys, give it a try instead of the command-line version. Also don’t forget to carefully read the manual (JetStress.doc) that comes with the tool where you’ll find much more in-depth technical stuff.

Additional Reading

JetStress Download Package
http://go.microsoft.com/fwlink/?LinkId=27883

Optimizing Storage for Exchange Server 2003
http://go.microsoft.com/fwlink/?linkid=49324

Exchange Server 2003 Performance and Scalability
http://go.microsoft.com/fwlink/?linkid=47576

Planning an Exchange Server 2003 Messaging System
http://go.microsoft.com/fwlink/?linkid=47584

Exchange Server 2003 Deployment Guide
http://go.microsoft.com/fwlink/?linkid=47569

Exchange Server 2003 Administration Guide
http://go.microsoft.com/fwlink/?linkid=47617

Troubleshooting Exchange Server 2003 Performance
http://go.microsoft.com/fwlink/?LinkId=22811

Blog about Exchange Pre-Deployment Testing and Sizing
https://blogs.msdn.com/hmlee

Rui Silva

Share
Published by
Rui Silva

Recent Posts

Qumulo raises $125M for cloud data management across a hybrid setup

Qumulo is an up-and-coming data management solution focusing on managing files in a hybrid setup.…

1 day ago

Why SMBs need a standalone solution for Windows 10 patch management

Is patch management for the Windows PCs at your business driving you crazy? Maybe there's…

2 days ago

Microsoft Teams guest access: How to enable and manage it

Two of the main factors that affect the total cost of an organization’s Microsoft 365…

2 days ago

Samsung Galaxy Unpacked 2020: Everything you need to know

Samsung rolled out the all-new Galaxy Z Fold 2, Note 20, Note 20 Ultra handsets…

3 days ago

SAN vs. NAS: Detailed comparison of these two storage technologies

SAN and NAS provide dedicated storage for a group of users using completely different approaches…

3 days ago

Generation 1 virtual machines: Modernize them and bring them up to date

In many companies, Generation 1 virtual machines have been superseded by Gen 2 VMs. But…

3 days ago