Exchange 2010 Mailbox Performance Testing

Introduction

I am currently in the process of designing a large (> 16,000 mailbox) Exchange 2010 environment for a customer. One of the issues is of course designing and testing the storage environment for the Mailbox Servers.

In Exchange Server 2010 Microsoft has made considerable changes to the Database architecture, compared to previous versions of Exchange Server the performance gain can be from 70% to 90%. This is huge… One of the design goals was to use large SATA disks where both the Mailbox Database as well as the log files can be stored. A prerequisite for using SATA disks in a configuration like this (JBOD, Just a Bunch of Disks) is that you have configured multiple copies of the Mailbox Databases in a Database Availability Group (DAG) of course. This just begs for testing!!

But, before starting to play with the available test tools I want to give some background information on the configuration I would like to test.

In my situation, we are thinking of locating 4,000 mailboxes on one Mailbox Server. We will implement a DAG for redundancy purposes, the Mailbox Servers will have Direct Attached Storage (DAS), but with a RAID-6 configuration. Why RAID-6? RAID-6 is like a RAID-5 configuration, but two disks will be used for storing parity information. If one disk fails the storage array will not start immediately with rebuilding the array and there is plenty of time for replacing the disk in the array. Also the idea of reseeding large Mailbox Databases in the DAG is not something the customer is fancying. But remember, this is a customer decision, driven out of Exchange Administration and not out of functionality!

In order to design a Mailbox Server configuration like this you have to use the Exchange 2010 Mailbox Server Role Requirements Calculator which can be found on the Exchange Team site. The following numbers are used as input for the Requirements calculator:

The requirements calculator will return the following data:

So, according to the requirements calculator each database will generate 32 IOPS, which totals to 234 IOPS per server (for the active databases). Now this is an interesting thing to test.

Microsoft has two tools available for testing your Exchange 2010 environment:

• Jetstress – Jetstress is a tool that can be used to stress test your storage subsystem for maximum performance. You use Jetstress before you build your Exchange environment. You can download Jetstress from Microsoft’s website.
  
• Loadgen – Loadgen is a tool that can be used to stress test your Exchange environment. You use Loadgen after you have built your Exchange environment. Loadgen is available for download from the Microsoft website as well. The current version (April 2010) is still in beta, but according to Microsoft the final version will not differ much from this beta version.

In order to test the Mailbox Servers, I have used the above mentioned numbers on a HP Lefthand P4100 iSCSI storage device. The Lefthand unit is configured with 2 RAID-5 sets (each set having 6 disks). The servers are running on a HP DL380G6, the iSCSI is connected using two network paths.

Each server is configured with 7 LUN’s, where both the Mailbox Database as well as the log files are stored. Since I was limited by time for this particular test, I did not have the option to test the DAG environment, so I only tested a single server environment (with 4,000 mailboxes).

JetStress

The first tests were conducted with Jetstress. As explained earlier, Jetstress is developed for testing the storage subsystem. Install Windows Server 2008 (or R2), configure the storage and install Jetstress on the server. Jetstress needs some ESE files for the database engine (ese.dll, eseperf.dll, eseperf.hxx and eseperf.ini) so copy these files from the Exchange installation DVD to the Jetstress directory on the server and you’re ready to go.

Start the JetStress tool and select “start a new test”. Follow the wizard to create a new test configuration. You can choose whether to test the disk system throughput or an Exchange Mailbox Profile. Select the “disk system throughput” and continue the wizard. In the “Select Capacity and Throughput” window there is the option to enable automating tuning of Jetstress, which is enabled by default. This option however fails quite some time for an unknown reason (unfortunately), so a better approach is to disable this function (just check the checkbox) and manually increase the number of threads.

Figure 1:  Suppres the autotuning and manually increase the number of threads

Follow the wizard and select the test duration. The minimum time you can select is 2 hours, but you can manually enter 1 hour to run JetStress. In the Database Configuration window enter the number of databases and enter each database’s location and log file path.

If this is the first time you are running JetStress, the tool needs to create the databases on the specified locations. Please be aware that this can take a tremendous amount of time, sometimes even more than 24 hours! In the Review and Execute Test window you can select “Prepare test”, in this case the databases will be created.  If the databases are already created you can select “Execute test” and Jetstress will start to run. To see what is going on you can start the Performance Monitor and monitor the physical disk counters of the disks you are running JetStress against. More information on monitoring the disk counters can be found on the Microsoft website.

When Jetstress finishes it shows whether the test failed or succeeded. If it failed the threshold of disk related counters are above the recommended values, if the test succeeded the thresholds are below the recommended values. Jetstress also shows an overview of these counters, and their values:

Figure 2: Part of the Jetstress results page.

Clearly visible in Figure 2 is the achieved transaction I/O and the performance counters monitored by JetStress. For readability purposes some counters have been removed. It is obvious that, although we have only tested one Mailbox Database, the storage subsystem is capable of handling more IOPS than we initially calculated.

LoadGen

Loadgen is a tool that can be used to test the complete Exchange environment, after you have built the Exchange environment. According to the Exchange Mailbox Server design the Mailbox Server is connected to 7 LUN’s and 7 Databases are created. For this test the Database and the accompanying Log Files are located on the same disks.

So, when you’ve determined that your storage subsystem is capable of handling the expected load you can build the Exchange Servers according to your design. Install LoadGen on a Client Server and when finished installing you can start the LoadGen utility. Follow the wizard to start a new test and create a new test configuration.

Enter the length of a “simulation day”, enter the credentials of the account the tool is using (by default the administrator account is used) and continue to the recipient management. Right here you should see the 7 databases we created earlier and enter the number of mailboxes per database. In our test environment we enter 571 mailboxes per database (a total of 3997 mailbox on the server).

If needed you can select the option to use Dynamic Distribution Groups, use contacts or use external contacts.

The “test user group” settings are important because it is here that you define the usage profile. You can select the type of client (for example Outlook 2007 cached mode), the usage profile and the Mailbox sizing. You can start with a light profile and increase the usage profile with subsequent tests. It is also possible to create multiple types of clients and use them simultaneously.

When you have multiple LoadGen clients you can configure remote configurations, and the LoadGen tests will be conducted from multiple client machines.

The last step is to initialize the test, in this initialization the 7 Mailbox Databases will be created. Like the JetStress tool this can take a tremendous amount of time, up to 24 hours. When the databases are created you can start the LoadGen tool and a working day will be simulated.

LoadGen does not monitor and present theperfmon counters like the JetStress tool, so you have to configure the Performance Monitor tool to log the various counters.

Figure 3: The performance logging results when running LoadGen. For readability purposes data from MDB05, MDB06 and MDB07 is removed

Conclusion

You can use the Storage Requirements Calculator to properly design a storage environment. For the purpose of this article I have tested one Mailbox Server with 4000 Mailboxes, spread across 7 Mailbox Databases. Using the Jetstress tool you can test the storage subsystem, using the LoadGen tool you can test the final Exchange Server installation.

Using these test tools you can clearly see, in advance, how your Mailbox Server will behave.