If you would like to read the first part in this article series please go to The Exchange 2007 Transport Dumpster (Part 1).
Introduction
This is the second and final part of a two-part article looking at the Exchange 2007 transport dumpster feature. In part one of this article series, we covered what the transport dumpster feature is, how to configure it, and how it looks from an operational perspective. To complete this series, we are going to look at how to re-size the transport dumpster and then finish off by looking at how it can be monitored in your environment.
Sizing The Transport Dumpster
Sizing of the transport dumpster has an impact on the design of your Hub Transport server, since you will have to take into consideration the size of the message queue in the transport dumpster as well as the additional disk I/O that will be seen when the transport dumpster is enabled. Let us look at how you can plan for these two factors.
Admittedly, the actual disk size of the transport dumpster is not necessarily that big unless the Hub Transport server is servicing a large number of storage groups. Remember that the transport dumpster sizing is configured on a per-storage group basis. Let us first take an example environment of two Hub Transport servers that service a single CCR environment that itself contains 10 storage groups. If the maximum message size within this organization is 30MB, we have already seen in part one of this article series that the MaxDumpsterSizePerStorageGroup parameter would be set to 45MB. Therefore, the actual amount of disk space that you would need to allow for would be 450MB, derived from 45MB x 10 storage groups. Naturally you would want to allow this amount of disk space on each Hub Transport server, since you would want to ensure full system operation in the event that you lost one of the two Hub Transport servers.
Let us consider another example environment consisting of two Hub Transport servers that both service three CCR environments that each contain a total of 15 storage groups with a maximum permissible message size of 50MB. In this case, the MaxDumpsterSizePerStorageGroup parameter would be set at 75MB, since 1.5 x 50MB is 75MB. Since there would be a total of 45 storage groups, you would need to plan for approximately 3.4GB of disk space for the transport dumpster, since 75MB x 45 storage groups is approximately 3.4GB.
Do not forget that the disk space requirements of the transport dumpster relate to the disk holding the Hub Transport server’s database file mail.que, since the transport dumpster is contained within this database. Since the Hub Transport database is an ESE database, it is normal to see the mail.que file moved to dedicated disk spindles for performance reasons.
For the effects on disk I/O caused by the enabling of the transport dumpster feature, it’s best to turn to the information supplied by Microsoft as a result of its internal testing. In the Transport Server Storage Design information, scroll down until you find the section called Transport Dumpster Sizing Example where you will see that the second table listed shows you the differences in I/O caused when the dumpster is enabled. In short, enabling the transport dumpster does increase the disk I/O so make sure that you plan for this.
Monitoring the Transport Dumpster
In a CCR environment, the Get-StorageGroupCopyStatus cmdlet can be used to obtain information on the overall health of continuous replication. There is an extra parameter, not used by default, that can be used to display information on the transport dumpster statistics. This parameter is DumpsterStatistics. For example, Figure 7 shows the Get-StorageGroupCopyStatus cmdlet run with the output piped to the format-list cmdlet and additionally filtered to show any parameters containing the string dumpster. As you will see from Figure 7, the parameters are repeated twice since there are two storage groups on this particular CCR environment.
Figure 7: Get-StorageGroupCopyStatus Cmdlet
What you will note from Figure 7 is that there are no dumpster statistics shown. To actually see these statistics, you will need to include the DumpsterStatistics parameter such as in this example cmdlet:
Get-StorageGroupCopyStatus –DumpsterStatistics | fl *dumpster*
The results of running this cmdlet are shown in Figure 8, where it may become apparent that this cmdlet has been run in a clean test environment. The DumpsterStatistics parameter shows 0 items in the dumpster for the Hub Transport server called SRV1, with these 0 items obviously totaling 0KB in size. Interestingly, you can also see the effects of when a Hub Transport server in the same Active Directory site as the CCR environment is not contactable. In this case the Hub Transport server called SRV2 is shown as being unavailable as it appears in the DumpsterServersNotAvailable list.
Figure 8: Get-StorageGroupCopyStatus Cmdlet With DumpsterStatistics
Let’s look again at the statistics on a slightly busier server. In Figure 9 you can see that the statistics for a single storage group have been retrieved and that the transport dumpster for the Hub Transport server SRV1 now contains 1,030 items at a total size of 819KB. You will also see the date and time of the oldest message in the transport dumpster queue.
Figure 9: Transport Dumpster Statistics
To get more detailed information on the transport dumpster statistics, we can use the performance monitor program. On a Hub Transport server, a performance object called MSExchangeTransport Dumpster can be found that contains 5 counters. You can see these counters in Figure 10.
Figure 10: Transport Dumpster Performance Counters
The counters are:
- Dumpster Deletes/sec – This is how many items per second that are deleted from the transport dumpster queue on this Hub Transport server
- Dumpster Inserts/sec – This is how many items per second that are submitted to the transport dumpster queue on this Hub Transport server
- Dumpster Item Count – This is the total number of items currently in the transport dumpster queue on the particular Hub Transport server. It is essentially the same information that you can obtain from the DumpsterStatistics parameter of the Get-StorageGroupCopyStatus cmdlet that we’ve already covered
- Dumpster Size – The Dumpster Size counter is the total size of the items in the transport dumpster on the particular Hub Transport server. This counter is measured in bytes. Again, this counter shows the same information that you can see via the Get-StorageGroupCopyStatus cmdlet
- Redelivery Count – This is simply a count of the number of items that were redelivered after the Exchange Replication Service requested a resubmit of messages.
For example, on a test CCR environment that I created, I forced a lossy failover situation by turning off the active node whilst messages were being processed. Once the Microsoft Exchange Replication Service on the remaining CCR node had requested a resubmit of messages from the Hub Transport server, I then ran the performance monitor tool on that Hub Transport server and measured the Dumpster Item Count, Dumpster Size and Redelivery Count objects as you can see from Figure 11. It is clear from this figure that the entire transport dumpster queue of 1540 items has been redelivered.
Figure 11: Measuring Transport Dumpster Performance Counters
So what actually happens during a lossy failover event in a CCR environment? Simply put, the previously passive node now becomes the active node and requests from the Hub Transport servers in the same Active Directory site that messages from the transport dumpster are resubmitted. To see this process in action, check the event log on the remaining CCR node for event 2099 that has a source of MSExchangeRepl. An example of this event log entry is shown in Figure 12. You can see from this figure that the CCR environment node called CCRB has requested that the Hub Transport server called SRV1 resubmits messages between the specific time periods. Note that this event will be recorded every 15 minutes if the particular Hub Transport server is unavailable.
Figure 12: Transport Dumpster Resubmission Request
Summary
That concludes this two-part article series on the Exchange 2007 transport dumpster feature. There is little doubt in my mind that this is an incredibly useful feature of Exchange 2007 as it can help an organization recover from data loss that may be experienced during an unscheduled outage of a CCR environment. As long as you have taken the time to configure the transport dumpster parameters, it should work with little administrator intervention.
If you would like to read the first part in this article series please go to The Exchange 2007 Transport Dumpster (Part 1).
