If you would like to read the next part in this article series please go to Deploying Exchange 2007 Multi-site CCR Clusters – Do’s and Don’ts (part 2).
Introduction
Back in 2006, from when Exchange 2007 RTM’d up until Exchange 2007 SP1 RTW’d, the Exchange Product group not only considered Cluster Continuous Replication (CCR) to be a local datacenter high availability solution but also a viable site resilience solution. Actually, CCR was the only native functionality that could be used to provide site resilience. Of course there were/are alternatives. You could/can use Single Copy Clusters (SCCs) combined with a software-based third party replication solution or have the storage vendor replicate data on the storage layer (typically block level). Several vendors also make it possible to combine CCR with storage layer replication, but why not take advantage of native functionality rather than investing in an expensive 3rd party solution?
Approximately a year later came Exchange 2007 SP1 and with it a brand new feature known as Standby Continuous Replication (SCR). SCR was developed primarily for site resilience scenarios. But Exchange 2007 SP1 also improved the existing CCR feature by making it possible to have the Windows failover cluster nodes located on separate subnets. So, although SCR does a pretty good job, some Exchange architects for some reason or the other still want to deploy multi-site CCR clusters. This is especially true now that you do not have to stretch the subnets. Quite frankly, a multi-site CCR cluster can be the right solution if you deploy it properly and have an infrastructure/topology that goes hand in hand with this type of resilience solution. Please understand, a multi-site CCR cluster can be an absolute nightmare if deployed in an infrastructure that does not live up to the requirements and/or the solution is not deployed properly. To any confusion, this article will explain the ramifications of multi-site CCR clusters as well as provide best practice recommendations.
Important:
Note this four part article isn’t a step by step guide on how you deploy multi-site CCR cluster. For general CCR cluster step by step guides see my Deploying CCR on Windows Server 2003/2008 article series:
What Operating System should I use?
So on which operating system should I install my multi-site CCR cluster nodes? Had you asked one year ago, I would have been tempted to recommend Windows Server 2003, but Windows Server 2008 includes some interesting improvements in regards to CCR clusters. Also, now that we’re at Exchange 2007 Rollup Update 7 (RU7), most of the bugs/issues that existed when installing Exchange 2007 SP1 on Windows Server 2008 have now been fixed.
Among the CCR related improvements are support for multi-subnet failover clusters and faster log file shipping. As you may know the Exchange Replication Service uses SMB for log file shipping. Windows Server 2008 introduces SMB v2, which does the job 30-40% faster than SMB v1 did.
The Windows 2008 Failover Cluster deployment experience is also much smoother than what we’re used to in Windows Server 2003. Move over, you should note that Windows Server 2003 will retire within the next couple of years (for exact details, check out the Microsoft Support Lifecycle website for Windows Server 2003 here). Finally, have in mind that you won’t be able to in-place upgrade Windows Server 2003 based Exchange 2007 servers to Windows Server 2008 at a later point in time should you decide to deploy Exchange 2007 on Windows Server 2003. Instead, you must replace the existing servers with new Windows Server 2008 based Exchange 2007 SP1 servers. Alternatively, you can uninstall Exchange 2007 from the existing machines and then in-place upgrade them to Windows Server 2008 followed by re-installing Exchange 2007 SP1. When Exchange 2007 SP1 has been installed, you can restore mailboxes/public folders from backup, re-connect disconnected LUNs, copy the EDB files back to the Mailbox servers or whatever strategy you prefer to use.
For more information on how to migrate Windows Server 2003 based Exchange 2007 server to Exchange 2007 SP1 on Windows Server 2008, see this article on Microsoft TechNet.
Based on the above, my recommendation is to install multi-site CCR cluster nodes (and any other Exchange 2007 server roles for that matter) on Windows Server 2008 machines.
Stretched or Non-Stretched Subnets?
If you’re installing the multi-site CCR cluster nodes on Windows Server 2003 based machines; the cluster nodes must be located on the same subnet. This is because of the fact that Windows Server 2003 Clusters doesn’t support locating cluster nodes on different subnets. In this case, you must stretch the subnets between the two datacenters for private (heartbeat), the public network as well as any other networks used by the cluster nodes. Stretching the subnets means that broadcast traffic will be sent over the WAN between the datacenters. Taking the folks in your organization’s network department into consideration, there’s a good chance this requirement is a showstopper. But personally I’ve worked on a few projects where this wasn’t an issue as the organization had 10GB links between the datacenters. We simply created a VLAN and stretched it between the datacenters.
With Windows Server 2008 stretched subnets are no longer a requirement as Windows Server 2008 has full support for routed networks. As you can see in the Create Cluster Wizard in Figure 1, we now have the option of specifying two IP addresses for the Windows Server 2008 Failover Cluster – one belonging to the subnet in the primary datacenter and one in the backup datacenter.
Figure 1: Specifying an IP address for the Windows Server 2008 Cluster name on two subnets
Note
If the interface for the private network isn’t residing in a stretched subnet, you must use routing between the two different networks on each physical site. When this is the case, there are special requirements when it comes to configuring the interface used for the private network. Since Tim McMichael (Enterprise Messaging Support Professional with MSFT) has written a great blog on this specific topic, I won’t go into the details in this articles series.
In Figure 2, you can see a dependency tree with a dependency structure based on “OR” for the two IP addresses (located on separate subnets) assigned to the cluster network name resource. Traditional clusters supported “AND”, but Windows Server 2008 Failover cluster also support “OR” which is used when multi-site clusters have IP addresses on separate subnets.
Figure 2: Windows Server 2008 Failover Cluster Dependency tree
To create a dependency report, right-click on the cluster network name resource and select Dependency Report in the context menu as shown in Figure 3.
Figure 3: Creating a Dependency Report
When the Windows Server 2008 Failover Cluster has been created and you install the Exchange 2007 Active Mailbox role on the cluster node in the primary datacenter, you have the option of specifying the IP address that should be assigned to the Clustered Mailbox Server (CMS) on each subnet (Figure 4). Although I don’t recommend it, you can even use DHCP assigned IP addresses and/or IPv6 address if you like.
Figure 4: Specifying an IP address for the CMS on both Subnets
Even though you can install the Active and Passive Mailbox server roles using the Exchange 2007 Setup wizard, you can run into a few issues when doing so. So instead, I recommend you use the command line setup command when you install a new CMS. To install the Active Clustered Mailbox role with Setup.com on the cluster node in the primary datacenter, use the following command:
Setup.com /Mode:install /Roles:mailbox /NewCMS /CMSName:<CMS name> -CMSIPv4Addresses:<IP1>,<IP2>
Figure 5: Installing the Active Clustered Mailbox Role using Setup.com
To install the Passive Mailbox role on the cluster node in the backup datacenter, use the following command:
Setup.com /Roles:mailbox
Figure 6: Installing the Passive Clustered Mailbox role
Network Cards
A best practice recommendation is to have 4 network cards (or more) installed in each CCR cluster node. Usually, I install 5 network cards in each node. Two network interfaces in a public team (configured in fault tolerance mode only. Don’t load balance them as experience has shown this can cause miscellaneous issues). Two network interfaces for log shipping (in order to accomplish log shipping redundancy). Finally one network interface for backup (disabled for cluster use) since you don’t want to perform backups of databases over the public network. This provides us with 3 independent paths for replication and 3 independent paths for heartbeat communications between the cluster nodes.
When you have this many network cards installed in each cluster node, it’s highly recommended you give them meaningful names in the Failover Cluster Management console as shown in Figure 7. Simply right-click on them and select the rename in the context menu.
Figure 7: Naming the networks with more meaningful names in the WFC console
Although not specifically minded CCR clusters, I also recommend that you disable IPv6 on all Exchange servers and domain controllers (unless you’re one of the few who actually use IPv6!). To disable IPv6 see the IPv6 FAQ here.
You should also be aware of the impact the Scalable Networking pack feature can have on any Windows 2003 based Exchange 2007 servers in your environment. For detailed information about the Scalable Networking Pack and how it affects Exchange, please see: http://msexchangeteam.com/archive/2007/07/18/446400.aspx. It’s usually a good idea to disable scalable networking pack features on all Exchange 2007 servers in your organization.
We have now reached the end of part 1, but you can look forward to part 2 in a very near future.
If you would like to read the next part in this article series please go to Deploying Exchange 2007 Multi-site CCR Clusters – Do’s and Don’ts (part 2).
