Had to perform an Exchange 2010 site switchover for a customer that had lost all servers in the primary datacenter this weekend (not only Exchange but everything!). 5 months ago their messaging solution was based on a non-redundant Exchange 2003 configuration (talk about good timing).
So the the customer uses an Exchange 2010 active/passive user distribution datacenter model with a shared namespace similar to scenario 1, which I describe in part 1 of my “Designing a Site Resilient Exchange 2010 Solution” articles series. Because the customer have LAN quality communication between the datacenters, they use a stretched Active Directory site/subnet. In addition, the DAG runs in DAC mode.
To get Exchange 2010 up and running in the failover datacenter, I basically went through the steps described in part 12 of my “Planning, Deploying, and Testing an Exchange 2010 Site-Resilient Solution sized for a Medium Organization” articles series.
First, I used “Stop-DatabaseAvailabilityGroup DAG01 –MailboxServer <DAG member server> –ConfigurationOnly” to stop the respective Mailbox servers located in the primary datacenter that participated in the DAG. I used the “ConfigurationOnly” parameter since all Domain Controllers in the primary datacenter was unavailable.
After having stopped the Mailbox servers in the primary datacenter, I used “Get-DatabaseAvailabilityGroup DAG01 | FL Name,StoppedMailboxServers,StartedMailboxServers” to verify the servers had been stopped and that the Mailbox servers in the failover datacenter was started.
Then I stopped the cluster service on the Mailbox servers in the failover datacenter as this is necessary prior to restoring the DAG using the “Restore-DatabaseAvailabilityGroup” cmdlet.
After having stopped the cluster service, I could concentrate on restoring the DAG in the failover datacenter. For this, I used the following command:
Restore-DatabaseAvailabilityGroup DAG01 –ActiveDirectorySite <AD site>
I used the “ActiveDirectorySite” parameter since a “MailboxServer” parameter doesn’t exist for the “Restore-DatabaseAvailabilityGroup” cmdlet (see http://technet.microsoft.com/en-us/library/dd351169.aspx).
The resulted in an error message similar to the following:
Confirm
Are you sure you want to perform this action?
Restoring Mailbox servers for Active Directory site “<AD site>” in database availability group “DAG01”.
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is “Y”): y
WARNING: Server ‘<Exchange Servers>’ was marked as stopped in database availability group ‘DAG01’ but couldn’t be removed
from the cluster. Error: A server-side database availability group administrative operation failed. Error: The
operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error
occurred while attempting a cluster operation. Error: Cluster API ‘”EvictClusterNodeEx(‘<Exchange Servers>’) failed
with 0x46. Error: The remote server has been paused or is in the process of being started”‘ failed. [Server: <Exchange Servers>]
WARNING: The operation wasn’t successful because an error was encountered. You may find more details in log file
“C:\ExchangeSetupLogs\DagTasks\dagtask_2011-10-08_21-22-17.549_restore-databaseavailabilitygroup.log”.
I suspected this was because Exchange tried to communicate with a Domain Controller in the failed datacenter, so I made sure Exchange used a domain controller in the failover datacenter using this command:
Set-ADServerSettings –PreferredServer <Domain Controller in Failover Datacenter>
Then I ran Restore-DatabaseAvailabilityGroup DAG01 –ActiveDirectorySite <AD site> again and voilá I got this result:
Confirm
Are you sure you want to perform this action?
Restoring Mailbox servers for Active Directory site “<AD site>” in database availability group “DAG01”.
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is “Y”): y
WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server
file02.domain.com.
And databases began to mount on DAG member servers in the failover datacenter.
So what have I learned? Remember to use the “Set-ADServerSettings” cmdlet to make sure Exchange tries to communicate with a Domain Controller in the failover datacenter, when you’re dealing with a stretched AD site/subnet.
Until later,
Henrik Walther
Technology Architect/Writer/MS Vendor
MCM: Exchange Server | MVP: Exchange Architecture