When your cluster goes “oops,” Using RecoverCMS

First, a quick note:  I’m posting this one from Windows Live Writer on Windows 7 RC1, which I’m happy to say is remarkably stable and much faster overall than Vista.  I’d recommend it wholeheartedly!

Funny story, I once had a client who swore that clustering was enough protection for their messaging environment, until an outage took out their entire cluster at once – causing them to be down for about week.  Now, that’s not the funny part, but what caused the outage is somewhat hilarious, more on that later.

Exchange 2003 and earlier had a pretty straight-forward method for recovering an entire MSCS cluster if one had failed on you.  You built one or more nodes of a brand new cluster, created an Exchange Virtual Server (EVS) Resource Group with the same parameters (names, IP’s etc) as the production system had, and Exchange would do the rest.

With Exchange 2007, the rules changed significantly, leaving many cluster users confused as to how the system now works if they suffer a cataclysmic failure of the production cluster.  Adding both Single Copy Cluster (SCC) and Continuous Cluster Replication (CCR) to the mix just makes things more confusing, so Microsoft created a new recovery method for Exchange 2007 clusters.  Called RecoverCMS, the system is really a setup task rather than a true failover system, but since your failover system just went belly-up, that’s not a bad thing.

If your Recovery Time Objectives are flexible enough to handle some downtime if an entire cluster fails, then you can leverage this system to get back up and running, either at the original production site, or at a new location.  There are some definite limits to what you can do with it which I’ll explain later, but he basics of how it works are pretty simple.

Step one is rebuild, repair or replace the original cluster hardware. If the repair works then you’re done, just restore any missing data from tape or other backup (due disclaimer, see below, I am biased on backup tools) and then resume normal operations. If you rebuild or replace completely, bring up a new server that is configured with Exchange 2007 in the Passive Cluster Node configuration.  You can find out how to do that:

Here for CCR clustering or,

Here for SCC Clustering

During that process you will also have installed the Exchange 2007 binaries on at least one node of the cluster system, so go to the directory that has the Exchange setup files and execute the following command:

Setup.com /recoverCMS /CMSName:<name> /CMSIPaddress:<ip>

Where <name> is the name of the EVS you’re restoring from, and <IP> is the IP address you want the recovered system to have – in theory the same IP as the original EVS had.

The rest of the procedure is pretty automated, and when finished, you will have a new EVS running on your new cluster node(s) that matches the original EVS and has all the users already assigned to it.  From there, you can restore your data if it was also lost to the disaster.

There are a few things that are extremely important to be aware of before you begin:

1 – Keep in mind that /recoverCMS is designed to restore a failed cluster only.  Attempting to use it for migration or for any other purpose will result in unpredictable behavior and is not supported by MSFT.

2 – You will need to manually create the volumes that existed on the failed cluster before you run /recoverCMS.  If volumes are missing then the recovery will fail.  They don’t have to be the same physical disk or size, just large enough to hold the data and with the same drive letters as the original cluster held.

3 – The System Attendant service will start and then immediately stop after you recover, this is normal, just bring the resource back online when you’re ready.

4 – Your databases are not mounted after a recovery, you must do this manually through PowerShell or the Exchange Management Console after you’re done with the restore.

5 – Do NOT try to use this across OS’s. If you started on Windows Server 2003, you must recover to Windows Server 2003, and 2008 to 2008.   It will not work if you try to go from one to the other.

6 – While you can pre-configure many portions of this system, it will still take some time to run through a /recoverCMS procedure from start to finish, so if you need a second-stage failover, /recoverCMS isn’t the best bet.  I’m quite biased on this (see disclaimer below), but unless you can be down for a few hours if both cluster nodes fail, you might want to go with another tool to provide remote site failover in addition to SCC or CCR clustering.

7 – Finally, SCR and CCR will not automatically work with /recoverCMS.  You will need to stop SCR if it’s running before you recover, and neither will resume automatically after the recovery is done.  Once you’re set up in the new node configuration, re-enable CCR and SCR manually as required.

/RecoverCMS is a great way to restore a failed cluster system to new hardware or rebuilt hardware after a fault.  You still need to back up your data to some device outside the cluster itself, but once you have that backup /recoverCMS can get your cluster back up and running much faster than the manual methodologies used in previous versions of Exchange.

As to the funny story I mentioned at the top of the blog, this particular client was in a hardened datacenter with UPS systems, 24/7 staff and a backup generator.  They were convinced that clustering was going to be more than enough for them.  After trying to explain that a shared-disk cluster (the only option at the time) had weak points, I finally gave up and let them be.  A few months later I got a great phone call.  Apparently – unbeknownst to the client – the datacenter crew had run all power connections through the UPS – including the generator.  The UPS was rated to handle the full power load of the datacenter on 1 of its 2 redundant circuit loops.  So far so good.  Well, this particular datacenter was in the middle of the dot-com boom (this was some time ago) and had grown exponentially in a short period of time.  What they had was well over half the full expected load on each of the two circuits, and one was failing.  So they diligently got replacement parts and moved the load over to the good circuit.  Since was over half the expected load, and circuit 2 was already under over half the load, they immediately overloaded the UPS, shorting it out.  The way it was explained to me, a solenoid shot through the casing of the UPS…and there was indeed a nice hole in the unit to back that up. No one was hurt, but needless to say, the whole datacenter was offline until they replaced the UPS, 4 days later, so they lost about one business week, without anything happening to the physical cluster at all.  Just goes to show you that anything that can go wrong, will.

The Dread Pirate Re-Seed – Part 2

Last week, we talked about re-seeding operations between two nodes of an Active/Passive CCR Cluster.  Since both nodes will most likely be inside the same logical network (though are not required to be in Server 2008), even unexpected re-seed operations on a CCR Cluster should be relatively painless.  Though it’s a full copy to target, it is happening over a LAN, and therefore faster and less resource-intensive than a WAN copy would be.

This week, let’s look at how the game changes when you work with Server (or Standby) Continuous Replication (SCR) when you have re-seed operations.  Just to be clear, re-seeds are not the normal method for data protection with SCR. Normally, when a log file is closed and a new prime log (usually E00 for the first Storage Group) is created, the closed log is shipped over to a target Exchange 2007 if SCR is enabled.  Once on the target, the log is held until the target reaches the number of logs specified in the log replay caching system – 50 by default.  After that, Exchange commits the logs to the database on the target, effectively providing SQL-like Log Shipping (or Continuous Data Mirroring, Asynchronous) between two Exchange 2007 Servers.

Whenever the two servers are not in communication with each other, there is the potential for operations to occur on the source that are not seen by the target, and vice versa.  Since this could easily result in a corrupted database, Exchange will check to ensure it knows what state the data on both servers is in before resuming normal SCR operations.  If the state is known, then SCR continues – transmitting all logs not yet on the target, committing all but the last 50 logs, and continuing on its way as normal.

If the state is unknown, a re-seed operation will need to occur.  More specifically if there is a gap in log file enumeration for some reason, you will require a re-seed.  You can see the reasons why re-seeds are required at the TechNet website listed here.

The one that is of most concern to Exchange Admins is that if the two servers are not in communication during a backup window, you will have to re-seed the database before normal SCR operations can continue.  When speaking of a local SCR pair this is not a big issue, as connectivity will be much more solid than WAN performance, and even if a re-seed is required, it will be relatively fast.  But across a WAN, it is more likely that you can occasionally suffer WAN outages that do not interrupt business operations.  If these outages extend past the time of the backup window, the backup tool will most likely truncate logs committed to tape without SCR being able to transmit those changes to the target Exchange Server.

Since minor network outages are a common (though hopefully not very common) issue with modern networks, the likelihood of requiring a re-seed due to this sequence of events is something that could be considered a normal part of Exchange SCR operations.  If your databases are small, then it won’t be an issue.  For larger databases, remember that a re-seed operation is a full copy of all data files to the target device from source, which could be problematic depending on your WAN throughput.

So with SCR, re-seeds do become a definite issue to contend with.  Since the occurrence of re-seed operations should be limited, you may be able to keep the systems successfully in sync without too much trouble. However, if you have larger databases or smaller WAN pipes, re-seeds can create problems for your network, especially if you use a backup tool that truncates your logs, or use circular logging for any reason.

Of course, there are alternatives to SCR for WAN protection and availability (see disclaimer below, I’m biased here), so you are not out of luck in terms of WAN operations with Exchange 2007.  Locally, CCR is a spectacular choice for nearly any sized Exchange system, with the exception of very large (over 1TB) datasets that may just take to long to re-seed even over a LAN.  Remotely, if you have properly planned for re-seeds then you will also be able to successfully utilize SCR, but if you do not – or cannot – plan for these required operations, then you’ll risk a failure in your failsafe system, which is not a great situation to be in.

In short, re-seeds are a normal part of SCR operations. They should not happen every day (or even every week) but you will most likely experience them over the course of your SCR lifetime, so plan for them now.

 

Next week is TechEd 2009!  I’ll be out in Los Angeles with the Double-Take Software crew, so please stop by and say hi. I’ll also be blogging from the event, so keep an eye on this column and follow me on Twitter at http://twitter.com/talonnyc.