Hyper-V Me (Yes, Even for Exchange)

Due to the release of Hyper-V R2, there has been a lot of buzz about using Microsoft’s Virtualization Platform to create virtual machines for everything in the datacenter from file servers to high-capacity databases.  There are quite a few systems that are clearly ok to use virtualization with, but Exchange Server has always been one of those platforms where virtual tech has been a questionable course.  Not the least of the reasons for this is that folks have been confused on if MSFT supports running Exchange in production on a virtual platform.  So, I’ll spell out the current support stance as found on official MSFT sites only.  Note:  There are many blog postings that point one way or the other, so I have hunted down official MSFT sites to quote here (i.e. TechNET, MSDN, etc).  While many blogs can be trusted (like the official Exchange Team Blog at http://msexchangeteam.com – and http://www.BeingExchanged.com of course), many of my readers have corporate policies about needing to see the documentation from a MSFT site directly; and so here it is:

Exchange 2000 and earlier:

Not supported for production environments.  See the end of the TechNET article found here: Link to TechNET

Exchange 2003:

Supported in production.  Warnings are given that you must use Virtual Server (2005 R2 or higher) and that there are some performance limitation to be considered. Link to TechNET

Exchange 2007

Supported for production environments on Hyper-V, but with some limitations. You must be running Server 2008 as the Guest OS, and you have to be running Exchange 2007 SP1. Unified Messaging may not be run on a Hyper-V Guest.  You cannot use both SCR and/or CCR and Hyper-V clustering at the same time. Finally, the virtual machine itself must meet all the hardware requirements (processors, RAM, etc) for Exchange 2007 Sp1. Link to TechNET

Exchange 2010:

Supported for production environments on Hyper-V, with some restrictions:  You must be running Server 2008 SP2 or R2 as the Guest OS, and Unified Communications Roles are not supported at all.  As with 2007, the virtual machine you install Exchange into must meet all the hardware requirements for Exchange 2010. You will also need to choose between DAG availability or Hyper-V failover solutions, you cannot run both at once. Link to TechNET (see section on Hardware Virtualization)

Common notes:

In all supported versions, certain functions of the virtualization systems are not supported with Exchange Server.  Specifically, dynamically expanding virtual disks are not allowed, you have to use pre-configured fixed-sized VHD’s for Exchange Server virtual machines. Likewise, you cannot leverage differencing or VSS snapshot disks if you’re running Exchange Server.

There are several rules about virtual hardware that you must follow as well, so read the related TechNET articles for guidance on things like disk types (SCSI, IDE, etc) and the like.

Hyper-V technology is bringing new opportunities to the virtualization technologies platform in the modern datacenter.  By clarifying what is and is not supported in a virtual environment, Microsoft has begun the process of allowing customers to safely start using virtualization for the Exchange Server platform.  Many users will still choose physical hardware for Exchange (either because they cannot meet one or more of the requirements, or just due to having hardware already provisioned), but these rules help clarify that options for virtualization exist and are supported.

Time Keeps on Slippin…Slippin…Slippin into the…past?

Once again, I grabbed the title of the post from a play on words on lyrics from “Fly Like an Eagle” by the Steve Miller Band.  For those of you who have no idea what I’m talking about, see this YouTube video, and know that you have made me feel very, very old.

The point to the title was that some of you may have noticed that calendar appointments, email time stamps and many other things seem to be off by 1 hour starting yesterday.  The reason is that unless you have all your updates and hot fixes from Microsoft, Windows 2000 and 2003 would believe that Daylight Savings Time changes occurred on Sunday, October 25th at 2am, and changed the clocks on any non-updated servers.  If that’s an Exchange Server, the incorrect change will flow over into all Exchange functions as well, causing quite a few problems.

Normally, we here in the US change our clocks twice per year.  The latter one used to happen on the last Sunday of October, when we all set the clocks back by one hour at 2am on that day.  The problem is that the US Government changed the rules late last year, changing the dates that these one-hour shifts take place on.  This year, it will be November 1st, but Windows wasn’t originally programmed to deal with that.

Most of us update Windows and Exchange regularly, so we got all the appropriate patches and ran the required updaters on the systems in question.  You’ll know you did it right if the clock did not change Sunday, and do change on November 1 at 2am.  You know you missed at least one if either your server time, or your email time stamps are all an hour off today.

Yet another reason to patch regularly, but everyone can miss one now and then, so visit this Microsoft Support site on DST changes if things are acting odd time-wise.  If thing are acting odd in other ways, throw me an email, I might do a column on it =)

Confirmed, No Server 2008 R2 for you!

Sorry for the horrible delay in posting, my loyal readers!  I’ve been trying to track down some official sources on this one, and of course Jeff Guillet from the EXPTA Blog beat me too the punch.  However, he’s a good guy and a great author, so I don’t mind giving him the credit.

Much to the dismay of a large number of Exchange Admins and Engineers, MSFT will not be supporting the 2007 version of Exchange on Server 2008 R2.  So, if you’re planning on upgrading from earlier versions of Exchange Server, you have only two choices:

1 – Use Server 2008 RTM, and be prepared to stay on that version of the OS for an extended period of time.  This will let you use Exchange Server 2007 SP1 and take advantage of Distributed Failover clustering and other nifty Exchange 2007-specific tool sets that rely on the Server 2008 architecture.

2 – Alternately, you can wait on your upgrade until Exchange 2010 is released to market, and become an early adopter on Server 2008 *or* Server 2008 R2. Interestingly enough, Exchange 2010 will indeed by backwards-compatible to run on Server 2008 RTM, even though 2007 won’t make the leap forwards onto R2.

As many clients were in the process of either rolling out 2007 to replace other email platforms or were upgrading from Exchange 2003 and earlier, this is a dramatic policy announcement from Redmond.  Current plans to continue toward Exchange 2007 on the latest OS (Server 2008 R2) will have to be altered or scrapped.  If abandoned, new plans have to be set up for Exchange 2010.  I can only believe this will lead to a slow adoption rate as Admins and Engineers try to figure out the combination of Exchange and Windows that will a) function and b) allow them to upgrade now and keep going for as long as possible. Since Exchange 2010 will have a lifecycle that goes beyond the end-of-life date for Exchange 2007, many clients are making hard decisions between moving forward with the Exchange 2007 rollout, or re-planning to use the newer version.

In the end, there are a lot of great reasons to move up to Exchange 2010, so from a purely technological perspective, moving up is a no-brainer.  The problem is that many folks have been planning for over a year to implement Exchange 2007.  Planning that started when the first Service Pack was released for both Windows 2008 and Exchange 2007 – traditionally signaling the point where most upgrades begin in earnest.

I fear that this will cause an additional hurdle to migration, both from other platforms and from Exchange 2003 and earlier.  I can only hope that Microsoft has some trick up their sleeves when Exchange 2010 launches to make the jump easier or simply more compelling.

Back to basics: Prepping a 2008 Server for Exchange 2007

Got a basic Exchange 2003 or 2007 topic you’d like covered in this series?  Email MikeTalonNYC@gmail.com and you might see your topic covered here!

A lot of my clients have been asking me to write about some more basic topics dealing with Exchange Server 2007 here on the blog, and so this week I’d like to talk about pre-requisites for installation of Exchange Server 2007 on Windows Server 2008.  These will change as Server 2008 R2 goes live later this year, so stay tuned for updates.

To start, run through the basic install of a Windows Server 2008 x64 box in either Standard or Enterprise Edition.  For a stand-alone Exchange server, you can use either version of Windows.  If, however, you need to cluster your Exchange 2007 systems with SCC or CCR clusters, then you must install Windows 2008 Server Enterprise Edition.  This article focuses on a stand-alone Exchange server, and therefore which version you install depends more on cost versus performance.  Check out this article for the details on how many processor cores, how much RAM and other factors that change based on versions of Windows Server 2008:

Windows Server 2008 Editions on Microsoft.com

Note: Since Exchange requires .NET, right now you cannot install Exchange to a Server Core installation.

This is a good time to install any tools your organization uses which are not Exchange-specific.  Monitoring tools, storage management, etc.

Once you have the basic Windows Server installed, make sure you run Windows Update to get the latest Service Pack and all the assorted hotfixes and patches.  Some of these are required for Exchange, so do not skip this step.  The most critical of these (beyond the Service Pack) is the .NET platform.  While you must have at least .NET 2 installed, the current version is 3.5.1, so you may wish to install that version as well.  No doubt future tools in Exchange 2007 will need some of those components.  Since most of the .NET platform shows up as Optional, you may need to manually check the boxes for these downloads before they will be installed by Windows Update.

Next up, install PowerShell to the machine if you haven’t already.  This shows up in the Server Manager Wizard as a Feature, and is a wizard-driven installation, so you do not need to download anything from Microsoft to do the install anymore.  The default parameters are fine unless you have a preferred setup for PowerShell.  (for those who will inevitably email me on this, according to MSFT, it is spelled “PowerShell,” not “Powershell.” Go yell at them if you don’t like odd capitalization.)

Exchange 2007 also requires that the IIS system be present on the server.  You can install this by bringing up the Server Manager and adding the Web Services (IIS) Role to the box. When you select this role, you will be asked if you wish to also add dependency features/roles.  Select Yes here to automatically add in quite a few feature sets that both IIS and Exchange need. 

A couple of notes: Two items that are not selected by default or by auto-selection are Dynamic Compression and the IIS 6 Admin Tools.  You should select both of those manually, as Exchange will need them.   Also, you have the option of selecting different authentication types for the Web Services in this wizard as well.  Be sure to select any and all that your organization will use.  As with most things, you can go back and add them later, but since you’re doing the install anyway, might as well get them on the box now.  Aside from these items, you can safely use the defaults for this wizard and allow it to perform the installation.

After you run through the IIS Role setup, you’ll need to install the remote AD configuration tool kit on the server.  The easiest way to perform this step is to open a command prompt and run the following command:

ServerManagerCmd -i RSAT-ADDS

When you run that command, the system will appear to be doing not much of anything for quite a few minutes.  If your experience is like mine, it will appear to be stuck at 10% for most of that time.  In my lab, with virtual machines that were resource starved, it took about 10 minutes.  Wait until you are returned to the command prompt (C:\>) before you move forward or the tools will not be installed and the Exchange validation tool will error out.

Next week, I’ll walk you through a basic installation of the Exchange 2007 server will the 3 main roles on one box.  Until then, have fun getting prepped.

OAB – Oh Boy! Yet another annoying error conquered

Exasperation no more!  OAB 0x80190194 error solved - finally

As many of you already know, I do a lot of Exchange configuration in my day to day job (see disclaimer below).  From time to time, I run into errors that seem to occur over and over, regardless of the environment, until someone can finally figure it out.  Outlook attempting to download the Offline Address Book (OAB) has usually been one of those odd errors, especially when a server is first created.

Starting in Exchange Server 2003, the OAB was only generated for the first time when the server entered the first maintenance window after installation.  So if you built a brand new Exchange environment, you would need to wait until somewhere between 1 and 3 AM before you had an OAB.  This meant that Outlook clients would exhibit a strange error about "File Not Found" during every synchronization.  Outlook went to look for an OAB to download, there was none there, and the sync would abnormally end.

This was easy to fix, however.  You simply located the OAB in the Exchange Management tools and forced a rebuild, which immediately cleared up the problem.  In rare circumstances you followed the TechNet articles and removed the OAB and created a new one, then forced a rebuild.  All in all, not a very difficult fix, but one that eludes many Exchange Engineers and Administrators for hours and hours until they either figure out the fix from reading about it on forums; or the give up in frustration and see the miracle of the error going away the next day (since the OAB auto-rebuilt during the night).

When Exchange Server 2007 came along, I had hoped that this would be a thing of the past.  After all, the least that Microsoft could do would be to put a message in the installer about having to rebuild the OAB. The best would be to have it auto-generate immediately on install.  Alas, as most of us in the Exchange world figured out, neither is the case.  On a new Exchange build, you still have to manually force an update of the Offline Address Book.

It is a lot easier now, though.  One Powershell command does it all:

Update-OfflineAddressBook -id "Default Offline Address Book"

Since the first OAB is always "Default Offline Address Book", the command is universal and can just be done immediately on finalizing your installation of the Exchange software.

All was well until I started getting a stream of calls from colleagues across the country about a new error cropping up in Exchange 2007 and Outlook 2007.  These folks had already regenerated the OAB manually or waited until after the first maintenance window, and now - unlike the File not Found errors before - they were getting a very odd error indeed:

Task 'Microsoft Exchange' reported error (0x80190194) : 'The operation failed.'

Google searches turned up suggestions for Microsoft Update, the Windows Update Site, and a few other red herrings.  Further investigation led only to the usual recommendations to regenerate the OAB.  This was driving many people, including me, nuts!  Remarkably, for several months none of us found an answer, but in all cases the problem went away on its own. Sometimes in a few days, sometimes in a few weeks, but always resolved itself in time. 

Now, my readers know me, that's not quite good enough for my palate.  So when time finally permitted I did some deeper investigation.  I found several things out that had nothing really to do with the error, but are fun:

- Avira Antivirus considers Experts-Exchange.com a phishing site.

- People tended to suggest everything short of lighting candles, standing on your head and typing mysterious incantations into Powershell (though frankly some of the stuff was pretty close)

- Many folks just let it go away on its own and never though about it again - until they had to build a new Exchange system.

Finally, I tripped over the MSServerAdmin site and to this article:

http://www.msserveradmin.com/how-to-overcome-the-exchange-2007-oab-issues/

Which got me to look at the Microsoft Exchange File Distribution service on my Client Access Role server (which in my case was on the same box as the Mailbox Role).  Once I stopped and started that service, Outlook immediately began to download the OAB and all was well once again.  That's really all there is to it.  This only seems to happen on Server 2008, which explains why I wasn't seeing it back when most folks were installing Exchange 2007 to Windows 2003.  I do not have an answer as to why this problem is so common with Server 2008, but if any of the Microsoft folks who read this can tell me, I'll happily post an update.

So, why did it go away mysteriously by itself over time?  That was a little more of a quandry, but I got that answer too.  These days, it's not usual to have to reboot an Exchange 2007 server after installation is finalized.  So folks were not restarting services regularly by rebooting the boxes regularly.  However, eventually, Windows Update Services would download a critical OS patch, and - left to its own defaults - automatically reboot the box.  This might happen the next day, or might be several weeks away, but eventually we'd reach Patch Tuesday and the boxes would reboot - restarting the service and clearing the problem.

Good things come to those who wait - this time.  Hopefully the answer will always be as simple as "Just wait and it'll fix itself," but until then I shall keep searching for answers and reporting them here.

When your cluster goes “oops,” Using RecoverCMS

First, a quick note:  I’m posting this one from Windows Live Writer on Windows 7 RC1, which I’m happy to say is remarkably stable and much faster overall than Vista.  I’d recommend it wholeheartedly!

Funny story, I once had a client who swore that clustering was enough protection for their messaging environment, until an outage took out their entire cluster at once – causing them to be down for about week.  Now, that’s not the funny part, but what caused the outage is somewhat hilarious, more on that later.

Exchange 2003 and earlier had a pretty straight-forward method for recovering an entire MSCS cluster if one had failed on you.  You built one or more nodes of a brand new cluster, created an Exchange Virtual Server (EVS) Resource Group with the same parameters (names, IP’s etc) as the production system had, and Exchange would do the rest.

With Exchange 2007, the rules changed significantly, leaving many cluster users confused as to how the system now works if they suffer a cataclysmic failure of the production cluster.  Adding both Single Copy Cluster (SCC) and Continuous Cluster Replication (CCR) to the mix just makes things more confusing, so Microsoft created a new recovery method for Exchange 2007 clusters.  Called RecoverCMS, the system is really a setup task rather than a true failover system, but since your failover system just went belly-up, that’s not a bad thing.

If your Recovery Time Objectives are flexible enough to handle some downtime if an entire cluster fails, then you can leverage this system to get back up and running, either at the original production site, or at a new location.  There are some definite limits to what you can do with it which I’ll explain later, but he basics of how it works are pretty simple.

Step one is rebuild, repair or replace the original cluster hardware. If the repair works then you’re done, just restore any missing data from tape or other backup (due disclaimer, see below, I am biased on backup tools) and then resume normal operations. If you rebuild or replace completely, bring up a new server that is configured with Exchange 2007 in the Passive Cluster Node configuration.  You can find out how to do that:

Here for CCR clustering or,

Here for SCC Clustering

During that process you will also have installed the Exchange 2007 binaries on at least one node of the cluster system, so go to the directory that has the Exchange setup files and execute the following command:

Setup.com /recoverCMS /CMSName:<name> /CMSIPaddress:<ip>

Where <name> is the name of the EVS you’re restoring from, and <IP> is the IP address you want the recovered system to have – in theory the same IP as the original EVS had.

The rest of the procedure is pretty automated, and when finished, you will have a new EVS running on your new cluster node(s) that matches the original EVS and has all the users already assigned to it.  From there, you can restore your data if it was also lost to the disaster.

There are a few things that are extremely important to be aware of before you begin:

1 – Keep in mind that /recoverCMS is designed to restore a failed cluster only.  Attempting to use it for migration or for any other purpose will result in unpredictable behavior and is not supported by MSFT.

2 – You will need to manually create the volumes that existed on the failed cluster before you run /recoverCMS.  If volumes are missing then the recovery will fail.  They don’t have to be the same physical disk or size, just large enough to hold the data and with the same drive letters as the original cluster held.

3 – The System Attendant service will start and then immediately stop after you recover, this is normal, just bring the resource back online when you’re ready.

4 – Your databases are not mounted after a recovery, you must do this manually through PowerShell or the Exchange Management Console after you’re done with the restore.

5 – Do NOT try to use this across OS’s. If you started on Windows Server 2003, you must recover to Windows Server 2003, and 2008 to 2008.   It will not work if you try to go from one to the other.

6 – While you can pre-configure many portions of this system, it will still take some time to run through a /recoverCMS procedure from start to finish, so if you need a second-stage failover, /recoverCMS isn’t the best bet.  I’m quite biased on this (see disclaimer below), but unless you can be down for a few hours if both cluster nodes fail, you might want to go with another tool to provide remote site failover in addition to SCC or CCR clustering.

7 – Finally, SCR and CCR will not automatically work with /recoverCMS.  You will need to stop SCR if it’s running before you recover, and neither will resume automatically after the recovery is done.  Once you’re set up in the new node configuration, re-enable CCR and SCR manually as required.

/RecoverCMS is a great way to restore a failed cluster system to new hardware or rebuilt hardware after a fault.  You still need to back up your data to some device outside the cluster itself, but once you have that backup /recoverCMS can get your cluster back up and running much faster than the manual methodologies used in previous versions of Exchange.

As to the funny story I mentioned at the top of the blog, this particular client was in a hardened datacenter with UPS systems, 24/7 staff and a backup generator.  They were convinced that clustering was going to be more than enough for them.  After trying to explain that a shared-disk cluster (the only option at the time) had weak points, I finally gave up and let them be.  A few months later I got a great phone call.  Apparently – unbeknownst to the client – the datacenter crew had run all power connections through the UPS – including the generator.  The UPS was rated to handle the full power load of the datacenter on 1 of its 2 redundant circuit loops.  So far so good.  Well, this particular datacenter was in the middle of the dot-com boom (this was some time ago) and had grown exponentially in a short period of time.  What they had was well over half the full expected load on each of the two circuits, and one was failing.  So they diligently got replacement parts and moved the load over to the good circuit.  Since was over half the expected load, and circuit 2 was already under over half the load, they immediately overloaded the UPS, shorting it out.  The way it was explained to me, a solenoid shot through the casing of the UPS…and there was indeed a nice hole in the unit to back that up. No one was hurt, but needless to say, the whole datacenter was offline until they replaced the UPS, 4 days later, so they lost about one business week, without anything happening to the physical cluster at all.  Just goes to show you that anything that can go wrong, will.

Quite a stretch for clustering in 2008

Microsoft Clustering Services (MSCS) have existed in one form or another since NT4, but have always suffered from a significant limitation.  All cluster resources had to exist within the same logical subnet, or else you couldn't create the cluster itself.  Windows Server 2008 allows for some flexibility in that regard, with the ability to create nodes of a contiguous cluster in different logical subnets.

Before we dive too far into that, you may want to see the official MSFT information here:

http://technet.microsoft.com/en-us/library/cc770625.aspx

So what does this mean for you and I?  It means that we can create CCR clusters on Exchange 2007 that stretch between physical locations and subnets.  However, to do this you'll need to be on Server 2008, the function just isn't available in Server 2003. This allows you to provide basic availability for Mailbox Role (MBX) servers in your Exchange environment, but doesn't take care of everything when it comes to DR planning.

First off, this applies only to Exchange 2007 Enterprise Edition, and then only to MBX role servers.  While most other roles are natively fault-tolerant with multiple servers installed with the same role able to stand in for each other, organizational or regulatory rules might not make that kind of redundancy possible.  Edge servers are the biggest example of this.  Since they're not tied into the domain structures, they don't contain any way to quickly flip traffic from one Edge server to another in different sites.  Third-party tools (see disclaimer below) can often take care of that function for you, as can working with your DNS provider to facilitate moving the MX records in the event of an emergency.

If you have legacy Exchange 2000/2003 servers, you're also not able to take advantage of this new MSCS feature set on those boxes.  The same goes with any non-Exchange tools, like SQL, anti-virus servers, anti-spam servers, etc.  Even if these servers run on Server 2008 clusters, they'll require some third-party intervention to handle the data replication for those systems.  This would include things like Blackberry servers, GoodLink systems and other non-Exchange remote email tools.

Finally, keep in mind that CCR clusters can only extend to Active/Passive, 2-node configurations. That will mean you can't use these solutions if you need to go beyond that model - which Exchange 2007 easily allows without CCR involved.

Server 2008 Failover Clustering is a great method for basic High Availability for Exchange 2007 CCR systems - even across subnets.  With some additional tools, it can become the center of an Exchange DR solution set that can help your organization withstand even site-wide emergencies.

Hyper-V-active Exchange

As most folks have already seen, Hyper-V - the newest Microsoft Virtual Server Solution - released to market yesterday.  This is a pretty big step forward for MSFT, adding a datacenter-ready virtualization solution to the Server 2008 platform.

I'm doing a little research now on the Exchange front with Hyper-V, but I was able to get an un-named source (ooooh spooooky) from MSFT to confirm that some MSFT Exchange 2007 servers at their facilities are running on Hyper-V, so support on that platform will most likely be forthcoming.  More info on this front as I can get my hands on it.

From the limited contact I've had with the betas, I can safely say that this product is a different animal than Virtual Server 2005.  The overall performance in both the Host and Guest components has ratcheted up by a factor of multiples, The management interface is also re-designed to allow for more fine-tuned control over the vitalized systems, and cluster support means the ability to move VM's between hosts is built-in.

We're still in the very early stages of this technology, so expect more reports as I run into it more and more in the field.  Within a short period of time, I'm also going to get to put Exchange 2007 through its paces on Hyper-V, and I'll be happy to report on that too!

It goes without saying that Double-Take is going to be supporting Hyper-V in the very near future with some amazing tool sets.  If course we can already replicate and fail over Exchange from Physical or Virtual (of any flavor) into a Hyper-V VM, but keep your eyes peeled to find out what new stuff we're working on over here especially for the Hyper-V platform.