Time Keeps on Slippin…Slippin…Slippin into the…past?

Once again, I grabbed the title of the post from a play on words on lyrics from “Fly Like an Eagle” by the Steve Miller Band.  For those of you who have no idea what I’m talking about, see this YouTube video, and know that you have made me feel very, very old.

The point to the title was that some of you may have noticed that calendar appointments, email time stamps and many other things seem to be off by 1 hour starting yesterday.  The reason is that unless you have all your updates and hot fixes from Microsoft, Windows 2000 and 2003 would believe that Daylight Savings Time changes occurred on Sunday, October 25th at 2am, and changed the clocks on any non-updated servers.  If that’s an Exchange Server, the incorrect change will flow over into all Exchange functions as well, causing quite a few problems.

Normally, we here in the US change our clocks twice per year.  The latter one used to happen on the last Sunday of October, when we all set the clocks back by one hour at 2am on that day.  The problem is that the US Government changed the rules late last year, changing the dates that these one-hour shifts take place on.  This year, it will be November 1st, but Windows wasn’t originally programmed to deal with that.

Most of us update Windows and Exchange regularly, so we got all the appropriate patches and ran the required updaters on the systems in question.  You’ll know you did it right if the clock did not change Sunday, and do change on November 1 at 2am.  You know you missed at least one if either your server time, or your email time stamps are all an hour off today.

Yet another reason to patch regularly, but everyone can miss one now and then, so visit this Microsoft Support site on DST changes if things are acting odd time-wise.  If thing are acting odd in other ways, throw me an email, I might do a column on it =)

Exchange 2003 only wins with WINS.

Since Microsoft released the Release Candidate for Exchange 2010, I figured that a look back at some things in Exchange 2003 are well in order.  After all, a lot of folks are currently on that platform, and waiting for 2010 to RTM before they upgrade.

So today, let’s take a look one of the components of Windows that Exchange 2003 needs to have in place, but that causes confusion and doubt every time it comes up in conversation.

The Windows Internet Naming Service (WINS) is a component of Windows Server NT4, 2000 and 2003 that – as its name implies – translates NetBIOS names into internet names and addresses.  And to be absolutely clear on the subject, Microsoft *requires* the installation of WINS for full functionality of Exchange 2003.  Here’s the proof.

The reasons for this insistence on having WINS present in Exchange 2003 are varied, but the controversy over the requirement has raged ever since Exchange 2003 was released.  The confusion stems from the apparent dichotomy between the fact that 2003 is supposed to be totally DNS integrated, and the view that NetBIOS dependencies make it look as though it is not.  Another reason for confusion is that even though Microsoft says that the installer will not run properly without WINS in place, many clients have managed to install Exchange 2003 without the WINS services running at all, or worse yet; with the WINS system improperly configured and malfunctioning.

Exchange 2003 leverages Active Directory (AD) DNS for nearly all name resolution functions, from finding other servers in the local environment to finding SMTP hosts for external mail destinations.  For internal resolution, Exchange can leverage AD and Global Catalog servers to find things, as long as everything in the domain in question is using Windows 2003 or higher, and configured appropriately.  MAPI systems in 2003 also can use DNS to find Exchange servers (Outlook 2003 and up) or to locate other resources.  So if you are in a pure Windows/Exchange/Outlook 2003 or higher environment, and you have only one Active Directory site, then you’re all set without WINS.

That last sentence is key to figuring out where the requirements for WINS come from, and where most of the confusion stems from as well.  Most environments that use Exchange 2003 still have mixed-mode domains, possibly even some Windows 2000 servers around.  They’ve also brought legacy systems up to Windows 2003 over the years, meaning that older names and objects may still be preserved in AD.  So if an Exchange 2003 server needs to find a resource that AD only has a “short name” (non-fully qualified domain name) for, the traditional method to find that resource would be WINS, not DNS.

This also comes into play if you have multiple AD sites across multiple physical locations.  Since short names don’t translate properly from site to site, or domain to domain in the same Forest, WINS is necessary to translate the resources name into a location for the resource itself. 

Finally, any legacy Outlook clients (XP and earlier) rely on WINS to find all resources, as the MAPI clients they contain don’t understand DNS lookups at all.  So if you have any older Outlook clients running around, you’ll need to ensure WINS is configured properly.  Similarly, ExMerge relies on WINS since it was a hold-over application from earlier Exchange versions, and therefore never designed to leverage DNS.

So, until your environment moves up to Exchange 2007 or 2010, and a Native 2003 or Native 2008 Domain architecture, you’re stuck with WINS.  The good news is that WINS is just another component of Windows in 2003, and therefore not a bear to implement at all.  Go to Add/Remove Programs and choose to install Windows Components.  You’ll need your OS CD’s to finish the install, and as always; be sure to Service Pack after you are done and run Windows Update.  There have been many changes and patches to WINS over the years, especially in light of recent attacks against it.

WINS is a legacy system that is – thankfully – going away as we move toward Exchange 2007 and 2010 native environments, and as legacy Outlook clients are phased out.  Until then, make sure you keep up with WINS patches and ensure it is installed and configured. Not doing so can cause resolution problems for Exchange 2003, and installing it is a requirement from Microsoft, so take the time now to make sure your WINS systems are operating full steam.

Confirmed, No Server 2008 R2 for you!

Sorry for the horrible delay in posting, my loyal readers!  I’ve been trying to track down some official sources on this one, and of course Jeff Guillet from the EXPTA Blog beat me too the punch.  However, he’s a good guy and a great author, so I don’t mind giving him the credit.

Much to the dismay of a large number of Exchange Admins and Engineers, MSFT will not be supporting the 2007 version of Exchange on Server 2008 R2.  So, if you’re planning on upgrading from earlier versions of Exchange Server, you have only two choices:

1 – Use Server 2008 RTM, and be prepared to stay on that version of the OS for an extended period of time.  This will let you use Exchange Server 2007 SP1 and take advantage of Distributed Failover clustering and other nifty Exchange 2007-specific tool sets that rely on the Server 2008 architecture.

2 – Alternately, you can wait on your upgrade until Exchange 2010 is released to market, and become an early adopter on Server 2008 *or* Server 2008 R2. Interestingly enough, Exchange 2010 will indeed by backwards-compatible to run on Server 2008 RTM, even though 2007 won’t make the leap forwards onto R2.

As many clients were in the process of either rolling out 2007 to replace other email platforms or were upgrading from Exchange 2003 and earlier, this is a dramatic policy announcement from Redmond.  Current plans to continue toward Exchange 2007 on the latest OS (Server 2008 R2) will have to be altered or scrapped.  If abandoned, new plans have to be set up for Exchange 2010.  I can only believe this will lead to a slow adoption rate as Admins and Engineers try to figure out the combination of Exchange and Windows that will a) function and b) allow them to upgrade now and keep going for as long as possible. Since Exchange 2010 will have a lifecycle that goes beyond the end-of-life date for Exchange 2007, many clients are making hard decisions between moving forward with the Exchange 2007 rollout, or re-planning to use the newer version.

In the end, there are a lot of great reasons to move up to Exchange 2010, so from a purely technological perspective, moving up is a no-brainer.  The problem is that many folks have been planning for over a year to implement Exchange 2007.  Planning that started when the first Service Pack was released for both Windows 2008 and Exchange 2007 – traditionally signaling the point where most upgrades begin in earnest.

I fear that this will cause an additional hurdle to migration, both from other platforms and from Exchange 2003 and earlier.  I can only hope that Microsoft has some trick up their sleeves when Exchange 2010 launches to make the jump easier or simply more compelling.

When your cluster goes “oops,” Using RecoverCMS

First, a quick note:  I’m posting this one from Windows Live Writer on Windows 7 RC1, which I’m happy to say is remarkably stable and much faster overall than Vista.  I’d recommend it wholeheartedly!

Funny story, I once had a client who swore that clustering was enough protection for their messaging environment, until an outage took out their entire cluster at once – causing them to be down for about week.  Now, that’s not the funny part, but what caused the outage is somewhat hilarious, more on that later.

Exchange 2003 and earlier had a pretty straight-forward method for recovering an entire MSCS cluster if one had failed on you.  You built one or more nodes of a brand new cluster, created an Exchange Virtual Server (EVS) Resource Group with the same parameters (names, IP’s etc) as the production system had, and Exchange would do the rest.

With Exchange 2007, the rules changed significantly, leaving many cluster users confused as to how the system now works if they suffer a cataclysmic failure of the production cluster.  Adding both Single Copy Cluster (SCC) and Continuous Cluster Replication (CCR) to the mix just makes things more confusing, so Microsoft created a new recovery method for Exchange 2007 clusters.  Called RecoverCMS, the system is really a setup task rather than a true failover system, but since your failover system just went belly-up, that’s not a bad thing.

If your Recovery Time Objectives are flexible enough to handle some downtime if an entire cluster fails, then you can leverage this system to get back up and running, either at the original production site, or at a new location.  There are some definite limits to what you can do with it which I’ll explain later, but he basics of how it works are pretty simple.

Step one is rebuild, repair or replace the original cluster hardware. If the repair works then you’re done, just restore any missing data from tape or other backup (due disclaimer, see below, I am biased on backup tools) and then resume normal operations. If you rebuild or replace completely, bring up a new server that is configured with Exchange 2007 in the Passive Cluster Node configuration.  You can find out how to do that:

Here for CCR clustering or,

Here for SCC Clustering

During that process you will also have installed the Exchange 2007 binaries on at least one node of the cluster system, so go to the directory that has the Exchange setup files and execute the following command:

Setup.com /recoverCMS /CMSName:<name> /CMSIPaddress:<ip>

Where <name> is the name of the EVS you’re restoring from, and <IP> is the IP address you want the recovered system to have – in theory the same IP as the original EVS had.

The rest of the procedure is pretty automated, and when finished, you will have a new EVS running on your new cluster node(s) that matches the original EVS and has all the users already assigned to it.  From there, you can restore your data if it was also lost to the disaster.

There are a few things that are extremely important to be aware of before you begin:

1 – Keep in mind that /recoverCMS is designed to restore a failed cluster only.  Attempting to use it for migration or for any other purpose will result in unpredictable behavior and is not supported by MSFT.

2 – You will need to manually create the volumes that existed on the failed cluster before you run /recoverCMS.  If volumes are missing then the recovery will fail.  They don’t have to be the same physical disk or size, just large enough to hold the data and with the same drive letters as the original cluster held.

3 – The System Attendant service will start and then immediately stop after you recover, this is normal, just bring the resource back online when you’re ready.

4 – Your databases are not mounted after a recovery, you must do this manually through PowerShell or the Exchange Management Console after you’re done with the restore.

5 – Do NOT try to use this across OS’s. If you started on Windows Server 2003, you must recover to Windows Server 2003, and 2008 to 2008.   It will not work if you try to go from one to the other.

6 – While you can pre-configure many portions of this system, it will still take some time to run through a /recoverCMS procedure from start to finish, so if you need a second-stage failover, /recoverCMS isn’t the best bet.  I’m quite biased on this (see disclaimer below), but unless you can be down for a few hours if both cluster nodes fail, you might want to go with another tool to provide remote site failover in addition to SCC or CCR clustering.

7 – Finally, SCR and CCR will not automatically work with /recoverCMS.  You will need to stop SCR if it’s running before you recover, and neither will resume automatically after the recovery is done.  Once you’re set up in the new node configuration, re-enable CCR and SCR manually as required.

/RecoverCMS is a great way to restore a failed cluster system to new hardware or rebuilt hardware after a fault.  You still need to back up your data to some device outside the cluster itself, but once you have that backup /recoverCMS can get your cluster back up and running much faster than the manual methodologies used in previous versions of Exchange.

As to the funny story I mentioned at the top of the blog, this particular client was in a hardened datacenter with UPS systems, 24/7 staff and a backup generator.  They were convinced that clustering was going to be more than enough for them.  After trying to explain that a shared-disk cluster (the only option at the time) had weak points, I finally gave up and let them be.  A few months later I got a great phone call.  Apparently – unbeknownst to the client – the datacenter crew had run all power connections through the UPS – including the generator.  The UPS was rated to handle the full power load of the datacenter on 1 of its 2 redundant circuit loops.  So far so good.  Well, this particular datacenter was in the middle of the dot-com boom (this was some time ago) and had grown exponentially in a short period of time.  What they had was well over half the full expected load on each of the two circuits, and one was failing.  So they diligently got replacement parts and moved the load over to the good circuit.  Since was over half the expected load, and circuit 2 was already under over half the load, they immediately overloaded the UPS, shorting it out.  The way it was explained to me, a solenoid shot through the casing of the UPS…and there was indeed a nice hole in the unit to back that up. No one was hurt, but needless to say, the whole datacenter was offline until they replaced the UPS, 4 days later, so they lost about one business week, without anything happening to the physical cluster at all.  Just goes to show you that anything that can go wrong, will.

Exchange on DC's

Smaller shops have always been faced with a fundamental quandary when the time has come for them to move off Small Business Server and into the world of the independent version of Microsoft Exchange.  One of the biggest considerations is where to put the Domain Controllers for Active Directory.  The first thought most of my clients have is to put it on the same server as the Exchange system, but outside of SBS that's a very bad idea for a handful of important reasons.

For example, if you're ready to make the move from a single server to a cluster, you cannot install Exchange 2003 on a cluster node that is also a DC.  That is just simply not supported by Microsoft, and therefore isn't an issue up for debate.

It's much more likely that you'll be moving to a single-server, however, as if you're just moving off of SBS then the likelihood you need a cluster is reduced quite a bit.  In those cases, while you could install Exchange on the DC, it's still not a good idea.  The Exchange Best Practices Analyzer will even flag this condition as going against best practices, which should be a giant red flag for anyone installing an Exchange Server.

Here's a few reasons it isn't a good idea:

1 - Since Exchange runs many services under Local System accounts with elevated privileges, a successful attack via the Exchange system could easily expose your AD controller to outside influences.  With the potential to gain access to all domain resources if the AD controller is compromised, reducing attack surface is vital, and installing Exchange (or any app, this is not an Exchange-specific issue) will *increase* attack surface.  If you get hit on an Exchange Server that is not a DC, the attacker has access to only that server, not the entire Domain.

2 - 3rd Party integration can suffer.  Since many 3rd Party tools may expect to use Local Accounts and Groups (which wouldn't exist on a DC) or may otherwise need to control aspects of the Exchange System that may not be available or accessible on a DC, integration with Anti-Virus, Backup and Recovery solutions can suffer if the server hosts both roles.

3 - Performance impacts will be clearly visible.  Since the Exchange Server constantly uses the DC and vice-versa, placing both on one physical server can effectively double the amount of work that server must do.  Some visible effects of this are slower overall Exchange performance; longer response times for Domain commands (such as logging in, changing passwords, etc.; and much longer reboot times.

Putting the Active Directory and Exchange Server systems on one physical machine may seem like a way to save money, but in reality it will cost more in downtime and headaches than it will save on server hardware.

 

[Note: Small Business Server contains a highly modified version of Microsoft Exchange Server that is specifically designed to run on an SBS server.  It does not permit all 3rd Party applications, limits functionality and is also limited to only being able to run on SBS.  Therefore, running the SBS version of Exchange on an SBS DC is both a supported installation and a best practice. This article specifically applied to non-SBS versions of Exchange Server.]