Hyper-V Me (Yes, Even for Exchange)

Due to the release of Hyper-V R2, there has been a lot of buzz about using Microsoft’s Virtualization Platform to create virtual machines for everything in the datacenter from file servers to high-capacity databases.  There are quite a few systems that are clearly ok to use virtualization with, but Exchange Server has always been one of those platforms where virtual tech has been a questionable course.  Not the least of the reasons for this is that folks have been confused on if MSFT supports running Exchange in production on a virtual platform.  So, I’ll spell out the current support stance as found on official MSFT sites only.  Note:  There are many blog postings that point one way or the other, so I have hunted down official MSFT sites to quote here (i.e. TechNET, MSDN, etc).  While many blogs can be trusted (like the official Exchange Team Blog at http://msexchangeteam.com – and http://www.BeingExchanged.com of course), many of my readers have corporate policies about needing to see the documentation from a MSFT site directly; and so here it is:

Exchange 2000 and earlier:

Not supported for production environments.  See the end of the TechNET article found here: Link to TechNET

Exchange 2003:

Supported in production.  Warnings are given that you must use Virtual Server (2005 R2 or higher) and that there are some performance limitation to be considered. Link to TechNET

Exchange 2007

Supported for production environments on Hyper-V, but with some limitations. You must be running Server 2008 as the Guest OS, and you have to be running Exchange 2007 SP1. Unified Messaging may not be run on a Hyper-V Guest.  You cannot use both SCR and/or CCR and Hyper-V clustering at the same time. Finally, the virtual machine itself must meet all the hardware requirements (processors, RAM, etc) for Exchange 2007 Sp1. Link to TechNET

Exchange 2010:

Supported for production environments on Hyper-V, with some restrictions:  You must be running Server 2008 SP2 or R2 as the Guest OS, and Unified Communications Roles are not supported at all.  As with 2007, the virtual machine you install Exchange into must meet all the hardware requirements for Exchange 2010. You will also need to choose between DAG availability or Hyper-V failover solutions, you cannot run both at once. Link to TechNET (see section on Hardware Virtualization)

Common notes:

In all supported versions, certain functions of the virtualization systems are not supported with Exchange Server.  Specifically, dynamically expanding virtual disks are not allowed, you have to use pre-configured fixed-sized VHD’s for Exchange Server virtual machines. Likewise, you cannot leverage differencing or VSS snapshot disks if you’re running Exchange Server.

There are several rules about virtual hardware that you must follow as well, so read the related TechNET articles for guidance on things like disk types (SCSI, IDE, etc) and the like.

Hyper-V technology is bringing new opportunities to the virtualization technologies platform in the modern datacenter.  By clarifying what is and is not supported in a virtual environment, Microsoft has begun the process of allowing customers to safely start using virtualization for the Exchange Server platform.  Many users will still choose physical hardware for Exchange (either because they cannot meet one or more of the requirements, or just due to having hardware already provisioned), but these rules help clarify that options for virtualization exist and are supported.

A word on Cluster Groups

Many clients are on Exchange 2003 or 2007 and will need to deal with Cluster Groups in Microsoft Cluster Services (MSCS) or Failover Clustering Services (FCS) including Cluster Continuous Replication (CCR).  So, it is important to understand one very critical restriction of Exchange Clustering that I’ve seen several clients trip over.

When installing a Microsoft Cluster of most flavors, you will configure Groups, which are logical units used to contain Resources like IP Addresses, Network Names, Disks and Services.  By default, a Cluster Group will be created that contains the name, IP address and Quorum Disk for the cluster itself.  It may also contain a networked Distributed Transaction Coordinator (DTC) resource for the cluster as a whole.  It is very tempting to place all other resources in this group, but you should avoid doing that at all costs for 2 significant reasons:

1 – It’s not supported by Microsoft.  For proof, I refer you to This TechNET article.  There is a long explanation of many thing having to do with configuring an Exchange Cluster, but here’s the specific info I’m referring to:

“It is an Exchange best practice to install the MSDTC resource into the default cluster group. However, the MSDTC resource is the only resource supported in the default cluster group. Exchange resources should not be added to the default cluster group, as that configuration is not supported.” [emphasis added]

TechNET and the Microsoft Sites have many other examples of this warning, and it is well documented by Microsoft and the Exchange Product Team.

2 – It makes life more difficult in day-to-day administration.  There may be instances where you want to perform operations on the Cluster Group without interrupting Exchange services for your organization.  You can normally accomplish this by moving the Cluster Group to a Cluster Node that isn’t hosting any Exchange Resource Groups, and perform your activities on that node.  If you have Exchange Resources in the Cluster Group, then this options disappears.  The same goes for many 3rd-Party products (see disclaimer at the end of the blog) which may not accept Exchange Resources that appear in the Cluster Group, as they must treat the Cluster Group and the Exchange Resource Group independently for administrative purposes.

So, as tempting as it is, avoid installing Exchange Resources into the Cluster Group at all costs.  If you already have put Exchange Resources into the Cluster Group, and you don’t plan on upgrading just yet, then seriously consider migrating to a supported cluster configuration when time permits.  Issues that arise from unsupported configuration and limited administration tend to hit without warning, and at the worst possible time.  Taking time to move to a supported platform will keep your organization in the safe zone, and make life a lot easier for you over time.

Look before you leap

There’s a great deal of talk these days about migrating directly to Exchange 2010 instead of jumping first to the 2007 version of the platform.  The arguments for and against this got more complicated when Microsoft did a reversal for support and announced that 2007 would be able to run (at some future point) on Server 2008 R2.  This effectively extended the support cycle of 2007, which means that a slightly older, and more thoroughly tested and widely installed, version is a legitimate option going forward.  That being said, many clients are opting to just go to Exchange 2010 from whatever they’re on now, skipping 2007 if it isn’t already in place.

I’m going to avoid discussing the pros and cons of going to Exchange 2010 directly in this particular post (that’s fodder for quite a few additional posts though).  Let’s just say that you’re jumping from an earlier version directly to 2010 and go on from there.  While most of the process isn’t too bad overall, there are a couple of sticking points that you’ll need to have sorted out before you make the move.  In smaller organizations, these are pretty easy issues to take care of, but on large-scale roll-outs, they can be a problem.

First, make sure your domains are at least at Server 2003 native functional domain levels.  Most of you are already there, but just in case you had a few NT servers hanging around, be ready for this one.  As long as all your servers are 2003 and up, making this change doesn’t have a lot of impact on your domain.  You can find out about the impact of raising Domain Functional Levels in this TechNET article.  So, what if you DO still have Exchange 2000/Windows 2000 servers in your environment?  You need to upgrade to Exchange 2007 as an interim step to 2010.  Since the Exchange 2000 life-cycle is definitely over (and has been for a while), Microsoft offers no method of co-existence between Exchange 2000 and Exchange 2010 – migration or otherwise.

Once your domain is prepped to 2003 or 2008 native mode, you then must plan to keep your old Exchange infrastructure long enough for the migration to occur.  That’s because the Mailbox systems for one version of Exchange can’t talk to any other in most cases.  So a 2003 Mailbox Server can’t talk directly to a 2010 Mailbox Server without some assistance.That assistance is in the form of Bridgehead servers (2003) or Hub/Transport and Client Access Servers (2007). You will need to have at least one of each for each Site you have set up in Exchange/AD.  One note, this has nothing to do with the migration itself, this is all about the period of co-existence you’ll need to go through until everyone is on the 2010 architecture.

After you have upgraded your Domain Functional Level and set up your front-end server systems, you can begin the process of installing the first of your Exchange 2010 server systems.  Start with the Hub/Transport and Client Access roles (unless you are installing them on the Mailbox Role server itself) to allow for message routing to get set up.  This lets mail flow between the 2010 and legacy front-end servers, so when you start moving mailboxes, you don’t lose mail connectivity.

Finally, once you’re sure mail is flowing correctly and that no one is about to get lost in the shuffle, take one more backup of the legacy systems (just to be safe) and start your migration.

You can find a guide to preparing for and deploying Exchange 2010 at this TechNET site.  Along with other blogs and Microsoft sites, this should be required reading before you even download the software and play with it in a lab.  With a migration that includes a period of co-existence with earlier versions of Exchange; careful, proper planning is not a nice-to-have thing.  It is a solid requirement that must not be overlooked or under-appreciated.

Something Old, Something New

As many of you know, Exchange Server 2010 Released To Manufacturing (RTM) today, with TechNET and MSDN releases to follow later this week.  There are a ton of new features, like Database Availability Groups and Archiving and compliance put in place in this version, so it’s a big step for Microsoft.  As you might expect, I’ll be writing quite a few articles on the new features as Exchange 2010 rolls out.

In the meantime, a recent 180 degree turn by Microsoft on support for Exchange 2007 has extended the theoretical useful lifetime of that platform by quite a bit.  It may not be time to start writing off Exchange 2007 just yet =)

As reported by Mary-Jo Foley in this ZDNet Blog post, MSFT has announced that they will continue to support Exchange 2007 into the Server 2008 R2 platform.  Prior to this point, official support for Exchange 2007 would end on the Server 2008 RTM platform, limiting the lifespan of the 2007 product to the 2008 RTM server. That wasn’t going to be tomorrow, or anything like that, but it certainly would be a shorter time-span than if Exchange 2007 got R2 support. 

Yielding to immense pressure from the end-user community, MSFT did acknowledge that not everyone was ready to upgrade to Exchange 2010.  There were no details on exactly when an update for Ex2007 would be available for R2, but suspicions are that it will be done via either a Roll Up or possibly even a new Service Pack early in calendar 2010.  Traditionally, this type of compatibility update had been done as part of a Service Pack, so my bet is riding on that idea.

Either way, the news that Exchange 2007 will live on to Server 2008 R2 is welcome.  This allows organizations who are in the middle of roll outs to not worry about if their servers are installed with Server 2008 RTM or R2, and will allow those migrations to complete on the 2007 platform.  This, in turn, allows upgrades to 2010 to occur without being rushed due to OS incompatibility issues.  There is something to be said for the fact that this decision will slow adoption of Exchange 2010 overall, but it will mean better, more structured roll-outs over time.  Safer, stronger and better planned upgrades are never a bad thing.

More Exchange 2007 OS Choice

A full article is coming early next week, but in the meantime:

Since Microsoft has announced that Exchange 2007 will indeed get support for Windows Server 2008 R2, you’ve got a new choice for running Ex 2007.  So what OS are you going with?

Can I get a Witness?

Continuous Cluster Replication in Exchange 2007 allows for two nodes of a Distributed Failover Cluster (DFC) for Exchange to be held in different physical locations and different physical network segments.  This is a good thing to leverage if you’re not concerned with local High Availability, but can lead to some interesting issues if something goes wrong.  The two nodes will use their quorum resources to find out which server should be in control of the cluster (and therefore assign resources accordingly) – but that doesn’t help if the nodes cannot see each other due to network failure.

One of two conditions would happen if you ran into this situation as described this far.  You could run into split brain, where both nodes thing they’re in charge and bring up Exchange resources.  This can take hours or even days of manual work to fix, and therefore Microsoft has taken steps to prohibit it.  If either node can’t figure out who’s supposed to be in charge, both go offline to prohibit split brain at all costs.

The second potential situation is the opposite, that neither node can figure out who is in control and both therefore shut down.  While this doesn’t put your data in danger, it does effectively shut off your Exchange system, stopping all messaging flow.  Neither situation is good, but by default, if arbitration is not possible via either quorum or other means, this safer situation occurs.  Luckily, there are “other means,” specifically the File Share Witness (FSW).

FSW is a file share (as its name implies) that both nodes can see under normal circumstances.  It must be placed on a server that isn’t part of the cluster.  Usually, you find it on a file server within the environment, but be aware that it will need to be at least Windows Server 2003 SP1 or better.  The FSW should also be placed either locally to the preferred node (the one you want to “win” in the event  of arbitration) or in an independent location that can be seen by both networks where CCR nodes reside.

In a CCR cluster, there are only two nodes, so right off the bat, if an arbitration event occurs, neither node could gain a majority and take over if there was some communication failure or other emergency.  The FSW acts as a third resource that can be polled to find out who is in control. Both nodes will attempt to take ownership of the FSW, but due to the physical placement of the Witness Server, only one will successfully do so.  That node stays online as owner of the cluster, the other node prohibits resources from going live until the emergency has been resolved.  As you can see, placement of the FSW becomes a critical component to the overall success of this arbitration system.

If you have only two physical locations, your best bet is to place the FSW on a server in the secondary site.  This allows the cluster to properly arbitrate to the remote site if the production site goes offline.  If you have more than two locations, then you can place the FSW on a server at a third location, just make sure connectivity to that site is stable and constant to and from both CCR servers.  If that link is unstable to one or more sites, you can create accidental arbitration events when they’re not really needed.  The benefit to putting the FSW at a 3rd site is that you can survive a link outage at either CCR node location without having to manually force one node or the other to take control (called a Force Quorum Operation). 

Here’s an example of what I mean.  If you have only two sites, and place the FSW at Site 2, a network link failure at Site 1 would force arbitration to Site 2 since the CCR node at Site 1 would not be able to communicate with either the node at Site 2 or the FSW hosted there.  In this scenario, there may be no value to failing over to Site 2, but you would automatically fail over anyway.  If, however, the FSW is hosted at a 3rd site, and both sites can see it, then a network fault between Site 1 and Site 2 would not flip everything to Site 2. Since Site 1 is the preferred owner, and can maintain control of the FSW, it will stay in control of the cluster.

You can find out a lot more about configuring FSW for Exchange 2007 via this TechNET article. The use of FSW technology is mandatory for CCR, and will continue to be a good idea for Exchange 2010 and Database Availability Groups as well.  Learning how this technology works today will allow you to create redundant solutions that last through your future Exchange solution sets.

Time Keeps on Slippin…Slippin…Slippin into the…past?

Once again, I grabbed the title of the post from a play on words on lyrics from “Fly Like an Eagle” by the Steve Miller Band.  For those of you who have no idea what I’m talking about, see this YouTube video, and know that you have made me feel very, very old.

The point to the title was that some of you may have noticed that calendar appointments, email time stamps and many other things seem to be off by 1 hour starting yesterday.  The reason is that unless you have all your updates and hot fixes from Microsoft, Windows 2000 and 2003 would believe that Daylight Savings Time changes occurred on Sunday, October 25th at 2am, and changed the clocks on any non-updated servers.  If that’s an Exchange Server, the incorrect change will flow over into all Exchange functions as well, causing quite a few problems.

Normally, we here in the US change our clocks twice per year.  The latter one used to happen on the last Sunday of October, when we all set the clocks back by one hour at 2am on that day.  The problem is that the US Government changed the rules late last year, changing the dates that these one-hour shifts take place on.  This year, it will be November 1st, but Windows wasn’t originally programmed to deal with that.

Most of us update Windows and Exchange regularly, so we got all the appropriate patches and ran the required updaters on the systems in question.  You’ll know you did it right if the clock did not change Sunday, and do change on November 1 at 2am.  You know you missed at least one if either your server time, or your email time stamps are all an hour off today.

Yet another reason to patch regularly, but everyone can miss one now and then, so visit this Microsoft Support site on DST changes if things are acting odd time-wise.  If thing are acting odd in other ways, throw me an email, I might do a column on it =)

Power(Shell)full Stuff

Windows PowerShell was introduced a few years back, but is still trying to find its way in the big, bright world of Windows even today.  This command-line interface allows users of all versions of Windows from XP SP2 on up to navigate through day-to-day operations without walking through layers of GUI interaction to get there.  While somewhat slow to take off in the mainstream Windows admin world, for Exchange Server (2007 and up) it has become an essential part of the Exchange Engineer’s toolkit.

With Exchange 2007, Microsoft removed a great deal of the control functionality from the Exchange Management Console (EMC) in favor of the extensions that Exchange makes to PowerShell, creating the Exchange Management Shell (EMS).  At first blush, this seems to be taking one giant leap backwards in terms of command and control on a Windows Server (DOS, anyone?) but once you dive a little deeper, there are a lot of advantages to be found.

Let’s start with speed.  EMC is slow, very slow.  Waiting for each command to finish loading up in the GUI just to figure out the piece of information you need is painful, to say the least.  The main reason for this is that the EMC doesn’t really do anything itself, it just displays the output of various PowerShell commands in a graphical format.  So each time you ask the EMC to do anything at all, what you’re really doing is waiting for it to create the corresponding PowerShell commands, evoke and run them, and then take the results and spit them back out on the screen for you.  For more complex tasks that are going to take some time anyway (like multi-mailbox move operations), this isn’t such a bad thing to deal with.  The GUI components make that job easier, and don’t add a huge amount of time to the overall process.  But for other operations, like getting a listing of all Storage Groups assigned to a particular server, the overhead in delays for getting the info and formatting it into the MMC3 GUI interface can mean the processes takes several times longer to perform in the EMC than it would at the command line.

Secondly, some techniques could never be performed inside of a GUI in any version of Exchange.  Database consistency checking and error correction were always done from the command line, and therefore it was logical to build those routines into PowerShell and the EMS as the newer versions of Exchange evolved.  By leveraging the way cmdlets (PowerShell code snippets) worked, much more complex database control and corrective action sequences could be piped through a single set of commands.  This lets administrators do more with fewer keystrokes, and opens up a whole new world to 3rd-Party software platforms who found that batch files just couldn’t cut it for what they needed to do.

Finally, the same way that cmdlets can expand what can be done within the framework of a script for things that were always command-line based; they can also allow administrators to automated a lot of their day-to-day work as well.  Prior to Exchange 2007, setting up users and mailboxes was a rather straight-forward process, but required that you interact with a series of GUI’s to get it done.  This took up extra time, and also introduced another level of potential errors every time you opened up another GUI.  Granted, this did not tend to lead to a lot of issues for experienced administrators, but even the most seasoned pro is going to slip up if they have to click in the same spot, over and over, several dozen times a week.  PowerShell allows for the creation of complex scripts that can leverage cmdlets, VB and C# commands and other attributes bound into single executable files.  This means that a set of operations can be crafted, tested and saved, then run over and over as required.  Less moving parts means less chance for errors, and portability means that a more experienced administrator can craft scripts for folks who may need to run repetitive tasks but not have the skills to work with cmdlets yet.  Entire online communities have sprung up to facilitate cooperative efforts on PowerShell scripts and to share what worked and what didn’t for various Exchange-related tasks.

Speed, efficiency and community interaction are just three components of what PowerShell can do to assist the average Exchange Engineer or Administrator.  Since the trend toward leveraging PowerShell and the EMS continues into Exchange 2010 (scheduled for release later this year), getting on-board with PowerShell tools now will build a knowledge-base that will grow with you as you continue to leverage new and better Exchange platforms.

Why use AD DNS?

My recent article on the need for (and use of) PTR Records in DNS have sparked quite a few questions on using DNS with Exchange Server in general.  The biggest one I get is “Do I need to use Active Directory DNS in order for Exchange Server to work?”  The answer to that one is a bit complicated, but in its simplest form, it boils down to, “No, but you really, truly should.”

Exchange Server 2000 and up required some form of DNS in order to function correctly.  This is mainly because the Windows Internet Naming System (WINS) was “depreciated” starting with that version of Exchange.  What that means is that MSFT officially asked the community to stop using it whenever possible, because it could be removed completely soon.  As it turns out, WINS was phased out in Exchange 2007, though it may still be required for certain Outlook functions.  That’s a topic for a whole different series of blog posts though.

As for DNS integration, it’s quite possible to install Exchange 2000-2007 without having Active Directory DNS configured in your domain, though it isn’t a best practice.  As long as your DNS system can handle Server Name Records (SRV type records), you can successfully use a 3rd-party DNS for your Exchange environment.  There are, however; some good reasons to go with the native Windows Active Directory Integrated DNS solutions:

1 – Exchange can natively talk to Active Directory DNS, and therefore can do some interesting tricks with that DNS platform that it can’t do with 3rd-party DNS.  Things like AutoDiscovery when you move a user to different mailbox servers, or after a recovery operation with Database Portability just don’t work the same way if you’re not using Active Directory DNS.

2 – Many 3rd-Party tools leverage AD DNS to figure out where Exchange resources are.  Note, I’m far from unbiased on this topic, so please see the disclaimer at the end of the blog.  Since many Windows-based tools will natively use AD DNS API calls (like DNSCMD and the newer variants in PowerShell), you may need to make manual updates to your 3rd-Party DNS, or may have to give up functionality.

3 – Many other non-mailbox objects are stored in AD DNS, and must be mapped manually in other DNS systems in order for Exchange to work properly.  You will have to track your Global Catalog servers, Domain Controllers and other resources in order for Exchange to function.

So, as you can see, there are some very good reasons to use Active Directory DNS if you plan on using Exchange Server.  While you may have external DNS records hosted with an ISP or other provider; internally you will be better off with the native DNS solutions in Windows unless you are ready and willing to fine tune your DNS systems and stay on top of it. 

If you are in doubt, you can use the Exchange Best Practices Analyzer to test your environment before you begin to install Exchange.  This tool will test for many things that Exchange needs, including properly configured AD or 3rd-Party DNS systems.

What is a PTR record and why should I care?

True story, I heard this exact question from a client not too long ago.  Through my trials and travails as an Exchange Engineer, DNS information is one of the very most confusing aspects of email systems that my clients have to deal with.  Just figuring out how to use normal DNS records tends to lead to a strong desire to give up on the whole project, so attempting to discuss reverse records can cause many to throw up their hands and run screaming from the server room.

Alright, that only happened once, but wow was it fun to watch.

PTR, or reverse lookup, records are used to allow external servers and systems a way to find out what identity a server has based on its IP address.  So, for example, we can look at the IP address information for Bing.com, Microsoft’s replacement for Live Search.

If I do an nslookup on bing.com, I get the following:

Name:    bing.com
Address:  64.4.8.147

So it appears that 64.4.8.147 is the IP address for that URL.  Now if I put the IP address into nslookup, my DNS server will attempt to backtrack to see what that server is identified as:

Name:    origin.bay.ux.search.live.com
Address:  64.4.8.147

Not a perfect return, as I was looking for it to reply that the IP was assigned to a bing.com address, but knowing that Bing Search replaced Live Search, the results are clearly traceable.

You can read up on what PTR records do, and how to create them at this Wikipedia page.

Now that you have some idea of what PTR records are used for, we can discuss why you will want to make sure that all mail domains you have authority over are properly configured to use them.

These days, you can do a quick Bing (or Google) search and find dozens of software packages designed to allow you to send out email as if you were someone else.  Sometimes there’s a legitimate reason to do this, such as if you manage email lists for multiple organizations through one mail domain.  In many cases, these tools are used toward nefarious ends, allowing a hacker to send email that appears to be from a domain in order to scam or infect the recipient.  Famous examples of this are phishing emails (see this Wikipedia page) where a scammer will send an email that appears to be from – say – a bank or other institution.  Your end user receives the mail, and since it appears to be from a trusted source, they’ll click the link and possibly enter in private information, leaving your organization open to attack.

To combat this, many SMTP servers either natively support, or can be configured to support, PTR lookups before accepting email.  This way, the header information can be examined to ensure that the email isn’t coming from a suspect source.  The end users don’t see much difference, but email that is from unknown domains, or domains known to be fraudulent, can be rejected before ever getting to their mailboxes. Exchange 2003 and 2007 do not natively reject email based on a bad reverse DNS lookup, so left to their own devices; you don’t have to worry about blocking incoming mail by accident.  This doesn’t mean you can ignore PTR records though.  Since many other mail server systems can be configured to reject non-verifiable mail, and since there are a host of 3rd-party systems that work with Exchange Server that can do the same, failure to properly configure a PTR record can cause your outgoing email to get bounced because you cannot verify your identity.  This means that you need to set up a PTR record for your Mail eXchanger (MX) records and your domain in general, or you risk having email returned as non-deliverable.  Having no PTR record is just as bad as an incorrect configuration here, as a lack of a reverse DNS lookup will cause the same results as an incorrect reverse DNS lookup.

There is a great debate over if this is a good or bad thing.  PTR responses could be forged, and so this is not a foolproof method of confirming identity.  Also, if multiple organizations all use the same SMTP server (thing about that multi-list email server from before) then which one gets the PTR record assigned to it?  If you have an email domain, then for now it is a good idea to ensure that PTR records are properly configured so you don’t run into the problem, but hopefully there will be better, more secure systems to rely on in future to confirm identity (such as the SMTP-AUTH proposed specification as part of E-SMTP).

To avoid blacklists, blocks and email black-holes, configure correct PTR records for your SMTP servers and domains.  It is by no means a perfect system, but it is one that you will run into, so an ounce of prevention is definitely worth a pound of cure here.

Net Neutrality and EAS

Mobile messaging and collaboration makes for a big, bright world in business today. What was once the domain of a single service provider (Research in Motion - RIM) has evolved into a robust set of platforms to convey email, appointments and contacts from one device to another.  Blackberries, Windows Mobile devices, iPhones, Android phones and so many other systems can communicate either directly or through a proxy to a Microsoft Exchange platform.  This unleashes the workforce and allows for your people to be where they need to be in order to work, instead of where they have to be just to talk to each other through the email system.

Net Neutrality is the idea that no matter what the network provider offers in terms of services and software, you should be able to use the devices of your choice and the platforms of your choice on those networks.  It’s a great theory, but putting it into practice is causing some issues along the way.  The FCC set forth a set of basic rules that they wanted carriers to follow, and in the greater sphere of comments, they were well received.  They recently added in two more proposed rules that directly impact cellular networks (digital broadband) and the services that run across it – including Exchange Active Sync (EAS).

You can get a rundown of the entire proposed rule set in this article, but the two that directly impact EAS most are:

5. Broadband providers cannot block or degrade lawful traffic over their networks, favor certain content or applications over others and cannot "disfavor an Internet service just because it competes with a similar service offered by that broadband provider."

On the surface, this looks like a standard anti-competitive rule.  In reality, however, many service providers in the cellular world are viciously blocking competing technologies, and their claim is that forcing neutrality will destroy their business.  EAS is a great example of this phenomenon, as not that long ago many providers didn’t allow that traffic on their mobile networks.  Mostly, this was due to the fact that they wanted to pus their own version of enterprise email synchronization (such as Sprint’s ill-fated attempt on the earlier Palm Treo devices).  Eventually, the need to allow this traffic or lose business to other devices and networks overrode the desire to use and sell their own platform, but that took a great deal of time, and lead to quite a bit of bad press and back-end attempts to circumvent the blocks.  By forcing mobile providers to allow all valid and legal traffic, the atmosphere for open communication standards will grow and more people will be able to take advantage of more technologies.

6.Broadband providers must be transparent about the service they are providing and how they are running their networks.

Proprietary networks are nothing new, but trying to create an EAS client for a phone on a network that actively blocks your ability to figure out how it sends and receives data makes this close to impossible.  Some providers have blocked all traffic they do not wish to have on their networks by simply making it very difficult – or nearly impossible – to figure out how a 3rd-party tool can possibly communicate with it.

I’m of two minds on these proposed rules.  On one side, EAS and other technologies require open, transparent communications platforms to work. Exchange can communicate with a whole world of different vendors’ mobile applications, but only if those apps can talk to the Exchange Server.  On the other side, competition drives better software and platforms.  If it wasn’t for all the things you can only do (or could only do) on an iPhone, RIM and Google would never have had the impetus to push their own platforms to new heights, and we’d still be staring at plain-text emails on black and white Blackberry devices.

It’s going to be a very loud fall season as the mobile providers and the FCC battle out these proposed rules.  The end result will have a huge effect, either good or bad, on how flexible and feasible your mobile Exchange platform plans will be. Competition is a good thing, but it cannot be forced on the market at the expense of profits.  There must be a way to balance these scales, and it will need to be found before Net Neutrality can be forged in the mobile marketplace.

 

 

Get back to where you once belonged (Failover Cluster version)

In honor of the re-release of the Beatles stuff all over the world (games, CD’s, maybe iTunes at some point), I took the title of today’s post from their song “Get Back” on the album Let It Be (Remastered).

I am, of course, going to tie this to something in Exchange; specifically Exchange 2007 Standby Clustering. Standby clustering refers to the theory of using a replication engine (like the native CCR or a 3rd-party system like Double-Take Availability – see disclaimer below) to place a copy of the data for the Storage Groups of the production cluster onto a secondary cluster.  Once the data is replicated, you can use the /RecoverCMS commands to recreate the production Exchange Cluster Mailbox Servers (CMS’s) on that secondary cluster.

The solution set for bringing up the Storage Groups and CMS’s on another physical cluster setup in the same or another location is fairly well established.  If a single node fails on a production cluster, other nodes take over the failed Storage Groups and work resumes in a very automated fashion.  If multiple nodes, or the entire cluster, fail you use /RecoverCMS and the associated protocols to manually get everything working on another system – so long as a copy of the data exists to work from.

The problem has traditionally been best expressed by the phrase, “And then what?”

If the original cluster failed completely, the answer was simple.  Rebuild the systems with the same node names, but prepare the systems as though they would be a new /RecoverCMS target system.  However, if you have not lost the production systems, and they’re stable enough to be used again, you would still have to reinstall them without some additional help.  The most common reasons for this kind of outage are routine testing of the failover systems and extended power failures that generators and UPS systems can’t handle.

Microsoft does offer a command set to fix this particular problem, but it is not well known or publicized.  As a matter of fact, during a recent client troubleshooting session, we had a couple or techs from Microsoft on the phone (Premier Support in this case) and they were not aware of this particular method for cluster restoration.

Once you have fixed whatever went wrong, if your production cluster is still viable (and is suitably stable for continued use), you can use a command set called /ClearLocalCMS to remove the original CMS entries from the original production cluster.  Doing so is not without risks, and you should familiarize yourself with this KB article on the subject before you try it. 

/ClearLocalCMS will remove the CMS components off the original production nodes, clean up AD, and disable the virtual computer object for the original cluster CMS.  This ensures that Exchange doesn’t accidentally address the original cluster system, even after the restore process begins.  Once the CMS is cleaned, you can go about restoration of the data using the same tools as you used to get it over to the standby cluster in the first place.

To get back to your original servers, use the /RecoverCMS command in the opposite direction (from DR back to production) and then use /ClearLocalCMS commands to re-prepare your DR cluster for use in the next emergency.

Jumping between clusters is not an automated or easy process, but it does work correctly if you follow all the steps in both directions.  This set of command suites (/RecoverCMS and /ClearLocalCMS) can allow you to get back to where you once belonged, every time.

CCR clustering is still clustering, and so is DAG

As more and more of my readers move to Exchange 2007 and 2010 from Exchange 2003 and earlier versions, I hear a lot about how using the new High Availability tools will finally free them from the yolk of clustering in Windows.  While both CCR and DAG are definite improvements over traditional shared-disk clustering, neither is a departure from clustering entirely.

We’ll be talking about the new HA stuff in Exchange 2010 (along with much more of course) in the webinar Double-Take Software and Microsoft are presenting tomorrow.  I’m the speaker for Double-Take, and Patrick Foley from Microsoft is going to be doing their portion. It’s September 9th at 11am, and you can still register for free by clicking here.

In the meantime, it is important to realize that both CCR (Continuous Cluster Replication) and DAG (Database Availability Groups) are offshoots of Windows Failover Clustering (WFC).  They both change the way WFC works, and by quite a lot, so you may never touch the underlying cluster technology, but it is still there.

CCR – as its name implies – works by allowing you to create a cluster during the installation of Exchange 2007.  This one is a bit easier to see as part of WFC, as you have to create a Failover Cluster first – specifically a Distributed Majority-Node File Share Witness Failover Cluster.  After that, when you install Exchange Server you can specify which server(s) will be the Active node(s) and which will be passive.  This creates the clustered Exchange resources for you, making the overall process of setting up clustering for Exchange a lot easier.  As this one has Cluster in the name, it’s easier to see the WFC roots.

DAG will permit you to create the cluster itself from Exchange 2010 command sets, eliminating the need to pre-create the Failover Cluster prior to getting the Exchange installation rolling.  While this makes the process even easier than in 2007, it still requires that you have two or more servers capable of running Distributed Failover Clustering.  This means that not every version of Windows 2008 is going to be suitable for DAG, but also means that – under the hood – you still need to know how Distributed Failover Clustering works to properly manage the DAG systems.

In both cases, the required level of understanding of clustering is greatly diminished from what was needed in Exchange 2003 and earlier versions.  Most of the guts of the cluster are controlled by Exchange itself, which is a double-edged sword.  On one side you have the fact that folks who don’t have a lot of cluster know-how can now set up HA solutions for Exchange.  On the other side, people who don’t have a lot of cluster know-how are facing troubleshooting clustered Exchange solutions they may not have realized were there.

Both solutions work great for Exchange.  While they don’t eliminate the need for 3rd-party products to help with overall HA (and I’m biased on this one, see disclaimer below), they do make mailbox server protection much more complete.  Just remember that you’re still running on a cluster, and arm yourself with the knowledge needed to keep it running smoothly.

Storage Groups, or, Keep Them Moving and Spread Them Out.

In a recent conversation with a colleague (James from VA in this case), he mentioned that there were a lot of folks who simply jammed all their users into one Storage Group in Exchange, even when they had the option to use multiple groups, and really should have.

Storage Groups are databases in Exchange Server.  They contain Information Stores and Log Files, and logically group together users and Public Folders that are related in some way.  There are a lot of different ways that you can choose to use allocate users to Storage Groups, but here are some of the more common methods:

1) Geography – Many organizations are centralizing Exchange servers to one or two datacenters, instead of putting a server at each location.  This saves money overall, allows for better message hygiene and, with the use of technologies like Outlook Anywhere, doesn’t alter the end-user experience to a any large degree.  Creating a Storage Group for each physical location allows you to better administer the Exchange system as a whole. If a new policy or procedure impacts only one office, you won’t be stuck combing through all the mailboxes and folders trying to find the users who are impacted by that policy. Just apply it to the Storage Group for that office and move on.

2) Department/Employment Level – This is a very popular method for splitting up users into Storage Groups, especially where there are a relatively small number of locations or other physical defining factors.  Many companies will define their Storage Groups by the departments that exist within the firm, mirroring their political structure into the messaging systems.  Sales, Support, Administration, Management and other logical groups can be translated into their own Storage Groups.  This has the dual benefit of not only allowing you to better control policies and procedures for each group, but also making it easy to put all managers into their own server which gets clustered, while the rest of the employees are on servers that will be restored from tape.  That’s just one example of the benefits of this segregation system for Storage Groups from the real world, so please don’t send hate mail. Of course, you can use a tool like Double-Take to give you even more options, but see the disclaimer at the bottom of the blog, as I’m far from unbiased on that topic.

4) Alphabetical – Where there may be only one or two locations for the business, or where everyone’s email is equal, then using something as simple as the alphabet can help figure out where to split out your Storage Groups.  A-M and N-Z or even 3 or 4 different breakouts can allow you to allocate your users across multiple Storage Groups without worrying about which users should go in which groups by other factors.

The reasons for splitting up Storage Groups go well beyond simple administrative tasks like policy and procedure enforcement. Storage groups can take an unmanageable situation, like 3000 users all in one gigantic glom, and make life easier to deal with by allowing you to manage smaller groups as required.  Database maintenance on a 500 user, 50GB Storage Group is a lot easier than database maintenance on a 3000 user, 1.2TB set of Stores.  It also means that the other 2500 users aren’t offline while you do that maintenance on the one Storage Group in question.

Physical limitations of Exchange could also force you to break things up into groups.  Granted, the built-in limit of 16TB per Storage Group (in Exchange 2007) is one most folks won’t ever hit without already breaking things out into multiple Storage Groups, but there are other size limitations.  Storage Groups must have their database files (.edb in 2003 and 2007, .stm in 2003 only) exist on a single logical volume.  This means that if you have 1TB of disk space, your Storage Group can’t grow past that point.  Multiple Storage Groups allows you to allocate your user data across multiple volumes, allowing easier growth and better storage management overall. This also leads to more spindles and read/write heads being used at once, which increases overall performance.

Even the Standard version of Exchange 2007 allows for up to 5 Storage Groups, so even smaller organizations can now split their users out across multiple Groups.  Exchange 2003 and earlier only supported one Storage Group, so if you’re on those versions, you will have to keep everyone together.  Otherwise, live by the simple mantra that more is better when it comes to the allocation of users to Storage Groups.

Getting Smart (Smarthosts that is)

No matter what version of Exchange Server you’re using – or even if you don’t use Exchange at all – you need to perform message hygiene on all mail in your organization.  Virus files, spam, phishing schemes and tons of other attacks are thrown at every email domain, every day.  If you’re not ready to deal with them, you’re dead in the water.  Smaller companies often just make due with hygiene software on the Exchange Server itself, but all sized firms need solution sets, and even smaller firms can take advantage of Smarthosts in a few different ways.

Smarthosts are servers or hosted solutions that are dedicated to performing message hygiene on all mail coming and going from your organization.  They come in many flavors, but tend to center around three types:

1 – Fully hosted solutions.  This method has no Smarthost hardware on-site, but instead contains everything at a hosting provider’s location.  Lowest up front cost, but highest ongoing (monthly, quarterly, etc.) costs.

2 – Hybrid solutions.  Here, you host a small appliance on-site to handle some of the workload, with the rest of the message hygiene functions handled by the service provider in their facility.  This isn’t quite as popular as the other two solutions, as the costs can quickly outweigh the benefits.

3 – Fully owned solutions.  Appliances and servers are in your datacenter, and totally controlled by you.  Start up costs are high, but ongoing costs are very low, as you’re only paying for updates to the virus/spam filter engines.

For smaller organizations, hosted solutions allow you to assign your Mail eXchanger (MX) record  to point to your Smarthost partner’s datacenter.  There, all incoming mail will go through the filters and checkers before it ever reaches your internal server set.  Most hosted providers allow you to update a list of “known good” recipients, so that any mail not destined for a valid employee is rejected immediately.  Virus scans are performed, and optionally spam filtration happens as well.  The remaining mail goes off to your internal mail servers for delivery to the end-users.

When mail leaves your organization, your internal servers are configured to send it first to the Smarthost provider, instead of directly to its destination SMTP servers.  The service provider performs all the same message hygiene on those messages, then acts as the SMTP gateway for your domain out to the rest of the world.

Hybrid solutions do everything that the fully hosted solutions do, but typically place a small appliance on-site to do periodic scans of the mail server itself.  They can also help speed up the whole process by making the “first hop” for email flow out of the organization be local instead of across the WAN.  It’s rare to see a hybrid Smarthost system that only does email hygiene, as the fully hosted or fully owned methods are much better for this kind of single-task solution.  Instead, the appliances on-site will typically handle all anti-malware processes including updating desktop protection and server scanning.

Fully owned solutions place appliances and/or other equipment at your production facility.  They act the same as the hosted solution, but you own all of the hardware and pay only for periodic updates to virus definitions, malware profiles and spam allow/deny lists.  This solution offers much more flexibility than a hosted solution, as you have complete and immediate control over any changes that might need to be put into effect.  The drawback is that if the hardware needs to be upgraded, you’ll be responsible for buying the new boxes.

Smarthosts are available from a large number of vendors.  Postini, Barracuda, and Microsoft Exchange Edge Services are just three that can be found with a Bing Search.  Most perform the same tasks, but each has their own selling points.  Your best bet is to talk with multiple vendors and see which one has the solution set closest to what you need.  Ask about things like ease of administration, web or application interfaces, ability to create custom allow/deny lists, and integration with your own Domain services to get updates to users and allowed accounts without compromising security on your AD servers.

No matter which vendor you choose, or which solution, using a Smarthost removes the message hygiene load off the Exchange Servers, which is always a good thing to do.  By dedicating either a service or server to handling message filtering and security, you can free up resources to provide a better end-user experience and a safer messaging environment.

Confirmed, No Server 2008 R2 for you!

Sorry for the horrible delay in posting, my loyal readers!  I’ve been trying to track down some official sources on this one, and of course Jeff Guillet from the EXPTA Blog beat me too the punch.  However, he’s a good guy and a great author, so I don’t mind giving him the credit.

Much to the dismay of a large number of Exchange Admins and Engineers, MSFT will not be supporting the 2007 version of Exchange on Server 2008 R2.  So, if you’re planning on upgrading from earlier versions of Exchange Server, you have only two choices:

1 – Use Server 2008 RTM, and be prepared to stay on that version of the OS for an extended period of time.  This will let you use Exchange Server 2007 SP1 and take advantage of Distributed Failover clustering and other nifty Exchange 2007-specific tool sets that rely on the Server 2008 architecture.

2 – Alternately, you can wait on your upgrade until Exchange 2010 is released to market, and become an early adopter on Server 2008 *or* Server 2008 R2. Interestingly enough, Exchange 2010 will indeed by backwards-compatible to run on Server 2008 RTM, even though 2007 won’t make the leap forwards onto R2.

As many clients were in the process of either rolling out 2007 to replace other email platforms or were upgrading from Exchange 2003 and earlier, this is a dramatic policy announcement from Redmond.  Current plans to continue toward Exchange 2007 on the latest OS (Server 2008 R2) will have to be altered or scrapped.  If abandoned, new plans have to be set up for Exchange 2010.  I can only believe this will lead to a slow adoption rate as Admins and Engineers try to figure out the combination of Exchange and Windows that will a) function and b) allow them to upgrade now and keep going for as long as possible. Since Exchange 2010 will have a lifecycle that goes beyond the end-of-life date for Exchange 2007, many clients are making hard decisions between moving forward with the Exchange 2007 rollout, or re-planning to use the newer version.

In the end, there are a lot of great reasons to move up to Exchange 2010, so from a purely technological perspective, moving up is a no-brainer.  The problem is that many folks have been planning for over a year to implement Exchange 2007.  Planning that started when the first Service Pack was released for both Windows 2008 and Exchange 2007 – traditionally signaling the point where most upgrades begin in earnest.

I fear that this will cause an additional hurdle to migration, both from other platforms and from Exchange 2003 and earlier.  I can only hope that Microsoft has some trick up their sleeves when Exchange 2010 launches to make the jump easier or simply more compelling.

Exchange 2007 Installation – continued (finally)

Some avid readers brought to my attention that I had promised an article on the minimum required for installation of Exchange 2007 in a default server config last week, and didn’t deliver.  Sorry about that, and here we go!

Once you’ve stepped through the pre-requisites for Exchange 2007 (see two articles back), you’ll be ready to run the installer proper for the Exchange system.  Today, we’ll focus on the setup for a single Exchange 2007 server holding all required roles.

Before you can install Exchange 2007, we’ll have to get AD ready to roll.  Previous versions of the Exchange installer had the /forestprep and /domainprep switches that could be run by Domain Admins who didn’t have Exchange permissions and didn’t want to give those domain privileges to the Exchange Admin. Exchange 2007 doesn’t have those switches, but instead segments out the different tasks into a set of 5 switches, each doing a specific prep job.  You have two choices here.  First you can get Enterprise Admin rights and just run the setup wizard.  Second, you can have someone with Enterprise Admin rights at the root domain and Domain Admin rights for each sub-domain to run the individual commands.  They can be found, and are explained, in this TechNet article. 

Once the domain and forest have been prepped, you can run the Exchange installer on the server where you want Exchange installed directly.  While there are various command-line and silent install methods, I’m going to focus on the wizard-based installation.

After you step through the welcome screens, you’ll be asked a few critical questions. From here on out, we’re going under the assumption that you’re running as an Enterprise Admin and that you’re doing everything (domain prep and all) at once.

You’ll need to define the Organization Name.  This is the Exchange Org, and not the company name or AD domain name.  Though the three names (Domain, Company and Exchange Org) may be related, the Domain Name and Exchange Org Name can’t be identical.  Choose something that makes sense, and doesn’t use any special characters – stick to numbers, letters and underscores.

You’ll also have to allow or deny permission for Microsoft to be informed about errors that Exchange sees, and you’ll tell Exchange if there are users on legacy Outlook clients (Outlook XP and 2003) and if there are 3rd-party MAPI clients (like Entourage) in the client-base.  Be careful here, if there is any chance that you’ll have non-Outlook 2007/2010 clients, use the legacy setting.  This creates a Public Folder hierarchy to handle administrative Public Folder tasks like the Offline Address Book distribution; even if you do not use Public Folders for anything else or you are setting up different Public-Folder Only servers.  Without the administrative folders, Outlook before 2007 will not be able to function correctly.

You will also need to choose what Roles to install.  The default is to create Mailbox (MBX), Hub/Transport (HT) and Client Access Services (CAS) roles, which are the three mandatory roles you must have in place for Exchange 2007 to run.  While you do not have to have these all on one server, each site has to have at least one of each of these roles running somewhere.  The default selection in the wizard will put these three roles onto the server you’re installing to, which is fine for smaller organizations.  If you have heavy MAPI users or a lot of users, you will want to install the MBX role only and put CAS and HT on one or two independent servers.  If that’s the case, select to install MBX role only here, and run the installer on the machine(s) that will host the HT and CAS roles and re-run the installer there, choosing the appropriate options as you go to install just the roles you need on each box.

The remainder of the installation is pretty much automated.  You’ll be able to watch the progress of each installation task, and as Exchange moves forward you will see a status report (with green check, yellow bang or red stop sign) as each is completed.  Hopefully, you’ll only see the green checks across the board.

While it is sometimes not required to reboot after the install, it’s not a bad idea to reboot anyway.  The reboot should be quick, and will ensure that resources used by the installers are freed up, and that all the services start properly.  Neither of those things is bad, and no one is yet using the server, so reboot, please.

For a default installation, that’s about it!  Next week, we’ll start talking about what the individual roles do, so you can decide if you want them on independent servers (or at all, for the non-required roles).  I’ll also endeavor to actually write up what I promise next week  And finally, for my non-Exchange-2007 users, I promise to do some articles on Exchange 2003 again in the very near future!

Big Brother is Watc…[Recalled]

The big kerfuffle over Amazon’s Kindle Team removing several books without warning from users’ devices (story and link at the end of the blog) has lead me to think about the various time clients have used the Recall Message feature. Typically, they have found that it behaved just slightly better than opening their office door and screaming “NOBODY READ THAT!”

There are a lot of great reasons to use Recall Message in Outlook and Exchange.  Perhaps something was sent before it was finished, accidentally clicking Send without typing in that last bit of information.  Maybe you meant to add an attachment but did not – in which case may I suggest some of the great tools at Sperry Software.  You may also have accidentally hit Reply All instead of Reply, or added in a group email list via AutoComplete instead of the one person you wanted to send email to.  All of these are legitimate reasons to really want to get the email out of everyone’s Inbox.

Years ago, Microsoft attempted to build the functionality of yanking back an errant email on command.  It required (and still requires) that you be using Outlook in Exchange Mode, and a Microsoft Exchange Server as your email platform.  When first introduced, this worked great.  There was no such thing as a preview pane, you were always using an Exchange Server and MAPI for communication, and folks were probably online and connected since there was no Cached Mode.

The issues arise when you look at what is required for a Recall attempt to succeed:

• The user must be online and connected to the Exchange Server
• They must not be in Cached or Offline Modes
• The message must be unread in the recipient’s Inbox (including unread by the Preview Pane)
• The recipient cannot have moved the message to another folder (unread or otherwise)
• They must be using MAPI. Even POP3 or IMAP on an Exchange Server won’t count.

So, as you can see, since the majority of email users these days do not meet one or many of these requirements, Recall Message will fail for the majority of your users.  What they will see is the original email (unchanged) and a notice saying you would like to recall the original email.  This can often be just as embarrassing as the original gaff, as it acts as a glaring exclamation point that someone made a goof.

One more note, Recall Message will not impact the copy of the information on a Blackberry or other 3rd-party Smartphones and mobile devices.  So even if the user’s desktop has all the requirements met, they’ll still have a copy of the message (and the recall request) sitting on their mobile device.

In the days of Exchange and Outlook with MAPI connections being the only conduit to email, Recall Message was a great way to manage errant information.  Today, with so many ways to connect to any mail system, including Exchange Server, this feature has lost its usefulness and created a false sense of security.  Remember, if there is a chance that someone’s using a non-Outlook client or not meeting all of the requirements above, you will not recall the mail.  An ounce of prevention is your only cure these days.

As for the Amazon gaff, for those who did not see the news:  Amazon recently pulled all the books by a major author and deleted them off all Kindle devices that held them.  This was due to someone posting the books on the Kindle store, without the publishing rights needed to earn money from them.  Once Amazon realized what was going on, they of course stopped selling all the books from that vendor.  Unfortunately they also removed all the digital copies that were stored on users’ Kindle devices without mentioning it to the Kindle owners.  They did refund everyone’s money, and they did send out an explanatory email, but not until several days later after the blogosphere detonated with Amazon hate postings.

Frankly, Amazon’s user agreement does say that they reserve the right to do exactly what they did.  I can’t fault them for heading off trouble (in the form of a copyright litigation) at the pass. However, it could have been handled a lot better, with more information supplied immediately to end users about what was going on and why.  Amazon has promised to make this process much more transparent so that next time there is no confusion.

The reason there was such an outcry about the process this time (as it has happened in the past, quietly and without fanfare or hate mail)?  The author in question was George Orwell, and the books were the likes of 1984 and Animal Farm.  Both of these books deal with subjects like retroactively removing information that didn’t meet the criteria of an overarching body of government or law.

From Dictionary.com:

Irony: an outcome of events contrary to what was, or might have been, expected.

Back to basics: Prepping a 2008 Server for Exchange 2007

Got a basic Exchange 2003 or 2007 topic you’d like covered in this series?  Email MikeTalonNYC@gmail.com and you might see your topic covered here!

A lot of my clients have been asking me to write about some more basic topics dealing with Exchange Server 2007 here on the blog, and so this week I’d like to talk about pre-requisites for installation of Exchange Server 2007 on Windows Server 2008.  These will change as Server 2008 R2 goes live later this year, so stay tuned for updates.

To start, run through the basic install of a Windows Server 2008 x64 box in either Standard or Enterprise Edition.  For a stand-alone Exchange server, you can use either version of Windows.  If, however, you need to cluster your Exchange 2007 systems with SCC or CCR clusters, then you must install Windows 2008 Server Enterprise Edition.  This article focuses on a stand-alone Exchange server, and therefore which version you install depends more on cost versus performance.  Check out this article for the details on how many processor cores, how much RAM and other factors that change based on versions of Windows Server 2008:

Windows Server 2008 Editions on Microsoft.com

Note: Since Exchange requires .NET, right now you cannot install Exchange to a Server Core installation.

This is a good time to install any tools your organization uses which are not Exchange-specific.  Monitoring tools, storage management, etc.

Once you have the basic Windows Server installed, make sure you run Windows Update to get the latest Service Pack and all the assorted hotfixes and patches.  Some of these are required for Exchange, so do not skip this step.  The most critical of these (beyond the Service Pack) is the .NET platform.  While you must have at least .NET 2 installed, the current version is 3.5.1, so you may wish to install that version as well.  No doubt future tools in Exchange 2007 will need some of those components.  Since most of the .NET platform shows up as Optional, you may need to manually check the boxes for these downloads before they will be installed by Windows Update.

Next up, install PowerShell to the machine if you haven’t already.  This shows up in the Server Manager Wizard as a Feature, and is a wizard-driven installation, so you do not need to download anything from Microsoft to do the install anymore.  The default parameters are fine unless you have a preferred setup for PowerShell.  (for those who will inevitably email me on this, according to MSFT, it is spelled “PowerShell,” not “Powershell.” Go yell at them if you don’t like odd capitalization.)

Exchange 2007 also requires that the IIS system be present on the server.  You can install this by bringing up the Server Manager and adding the Web Services (IIS) Role to the box. When you select this role, you will be asked if you wish to also add dependency features/roles.  Select Yes here to automatically add in quite a few feature sets that both IIS and Exchange need. 

A couple of notes: Two items that are not selected by default or by auto-selection are Dynamic Compression and the IIS 6 Admin Tools.  You should select both of those manually, as Exchange will need them.   Also, you have the option of selecting different authentication types for the Web Services in this wizard as well.  Be sure to select any and all that your organization will use.  As with most things, you can go back and add them later, but since you’re doing the install anyway, might as well get them on the box now.  Aside from these items, you can safely use the defaults for this wizard and allow it to perform the installation.

After you run through the IIS Role setup, you’ll need to install the remote AD configuration tool kit on the server.  The easiest way to perform this step is to open a command prompt and run the following command:

ServerManagerCmd -i RSAT-ADDS

When you run that command, the system will appear to be doing not much of anything for quite a few minutes.  If your experience is like mine, it will appear to be stuck at 10% for most of that time.  In my lab, with virtual machines that were resource starved, it took about 10 minutes.  Wait until you are returned to the command prompt (C:\>) before you move forward or the tools will not be installed and the Exchange validation tool will error out.

Next week, I’ll walk you through a basic installation of the Exchange 2007 server will the 3 main roles on one box.  Until then, have fun getting prepped.

OAB – Oh Boy! Yet another annoying error conquered

Exasperation no more!  OAB 0x80190194 error solved - finally

As many of you already know, I do a lot of Exchange configuration in my day to day job (see disclaimer below).  From time to time, I run into errors that seem to occur over and over, regardless of the environment, until someone can finally figure it out.  Outlook attempting to download the Offline Address Book (OAB) has usually been one of those odd errors, especially when a server is first created.

Starting in Exchange Server 2003, the OAB was only generated for the first time when the server entered the first maintenance window after installation.  So if you built a brand new Exchange environment, you would need to wait until somewhere between 1 and 3 AM before you had an OAB.  This meant that Outlook clients would exhibit a strange error about "File Not Found" during every synchronization.  Outlook went to look for an OAB to download, there was none there, and the sync would abnormally end.

This was easy to fix, however.  You simply located the OAB in the Exchange Management tools and forced a rebuild, which immediately cleared up the problem.  In rare circumstances you followed the TechNet articles and removed the OAB and created a new one, then forced a rebuild.  All in all, not a very difficult fix, but one that eludes many Exchange Engineers and Administrators for hours and hours until they either figure out the fix from reading about it on forums; or the give up in frustration and see the miracle of the error going away the next day (since the OAB auto-rebuilt during the night).

When Exchange Server 2007 came along, I had hoped that this would be a thing of the past.  After all, the least that Microsoft could do would be to put a message in the installer about having to rebuild the OAB. The best would be to have it auto-generate immediately on install.  Alas, as most of us in the Exchange world figured out, neither is the case.  On a new Exchange build, you still have to manually force an update of the Offline Address Book.

It is a lot easier now, though.  One Powershell command does it all:

Update-OfflineAddressBook -id "Default Offline Address Book"

Since the first OAB is always "Default Offline Address Book", the command is universal and can just be done immediately on finalizing your installation of the Exchange software.

All was well until I started getting a stream of calls from colleagues across the country about a new error cropping up in Exchange 2007 and Outlook 2007.  These folks had already regenerated the OAB manually or waited until after the first maintenance window, and now - unlike the File not Found errors before - they were getting a very odd error indeed:

Task 'Microsoft Exchange' reported error (0x80190194) : 'The operation failed.'

Google searches turned up suggestions for Microsoft Update, the Windows Update Site, and a few other red herrings.  Further investigation led only to the usual recommendations to regenerate the OAB.  This was driving many people, including me, nuts!  Remarkably, for several months none of us found an answer, but in all cases the problem went away on its own. Sometimes in a few days, sometimes in a few weeks, but always resolved itself in time. 

Now, my readers know me, that's not quite good enough for my palate.  So when time finally permitted I did some deeper investigation.  I found several things out that had nothing really to do with the error, but are fun:

- Avira Antivirus considers Experts-Exchange.com a phishing site.

- People tended to suggest everything short of lighting candles, standing on your head and typing mysterious incantations into Powershell (though frankly some of the stuff was pretty close)

- Many folks just let it go away on its own and never though about it again - until they had to build a new Exchange system.

Finally, I tripped over the MSServerAdmin site and to this article:

http://www.msserveradmin.com/how-to-overcome-the-exchange-2007-oab-issues/

Which got me to look at the Microsoft Exchange File Distribution service on my Client Access Role server (which in my case was on the same box as the Mailbox Role).  Once I stopped and started that service, Outlook immediately began to download the OAB and all was well once again.  That's really all there is to it.  This only seems to happen on Server 2008, which explains why I wasn't seeing it back when most folks were installing Exchange 2007 to Windows 2003.  I do not have an answer as to why this problem is so common with Server 2008, but if any of the Microsoft folks who read this can tell me, I'll happily post an update.

So, why did it go away mysteriously by itself over time?  That was a little more of a quandry, but I got that answer too.  These days, it's not usual to have to reboot an Exchange 2007 server after installation is finalized.  So folks were not restarting services regularly by rebooting the boxes regularly.  However, eventually, Windows Update Services would download a critical OS patch, and - left to its own defaults - automatically reboot the box.  This might happen the next day, or might be several weeks away, but eventually we'd reach Patch Tuesday and the boxes would reboot - restarting the service and clearing the problem.

Good things come to those who wait - this time.  Hopefully the answer will always be as simple as "Just wait and it'll fix itself," but until then I shall keep searching for answers and reporting them here.

Anyway, Anyhow, (Outlook) Anywhere

For some strange reason on my a classic rock kick these last few weeks, so today’s posting title comes from another song by The Who.

No clue?  Watch this YouTube video.

And listen for this lyric:

Nothing gets in my way / Not even locked doors / Don't follow the lines
That been laid before / I get along anyway I dare / Anyway, anyhow, anywhere

Outlook 2003 and 2007 offered a new approach to Exchange Server connectivity that did not exist in the wildest dreams of earlier versions of Exchange and Outlook.  These two versions of Outlook (and the upcoming 2010 version as well) have a feature set called Outlook Anywhere, previously known as RPC over HTTPS.

Since the dawn of distributed email systems, there has been a schism between the need for connectivity and the need for security.  Most servers can be isolated from the outside world in some way.  Many can be completely locked down and held within the corporate network.  Others can have limited web-based connectivity via tightly controlled front-end servers, allowing the sensitive back-end solutions to be protected inside the network.  The major issue for email was that the systems are inherently designed to communicate with the outside world.  In fact, it’s the whole point of the exercise for the majority of Exchange users.

Microsoft, to their credit, has created a series of systems that allow Exchange 2007 server to only expose the bare necessities to the outside world with the new Edge Server Role on that platform.  But not every company can afford that type of infrastructure, and it wasn’t available before the Exchange 2007 Server platform.  So most companies these days cannot take advantage of that role, and even when they do, the limitations of the Edge Server may not make it the best chose for remote users.

Before Exchange 2003, the alternative was to limit connectivity between the Exchange system and the world at large by using port control on the server side, and VPN connectivity on the user side.  This allowed for Exchange to see only what it needed to on the net, but also required administrators to set up and manage VPN software on all their mobile client laptops, home-office desktops, etc.  It was definitely a trade-off if all the users really needed to use the VPN for was Outlook.

The came RPC over HTTPS.  This technology lets MAPI RPC calls (the communications protocol Outlook uses natively until the 2010 version) to go over the public network in a secure manner.  Essentially, the Outlook 2003 or 2007 client will use HTTPS security – much like you would in a web browser to communicate with a bank or other secure website – but in this case it is Outlook talking to Exchange.  It is secure, reliable, and requires no other software on either server or client side.  It does require enterprise CAL’s in Exchange 2007, so keep that in mind, but it doesn’t need to have anything outside of Exchange Server 2003 or 2007 installed on the servers themselves.

With the advent of Exchange 2007, RPC over HTTPS became a more accepted part of Exchange and gained a formal name – Outlook Anywhere.  The system is simple. Outlook is configured to make a secure HTTPS connection to the Exchange Server or (in 2007) the Client Access Server/Hub Transport Server.  This lets the administrator strictly control access both via security certificates and limiting of what ports have to be exposed, but let end users function with just Outlook and no 3rd party or other VPN client.

To configure, you alter your Outlook Profile, under Account Settings|Microsoft Exchange account|Advanced Settings in Outlook 2007.  Once there, open the Connection tab and click the check box near the bottom to enable HTTP connections.  You will also need to click the button under the checkbox and supply the connection URL for your server. Typically, this is your Outlook Web Access URL, though you can configure a specific record if you prefer.

There are several settings that can help improve network function over slower links, but the defaults should be fine for most networks.  Note that you MUST have the ability to connect to webmail via SSL (in other words https:\\mail.yourdomain.com and not just http:\\mail.yourdomain.com).  Since the data needs to be encrypted, you have to have security set up beforehand.  Though, you can use a self-signed certificate, that will cause your end-users to have to click through a security warning when they log in, so obtaining a public SSL certificate is a good idea.

Once configured, Outlook Anywhere will allow normal Outlook functions to occur as if the users were on your corporate network, but with all information between them and your system secured and limited to avoid potential attack.  It is a great solution where setting up a VPN communication tunnel for each remote user just isn’t feasible or where the only reason to do so is for Outlook communication.

Microsoft has made a lot of good progress in helping secure an application platform that – by its nature – must communicate with the outside world in order to function properly.  Outlook Anywhere is one way that this type of communication can occur, and is a great way to allow your users freedom to get their work down without requiring specialized network technology to be put into play.  Anyhow, anyway, anywhere – a thought never more true for Exchange and Outlook.

Avoid bumpy rides with Exchange Online

I’m posting from a hotel room in Hartford, CT, getting ready for a meeting with a client tomorrow.  Getting here reminded me of why you might want to choose to not try to go it alone when you set up Exchange Server for your organization.  More details on the trip later.

One of the biggest problems I see clients facing when they deploy Exchange Server is the simple fact that it’s a huge software package.  In reality, it’s more of a suite of tools than one application, and therefore requires either pre-existing expertise, or a very fast ramp-up to be able to properly install, manage and maintain the solution set.  Luckily, there are options today that were simply not available in the past, namely Hosted Exchange Services (a.k.a.. Exchange Online).

Hosted Exchange is – at its heart – renting space on a shared or dedicated Exchange Server that is managed and maintained by a 3rd party.  These are hosting providers vetted by Microsoft (or at least they should be) and skilled in creating, deploying and managing Exchange Server solutions. No hardware on-site, and no major management needed by your own tech people.  So what are the pros and cons of using a solution like this.

The major con is that you have to keep your data on another server system that doesn’t reside within your firewall and security network.  Though the reality is that anything which requires connectivity to the outside world is technically open to attempted attack, some companies may not be able to tolerate hosted data.  Either corporate or external compliance policies and laws may prohibit this, so be sure to check with your compliance staff to make sure it’s ok before you ship the email off-site.

Barring compliance issues, security may still be a concern.  Make sure you completely check out your hosting provider, make sure they are a Microsoft Partner and make sure to get references before you sign over your digital life.  It’s totally fair to put the hard questions to your hosting provider, as you’re trusting them with a tremendous amount of sensitive data.

Beyond those two potential cons, the pros are pretty deep.  Many small to mid-sized organizations may attempt to host an Exchange Server in their own network.  Due to inexperience or just plain lack of time, patches are missed, settings are mis-configured, security concerns are overlooked.  This leads to either a poorly functioning Exchange system, or worse, an attack against the organization using the messaging platform as the attack vector.  Larger organizations who have dedicated messaging staff can afford to have people who look after these details, but smaller shops can’t, and often the company infrastructure suffers because of it.  I’ve discussed a lot of potential pitfalls to running an Exchange environment – this is a well-rounded way beyond most of them.

As a matter of fact, when talking with Jon Orton – one of the TPM’s that works with Exchange Online for Microsoft – he related to me that “we’re seeing more and more people, especially in the “small Exchange deployment, not much expertise” category, signing up for Exchange Online and partner-hosted Exchange.”  By placing the infrastructure in the hands of a team of experienced Exchange engineers, you remove that burden from your IT staffers.  Don’t get me wrong, I believe anyone can indeed learn to manage an Exchange Server messaging and collaboration environment However, if you do not need to, then this is one thing you can get off your plate and off your mind – safely.

So, even though it might be very tempting to set up Exchange Server within your network, the facts that it is readily available for purchase and that server hardware is getting cheaper should not be the major factors in decision.  Hosted Exchange Services with a trusted provider are easier to work with and just as readily available, and with a little legwork ahead of time, will be a better solution for many small and mid-sized organizations.

Which leads me to my harrowing journey today.  When considering how to get to Hartford from my native New York City (Queens, to be exact),  I had a few choices.  All were pretty easy.  I could take a train to a mid-way city and hitch a ride with a co-worker going to the same meeting from there.  I could take a different train and then a quick shuttle here.  But, because it was readily available and easy to set up, I chose to take a bus directly out to Hartford.  Much as with the decision to leap before you look and set up an Exchange Infrastructure on-site, I nabbed the tickets and got on the bus without a lot of pre-existing experience with the bus route.  After getting bounced around the bus for about 3 hours, and nearly losing the sandwich I had grabbed at the terminal, I finally got off the bus from hell and collapsed into the hotel room.  Had I researched my options a little better – and sought out some experience from others – I would have realized that a little more legwork would have led to a much better experience for me.

So do a little legwork before you put in your Exchange Server.  Know what you’re getting into and know that there are great options for getting out of the situation.  Exchange Online with a trusted provider can offer a convenient, safe and workable option for those who are just not quite ready to roll their own.

When your cluster goes “oops,” Using RecoverCMS

First, a quick note:  I’m posting this one from Windows Live Writer on Windows 7 RC1, which I’m happy to say is remarkably stable and much faster overall than Vista.  I’d recommend it wholeheartedly!

Funny story, I once had a client who swore that clustering was enough protection for their messaging environment, until an outage took out their entire cluster at once – causing them to be down for about week.  Now, that’s not the funny part, but what caused the outage is somewhat hilarious, more on that later.

Exchange 2003 and earlier had a pretty straight-forward method for recovering an entire MSCS cluster if one had failed on you.  You built one or more nodes of a brand new cluster, created an Exchange Virtual Server (EVS) Resource Group with the same parameters (names, IP’s etc) as the production system had, and Exchange would do the rest.

With Exchange 2007, the rules changed significantly, leaving many cluster users confused as to how the system now works if they suffer a cataclysmic failure of the production cluster.  Adding both Single Copy Cluster (SCC) and Continuous Cluster Replication (CCR) to the mix just makes things more confusing, so Microsoft created a new recovery method for Exchange 2007 clusters.  Called RecoverCMS, the system is really a setup task rather than a true failover system, but since your failover system just went belly-up, that’s not a bad thing.

If your Recovery Time Objectives are flexible enough to handle some downtime if an entire cluster fails, then you can leverage this system to get back up and running, either at the original production site, or at a new location.  There are some definite limits to what you can do with it which I’ll explain later, but he basics of how it works are pretty simple.

Step one is rebuild, repair or replace the original cluster hardware. If the repair works then you’re done, just restore any missing data from tape or other backup (due disclaimer, see below, I am biased on backup tools) and then resume normal operations. If you rebuild or replace completely, bring up a new server that is configured with Exchange 2007 in the Passive Cluster Node configuration.  You can find out how to do that:

Here for CCR clustering or,

Here for SCC Clustering

During that process you will also have installed the Exchange 2007 binaries on at least one node of the cluster system, so go to the directory that has the Exchange setup files and execute the following command:

Setup.com /recoverCMS /CMSName:<name> /CMSIPaddress:<ip>

Where <name> is the name of the EVS you’re restoring from, and <IP> is the IP address you want the recovered system to have – in theory the same IP as the original EVS had.

The rest of the procedure is pretty automated, and when finished, you will have a new EVS running on your new cluster node(s) that matches the original EVS and has all the users already assigned to it.  From there, you can restore your data if it was also lost to the disaster.

There are a few things that are extremely important to be aware of before you begin:

1 – Keep in mind that /recoverCMS is designed to restore a failed cluster only.  Attempting to use it for migration or for any other purpose will result in unpredictable behavior and is not supported by MSFT.

2 – You will need to manually create the volumes that existed on the failed cluster before you run /recoverCMS.  If volumes are missing then the recovery will fail.  They don’t have to be the same physical disk or size, just large enough to hold the data and with the same drive letters as the original cluster held.

3 – The System Attendant service will start and then immediately stop after you recover, this is normal, just bring the resource back online when you’re ready.

4 – Your databases are not mounted after a recovery, you must do this manually through PowerShell or the Exchange Management Console after you’re done with the restore.

5 – Do NOT try to use this across OS’s. If you started on Windows Server 2003, you must recover to Windows Server 2003, and 2008 to 2008.   It will not work if you try to go from one to the other.

6 – While you can pre-configure many portions of this system, it will still take some time to run through a /recoverCMS procedure from start to finish, so if you need a second-stage failover, /recoverCMS isn’t the best bet.  I’m quite biased on this (see disclaimer below), but unless you can be down for a few hours if both cluster nodes fail, you might want to go with another tool to provide remote site failover in addition to SCC or CCR clustering.

7 – Finally, SCR and CCR will not automatically work with /recoverCMS.  You will need to stop SCR if it’s running before you recover, and neither will resume automatically after the recovery is done.  Once you’re set up in the new node configuration, re-enable CCR and SCR manually as required.

/RecoverCMS is a great way to restore a failed cluster system to new hardware or rebuilt hardware after a fault.  You still need to back up your data to some device outside the cluster itself, but once you have that backup /recoverCMS can get your cluster back up and running much faster than the manual methodologies used in previous versions of Exchange.

As to the funny story I mentioned at the top of the blog, this particular client was in a hardened datacenter with UPS systems, 24/7 staff and a backup generator.  They were convinced that clustering was going to be more than enough for them.  After trying to explain that a shared-disk cluster (the only option at the time) had weak points, I finally gave up and let them be.  A few months later I got a great phone call.  Apparently – unbeknownst to the client – the datacenter crew had run all power connections through the UPS – including the generator.  The UPS was rated to handle the full power load of the datacenter on 1 of its 2 redundant circuit loops.  So far so good.  Well, this particular datacenter was in the middle of the dot-com boom (this was some time ago) and had grown exponentially in a short period of time.  What they had was well over half the full expected load on each of the two circuits, and one was failing.  So they diligently got replacement parts and moved the load over to the good circuit.  Since was over half the expected load, and circuit 2 was already under over half the load, they immediately overloaded the UPS, shorting it out.  The way it was explained to me, a solenoid shot through the casing of the UPS…and there was indeed a nice hole in the unit to back that up. No one was hurt, but needless to say, the whole datacenter was offline until they replaced the UPS, 4 days later, so they lost about one business week, without anything happening to the physical cluster at all.  Just goes to show you that anything that can go wrong, will.

Outlook, can you hear me? Can you feel me near you?

Might be showing my age and/or taste in music with that particular title (and if you’re totally confused by it, check out This YouTube video), but I think that it’s a great way to describe an annoyance that can happen if you’re using versions of Outlook before Outlook 2007.  Since a large portion of the users of Office are on the 2003 version (and many even earlier than that), resolution to a new server in the event of a disaster recovery event is a subject that is just as confusing as the famous rock opera I’m making use of in my title today.

When Outlook 2007 was introduced to the world with Exchange 2007, a lot was made (and rightfully so) of the new AutoDiscover features that this platform brought into the Enterprise Email marketplace.  The long and short of the AutoDiscover solution set is this:

When an Outlook 2007 client cannot find its home server – either because it is a brand new install of Outlook or because the home server has moved or been replaced – the Exchange 2007 AutoDiscover system can help Outlook 2007 find its home.  If the Outlook client can see an Exchange Server (or be directed to one by Active Directory), the Server can tell Outlook where the mailbox information for the user’s profile exists, and direct Outlook to connect to the appropriate CAS or Mailbox systems and get connected.  All the user/Admin has to do is tell Outlook the user’s email address and password, and AD with Exchange 2007 will handle it from there.  So if you’re installing Outlook for the first time, you don’t have to manually configure the Profile anymore – a great boon to Admins everywhere.

This system also kicks in if you perform Database Portability during a disaster, and have replicated the database with SCR; or have used a 3rd party disaster recovery/availability solution (see disclaimer below for all my bias information on that one =).  Once the Exchange system is responding again, AD can ferry the Outlook 2007 client to the new home for that mailbox, requiring only that the end-user close and reopen Outlook to complete the process.

However, what many folks do not realize right off the bat is that this solution set is ONLY available if you have both Outlook 2007 and Exchange 2007 as your messaging platform.  All users who need to take advantage of AutoDiscover must be using that combination of tools, and no other.  As you might expect, POP3 and IMAP systems do not AutoDiscover, but the majority of my clients were unaware that Outlook 2003 and earlier also cannot take advantage of this system, even if you have upgraded to Exchange 2007 as the messaging platform of choice.  It’s also worth noting that AutoDiscover doesn’t officially work in Exchange 2003 – no matter what Outlook version you are on.  Before I get blasted by mail on this one, I know some folks have sometimes seen it to work on Outlook 2003 with Exchange 2007, but it bombs more than it works, and officially it’s not supported.  For proof, I direct you to this article by the MS Exchange Team.

Since the code to perform AutoDiscover wasn’t in Outlook 2003, users on that client software will not be able to dynamically re-link to the new Exchange server unless the original mailbox server is still responding.  If it is, then Outlook can find the new server via the original server and re-home itself.  If not, Outlook must be manually re-directed to the new server.

Of course, there are ways around this.  You could update DNS to re-direct anyone calling for “Server 1” to the IP Address of “Server 2” – effectively re-routing all client software including POP and IMAP.  Outlook 2003 will still need to be re-profiled unless you take over the Service Principle Name (SPN) of “Server 1” on “Server 2,” but it will be a smoother transition.  Using a 3rd party tool (see disclaimer below) you may have the option of automated DNS and SPN updates, which will allow even legacy Outlook clients to jump to the new server with no more intervention than is required on Outlook 2007 with Exchange 2007 – even if you’re whole system in on the 2003 versions of those software platforms or earlier.

So, you are not without lots of options if you have any legacy servers and/or clients – or non MAPI clients – in your environment. You just need to be aware that the Exchange 2007/Outlook 2007 solutions for AutoDiscover services are not backward compatible, and plan accordingly.  Right now it looks like Exchange 2010 will have AutoDiscover that is backward compatible to Outlook 2007 only, so this soon-to-be-released platform is not going to solve this particular problem unless you’re planning on upgrading everything else in the environment to at least Outlook 2007 first.

Since I like to let folks continue to discover on their own, here’s a link to the White Paper from MSFT on AutoDiscover.

Finally, Update Roll-Up 8 for Exchange 2007 is out there, which can make life easier if you’re doing a fresh install of Exchange 2007 and want to get up to date with patches and fixes post SP1 quickly.  You can get it at this link.

The Dread Pirate Re-Seed – Part 2

Last week, we talked about re-seeding operations between two nodes of an Active/Passive CCR Cluster.  Since both nodes will most likely be inside the same logical network (though are not required to be in Server 2008), even unexpected re-seed operations on a CCR Cluster should be relatively painless.  Though it’s a full copy to target, it is happening over a LAN, and therefore faster and less resource-intensive than a WAN copy would be.

This week, let’s look at how the game changes when you work with Server (or Standby) Continuous Replication (SCR) when you have re-seed operations.  Just to be clear, re-seeds are not the normal method for data protection with SCR. Normally, when a log file is closed and a new prime log (usually E00 for the first Storage Group) is created, the closed log is shipped over to a target Exchange 2007 if SCR is enabled.  Once on the target, the log is held until the target reaches the number of logs specified in the log replay caching system – 50 by default.  After that, Exchange commits the logs to the database on the target, effectively providing SQL-like Log Shipping (or Continuous Data Mirroring, Asynchronous) between two Exchange 2007 Servers.

Whenever the two servers are not in communication with each other, there is the potential for operations to occur on the source that are not seen by the target, and vice versa.  Since this could easily result in a corrupted database, Exchange will check to ensure it knows what state the data on both servers is in before resuming normal SCR operations.  If the state is known, then SCR continues – transmitting all logs not yet on the target, committing all but the last 50 logs, and continuing on its way as normal.

If the state is unknown, a re-seed operation will need to occur.  More specifically if there is a gap in log file enumeration for some reason, you will require a re-seed.  You can see the reasons why re-seeds are required at the TechNet website listed here.

The one that is of most concern to Exchange Admins is that if the two servers are not in communication during a backup window, you will have to re-seed the database before normal SCR operations can continue.  When speaking of a local SCR pair this is not a big issue, as connectivity will be much more solid than WAN performance, and even if a re-seed is required, it will be relatively fast.  But across a WAN, it is more likely that you can occasionally suffer WAN outages that do not interrupt business operations.  If these outages extend past the time of the backup window, the backup tool will most likely truncate logs committed to tape without SCR being able to transmit those changes to the target Exchange Server.

Since minor network outages are a common (though hopefully not very common) issue with modern networks, the likelihood of requiring a re-seed due to this sequence of events is something that could be considered a normal part of Exchange SCR operations.  If your databases are small, then it won’t be an issue.  For larger databases, remember that a re-seed operation is a full copy of all data files to the target device from source, which could be problematic depending on your WAN throughput.

So with SCR, re-seeds do become a definite issue to contend with.  Since the occurrence of re-seed operations should be limited, you may be able to keep the systems successfully in sync without too much trouble. However, if you have larger databases or smaller WAN pipes, re-seeds can create problems for your network, especially if you use a backup tool that truncates your logs, or use circular logging for any reason.

Of course, there are alternatives to SCR for WAN protection and availability (see disclaimer below, I’m biased here), so you are not out of luck in terms of WAN operations with Exchange 2007.  Locally, CCR is a spectacular choice for nearly any sized Exchange system, with the exception of very large (over 1TB) datasets that may just take to long to re-seed even over a LAN.  Remotely, if you have properly planned for re-seeds then you will also be able to successfully utilize SCR, but if you do not – or cannot – plan for these required operations, then you’ll risk a failure in your failsafe system, which is not a great situation to be in.

In short, re-seeds are a normal part of SCR operations. They should not happen every day (or even every week) but you will most likely experience them over the course of your SCR lifetime, so plan for them now.

 

Next week is TechEd 2009!  I’ll be out in Los Angeles with the Double-Take Software crew, so please stop by and say hi. I’ll also be blogging from the event, so keep an eye on this column and follow me on Twitter at http://twitter.com/talonnyc.

The Dread Pirate Re-Seed – Part 1

Among the most common questions I get from clients about the new data-protection features in Exchange 2007 (and the soon-to-be-released Exchange 2010) is, “What is a re-seed and why does it happen?”  This mostly falls into the category of “fear of the unknown” since the technology is new, and documentation on how it works is somewhat scarce.

Re-seeds are a commonly confusing part of most protection methods, though they fall under different names and methodologies.  In a solution like Double-Take (see disclaimer below), they’re called re-mirrors or re-synchronization operations – and are typically differences only.  In a tape-backup solution it’s a restore operation, and might be everything, incremental pieces, or some combination thereof.  In Exchange 2007 these operations are called re-seeds, basically the replay of data from a server that has a “correct” copy to one that does not.  Today, we see these operations in Exchange 2007 Local Continuous Replication (LCR), Cluster Continuous Replication (CCR) and Server – or Standby – Continuous Replication (SCR).  Today, we’ll talk about CCR, and address LCR/SCR next week.

CCR allows an active node of a 2-node Active/Passive Exchange Cluster to replicate a copy its data to the passive node.  This allows the passive node to take over with a nearly-current copy of the data if the production system fails due to hardware or software failure.  There is a log replay lag to be considered, but it’s only 50 logs that need to be applied to the passive node during a rollover event, and that does give you some measure of protection against corruption if you catch it fast enough.  Otherwise, the system acts much like a traditional Single Copy Cluster (formerly Shared Disk Cluster) in behavior, and is controlled with a combination of Windows cluster tools and PowerShell.

Whenever a log file is committed, and a new prime log (usually E00) is created, the closed log is copied over to the passive node via an SMB share, where it is held until it passes the 50 log replay limit and is then committed to the database, or a rollover occurs and the logs are committed immediately.  Exchange 2010 will move away from the SMB share, but will utilize a similar methodology overall, if the beta is to be taken at face value.

In order to get the passive node in sync with the active data, the CCR system starts with a re-seed operation.  All data from the database is copied from the active node to the passive node, as well as any non-truncated logs.  From then on, only log files are copied, as they are committed on the active node.  If all goes well, this will probably be the only re-seed you see unless you have a rollover.

If you do flip nodes – let’s say from Node A to Node B – then Node B will re-seed back to Node A if Node A becomes divergent. In other words, if Exchange cannot determine what logs still exist on Node A, or if the logs are inconsistent, or if some are missing.  A graceful rollover will not cause a re-seed, but most emergency rollovers will require it.

The same will happen if you haven’t rolled over, but instead Node B was offline for some other reason.  When Node B comes back online, Node A will see if all the required logs are on both machines, and then either just continue CCR protection or else initiate a re-seed to copy the data over again if anything is amiss.  The only issue here is if your backup tools purge logs while Node B is still offline.  In that case the servers will be considered divergent and need a re-seed to get back up and running properly.

Finally, if a cluster is restored from a backup (tape or otherwise) to the active node, then a re-seed must be manually initiated to re-sync the nodes properly.  You will see errors telling you to do this after the restore is complete and you bring Node A back online.

One other condition exists, but it is a manually created condition. If you perform Offline Defragmentation of the database, you will trigger a re-seed operation when Node A is brought back online.  As long as the first Exchange log is still present (which it should be) then this will happen automatically. Otherwise, it will need to be initiated manually.

So, why is this an issue?  Normally, it’s not, but keep in mind that re-seed operations are *full* copies of the entire database.  So if you have relatively small databases and only a few of them, this isn’t a problem.  But let’s say you have over 1 Terabyte of data in your Exchange cluster.  Re-seeding that much data locally will be time and resource consuming, and doing it over a WAN (for distributed failover clustering) could be problematic – to say the least.  So you want to avoid re-seed operations at all costs and wherever possible, which means treating the CCR cluster very carefully, and following all the best practices from Microsoft on Exchange 2007 Clustering in general.

For information on when re-seeds occur, take a look at this TechNet article.  They’re not an everyday occurrence, but you will need to be sure you know when and why they will happen to avoid confusion and frustration.

Public airing of Exchange laundry...

Last week we finished up talking about what happens to Private Mail Stores during Online Maintenance (OM) on your Exchange servers.  While most people really tend to focus on what happens in those Private Stores the Public Stores are just as important, especially if you're not on Exchange 2007 yet.

For all versions of Exchange, Public Folders experience most of the same processes that their Private Mailbox cousins do, with the Exchange system scavenging items that are marked for deletion and have passed deleted item retention to start.  The system will also get rid of expired folders - a similar process to deleted mailbox systems in the Private Mailbox Stores - and perform an online defragmentation of the databases.  General maintenance and cleanup for Public Folders should be considered to be the same priority as OM on your Private Mailbox Stores, even if you're on Exchange 2007.

With Public Folders in Exchange 2000 and 2003, and even in 2007 in some circumstances, there is another vital process that goes on each day. It is not technically part of OM, but should be considered on par with the OM process.

As many people notice, the first time you boot up a newly-installed Exchange server, Outlook will complain about not being able to find "a required file" when you do a send/receive operation.  This is because - unless you're using a native Exchange 2007/Outlook2007 environment - Outlook cannot find the Offline Address Book (OAB) which doesn't get created in the Administrative Public Folders until around 5am the morning after the Exchange server goes online.  While not officially part of the OM process, the creation and subsequent updates of the OAB do happen overnight - 5am by default, just after the OM process is typically completed. 

OABgen (the process that - wait for it - generates the OAB) should be considered part of your OM systems. If you do not allow this to occur regularly, your OAB doesn't get updated and therefore your end users don't have the latest information - causing no end of frustration for you and your technical staff.  You could manually create/update the OAB using tools inside of Exchange, but the automated process is still your best bet for making sure these critical systems get updated regularly.

And one final note.  For those who believe they can get away with no Public Folders and/or PF maintenance because you are on Exchange 2007, be aware of one very important caveat.  All Outlook clients must be on Outlook 2007, and no Entourage or other MAPI clients can be used. That's because all MAPI systems before Outlook 2007 required the use of Public Folders for administrative information, like the OAB and other components.  In Outlook 2007, that's handled by Active Directory and Exchange itself, but if there is even one legacy client in your environment, you'll still need at least one PF tree, and therefore need to perform regular Online Maintenance on those Public Folder Stores.

Next week, back to regular blog topics - and speaking of which, if you have a topic you'd like me to cover, go ahead and drop me a line at MikeTalonNYC@yahoo.com

Mailbox Separation Anxiety

As we have been discussing over the last few weeks, Exchange 2000-2007 runs through quite a few operations during the Online Maintenance (OM) process.  We've talked about how databases get cleaned up, and how messages get deleted (temporarily and then permanently).  This week, let's look at Deleted Mailbox Retention (DMR) and what it can do for you.

DMR is the process that Exchange uses to remove mailboxes that have aged past their expiry date in the Private Database.  These expired mailboxes can be created by different operations within Exchange, either directly by the administrator of Exchange, or by operations in Active Directory that inadvertently cause a mailbox disconnect.  Disconnected mailboxes are simply mailboxes that not longer have an associated user or object in Active Directory. The simplest way to do this is to delete a user account in AD, which will disconnect the mailbox assigned to that user as part of the process.  However, this isn't the only operation that can cause mailboxes to become marked for deletion.  There is an odd side-effect to moving mailboxes that can happen if you jump between AD sites and don't finish fast enough (You can read about it here), and of course; if you get hit with AD corruption it could disconnect mailboxes as well.

Once a mailbox is disconnected, the first OM run that happens after the disconnect will mark the mailbox for expiry in a user-defined number of days. The default is 30 days, so if you run maintenance daily, the mailbox will stay in the database, but disconnected, for 30 days after it is disconnected from the mail-enabled object (like a user) in AD.  It also means you have to take when you do OM into account for your deleted item retention numbers.  For example, if you only do OM once per week, then you will need to add up to 7 days to your overall retention time for planning purposes.  Since the 30 day clock doesn't start ticking until the mailbox is marked during OM, you will extend the amount of time data remains in the database.  This is critical for regulatory compliance, as holding data for an extra week could put you over the line in terms of meeting your regulatory guidelines.

The other part of this OM process happens when Exchange scavenges the database for expired mailboxes.  If it finds one, and the mailbox hasn't been re-assigned to another AD object, Exchange will delete the data from the database permanently and create white-space in the database that gets cleaned up by OM to be reused by Exchange.

So, here again, it becomes vital that OM is permitted to run on a regular basis.  If not, deleted mailboxes will not be removed from the database unless you manually launch the cleanup wizards.  This is more than just a regulatory problem, as without removing the old information and freeing up white space, your databases will grow much more rapidly than if you are regularly cleaning up your mail stores.

Next week... Public Folder operations during Online Maintenance, so stay tuned!

Update now - no I mean it, NOW

Though I'm known for keeping a level head during these kinds of scenarios, there is an emerging vulnerability in both Exchange Server and Internet Explorer that you'll have to address immediately to keep your enterprise safe.  I wish I was kidding, but no, it's true.

The long and short of it is that specially crafted messages could be sent to an Exchange server that - even if only previewed by a user - can allow an attacker to take control of the Exchange server itself with full Exchange Admin privileges.  The details can be found at this link, but I'll summarize here.

If an attacker crafts a message in certain ways via TNEF (Transport Neutral Encapsulation Format), and that message is then either previewed or opened by an end-user (which is the point of email after all), then the attacker can gain elevated privileges on the Exchange server and wreak havoc. While that sounds like a major problem, there is good news.  The attack would require a sophisticated attacker creating just the right message and then finding a live mailbox to send it to.  Not only that, but it would need to get past the SPAM and CAPTCHA systems if you use them.  Finally, eventually the anti-virus vendors will jump in as well, scanning for messages crafted with the exploit.

The really bad news is that end-users do NOT have to open an attachment this time.  They can unwittingly infect the mail servers by previewing the message body or opening the message itself.  Since Outlook clients and now even Outlook Web Access both Auto-Preview by default, end-users can follow all best-practice safety protocols and still end up infecting the organization through no fault of their own.  Add to this that some anti-virus tools may use MAPI viewers to scan mail, which means that the system designed to shield you could accidentally infect you instead.  The same goes for backup agents that use MAPI clients to do brick-level backups.

There is one more bit of scariness to be told, though information on this last peice is more sketchy than the other exploit vectors. It would appear that the TNEF message may not need to be even previewed by an end user, but could in fact attack a server just be being received by the server itself.  Even until we know more for sure on this last point, I think the previous points - which are verified by this author and others - are bad enough that you should jump on this right now.

Microsoft has released a patch - also available via this same link, under "Security Update Deployment" and so you can protect yourself and your organization.  I strongly urge you all to do this immediately, and to have your end-users run Windows Update (Start|Windows Update on Vista, Microsoft Windows Update Online for everyone else) to get critical Internet Explorer patches as well.

Now, this is not only your Exchange server that you are protecting.  If an attacker gains privileges on your mail server, they can use it as a launching pad for attacking other servers with the same exploit or any other they'd care to try.  That means that if you don't patch your systems, you could inadvertently be responsible for attacks against other servers all over the net.

And so, I am asking my readers to protect the community at large as much as they can, by spending the time necessary to download and install the appropriate patches.  Yes, this will mean a maintenance window for a reboot.  Yes, it's annoying that we've gotten hit with yet another critical vulnerability.  But, yes, it is our responsibility to make sure that the attacks stop at our own borders, and that our own mail servers do not become the launching point for the next Code Red.

If I cannot bring you comfort, then at least I bring you hope

As we get ready to celebrate the beginning of a new year (in many countries), now is a good time to plan for your Exchange Servers and what you'll be doing in the year ahead.  While this is not the best time to plan out new infrastructure or massive overhauls immediately, you can start planning what you'll do later in the year on that scale.  You can also plan for what you'll do right off the bat in January for things like maintenance and updates/upgrades.

I plan to dedicate entire columns to these topics in the next few weeks, but here's the overview to get you thinking (once the champagne wears off...):

Offline Maintenance is a good thing to do regularly for Microsoft Exchange.  About once or twice a year you will probably want to take the time and walk through the process.  Microsoft says that this is not a solid requirement, but it can help identify long-trending problems deep within a database, so doing it once or twice per year is time-consuming, but worth it.  Plan now for downtime, so that you don't have to worry about when you'll have the ability to do it later. 

Online Maintenance is automatic, but you may want to look at your Exchange Server profile and see when the best time to have it done really is.  By default, the online maintenance system will run every night, but if you're also performing backups overnight, the two systems can overlap, causing the maintenance to be halted and never finish.  Take a look at your event logs to see if both systems are operating normally, or if the online maintenance run is terminating early.  If it is, you may want to change the times it runs, or even only run it on certain days.  Unlike with Exchange 2000, later versions of Exchange can hold up admirably well with even weekly maintenance runs, but be aware of what won't get done until maintenance by checking out Microsoft's Knowledge Base and/or waiting until my upcoming column on the topic.

Test, test, test, test, test......then test again! Now is the perfect time to set out your goals for testing of recovery/availability solutions (see disclaimer at the bottom of the page).  Backup systems, failover systems and other recovery solutions MUST be tested, otherwise how do you know they're doing what they say they're doing.  Best bets: do a test restore of Exchange data (at all the levels available from your tool like database, mailbox, mail item) to a different directory on the Exchange server and mount the DB in a Recovery Storage Group to ensure it is working properly.  This should most likely be done once per month, but at least once per quarter.  It doesn't require downtime, and doesn't require end-users to do anything.  Perform a failover test (if you use a failover tool) or a total restore test at *least* once per year.  2x per year or once per quarter is better, but of course since this type of testing will impact end-users, scheduling can be an issue.

Housekeeping is never a bad thing to look at after the clock strikes midnight on January 1.  Are you reaching your database limits in Exchange Standard edition?  If so it's time to either start ratcheting down mailbox limits or else starting to plan your upgrade strategy.  If you're already on Enterprise edition, are you pushing the limits of the free disk space you have on your drives?  Plan now to move the databases to another partition with more space.  Have your Stores grown to alarming proportions?  Might be time to start planning out new Stores and even Storage Groups to distribute the load.

Finally, give a look through the event logs and such to see if there are repetitive errors or other trends that can indicate problems brewing.  One more common example of this is recurring event ID's 1018, 1019, and 1022.  These indicate that there could be problems brewing in the database itself, though it is very possible that the error events are the only indication you see until the DB crashes.  If you're looking for a tool to help wade through the log information, check out Splunk, an IT search solution set that can make your life a lot easier.  [Due Disclosure: Splunk is currently working on partnerships with Double-Take Software.]

Have a healthy, happy and safe New Year's celebration.  Here's hoping 2009 exceeds everyone's wildest dreams in terms of greatness, and that Exchange never ceases to help us all live in these very interesting times.

Until next year.....

Mike Talon, Chief Columnist, BeingExchanged.com

PS: The title of this posting is taken from the song "Closing of the Year" from the movie soundtrack to "Toys."  The lyrics of the first verse are incredibly relevant to our combined experience this year, and I offer them here:

If I cannot bring you comfort,
then at least I bring you hope
For nothing is more precious than the time we have, and so
We all must learn from small misfortunes
Count the blessings that are real
Let the bells ring out for Christmas
At the closing of the year

- The Musical Cast of Toys Featuring Wendy and Lisa

Under Pressure (da da dum diddy dum dum)...

First off, welcome to the new subscribers who hopped on-board with BeingExchanged!  Also, congratulations to everyone who recognized the riff I used in my Subject line, and for those who don't, it's referring to This Song, and don't we all feel old?

Back Pressure is a phenomenon in Exchange 2007 that can grind your email flow to a halt with little or no warning, so it's a great thing to begin to understand now, before you run into it in the real world.  The basic idea is that Exchange Servers through the years were famous (sorry in advance to the guys from Microsoft) for collapsing under heavy I/O load and/or lack of disk space.  Back Pressure allows the Exchange 2007 system to halt some of the processing of data temporarily to allow the system to keep running without the danger of I/O overload.

Specifically, Back Pressure allows Exchange Servers to stop accepting new messages into transport queues, thereby allowing system resources to free up over time and for those queues to empty out.  This translates into no mail coming or going from your organization while Back Pressure is in effect, which will have quite an impact on your operations.

While the Back Pressure system could impact your mail organization as a whole, it's good to point out that only Hub/Transport and Edge Services servers are impacted.  Messaging queues and the disks they sit on are what the Back Pressure systems monitor, not database locations.  Pure Mailbox Role (MBX) servers won't experience Back Pressure, but they will be impacted by it if mail cannot flow in or out of the MBX servers due to mail being unable to transverse the other roles.

When the Back Pressure system kicks into gear, the most noticeable impact is that Outlook clients will have all their outgoing mail stuck in either the Outbox or Drafts folders.  OWA will show everything in Drafts and other client software will show that mail isn't being sent.  Of course, no mail is coming in either, but that's a little more difficult to "see" unless the user was expected a message.  On further examination, you will see that no mail is coming into or going out of mail queues on the Hub Transport or Edge servers, a dead giveaway that Back Pressure kicked in if your connectivity is working just fine but no mail is moving.

There are three ways Back Pressure can be brought online on an H/T or Edge server:

First off, if the logical disk that contains logging and other information for either role runs below a set level of available space, Back Pressure will be invoked.  This is done to keep Exchange from "crashing" a disk, or filling up all space available and causing problems within the Exchange system or (if it's also the system drive) Windows itself.

Next, if the amount of physical memory or other system resources run low; Back Pressure comes online.  Since mail isn't coming in or going out, the resources will return to a normal level once whatever process is spiking finishes or is terminated.

Third, the number of pending operations in the memory queue (as opposed to the overall amount of memory in use) can build up if something is slowing down Exchange like a virus, DDOS attack or just a sudden influx of mail transfer.  If the depth of these transactions exceeds a limit set within Exchange, the result is Back Pressure stopping mail flow, allowing the existing transactions in memory to commit through to disk without new transactions coming in.

Each of these metrics has a different number of resources assigned as the "break point" where Back Pressure kicks in.

For disks, Exchange uses a formula to determine what space must be present and free before Back Pressure is invoked.  Specifically the formula is 100*(hard disk drive size - fixed constant) / hard disk drive size. The "fixed constant" number is 4GB for the release version of Exchange 2007, 500MB for the SP1 version. If the total available free space on the drives which contain queuing systems drops below that number, new mail flow stops and Exchange system spools out the queued data until the disk systems free up enough space to get above the threshold.

Finding the amount of memory used to reach the "high" utilization number, or the depth of the queues is a bit trickier.  There are several calculations and settings used to figure those out, and a complete description can be found in this article about Back Pressure technology in Exchange 2007.

The end result is the same.  Build an Exchange 2007 server with the wrong sized resources (disks, memory or server load), and the system will effectively halt itself to make sure that those limited resources don't crash.  So it is vital to properly size out both your expected user load, and the resources (physical and otherwise) that you're planning on running that load on.  You can change the thresholds for the Back Pressure triggers (see this link) but that's designed to be more of a temporary fix than a permanent change.  Instead, work with the publicly available sizing guides from Microsoft and other sources to make sure your hardware can handle the mail transfer without feeling the pressure.

MTC Part Two

Back at the Microsoft Technology Center here in NYC for another day of hands-on Exchange 2007 setup work.  We're presenting tomorrow on both Exchange and SharePoint protection with Double-Take Software (see disclaimer below), and today I got to finalize the environment and test failover and failback.  Of course, failover and failback worked perfectly, but I was really impressed with the new Hyper-V systems that we're using to create and run the demos.

So far they've handled all of the Exchange stuff the exact same way as physical servers, including the failover and failback processes that call for Active Directory and WMI updates along with starting and stopping Exchange services on one or both machines.  Since the system is automated, the changes happen rapidly.  On physical servers, this doesn't pose a problem, but sometimes virtual machines (sharing processors and RAM) can get bogged down.  That has not been the case here, with the Hyper-V VM's staying responsive and moving fluidly through the entire process.

Granted, you probably wouldn't put the DC/DNS Server and both the production and failover Exchange servers on the same Hyper-V host in a production environment.  But, it is nice to know that it can be done.

Hyper-V is relatively easy to use if you're familiar with the current virtualization tools on the market from a variety of vendors.  For Exchange 2007 on Server 2003, the VM's behave exactly the same as a physical box would with the same software set - as you would expect from a VM tool.  What is different is the speed at which they reboot.  Even with Exchange 2007's extended boot-up time, the average reboot took only 1-2 minutes, as compared to 3-5 minutes for physical boxes and other VM platforms.  This could just be a quirk of the environment we're building in, but that's still a fast boot up.

In terms of configuration, the GUI interface for Hyper-V allows for the standard set of server definition variables.  You select file locations for the definition files and the virtual hard disks, set the amount of RAM, processors, etc, and then boot the VM.  For these servers, we used cloned Windows 2003 Enterprise installs taken from other VM's already built, but you can boot from a virtual CD-ROM or any physical boot media the host can see attached to itself.

From there, as expected, the server behaves much like a physical box, loading the OS and rebooting as required.  Once you're up and running on a Supported OS, you can install the Hyper-V Integration Services via a virtual CD-ROM.  This enhances mouse and keyboard integration and opens up several dozen options - such as the ability to share folders between the host and the guest.

Networking options follow the standard set of VM options, including the ability to create Host-Only networks that can't see the outside world, or bridge physical adapters from the host to the guests.  This is especially helpful for Exchange, as you could create a Host-Only network between the Mailbox role servers and the Hub/Transport and CAS role servers, while giving the latter another interface to the outside world.

We're using a combination of virtual disks and physical disks, so I did not get a chance to play around with the Snapshot features - which require that all volumes be on virtual disks to function properly.  I've done snaps of Exchange 2007 servers with other tools with a good success record, and I was hoping to try it with Hyper-V too.  That'll have to wait for next time.

Oh, before I forget, though we'll be in the Grand Central Briefing Center, this MTC has an Envisioning Center as well.   It's a theater-like room that showcases all the latest (and even some upcoming) Microsoft products.  And sure enough, they have a working Surface unit that I got to play with.

You can find more info on what Surface is (and I really suggest checking it out) at this link.

Get the message? If not, it may not be your fault.

Microsoft Exchange engineers have always come under fire when a message that was sent to someone in their organization didn't get delivered, for whatever the reason may be.  The same goes for messages that bounce back for no apparent reason.  It could be a problem with the user's account (disabled, misspelled, etc) or it could be a problem with the server (offline, not connected to the net, etc).  It could also be a problem totally outside your organization, and that's what I'd like to talk about today.

In this ever-more interconnected world, routing of email can often depend more on external factors than internal ones.  Email messages can get mis-routed or even totally lost well before they reach your system, or can get purposely removed from the Internet somewhere along the way through no fault of your own.  If you're finding that more and more outgoing email is lost without any visible reason, the most common problem that is beyond your immediate control is getting accidentally blacklisted.

Blacklists are specific shared lists of either email servers or - in some cases - individual email addresses that spam control companies make available to end-users.  The most common of these is the DNS BlackList (DNSBL) which can either track IP addresses of servers known (or thought to be known) to send spam, or can track the domain names of such servers.  You can find out more about DNS BlackLists from Wikipedia at this link.  These systems collect information on servers and domains that send out spam and/or dangerous files (like viruses), and share that information with anti-malware companies (like McAfee and Kaspersky among others).  The anti-malware companies then use those lists to update their anti-spam and anti-virus solutions to protect consumers.

So if your server's IP address or domain accidentally ends up on these lists, suddenly external servers won't send mail to your servers anymore and won't accept incoming mail from you either.  It isn't easy to get on these lists by accident, but it could happen to you.  One or more internal users may get hit with a virus that sends tons of spam mails out from your email server, or someone totally outside your sphere of influence gets their computer taken over by a virus that forges headers to make it look like your servers are sending spam - and that's just two examples of things that could get you accidentally listed.

Once you're on these lists, it can be a very time-consuming process to get yourself un-listed.  Generally speaking, it will be a manual process of contacting each organization that has you listed on their blacklist system, and proving that you don't deserve to be there.  You can start your search at w3dt.net, which will assist you in figuring out which companies and filters have got your IP listed as up to no good.  Once you have that information, contact those organizations directly and either ask what caused you to get blacklisted or - if you already know that info - what you've done to stop the problem.

Be prepared to show that you identified who was sending the spam and why (and that you stopped it), or that you've isolated and removed the virus.  You will need to be patient and persistent, and will probably need to make several calls over a significant period of time (days to even weeks) to make sure that the problem gets solved.  Each organization that has you blacklisted will want their own proof that you're not a threat anymore, and in reality - provided they're not abusing their power - I can understand why they're being so careful.  Spam and viruses are a huge problem in the net today, and making sure they protect their clients is their business model.

Once you have the blacklist fixed to show you're not a spammer anymore, it can still take several days or longer for end-users of those services to get the updates to their spam and virus control systems.  So, if you do end up on a blacklist, you have quite a road ahead of you, but well worth it to clear your good name!

Getting blacklisted accidentally is a horrible thing.  Blacklists are a good thing overall for the benefit of email users all over the world, but when they suddenly stop you from emailing anyone (or getting email yourself), they can be a giant headache that you'll have to deal with effectively to get back in business.

Update on Wildcard certificates

Not that long ago I talked about using wildcard certificates to allow you to move OWA and ActiveSync services from one physical server to another.  Since single certificates are assigned to a single server, failing over or moving to another server would cause the clients to suddenly lose SSL connectivity, as the certificate would not match up, and ActiveSync devices cannot pop up the error about a non-secure connection. OWA can, but it can be troublesome with end-users who suddenly start seeing security warnings.

Following up on this theory of using a domain-assigned wildcard certificate, research has shown that older Windows Mobile devices (WM 5 or earlier) cannot leverage these types of certificates at all.  WM 6 and higher can leverage this technology, but earlier versions were not coded with the required information to recognize that a certificate could possibly be assigned to more than one physical server or networked device.

So, for those using WM 6, OWA and Outlook Anywhere, you can use wildcard certificates to allow for services to move from server to server as required.  For those using earlier versions of WM (or the 3G iPhone - though the jury is still out on that one), you must use server-specific certificates, and re-configure the devices' ActiveSync connection if the server itself moves. 

None of this impacts Blackberries, as they authenticate via the RIM network, and not directly to the Exchange servers.  If you're using Blackberries, wildcard certificates offer the ability for all other mobile systems (OWA, Outlook Anywhere, etc) to move with your servers in the event of a loss of a particular physical machine, while RIM will handle moving the Blackberry devices.

Long story short, if you're on WM 5 or earlier, be ready for a few support calls when you need to move the services or fail over between servers - even if you use wildcard certificates.  If your users are on the new iPhone systems, be sure to keep a close watch on Apple's forums, as new information is being discovered every day.

I Luv Exchange.

Over the years that I've been working with Exchange Server, one of the more common questions I get asked is "Why Exchange?"  With all the other messaging systems out there, it's a valid inquiry.  The short answer is, "I am a masochist."

Yes, that was a joke, please do not send any more of those really weird emails my way...

So why do I stand by Exchange Server year after year, even as yet another version releases without a SQL back-end, forces hardware upgrades and has a difficult migration path?  Because - at its heart - it does what I and my clients need it to do.

Exchange Server is a complex animal, but one that other vendors have not yet been able to completely match in terms of what it does.  There are tools that do messaging and shared calendars well - some might even say better.  There are collaboration tools that are superb!  There are even Unified Messaging platforms that rival the best efforts of the guys in Redmond.  There are not, however, many total packages that can do it all, and do it well enough that you can rely on them.

The combination of Exchange Server, Office Communication Server, SharePoint Server and Outlook (well, all of Office) creates a solution set that does everything that most businesses would require for messaging and collaboration.  Add in the new Groove systems and now you're moving well past what most Enterprise Messaging solutions can provide.

Yes, it is true that Exchange is challenging. It is also true that the Messaging and Collaboration Platform is complex and overwhelming for inexperienced users.  However, I challenge a novice to take Lotus or SendMail and get it working without either a little help or a lot of reading.

So, enough with the Exchange bashing!  Reach out to others in the community, search knowledge bases and web forums, and generally let others who have been in your shoes before help you.  In most cases you can find someone who has experienced the same problem you have, and who found a way to fix it.  And in any case you'll learn a lot more about how the platform works.

No, it's not perfect - not by a long shot - but Exchange is the best Messaging and Collaboration platform (with its eco-system software tools) that I have used in my time as a Systems Consultant and Engineer.  So yea, I do enjoy working with it, and there might even be some luv in there too.

Quite a stretch for clustering in 2008

Microsoft Clustering Services (MSCS) have existed in one form or another since NT4, but have always suffered from a significant limitation.  All cluster resources had to exist within the same logical subnet, or else you couldn't create the cluster itself.  Windows Server 2008 allows for some flexibility in that regard, with the ability to create nodes of a contiguous cluster in different logical subnets.

Before we dive too far into that, you may want to see the official MSFT information here:

http://technet.microsoft.com/en-us/library/cc770625.aspx

So what does this mean for you and I?  It means that we can create CCR clusters on Exchange 2007 that stretch between physical locations and subnets.  However, to do this you'll need to be on Server 2008, the function just isn't available in Server 2003. This allows you to provide basic availability for Mailbox Role (MBX) servers in your Exchange environment, but doesn't take care of everything when it comes to DR planning.

First off, this applies only to Exchange 2007 Enterprise Edition, and then only to MBX role servers.  While most other roles are natively fault-tolerant with multiple servers installed with the same role able to stand in for each other, organizational or regulatory rules might not make that kind of redundancy possible.  Edge servers are the biggest example of this.  Since they're not tied into the domain structures, they don't contain any way to quickly flip traffic from one Edge server to another in different sites.  Third-party tools (see disclaimer below) can often take care of that function for you, as can working with your DNS provider to facilitate moving the MX records in the event of an emergency.

If you have legacy Exchange 2000/2003 servers, you're also not able to take advantage of this new MSCS feature set on those boxes.  The same goes with any non-Exchange tools, like SQL, anti-virus servers, anti-spam servers, etc.  Even if these servers run on Server 2008 clusters, they'll require some third-party intervention to handle the data replication for those systems.  This would include things like Blackberry servers, GoodLink systems and other non-Exchange remote email tools.

Finally, keep in mind that CCR clusters can only extend to Active/Passive, 2-node configurations. That will mean you can't use these solutions if you need to go beyond that model - which Exchange 2007 easily allows without CCR involved.

Server 2008 Failover Clustering is a great method for basic High Availability for Exchange 2007 CCR systems - even across subnets.  With some additional tools, it can become the center of an Exchange DR solution set that can help your organization withstand even site-wide emergencies.

More than zero?

There are a ton of interesting settings and switches in Exchange 2000/2003, but one of the more unusual is the "Zero Out Deleted Database Pages."

Exchange 2000 and higher all have several levels of potential access points for un-delete operations.  When anything is deleted from a mailbox, the data isn't truly removed, but rather marked as scheduled for delete at some later date.  The default settings keep mail items for 15 days and mailboxes for 30, after which point they're removed from the server.

Data pages within the ESE database are another story; however.  They get deleted all the time, but not removed from disk.  They're simply marked as deleted, ready to be overwritten as needed.  Normally, this doesn't pose any specific security threat, as File servers and most other Windows servers do the same thing when data is deleted.  In higher-security environments; however, this could be a policy violation, as the data would be stored "in the clear" and vulnerable  to anyone able to get their hands on the physical disk devices.

Microsoft has taken steps to natively remove the potential for security breaches, but at a cost.  Built into every version of Exchange supported today is the ability to literally overwrite each deleted ESE page with zeros.  The processes is therefore referred to as "zeroing out" the pages, and the "Zero Out Deleted Database Pages" setting controls this behavior.  By default, the setting is not turned on, but can be enabled via the Exchange System Manager GUI, Powershell or via Active Directory.

The problem is that the benefits rarely outweigh the costs in the real world.  Zeroing out pages takes up a good chunk of processor time, and Exchange deletes pages - quite literally - all the time.  Zeroing out will also have a huge impact on any form of replication, just because a large amount of data change is required for the process to work.  In addition, this method of secure wipe meets only very low-level security concerns.  Most regulatory requirements require multiple overwrite passes (7, 15, sometimes even more) and mandate random data be used, not just zeroes. 

So, while you might feel like you'll gain some security by using this setting, the extra overhead and falling short of regulatory compliance should make you think twice before you turn it on.

Get wild with wildcard certificates

Most organizations that are considering securing their email systems are also looking at SSL certificates to aid in that effort.  Security certificates allow for encrypted communications using Outlook Web Access, Outlook Anywhere (RPC over HTTPS) and ActiveSync devices like Windows Mobile Smartphones. The issue is that a normal SSL certificate is bound to a single server, which means you'd have to obtain a signed certificate from a provider for each server you want to put into the solution.

You could use self-signed certificates to get around the issue of having to purchase individual certificates, but only at a cost.  Many applications will not accept self-signed certificates, and even Microsoft-controlled systems like OWA will kick up several layers of security errors - which will increase support headaches for you and your staff.

Alternately, you can obtain a Wildcard Certificate from a provider, which will allow you to use one certificate for multiple servers in your organization.  These certificates are more expensive than single-server options, but can give you much more flexibility with things like server rebuilds, migrations, and failover.  The only restriction is that the servers will have to be part of the same organization. That means that all the servers will have to be part of the same domain and/or forest in the sense of Active Directory.

Wildcard certificates can be defined at either the organizational level - such as *.wildcarddomain.com - or masked at another level, www*.wildcarddomain.com for example.  With the first example, all server identities that are part of wilddcarddomain.com will be able to share the certificate.  In the second, www1, www2 and even wwweb.wildcarddomain.com will be able to share the masked certificate.  Which you use will depend on the depth of security you are looking to implement, as more specific masks allow for less chance of someone spoofing the certificate for a foreign machine.

Once you install the wildcard certificates, you can allow services to move between servers within the same organization without breaking security. This can be extremely helpful for both migrations and disaster recovery, as you can have both of the servers shielded under the same certificate, and therefore you will not get naming errors when the users move or fail over. 

In a worst-case scenario, you can use single-server certificates on a migration or failover target, but your end users will get errors about the certificate being valid, but assigned to a different server name than the one they are connecting to.  You can then purchase another certificate after the fact and assign it to the target server to remove the error if/when needed.  However, wildcard certificates give a much better option for flexibility to you and your team.

Dial-tone revisited

The theory of Dial-Tone Recovery (DTR) is one that has often been overlooked in the world of Disaster Recovery (DR) for Exchange Server.  However, even in Exchange 2007, DTR can provide a great method for immediate restoration of email services, though with a few things to keep in mind.

For those who haven't heard of DTR before, here's a primer:

If a primary Exchange 2000, 2003 or 2007 server fails, you can attempt to restore services by deleting the corrupted databases and re-starting Exchange services.  This will create blank databases and allow users to send and receive new mail, access new calendar items and access all shared contact information.  Running a /disasterrecovery install of Exchange on a rebuilt box with no data will do the same thing.  Though end-users can access their email systems again, there will be no historical data, so this isn't a total solution set for true DR, but gives you some options for immediate availability.

In an emergency, this can give you time to perform restoration steps - which could take quite a while to finish - without making everyone wait to get back basic send/receive capability. You can restore a copy of the historical data via several methods, taking the time you need to do it right.

If you used a brick-level tape or disk backup solution, you can restore mailboxes via that tool's recovery system. Archiving solutions and Continuous Data Recovery systems (like TimeData from Double-Take - see disclaimer below), can let you move mailbox, folder and other data back over time as well.  If neither of those tools are at your disposal, but you to have a backup of the database and logs, you can restore those to a Recovery Storage Group, and use ExMerge or Exchange 2007 tools to bring back mailbox data and merge it with the new information on the DTR-recovered server.

If no backup is available at all, you can still provide Exchange services from the point of DTR onward.  While no historical info will be available to them, the end-users will be able to send and receive new email, calendar entries and Public Folder data.

Hyper-V-active Exchange

As most folks have already seen, Hyper-V - the newest Microsoft Virtual Server Solution - released to market yesterday.  This is a pretty big step forward for MSFT, adding a datacenter-ready virtualization solution to the Server 2008 platform.

I'm doing a little research now on the Exchange front with Hyper-V, but I was able to get an un-named source (ooooh spooooky) from MSFT to confirm that some MSFT Exchange 2007 servers at their facilities are running on Hyper-V, so support on that platform will most likely be forthcoming.  More info on this front as I can get my hands on it.

From the limited contact I've had with the betas, I can safely say that this product is a different animal than Virtual Server 2005.  The overall performance in both the Host and Guest components has ratcheted up by a factor of multiples, The management interface is also re-designed to allow for more fine-tuned control over the vitalized systems, and cluster support means the ability to move VM's between hosts is built-in.

We're still in the very early stages of this technology, so expect more reports as I run into it more and more in the field.  Within a short period of time, I'm also going to get to put Exchange 2007 through its paces on Hyper-V, and I'll be happy to report on that too!

It goes without saying that Double-Take is going to be supporting Hyper-V in the very near future with some amazing tool sets.  If course we can already replicate and fail over Exchange from Physical or Virtual (of any flavor) into a Hyper-V VM, but keep your eyes peeled to find out what new stuff we're working on over here especially for the Hyper-V platform.

Legally important

Providing Disaster Recovery (DR) for Exchange servers has always had strong arguments in its favor.  As an overall requirement, being able to get the email systems, calendars and contacts back up and running ranks pretty high up the list.  But there are other reasons to look at Dynamic Infrastructure solutions for Exchange that go beyond the convenience of Exchange end-users.

When email is stored only on the production Exchange server, it can be altered or destroyed by anyone with access to that server, which means anyone with an email account that allows them to see that particular user's mailbox.  This leaves you with a whopping legal liability (check local listings on exactly what that is for you), but one that can be avoided in most cases.

Using an Operational Recovery tool, like Double-Take TimeData (see disclaimer below), will allow you to ensure that another copy of the data is not only held off-site, but held in a repository that tracks all changes to the email, calendars, contacts and all other information.  This way if a critical change is accidentally applied, or if someone maliciously attacks the data on the production server, you have the ability to revert either individual items or the entire Store or Storage Group as required.

This means that if the data is required for a legal requirement (like court-ordered discovery), you can be sure that not only will the system be available, but also that any information that could be lost without impacting the server can also be quickly and efficiently restored.  Of course, being able to get that information back to some other server or even to a desktop or laptop is best, so look for tools that give you that flexibility.  After all, if you don't happen to have the original server running to perform the restore of information, you'll need to be able to designate someplace else to receive the data instead.

This doesn't change anything you might be doing with DR and Dynamic Infrastructure in your Exchange Environment, but gives you a deeper level of protection and flexibility that most standard DR tool-sets just can't natively offer.