Hyper-V Me (Yes, Even for Exchange)

Due to the release of Hyper-V R2, there has been a lot of buzz about using Microsoft’s Virtualization Platform to create virtual machines for everything in the datacenter from file servers to high-capacity databases.  There are quite a few systems that are clearly ok to use virtualization with, but Exchange Server has always been one of those platforms where virtual tech has been a questionable course.  Not the least of the reasons for this is that folks have been confused on if MSFT supports running Exchange in production on a virtual platform.  So, I’ll spell out the current support stance as found on official MSFT sites only.  Note:  There are many blog postings that point one way or the other, so I have hunted down official MSFT sites to quote here (i.e. TechNET, MSDN, etc).  While many blogs can be trusted (like the official Exchange Team Blog at http://msexchangeteam.com – and http://www.BeingExchanged.com of course), many of my readers have corporate policies about needing to see the documentation from a MSFT site directly; and so here it is:

Exchange 2000 and earlier:

Not supported for production environments.  See the end of the TechNET article found here: Link to TechNET

Exchange 2003:

Supported in production.  Warnings are given that you must use Virtual Server (2005 R2 or higher) and that there are some performance limitation to be considered. Link to TechNET

Exchange 2007

Supported for production environments on Hyper-V, but with some limitations. You must be running Server 2008 as the Guest OS, and you have to be running Exchange 2007 SP1. Unified Messaging may not be run on a Hyper-V Guest.  You cannot use both SCR and/or CCR and Hyper-V clustering at the same time. Finally, the virtual machine itself must meet all the hardware requirements (processors, RAM, etc) for Exchange 2007 Sp1. Link to TechNET

Exchange 2010:

Supported for production environments on Hyper-V, with some restrictions:  You must be running Server 2008 SP2 or R2 as the Guest OS, and Unified Communications Roles are not supported at all.  As with 2007, the virtual machine you install Exchange into must meet all the hardware requirements for Exchange 2010. You will also need to choose between DAG availability or Hyper-V failover solutions, you cannot run both at once. Link to TechNET (see section on Hardware Virtualization)

Common notes:

In all supported versions, certain functions of the virtualization systems are not supported with Exchange Server.  Specifically, dynamically expanding virtual disks are not allowed, you have to use pre-configured fixed-sized VHD’s for Exchange Server virtual machines. Likewise, you cannot leverage differencing or VSS snapshot disks if you’re running Exchange Server.

There are several rules about virtual hardware that you must follow as well, so read the related TechNET articles for guidance on things like disk types (SCSI, IDE, etc) and the like.

Hyper-V technology is bringing new opportunities to the virtualization technologies platform in the modern datacenter.  By clarifying what is and is not supported in a virtual environment, Microsoft has begun the process of allowing customers to safely start using virtualization for the Exchange Server platform.  Many users will still choose physical hardware for Exchange (either because they cannot meet one or more of the requirements, or just due to having hardware already provisioned), but these rules help clarify that options for virtualization exist and are supported.

A word on Cluster Groups

Many clients are on Exchange 2003 or 2007 and will need to deal with Cluster Groups in Microsoft Cluster Services (MSCS) or Failover Clustering Services (FCS) including Cluster Continuous Replication (CCR).  So, it is important to understand one very critical restriction of Exchange Clustering that I’ve seen several clients trip over.

When installing a Microsoft Cluster of most flavors, you will configure Groups, which are logical units used to contain Resources like IP Addresses, Network Names, Disks and Services.  By default, a Cluster Group will be created that contains the name, IP address and Quorum Disk for the cluster itself.  It may also contain a networked Distributed Transaction Coordinator (DTC) resource for the cluster as a whole.  It is very tempting to place all other resources in this group, but you should avoid doing that at all costs for 2 significant reasons:

1 – It’s not supported by Microsoft.  For proof, I refer you to This TechNET article.  There is a long explanation of many thing having to do with configuring an Exchange Cluster, but here’s the specific info I’m referring to:

“It is an Exchange best practice to install the MSDTC resource into the default cluster group. However, the MSDTC resource is the only resource supported in the default cluster group. Exchange resources should not be added to the default cluster group, as that configuration is not supported.” [emphasis added]

TechNET and the Microsoft Sites have many other examples of this warning, and it is well documented by Microsoft and the Exchange Product Team.

2 – It makes life more difficult in day-to-day administration.  There may be instances where you want to perform operations on the Cluster Group without interrupting Exchange services for your organization.  You can normally accomplish this by moving the Cluster Group to a Cluster Node that isn’t hosting any Exchange Resource Groups, and perform your activities on that node.  If you have Exchange Resources in the Cluster Group, then this options disappears.  The same goes for many 3rd-Party products (see disclaimer at the end of the blog) which may not accept Exchange Resources that appear in the Cluster Group, as they must treat the Cluster Group and the Exchange Resource Group independently for administrative purposes.

So, as tempting as it is, avoid installing Exchange Resources into the Cluster Group at all costs.  If you already have put Exchange Resources into the Cluster Group, and you don’t plan on upgrading just yet, then seriously consider migrating to a supported cluster configuration when time permits.  Issues that arise from unsupported configuration and limited administration tend to hit without warning, and at the worst possible time.  Taking time to move to a supported platform will keep your organization in the safe zone, and make life a lot easier for you over time.

When it’s time to say goodbye to 2003…

Let’s face facts, 2010 is out, 2007 has been out, and that means that 2003 is slowly but surely fading into obscurity.  Make no mistake, we’ll see lots of 2003 in the real world for some time to come, but many of my clients are starting their migrations to the newer Exchange Server platforms.  There’s a lot of good reasons to migrate, and a lot of great native and 3rd-Party tools to help, but when you are at the point where you want to turn off the Exchange 2003 servers for good, don’t forget you need to do a few things first.

Exchange 2003 servers may seem to be alike, but the first server you installed in your environment holds special significance in the Exchange Organization.  So as you migrate users off and into the new systems, make sure you don’t simply shut down the first Exchange 2003 server and walk away.  Some of the things you need to do to that first server are critical!

If you use Public Folders, you need to replicate any folders that reside on the first Exchange server of the site. Chances are that you have already done this for your more common folders, but have a long, hard look at the PF tree and make sure that there are no folders homed only on that Exchange 2003 box.

The Exchange 2003 Offline Address Book will be homed on that first server as well.  If you are upgrading, make sure to create multiple versions of the OAB on one or more of the new servers (OAB3 OAB4, etc).  Make sure your users are then getting their OAB info from the new servers instead of the first 2003 server in the site.

Ensure that another server holds all necessary Master Roles for Exchange.  Many of these roles aren’t required for newer versions of Exchange, but it pays to do the homework to make sure.

If you use custom connectors that must continue to work in the new versions of Exchange; make sure you get them set up on your Hub/Transport and Edge servers to ensure communication continues.  SMTP is native, but others are not.

Once you step through these items (and possibly a few others like Schedule+ info which needs to be migrated, not simply moved), you can safely remove that first Exchange 2003 server in your site.  If you need to figure out which one was the first, or if you need more details, check out the KB article from Microsoft on how to go about doing this:

http://support.microsoft.com/kb/822931

Migration is a fact of life that most of us will face in the next year or two.  Don’t get caught short because you missed one or more critical steps for that one last Exchange 2003 box.

Look before you leap

There’s a great deal of talk these days about migrating directly to Exchange 2010 instead of jumping first to the 2007 version of the platform.  The arguments for and against this got more complicated when Microsoft did a reversal for support and announced that 2007 would be able to run (at some future point) on Server 2008 R2.  This effectively extended the support cycle of 2007, which means that a slightly older, and more thoroughly tested and widely installed, version is a legitimate option going forward.  That being said, many clients are opting to just go to Exchange 2010 from whatever they’re on now, skipping 2007 if it isn’t already in place.

I’m going to avoid discussing the pros and cons of going to Exchange 2010 directly in this particular post (that’s fodder for quite a few additional posts though).  Let’s just say that you’re jumping from an earlier version directly to 2010 and go on from there.  While most of the process isn’t too bad overall, there are a couple of sticking points that you’ll need to have sorted out before you make the move.  In smaller organizations, these are pretty easy issues to take care of, but on large-scale roll-outs, they can be a problem.

First, make sure your domains are at least at Server 2003 native functional domain levels.  Most of you are already there, but just in case you had a few NT servers hanging around, be ready for this one.  As long as all your servers are 2003 and up, making this change doesn’t have a lot of impact on your domain.  You can find out about the impact of raising Domain Functional Levels in this TechNET article.  So, what if you DO still have Exchange 2000/Windows 2000 servers in your environment?  You need to upgrade to Exchange 2007 as an interim step to 2010.  Since the Exchange 2000 life-cycle is definitely over (and has been for a while), Microsoft offers no method of co-existence between Exchange 2000 and Exchange 2010 – migration or otherwise.

Once your domain is prepped to 2003 or 2008 native mode, you then must plan to keep your old Exchange infrastructure long enough for the migration to occur.  That’s because the Mailbox systems for one version of Exchange can’t talk to any other in most cases.  So a 2003 Mailbox Server can’t talk directly to a 2010 Mailbox Server without some assistance.That assistance is in the form of Bridgehead servers (2003) or Hub/Transport and Client Access Servers (2007). You will need to have at least one of each for each Site you have set up in Exchange/AD.  One note, this has nothing to do with the migration itself, this is all about the period of co-existence you’ll need to go through until everyone is on the 2010 architecture.

After you have upgraded your Domain Functional Level and set up your front-end server systems, you can begin the process of installing the first of your Exchange 2010 server systems.  Start with the Hub/Transport and Client Access roles (unless you are installing them on the Mailbox Role server itself) to allow for message routing to get set up.  This lets mail flow between the 2010 and legacy front-end servers, so when you start moving mailboxes, you don’t lose mail connectivity.

Finally, once you’re sure mail is flowing correctly and that no one is about to get lost in the shuffle, take one more backup of the legacy systems (just to be safe) and start your migration.

You can find a guide to preparing for and deploying Exchange 2010 at this TechNET site.  Along with other blogs and Microsoft sites, this should be required reading before you even download the software and play with it in a lab.  With a migration that includes a period of co-existence with earlier versions of Exchange; careful, proper planning is not a nice-to-have thing.  It is a solid requirement that must not be overlooked or under-appreciated.

Time Keeps on Slippin…Slippin…Slippin into the…past?

Once again, I grabbed the title of the post from a play on words on lyrics from “Fly Like an Eagle” by the Steve Miller Band.  For those of you who have no idea what I’m talking about, see this YouTube video, and know that you have made me feel very, very old.

The point to the title was that some of you may have noticed that calendar appointments, email time stamps and many other things seem to be off by 1 hour starting yesterday.  The reason is that unless you have all your updates and hot fixes from Microsoft, Windows 2000 and 2003 would believe that Daylight Savings Time changes occurred on Sunday, October 25th at 2am, and changed the clocks on any non-updated servers.  If that’s an Exchange Server, the incorrect change will flow over into all Exchange functions as well, causing quite a few problems.

Normally, we here in the US change our clocks twice per year.  The latter one used to happen on the last Sunday of October, when we all set the clocks back by one hour at 2am on that day.  The problem is that the US Government changed the rules late last year, changing the dates that these one-hour shifts take place on.  This year, it will be November 1st, but Windows wasn’t originally programmed to deal with that.

Most of us update Windows and Exchange regularly, so we got all the appropriate patches and ran the required updaters on the systems in question.  You’ll know you did it right if the clock did not change Sunday, and do change on November 1 at 2am.  You know you missed at least one if either your server time, or your email time stamps are all an hour off today.

Yet another reason to patch regularly, but everyone can miss one now and then, so visit this Microsoft Support site on DST changes if things are acting odd time-wise.  If thing are acting odd in other ways, throw me an email, I might do a column on it =)

Why use AD DNS?

My recent article on the need for (and use of) PTR Records in DNS have sparked quite a few questions on using DNS with Exchange Server in general.  The biggest one I get is “Do I need to use Active Directory DNS in order for Exchange Server to work?”  The answer to that one is a bit complicated, but in its simplest form, it boils down to, “No, but you really, truly should.”

Exchange Server 2000 and up required some form of DNS in order to function correctly.  This is mainly because the Windows Internet Naming System (WINS) was “depreciated” starting with that version of Exchange.  What that means is that MSFT officially asked the community to stop using it whenever possible, because it could be removed completely soon.  As it turns out, WINS was phased out in Exchange 2007, though it may still be required for certain Outlook functions.  That’s a topic for a whole different series of blog posts though.

As for DNS integration, it’s quite possible to install Exchange 2000-2007 without having Active Directory DNS configured in your domain, though it isn’t a best practice.  As long as your DNS system can handle Server Name Records (SRV type records), you can successfully use a 3rd-party DNS for your Exchange environment.  There are, however; some good reasons to go with the native Windows Active Directory Integrated DNS solutions:

1 – Exchange can natively talk to Active Directory DNS, and therefore can do some interesting tricks with that DNS platform that it can’t do with 3rd-party DNS.  Things like AutoDiscovery when you move a user to different mailbox servers, or after a recovery operation with Database Portability just don’t work the same way if you’re not using Active Directory DNS.

2 – Many 3rd-Party tools leverage AD DNS to figure out where Exchange resources are.  Note, I’m far from unbiased on this topic, so please see the disclaimer at the end of the blog.  Since many Windows-based tools will natively use AD DNS API calls (like DNSCMD and the newer variants in PowerShell), you may need to make manual updates to your 3rd-Party DNS, or may have to give up functionality.

3 – Many other non-mailbox objects are stored in AD DNS, and must be mapped manually in other DNS systems in order for Exchange to work properly.  You will have to track your Global Catalog servers, Domain Controllers and other resources in order for Exchange to function.

So, as you can see, there are some very good reasons to use Active Directory DNS if you plan on using Exchange Server.  While you may have external DNS records hosted with an ISP or other provider; internally you will be better off with the native DNS solutions in Windows unless you are ready and willing to fine tune your DNS systems and stay on top of it. 

If you are in doubt, you can use the Exchange Best Practices Analyzer to test your environment before you begin to install Exchange.  This tool will test for many things that Exchange needs, including properly configured AD or 3rd-Party DNS systems.

What is a PTR record and why should I care?

True story, I heard this exact question from a client not too long ago.  Through my trials and travails as an Exchange Engineer, DNS information is one of the very most confusing aspects of email systems that my clients have to deal with.  Just figuring out how to use normal DNS records tends to lead to a strong desire to give up on the whole project, so attempting to discuss reverse records can cause many to throw up their hands and run screaming from the server room.

Alright, that only happened once, but wow was it fun to watch.

PTR, or reverse lookup, records are used to allow external servers and systems a way to find out what identity a server has based on its IP address.  So, for example, we can look at the IP address information for Bing.com, Microsoft’s replacement for Live Search.

If I do an nslookup on bing.com, I get the following:

Name:    bing.com
Address:  64.4.8.147

So it appears that 64.4.8.147 is the IP address for that URL.  Now if I put the IP address into nslookup, my DNS server will attempt to backtrack to see what that server is identified as:

Name:    origin.bay.ux.search.live.com
Address:  64.4.8.147

Not a perfect return, as I was looking for it to reply that the IP was assigned to a bing.com address, but knowing that Bing Search replaced Live Search, the results are clearly traceable.

You can read up on what PTR records do, and how to create them at this Wikipedia page.

Now that you have some idea of what PTR records are used for, we can discuss why you will want to make sure that all mail domains you have authority over are properly configured to use them.

These days, you can do a quick Bing (or Google) search and find dozens of software packages designed to allow you to send out email as if you were someone else.  Sometimes there’s a legitimate reason to do this, such as if you manage email lists for multiple organizations through one mail domain.  In many cases, these tools are used toward nefarious ends, allowing a hacker to send email that appears to be from a domain in order to scam or infect the recipient.  Famous examples of this are phishing emails (see this Wikipedia page) where a scammer will send an email that appears to be from – say – a bank or other institution.  Your end user receives the mail, and since it appears to be from a trusted source, they’ll click the link and possibly enter in private information, leaving your organization open to attack.

To combat this, many SMTP servers either natively support, or can be configured to support, PTR lookups before accepting email.  This way, the header information can be examined to ensure that the email isn’t coming from a suspect source.  The end users don’t see much difference, but email that is from unknown domains, or domains known to be fraudulent, can be rejected before ever getting to their mailboxes. Exchange 2003 and 2007 do not natively reject email based on a bad reverse DNS lookup, so left to their own devices; you don’t have to worry about blocking incoming mail by accident.  This doesn’t mean you can ignore PTR records though.  Since many other mail server systems can be configured to reject non-verifiable mail, and since there are a host of 3rd-party systems that work with Exchange Server that can do the same, failure to properly configure a PTR record can cause your outgoing email to get bounced because you cannot verify your identity.  This means that you need to set up a PTR record for your Mail eXchanger (MX) records and your domain in general, or you risk having email returned as non-deliverable.  Having no PTR record is just as bad as an incorrect configuration here, as a lack of a reverse DNS lookup will cause the same results as an incorrect reverse DNS lookup.

There is a great debate over if this is a good or bad thing.  PTR responses could be forged, and so this is not a foolproof method of confirming identity.  Also, if multiple organizations all use the same SMTP server (thing about that multi-list email server from before) then which one gets the PTR record assigned to it?  If you have an email domain, then for now it is a good idea to ensure that PTR records are properly configured so you don’t run into the problem, but hopefully there will be better, more secure systems to rely on in future to confirm identity (such as the SMTP-AUTH proposed specification as part of E-SMTP).

To avoid blacklists, blocks and email black-holes, configure correct PTR records for your SMTP servers and domains.  It is by no means a perfect system, but it is one that you will run into, so an ounce of prevention is definitely worth a pound of cure here.

Storage Groups, or, Keep Them Moving and Spread Them Out.

In a recent conversation with a colleague (James from VA in this case), he mentioned that there were a lot of folks who simply jammed all their users into one Storage Group in Exchange, even when they had the option to use multiple groups, and really should have.

Storage Groups are databases in Exchange Server.  They contain Information Stores and Log Files, and logically group together users and Public Folders that are related in some way.  There are a lot of different ways that you can choose to use allocate users to Storage Groups, but here are some of the more common methods:

1) Geography – Many organizations are centralizing Exchange servers to one or two datacenters, instead of putting a server at each location.  This saves money overall, allows for better message hygiene and, with the use of technologies like Outlook Anywhere, doesn’t alter the end-user experience to a any large degree.  Creating a Storage Group for each physical location allows you to better administer the Exchange system as a whole. If a new policy or procedure impacts only one office, you won’t be stuck combing through all the mailboxes and folders trying to find the users who are impacted by that policy. Just apply it to the Storage Group for that office and move on.

2) Department/Employment Level – This is a very popular method for splitting up users into Storage Groups, especially where there are a relatively small number of locations or other physical defining factors.  Many companies will define their Storage Groups by the departments that exist within the firm, mirroring their political structure into the messaging systems.  Sales, Support, Administration, Management and other logical groups can be translated into their own Storage Groups.  This has the dual benefit of not only allowing you to better control policies and procedures for each group, but also making it easy to put all managers into their own server which gets clustered, while the rest of the employees are on servers that will be restored from tape.  That’s just one example of the benefits of this segregation system for Storage Groups from the real world, so please don’t send hate mail. Of course, you can use a tool like Double-Take to give you even more options, but see the disclaimer at the bottom of the blog, as I’m far from unbiased on that topic.

4) Alphabetical – Where there may be only one or two locations for the business, or where everyone’s email is equal, then using something as simple as the alphabet can help figure out where to split out your Storage Groups.  A-M and N-Z or even 3 or 4 different breakouts can allow you to allocate your users across multiple Storage Groups without worrying about which users should go in which groups by other factors.

The reasons for splitting up Storage Groups go well beyond simple administrative tasks like policy and procedure enforcement. Storage groups can take an unmanageable situation, like 3000 users all in one gigantic glom, and make life easier to deal with by allowing you to manage smaller groups as required.  Database maintenance on a 500 user, 50GB Storage Group is a lot easier than database maintenance on a 3000 user, 1.2TB set of Stores.  It also means that the other 2500 users aren’t offline while you do that maintenance on the one Storage Group in question.

Physical limitations of Exchange could also force you to break things up into groups.  Granted, the built-in limit of 16TB per Storage Group (in Exchange 2007) is one most folks won’t ever hit without already breaking things out into multiple Storage Groups, but there are other size limitations.  Storage Groups must have their database files (.edb in 2003 and 2007, .stm in 2003 only) exist on a single logical volume.  This means that if you have 1TB of disk space, your Storage Group can’t grow past that point.  Multiple Storage Groups allows you to allocate your user data across multiple volumes, allowing easier growth and better storage management overall. This also leads to more spindles and read/write heads being used at once, which increases overall performance.

Even the Standard version of Exchange 2007 allows for up to 5 Storage Groups, so even smaller organizations can now split their users out across multiple Groups.  Exchange 2003 and earlier only supported one Storage Group, so if you’re on those versions, you will have to keep everyone together.  Otherwise, live by the simple mantra that more is better when it comes to the allocation of users to Storage Groups.

Getting Smart (Smarthosts that is)

No matter what version of Exchange Server you’re using – or even if you don’t use Exchange at all – you need to perform message hygiene on all mail in your organization.  Virus files, spam, phishing schemes and tons of other attacks are thrown at every email domain, every day.  If you’re not ready to deal with them, you’re dead in the water.  Smaller companies often just make due with hygiene software on the Exchange Server itself, but all sized firms need solution sets, and even smaller firms can take advantage of Smarthosts in a few different ways.

Smarthosts are servers or hosted solutions that are dedicated to performing message hygiene on all mail coming and going from your organization.  They come in many flavors, but tend to center around three types:

1 – Fully hosted solutions.  This method has no Smarthost hardware on-site, but instead contains everything at a hosting provider’s location.  Lowest up front cost, but highest ongoing (monthly, quarterly, etc.) costs.

2 – Hybrid solutions.  Here, you host a small appliance on-site to handle some of the workload, with the rest of the message hygiene functions handled by the service provider in their facility.  This isn’t quite as popular as the other two solutions, as the costs can quickly outweigh the benefits.

3 – Fully owned solutions.  Appliances and servers are in your datacenter, and totally controlled by you.  Start up costs are high, but ongoing costs are very low, as you’re only paying for updates to the virus/spam filter engines.

For smaller organizations, hosted solutions allow you to assign your Mail eXchanger (MX) record  to point to your Smarthost partner’s datacenter.  There, all incoming mail will go through the filters and checkers before it ever reaches your internal server set.  Most hosted providers allow you to update a list of “known good” recipients, so that any mail not destined for a valid employee is rejected immediately.  Virus scans are performed, and optionally spam filtration happens as well.  The remaining mail goes off to your internal mail servers for delivery to the end-users.

When mail leaves your organization, your internal servers are configured to send it first to the Smarthost provider, instead of directly to its destination SMTP servers.  The service provider performs all the same message hygiene on those messages, then acts as the SMTP gateway for your domain out to the rest of the world.

Hybrid solutions do everything that the fully hosted solutions do, but typically place a small appliance on-site to do periodic scans of the mail server itself.  They can also help speed up the whole process by making the “first hop” for email flow out of the organization be local instead of across the WAN.  It’s rare to see a hybrid Smarthost system that only does email hygiene, as the fully hosted or fully owned methods are much better for this kind of single-task solution.  Instead, the appliances on-site will typically handle all anti-malware processes including updating desktop protection and server scanning.

Fully owned solutions place appliances and/or other equipment at your production facility.  They act the same as the hosted solution, but you own all of the hardware and pay only for periodic updates to virus definitions, malware profiles and spam allow/deny lists.  This solution offers much more flexibility than a hosted solution, as you have complete and immediate control over any changes that might need to be put into effect.  The drawback is that if the hardware needs to be upgraded, you’ll be responsible for buying the new boxes.

Smarthosts are available from a large number of vendors.  Postini, Barracuda, and Microsoft Exchange Edge Services are just three that can be found with a Bing Search.  Most perform the same tasks, but each has their own selling points.  Your best bet is to talk with multiple vendors and see which one has the solution set closest to what you need.  Ask about things like ease of administration, web or application interfaces, ability to create custom allow/deny lists, and integration with your own Domain services to get updates to users and allowed accounts without compromising security on your AD servers.

No matter which vendor you choose, or which solution, using a Smarthost removes the message hygiene load off the Exchange Servers, which is always a good thing to do.  By dedicating either a service or server to handling message filtering and security, you can free up resources to provide a better end-user experience and a safer messaging environment.

Exchange 2003 only wins with WINS.

Since Microsoft released the Release Candidate for Exchange 2010, I figured that a look back at some things in Exchange 2003 are well in order.  After all, a lot of folks are currently on that platform, and waiting for 2010 to RTM before they upgrade.

So today, let’s take a look one of the components of Windows that Exchange 2003 needs to have in place, but that causes confusion and doubt every time it comes up in conversation.

The Windows Internet Naming Service (WINS) is a component of Windows Server NT4, 2000 and 2003 that – as its name implies – translates NetBIOS names into internet names and addresses.  And to be absolutely clear on the subject, Microsoft *requires* the installation of WINS for full functionality of Exchange 2003.  Here’s the proof.

The reasons for this insistence on having WINS present in Exchange 2003 are varied, but the controversy over the requirement has raged ever since Exchange 2003 was released.  The confusion stems from the apparent dichotomy between the fact that 2003 is supposed to be totally DNS integrated, and the view that NetBIOS dependencies make it look as though it is not.  Another reason for confusion is that even though Microsoft says that the installer will not run properly without WINS in place, many clients have managed to install Exchange 2003 without the WINS services running at all, or worse yet; with the WINS system improperly configured and malfunctioning.

Exchange 2003 leverages Active Directory (AD) DNS for nearly all name resolution functions, from finding other servers in the local environment to finding SMTP hosts for external mail destinations.  For internal resolution, Exchange can leverage AD and Global Catalog servers to find things, as long as everything in the domain in question is using Windows 2003 or higher, and configured appropriately.  MAPI systems in 2003 also can use DNS to find Exchange servers (Outlook 2003 and up) or to locate other resources.  So if you are in a pure Windows/Exchange/Outlook 2003 or higher environment, and you have only one Active Directory site, then you’re all set without WINS.

That last sentence is key to figuring out where the requirements for WINS come from, and where most of the confusion stems from as well.  Most environments that use Exchange 2003 still have mixed-mode domains, possibly even some Windows 2000 servers around.  They’ve also brought legacy systems up to Windows 2003 over the years, meaning that older names and objects may still be preserved in AD.  So if an Exchange 2003 server needs to find a resource that AD only has a “short name” (non-fully qualified domain name) for, the traditional method to find that resource would be WINS, not DNS.

This also comes into play if you have multiple AD sites across multiple physical locations.  Since short names don’t translate properly from site to site, or domain to domain in the same Forest, WINS is necessary to translate the resources name into a location for the resource itself. 

Finally, any legacy Outlook clients (XP and earlier) rely on WINS to find all resources, as the MAPI clients they contain don’t understand DNS lookups at all.  So if you have any older Outlook clients running around, you’ll need to ensure WINS is configured properly.  Similarly, ExMerge relies on WINS since it was a hold-over application from earlier Exchange versions, and therefore never designed to leverage DNS.

So, until your environment moves up to Exchange 2007 or 2010, and a Native 2003 or Native 2008 Domain architecture, you’re stuck with WINS.  The good news is that WINS is just another component of Windows in 2003, and therefore not a bear to implement at all.  Go to Add/Remove Programs and choose to install Windows Components.  You’ll need your OS CD’s to finish the install, and as always; be sure to Service Pack after you are done and run Windows Update.  There have been many changes and patches to WINS over the years, especially in light of recent attacks against it.

WINS is a legacy system that is – thankfully – going away as we move toward Exchange 2007 and 2010 native environments, and as legacy Outlook clients are phased out.  Until then, make sure you keep up with WINS patches and ensure it is installed and configured. Not doing so can cause resolution problems for Exchange 2003, and installing it is a requirement from Microsoft, so take the time now to make sure your WINS systems are operating full steam.

Confirmed, No Server 2008 R2 for you!

Sorry for the horrible delay in posting, my loyal readers!  I’ve been trying to track down some official sources on this one, and of course Jeff Guillet from the EXPTA Blog beat me too the punch.  However, he’s a good guy and a great author, so I don’t mind giving him the credit.

Much to the dismay of a large number of Exchange Admins and Engineers, MSFT will not be supporting the 2007 version of Exchange on Server 2008 R2.  So, if you’re planning on upgrading from earlier versions of Exchange Server, you have only two choices:

1 – Use Server 2008 RTM, and be prepared to stay on that version of the OS for an extended period of time.  This will let you use Exchange Server 2007 SP1 and take advantage of Distributed Failover clustering and other nifty Exchange 2007-specific tool sets that rely on the Server 2008 architecture.

2 – Alternately, you can wait on your upgrade until Exchange 2010 is released to market, and become an early adopter on Server 2008 *or* Server 2008 R2. Interestingly enough, Exchange 2010 will indeed by backwards-compatible to run on Server 2008 RTM, even though 2007 won’t make the leap forwards onto R2.

As many clients were in the process of either rolling out 2007 to replace other email platforms or were upgrading from Exchange 2003 and earlier, this is a dramatic policy announcement from Redmond.  Current plans to continue toward Exchange 2007 on the latest OS (Server 2008 R2) will have to be altered or scrapped.  If abandoned, new plans have to be set up for Exchange 2010.  I can only believe this will lead to a slow adoption rate as Admins and Engineers try to figure out the combination of Exchange and Windows that will a) function and b) allow them to upgrade now and keep going for as long as possible. Since Exchange 2010 will have a lifecycle that goes beyond the end-of-life date for Exchange 2007, many clients are making hard decisions between moving forward with the Exchange 2007 rollout, or re-planning to use the newer version.

In the end, there are a lot of great reasons to move up to Exchange 2010, so from a purely technological perspective, moving up is a no-brainer.  The problem is that many folks have been planning for over a year to implement Exchange 2007.  Planning that started when the first Service Pack was released for both Windows 2008 and Exchange 2007 – traditionally signaling the point where most upgrades begin in earnest.

I fear that this will cause an additional hurdle to migration, both from other platforms and from Exchange 2003 and earlier.  I can only hope that Microsoft has some trick up their sleeves when Exchange 2010 launches to make the jump easier or simply more compelling.

Big Brother is Watc…[Recalled]

The big kerfuffle over Amazon’s Kindle Team removing several books without warning from users’ devices (story and link at the end of the blog) has lead me to think about the various time clients have used the Recall Message feature. Typically, they have found that it behaved just slightly better than opening their office door and screaming “NOBODY READ THAT!”

There are a lot of great reasons to use Recall Message in Outlook and Exchange.  Perhaps something was sent before it was finished, accidentally clicking Send without typing in that last bit of information.  Maybe you meant to add an attachment but did not – in which case may I suggest some of the great tools at Sperry Software.  You may also have accidentally hit Reply All instead of Reply, or added in a group email list via AutoComplete instead of the one person you wanted to send email to.  All of these are legitimate reasons to really want to get the email out of everyone’s Inbox.

Years ago, Microsoft attempted to build the functionality of yanking back an errant email on command.  It required (and still requires) that you be using Outlook in Exchange Mode, and a Microsoft Exchange Server as your email platform.  When first introduced, this worked great.  There was no such thing as a preview pane, you were always using an Exchange Server and MAPI for communication, and folks were probably online and connected since there was no Cached Mode.

The issues arise when you look at what is required for a Recall attempt to succeed:

• The user must be online and connected to the Exchange Server
• They must not be in Cached or Offline Modes
• The message must be unread in the recipient’s Inbox (including unread by the Preview Pane)
• The recipient cannot have moved the message to another folder (unread or otherwise)
• They must be using MAPI. Even POP3 or IMAP on an Exchange Server won’t count.

So, as you can see, since the majority of email users these days do not meet one or many of these requirements, Recall Message will fail for the majority of your users.  What they will see is the original email (unchanged) and a notice saying you would like to recall the original email.  This can often be just as embarrassing as the original gaff, as it acts as a glaring exclamation point that someone made a goof.

One more note, Recall Message will not impact the copy of the information on a Blackberry or other 3rd-party Smartphones and mobile devices.  So even if the user’s desktop has all the requirements met, they’ll still have a copy of the message (and the recall request) sitting on their mobile device.

In the days of Exchange and Outlook with MAPI connections being the only conduit to email, Recall Message was a great way to manage errant information.  Today, with so many ways to connect to any mail system, including Exchange Server, this feature has lost its usefulness and created a false sense of security.  Remember, if there is a chance that someone’s using a non-Outlook client or not meeting all of the requirements above, you will not recall the mail.  An ounce of prevention is your only cure these days.

As for the Amazon gaff, for those who did not see the news:  Amazon recently pulled all the books by a major author and deleted them off all Kindle devices that held them.  This was due to someone posting the books on the Kindle store, without the publishing rights needed to earn money from them.  Once Amazon realized what was going on, they of course stopped selling all the books from that vendor.  Unfortunately they also removed all the digital copies that were stored on users’ Kindle devices without mentioning it to the Kindle owners.  They did refund everyone’s money, and they did send out an explanatory email, but not until several days later after the blogosphere detonated with Amazon hate postings.

Frankly, Amazon’s user agreement does say that they reserve the right to do exactly what they did.  I can’t fault them for heading off trouble (in the form of a copyright litigation) at the pass. However, it could have been handled a lot better, with more information supplied immediately to end users about what was going on and why.  Amazon has promised to make this process much more transparent so that next time there is no confusion.

The reason there was such an outcry about the process this time (as it has happened in the past, quietly and without fanfare or hate mail)?  The author in question was George Orwell, and the books were the likes of 1984 and Animal Farm.  Both of these books deal with subjects like retroactively removing information that didn’t meet the criteria of an overarching body of government or law.

From Dictionary.com:

Irony: an outcome of events contrary to what was, or might have been, expected.

Anyway, Anyhow, (Outlook) Anywhere

For some strange reason on my a classic rock kick these last few weeks, so today’s posting title comes from another song by The Who.

No clue?  Watch this YouTube video.

And listen for this lyric:

Nothing gets in my way / Not even locked doors / Don't follow the lines
That been laid before / I get along anyway I dare / Anyway, anyhow, anywhere

Outlook 2003 and 2007 offered a new approach to Exchange Server connectivity that did not exist in the wildest dreams of earlier versions of Exchange and Outlook.  These two versions of Outlook (and the upcoming 2010 version as well) have a feature set called Outlook Anywhere, previously known as RPC over HTTPS.

Since the dawn of distributed email systems, there has been a schism between the need for connectivity and the need for security.  Most servers can be isolated from the outside world in some way.  Many can be completely locked down and held within the corporate network.  Others can have limited web-based connectivity via tightly controlled front-end servers, allowing the sensitive back-end solutions to be protected inside the network.  The major issue for email was that the systems are inherently designed to communicate with the outside world.  In fact, it’s the whole point of the exercise for the majority of Exchange users.

Microsoft, to their credit, has created a series of systems that allow Exchange 2007 server to only expose the bare necessities to the outside world with the new Edge Server Role on that platform.  But not every company can afford that type of infrastructure, and it wasn’t available before the Exchange 2007 Server platform.  So most companies these days cannot take advantage of that role, and even when they do, the limitations of the Edge Server may not make it the best chose for remote users.

Before Exchange 2003, the alternative was to limit connectivity between the Exchange system and the world at large by using port control on the server side, and VPN connectivity on the user side.  This allowed for Exchange to see only what it needed to on the net, but also required administrators to set up and manage VPN software on all their mobile client laptops, home-office desktops, etc.  It was definitely a trade-off if all the users really needed to use the VPN for was Outlook.

The came RPC over HTTPS.  This technology lets MAPI RPC calls (the communications protocol Outlook uses natively until the 2010 version) to go over the public network in a secure manner.  Essentially, the Outlook 2003 or 2007 client will use HTTPS security – much like you would in a web browser to communicate with a bank or other secure website – but in this case it is Outlook talking to Exchange.  It is secure, reliable, and requires no other software on either server or client side.  It does require enterprise CAL’s in Exchange 2007, so keep that in mind, but it doesn’t need to have anything outside of Exchange Server 2003 or 2007 installed on the servers themselves.

With the advent of Exchange 2007, RPC over HTTPS became a more accepted part of Exchange and gained a formal name – Outlook Anywhere.  The system is simple. Outlook is configured to make a secure HTTPS connection to the Exchange Server or (in 2007) the Client Access Server/Hub Transport Server.  This lets the administrator strictly control access both via security certificates and limiting of what ports have to be exposed, but let end users function with just Outlook and no 3rd party or other VPN client.

To configure, you alter your Outlook Profile, under Account Settings|Microsoft Exchange account|Advanced Settings in Outlook 2007.  Once there, open the Connection tab and click the check box near the bottom to enable HTTP connections.  You will also need to click the button under the checkbox and supply the connection URL for your server. Typically, this is your Outlook Web Access URL, though you can configure a specific record if you prefer.

There are several settings that can help improve network function over slower links, but the defaults should be fine for most networks.  Note that you MUST have the ability to connect to webmail via SSL (in other words https:\\mail.yourdomain.com and not just http:\\mail.yourdomain.com).  Since the data needs to be encrypted, you have to have security set up beforehand.  Though, you can use a self-signed certificate, that will cause your end-users to have to click through a security warning when they log in, so obtaining a public SSL certificate is a good idea.

Once configured, Outlook Anywhere will allow normal Outlook functions to occur as if the users were on your corporate network, but with all information between them and your system secured and limited to avoid potential attack.  It is a great solution where setting up a VPN communication tunnel for each remote user just isn’t feasible or where the only reason to do so is for Outlook communication.

Microsoft has made a lot of good progress in helping secure an application platform that – by its nature – must communicate with the outside world in order to function properly.  Outlook Anywhere is one way that this type of communication can occur, and is a great way to allow your users freedom to get their work down without requiring specialized network technology to be put into play.  Anyhow, anyway, anywhere – a thought never more true for Exchange and Outlook.

Exchange gets tired.

I’m not talking about getting old here, but rather about exhaustion, which can happen in even newer versions of Exchange Server. Checkpoint exhaustion is a function of the ESE database system set used in Exchange, and designed to stop the system from corrupting itself if too many database transactions get held up in a log file by anything.

The normal sequence of events is well known for most relational databases.  Information added to a database is sent to a so-called Lazy Page Writer to minimize disk access by aggregating I/O operations.  When this process happens, a write-ahead logging system tracks the operations, so that if they need to be replayed, they can be safely re-generated if the Lazy Writer data is lost.  As the transactions are committed to the database itself, the log systems note this by checkpointing those transactions, essentially noting that they’ve been safely committed to the database.  For most versions of Exchange Server this happens every 20MB or so (that’s 20MB of log data, not database writes).

When things do not go according to schedule, the logging systems can recover data that was in the Lazy Writer but not yet committed to the database with emergency systems.  Some, like the normal startup process for an Exchange database, are done automatically.  Others, like ESEUtil rescue operations are done manually, and hopefully avoided entirely.  But, the point at which the logs logjam up so much that the system forces emergency procedures can be more difficult to explain.

The short answer is that Exchange can log up to 1,008 transactions before Checkpoint Exhaustion hits.  This is the point where Exchange can no longer just keep writing to the logs and still know that those transactions can be safely recovered even if the Lazy Writer doesn’t have the transactional data anymore.  If the system sees that the logs are tracking more than 1,008 transactions that have not yet been checkpointed, it will forcibly dismount that database – in this case Storage Group and related Stores.

Now, when does Checkpoint Exhaustion happen?  That’s a bit trickier to explain, but there are a few definite times you might see it.

1 – The system is horrifically under-powered in terms of how fast it can get data out of the Lazy Writer and onto disk.  This is rare, but can happen if there is a failing RAID controller or other disk-device.

2 – Something has placed a lock on log files or databases.  In this case, the offending application is stopping Exchange from writing to either a log or database, or at the very least stopping the checkpointing process from happening regularly. Since this means that Exchange cannot successfully checkpoint the database, logs continue to build and build without being “caught up.” This is also not common, but can occur with management tools that go awry.

3 – Backup systems don’t work as expected.  With most Exchange-aware backup tools, the backup system prevents Exchange from checkpointing during the backup operation.  Normally, this is a good thing.  Checkpoint prevention means that no new data will be officially committed while the backup is in the process of…well…backing up.  This means that a point-in-time tape or disk backup can be transactionally intact, even if it cannot maintain strict write integrity native to the backup solution.  Since the Lazy Writer and the log files both have copies of the information, as long as the block is lifted at the end of the backup, all is safe and all is well.

However, if the backup barfs and never lifts the lock, you could have a major issue.  Since no new checkpoints can be set, unchecked log data starts piling up, and eventually you’ll hit the 1,008 transaction limit and see a forced-dismount happen on that Storage Group.  Luckily, restarting the Exchange services nearly always clears this problem, and since the logs still have a copy of all non-checked data, you’re not going to lose anything but time.

Checkpoint Exhaustion was built to solve a particularly thorny problem.  The earlier versions of Exchange Server that are still supported had a hard-coded limit to 1,024 transactions before the database would fault (not just go offline).  That meant that if the Checkpoint Depth hit that level, you could lose data.  So the exhaustion systems allow the database to safely shut down without losing track of any data – which is a not-so-great but definitely better than data loss type of solution set.

So, to avoid exhaustion, there are several things you can do.

1 – Make sure you’re not overloading your Exchange Servers.  Stay within best-practice guidelines for Microsoft Exchange versions, which can be found on TechNET. and be sure you periodically check for hardware issues.

2 – Try to minimize the use of applications that could lock the Exchange Server file sets.  3rd-party non-Exchange indexing for example

3 – Make sure your backup runs are completing properly. One bad run that never got caught can snowball into an exhaustion situation easily.  If you do have a bad backup run, declare a maintenance window of 20-30 minutes and restart the Exchange services, or even gracefully bounce the box.

 

Don’t forget, if you have an Exchange related topic you’d like to ask about, drop an email to miketalonnyc@gmail.com .  Next update should be on Monday or Tuesday of next week, back to my usual schedule. 

A special hello to Larry Meister at Mercury Networks in Rochester, NY, thanks for inviting me to speak at the White Hat Security Day conference.  It was a great event, as usual!  If you’re looking for a partner for your IT needs in Western NY State, Larry is your guy.

Outlook, can you hear me? Can you feel me near you?

Might be showing my age and/or taste in music with that particular title (and if you’re totally confused by it, check out This YouTube video), but I think that it’s a great way to describe an annoyance that can happen if you’re using versions of Outlook before Outlook 2007.  Since a large portion of the users of Office are on the 2003 version (and many even earlier than that), resolution to a new server in the event of a disaster recovery event is a subject that is just as confusing as the famous rock opera I’m making use of in my title today.

When Outlook 2007 was introduced to the world with Exchange 2007, a lot was made (and rightfully so) of the new AutoDiscover features that this platform brought into the Enterprise Email marketplace.  The long and short of the AutoDiscover solution set is this:

When an Outlook 2007 client cannot find its home server – either because it is a brand new install of Outlook or because the home server has moved or been replaced – the Exchange 2007 AutoDiscover system can help Outlook 2007 find its home.  If the Outlook client can see an Exchange Server (or be directed to one by Active Directory), the Server can tell Outlook where the mailbox information for the user’s profile exists, and direct Outlook to connect to the appropriate CAS or Mailbox systems and get connected.  All the user/Admin has to do is tell Outlook the user’s email address and password, and AD with Exchange 2007 will handle it from there.  So if you’re installing Outlook for the first time, you don’t have to manually configure the Profile anymore – a great boon to Admins everywhere.

This system also kicks in if you perform Database Portability during a disaster, and have replicated the database with SCR; or have used a 3rd party disaster recovery/availability solution (see disclaimer below for all my bias information on that one =).  Once the Exchange system is responding again, AD can ferry the Outlook 2007 client to the new home for that mailbox, requiring only that the end-user close and reopen Outlook to complete the process.

However, what many folks do not realize right off the bat is that this solution set is ONLY available if you have both Outlook 2007 and Exchange 2007 as your messaging platform.  All users who need to take advantage of AutoDiscover must be using that combination of tools, and no other.  As you might expect, POP3 and IMAP systems do not AutoDiscover, but the majority of my clients were unaware that Outlook 2003 and earlier also cannot take advantage of this system, even if you have upgraded to Exchange 2007 as the messaging platform of choice.  It’s also worth noting that AutoDiscover doesn’t officially work in Exchange 2003 – no matter what Outlook version you are on.  Before I get blasted by mail on this one, I know some folks have sometimes seen it to work on Outlook 2003 with Exchange 2007, but it bombs more than it works, and officially it’s not supported.  For proof, I direct you to this article by the MS Exchange Team.

Since the code to perform AutoDiscover wasn’t in Outlook 2003, users on that client software will not be able to dynamically re-link to the new Exchange server unless the original mailbox server is still responding.  If it is, then Outlook can find the new server via the original server and re-home itself.  If not, Outlook must be manually re-directed to the new server.

Of course, there are ways around this.  You could update DNS to re-direct anyone calling for “Server 1” to the IP Address of “Server 2” – effectively re-routing all client software including POP and IMAP.  Outlook 2003 will still need to be re-profiled unless you take over the Service Principle Name (SPN) of “Server 1” on “Server 2,” but it will be a smoother transition.  Using a 3rd party tool (see disclaimer below) you may have the option of automated DNS and SPN updates, which will allow even legacy Outlook clients to jump to the new server with no more intervention than is required on Outlook 2007 with Exchange 2007 – even if you’re whole system in on the 2003 versions of those software platforms or earlier.

So, you are not without lots of options if you have any legacy servers and/or clients – or non MAPI clients – in your environment. You just need to be aware that the Exchange 2007/Outlook 2007 solutions for AutoDiscover services are not backward compatible, and plan accordingly.  Right now it looks like Exchange 2010 will have AutoDiscover that is backward compatible to Outlook 2007 only, so this soon-to-be-released platform is not going to solve this particular problem unless you’re planning on upgrading everything else in the environment to at least Outlook 2007 first.

Since I like to let folks continue to discover on their own, here’s a link to the White Paper from MSFT on AutoDiscover.

Finally, Update Roll-Up 8 for Exchange 2007 is out there, which can make life easier if you’re doing a fresh install of Exchange 2007 and want to get up to date with patches and fixes post SP1 quickly.  You can get it at this link.

Public airing of Exchange laundry...

Last week we finished up talking about what happens to Private Mail Stores during Online Maintenance (OM) on your Exchange servers.  While most people really tend to focus on what happens in those Private Stores the Public Stores are just as important, especially if you're not on Exchange 2007 yet.

For all versions of Exchange, Public Folders experience most of the same processes that their Private Mailbox cousins do, with the Exchange system scavenging items that are marked for deletion and have passed deleted item retention to start.  The system will also get rid of expired folders - a similar process to deleted mailbox systems in the Private Mailbox Stores - and perform an online defragmentation of the databases.  General maintenance and cleanup for Public Folders should be considered to be the same priority as OM on your Private Mailbox Stores, even if you're on Exchange 2007.

With Public Folders in Exchange 2000 and 2003, and even in 2007 in some circumstances, there is another vital process that goes on each day. It is not technically part of OM, but should be considered on par with the OM process.

As many people notice, the first time you boot up a newly-installed Exchange server, Outlook will complain about not being able to find "a required file" when you do a send/receive operation.  This is because - unless you're using a native Exchange 2007/Outlook2007 environment - Outlook cannot find the Offline Address Book (OAB) which doesn't get created in the Administrative Public Folders until around 5am the morning after the Exchange server goes online.  While not officially part of the OM process, the creation and subsequent updates of the OAB do happen overnight - 5am by default, just after the OM process is typically completed. 

OABgen (the process that - wait for it - generates the OAB) should be considered part of your OM systems. If you do not allow this to occur regularly, your OAB doesn't get updated and therefore your end users don't have the latest information - causing no end of frustration for you and your technical staff.  You could manually create/update the OAB using tools inside of Exchange, but the automated process is still your best bet for making sure these critical systems get updated regularly.

And one final note.  For those who believe they can get away with no Public Folders and/or PF maintenance because you are on Exchange 2007, be aware of one very important caveat.  All Outlook clients must be on Outlook 2007, and no Entourage or other MAPI clients can be used. That's because all MAPI systems before Outlook 2007 required the use of Public Folders for administrative information, like the OAB and other components.  In Outlook 2007, that's handled by Active Directory and Exchange itself, but if there is even one legacy client in your environment, you'll still need at least one PF tree, and therefore need to perform regular Online Maintenance on those Public Folder Stores.

Next week, back to regular blog topics - and speaking of which, if you have a topic you'd like me to cover, go ahead and drop me a line at MikeTalonNYC@yahoo.com

Mailbox Separation Anxiety

As we have been discussing over the last few weeks, Exchange 2000-2007 runs through quite a few operations during the Online Maintenance (OM) process.  We've talked about how databases get cleaned up, and how messages get deleted (temporarily and then permanently).  This week, let's look at Deleted Mailbox Retention (DMR) and what it can do for you.

DMR is the process that Exchange uses to remove mailboxes that have aged past their expiry date in the Private Database.  These expired mailboxes can be created by different operations within Exchange, either directly by the administrator of Exchange, or by operations in Active Directory that inadvertently cause a mailbox disconnect.  Disconnected mailboxes are simply mailboxes that not longer have an associated user or object in Active Directory. The simplest way to do this is to delete a user account in AD, which will disconnect the mailbox assigned to that user as part of the process.  However, this isn't the only operation that can cause mailboxes to become marked for deletion.  There is an odd side-effect to moving mailboxes that can happen if you jump between AD sites and don't finish fast enough (You can read about it here), and of course; if you get hit with AD corruption it could disconnect mailboxes as well.

Once a mailbox is disconnected, the first OM run that happens after the disconnect will mark the mailbox for expiry in a user-defined number of days. The default is 30 days, so if you run maintenance daily, the mailbox will stay in the database, but disconnected, for 30 days after it is disconnected from the mail-enabled object (like a user) in AD.  It also means you have to take when you do OM into account for your deleted item retention numbers.  For example, if you only do OM once per week, then you will need to add up to 7 days to your overall retention time for planning purposes.  Since the 30 day clock doesn't start ticking until the mailbox is marked during OM, you will extend the amount of time data remains in the database.  This is critical for regulatory compliance, as holding data for an extra week could put you over the line in terms of meeting your regulatory guidelines.

The other part of this OM process happens when Exchange scavenges the database for expired mailboxes.  If it finds one, and the mailbox hasn't been re-assigned to another AD object, Exchange will delete the data from the database permanently and create white-space in the database that gets cleaned up by OM to be reused by Exchange.

So, here again, it becomes vital that OM is permitted to run on a regular basis.  If not, deleted mailboxes will not be removed from the database unless you manually launch the cleanup wizards.  This is more than just a regulatory problem, as without removing the old information and freeing up white space, your databases will grow much more rapidly than if you are regularly cleaning up your mail stores.

Next week... Public Folder operations during Online Maintenance, so stay tuned!

Update now - no I mean it, NOW

Though I'm known for keeping a level head during these kinds of scenarios, there is an emerging vulnerability in both Exchange Server and Internet Explorer that you'll have to address immediately to keep your enterprise safe.  I wish I was kidding, but no, it's true.

The long and short of it is that specially crafted messages could be sent to an Exchange server that - even if only previewed by a user - can allow an attacker to take control of the Exchange server itself with full Exchange Admin privileges.  The details can be found at this link, but I'll summarize here.

If an attacker crafts a message in certain ways via TNEF (Transport Neutral Encapsulation Format), and that message is then either previewed or opened by an end-user (which is the point of email after all), then the attacker can gain elevated privileges on the Exchange server and wreak havoc. While that sounds like a major problem, there is good news.  The attack would require a sophisticated attacker creating just the right message and then finding a live mailbox to send it to.  Not only that, but it would need to get past the SPAM and CAPTCHA systems if you use them.  Finally, eventually the anti-virus vendors will jump in as well, scanning for messages crafted with the exploit.

The really bad news is that end-users do NOT have to open an attachment this time.  They can unwittingly infect the mail servers by previewing the message body or opening the message itself.  Since Outlook clients and now even Outlook Web Access both Auto-Preview by default, end-users can follow all best-practice safety protocols and still end up infecting the organization through no fault of their own.  Add to this that some anti-virus tools may use MAPI viewers to scan mail, which means that the system designed to shield you could accidentally infect you instead.  The same goes for backup agents that use MAPI clients to do brick-level backups.

There is one more bit of scariness to be told, though information on this last peice is more sketchy than the other exploit vectors. It would appear that the TNEF message may not need to be even previewed by an end user, but could in fact attack a server just be being received by the server itself.  Even until we know more for sure on this last point, I think the previous points - which are verified by this author and others - are bad enough that you should jump on this right now.

Microsoft has released a patch - also available via this same link, under "Security Update Deployment" and so you can protect yourself and your organization.  I strongly urge you all to do this immediately, and to have your end-users run Windows Update (Start|Windows Update on Vista, Microsoft Windows Update Online for everyone else) to get critical Internet Explorer patches as well.

Now, this is not only your Exchange server that you are protecting.  If an attacker gains privileges on your mail server, they can use it as a launching pad for attacking other servers with the same exploit or any other they'd care to try.  That means that if you don't patch your systems, you could inadvertently be responsible for attacks against other servers all over the net.

And so, I am asking my readers to protect the community at large as much as they can, by spending the time necessary to download and install the appropriate patches.  Yes, this will mean a maintenance window for a reboot.  Yes, it's annoying that we've gotten hit with yet another critical vulnerability.  But, yes, it is our responsibility to make sure that the attacks stop at our own borders, and that our own mail servers do not become the launching point for the next Code Red.

If I cannot bring you comfort, then at least I bring you hope

As we get ready to celebrate the beginning of a new year (in many countries), now is a good time to plan for your Exchange Servers and what you'll be doing in the year ahead.  While this is not the best time to plan out new infrastructure or massive overhauls immediately, you can start planning what you'll do later in the year on that scale.  You can also plan for what you'll do right off the bat in January for things like maintenance and updates/upgrades.

I plan to dedicate entire columns to these topics in the next few weeks, but here's the overview to get you thinking (once the champagne wears off...):

Offline Maintenance is a good thing to do regularly for Microsoft Exchange.  About once or twice a year you will probably want to take the time and walk through the process.  Microsoft says that this is not a solid requirement, but it can help identify long-trending problems deep within a database, so doing it once or twice per year is time-consuming, but worth it.  Plan now for downtime, so that you don't have to worry about when you'll have the ability to do it later. 

Online Maintenance is automatic, but you may want to look at your Exchange Server profile and see when the best time to have it done really is.  By default, the online maintenance system will run every night, but if you're also performing backups overnight, the two systems can overlap, causing the maintenance to be halted and never finish.  Take a look at your event logs to see if both systems are operating normally, or if the online maintenance run is terminating early.  If it is, you may want to change the times it runs, or even only run it on certain days.  Unlike with Exchange 2000, later versions of Exchange can hold up admirably well with even weekly maintenance runs, but be aware of what won't get done until maintenance by checking out Microsoft's Knowledge Base and/or waiting until my upcoming column on the topic.

Test, test, test, test, test......then test again! Now is the perfect time to set out your goals for testing of recovery/availability solutions (see disclaimer at the bottom of the page).  Backup systems, failover systems and other recovery solutions MUST be tested, otherwise how do you know they're doing what they say they're doing.  Best bets: do a test restore of Exchange data (at all the levels available from your tool like database, mailbox, mail item) to a different directory on the Exchange server and mount the DB in a Recovery Storage Group to ensure it is working properly.  This should most likely be done once per month, but at least once per quarter.  It doesn't require downtime, and doesn't require end-users to do anything.  Perform a failover test (if you use a failover tool) or a total restore test at *least* once per year.  2x per year or once per quarter is better, but of course since this type of testing will impact end-users, scheduling can be an issue.

Housekeeping is never a bad thing to look at after the clock strikes midnight on January 1.  Are you reaching your database limits in Exchange Standard edition?  If so it's time to either start ratcheting down mailbox limits or else starting to plan your upgrade strategy.  If you're already on Enterprise edition, are you pushing the limits of the free disk space you have on your drives?  Plan now to move the databases to another partition with more space.  Have your Stores grown to alarming proportions?  Might be time to start planning out new Stores and even Storage Groups to distribute the load.

Finally, give a look through the event logs and such to see if there are repetitive errors or other trends that can indicate problems brewing.  One more common example of this is recurring event ID's 1018, 1019, and 1022.  These indicate that there could be problems brewing in the database itself, though it is very possible that the error events are the only indication you see until the DB crashes.  If you're looking for a tool to help wade through the log information, check out Splunk, an IT search solution set that can make your life a lot easier.  [Due Disclosure: Splunk is currently working on partnerships with Double-Take Software.]

Have a healthy, happy and safe New Year's celebration.  Here's hoping 2009 exceeds everyone's wildest dreams in terms of greatness, and that Exchange never ceases to help us all live in these very interesting times.

Until next year.....

Mike Talon, Chief Columnist, BeingExchanged.com

PS: The title of this posting is taken from the song "Closing of the Year" from the movie soundtrack to "Toys."  The lyrics of the first verse are incredibly relevant to our combined experience this year, and I offer them here:

If I cannot bring you comfort,
then at least I bring you hope
For nothing is more precious than the time we have, and so
We all must learn from small misfortunes
Count the blessings that are real
Let the bells ring out for Christmas
At the closing of the year

- The Musical Cast of Toys Featuring Wendy and Lisa

Get the message? If not, it may not be your fault.

Microsoft Exchange engineers have always come under fire when a message that was sent to someone in their organization didn't get delivered, for whatever the reason may be.  The same goes for messages that bounce back for no apparent reason.  It could be a problem with the user's account (disabled, misspelled, etc) or it could be a problem with the server (offline, not connected to the net, etc).  It could also be a problem totally outside your organization, and that's what I'd like to talk about today.

In this ever-more interconnected world, routing of email can often depend more on external factors than internal ones.  Email messages can get mis-routed or even totally lost well before they reach your system, or can get purposely removed from the Internet somewhere along the way through no fault of your own.  If you're finding that more and more outgoing email is lost without any visible reason, the most common problem that is beyond your immediate control is getting accidentally blacklisted.

Blacklists are specific shared lists of either email servers or - in some cases - individual email addresses that spam control companies make available to end-users.  The most common of these is the DNS BlackList (DNSBL) which can either track IP addresses of servers known (or thought to be known) to send spam, or can track the domain names of such servers.  You can find out more about DNS BlackLists from Wikipedia at this link.  These systems collect information on servers and domains that send out spam and/or dangerous files (like viruses), and share that information with anti-malware companies (like McAfee and Kaspersky among others).  The anti-malware companies then use those lists to update their anti-spam and anti-virus solutions to protect consumers.

So if your server's IP address or domain accidentally ends up on these lists, suddenly external servers won't send mail to your servers anymore and won't accept incoming mail from you either.  It isn't easy to get on these lists by accident, but it could happen to you.  One or more internal users may get hit with a virus that sends tons of spam mails out from your email server, or someone totally outside your sphere of influence gets their computer taken over by a virus that forges headers to make it look like your servers are sending spam - and that's just two examples of things that could get you accidentally listed.

Once you're on these lists, it can be a very time-consuming process to get yourself un-listed.  Generally speaking, it will be a manual process of contacting each organization that has you listed on their blacklist system, and proving that you don't deserve to be there.  You can start your search at w3dt.net, which will assist you in figuring out which companies and filters have got your IP listed as up to no good.  Once you have that information, contact those organizations directly and either ask what caused you to get blacklisted or - if you already know that info - what you've done to stop the problem.

Be prepared to show that you identified who was sending the spam and why (and that you stopped it), or that you've isolated and removed the virus.  You will need to be patient and persistent, and will probably need to make several calls over a significant period of time (days to even weeks) to make sure that the problem gets solved.  Each organization that has you blacklisted will want their own proof that you're not a threat anymore, and in reality - provided they're not abusing their power - I can understand why they're being so careful.  Spam and viruses are a huge problem in the net today, and making sure they protect their clients is their business model.

Once you have the blacklist fixed to show you're not a spammer anymore, it can still take several days or longer for end-users of those services to get the updates to their spam and virus control systems.  So, if you do end up on a blacklist, you have quite a road ahead of you, but well worth it to clear your good name!

Getting blacklisted accidentally is a horrible thing.  Blacklists are a good thing overall for the benefit of email users all over the world, but when they suddenly stop you from emailing anyone (or getting email yourself), they can be a giant headache that you'll have to deal with effectively to get back in business.

Update on Wildcard certificates

Not that long ago I talked about using wildcard certificates to allow you to move OWA and ActiveSync services from one physical server to another.  Since single certificates are assigned to a single server, failing over or moving to another server would cause the clients to suddenly lose SSL connectivity, as the certificate would not match up, and ActiveSync devices cannot pop up the error about a non-secure connection. OWA can, but it can be troublesome with end-users who suddenly start seeing security warnings.

Following up on this theory of using a domain-assigned wildcard certificate, research has shown that older Windows Mobile devices (WM 5 or earlier) cannot leverage these types of certificates at all.  WM 6 and higher can leverage this technology, but earlier versions were not coded with the required information to recognize that a certificate could possibly be assigned to more than one physical server or networked device.

So, for those using WM 6, OWA and Outlook Anywhere, you can use wildcard certificates to allow for services to move from server to server as required.  For those using earlier versions of WM (or the 3G iPhone - though the jury is still out on that one), you must use server-specific certificates, and re-configure the devices' ActiveSync connection if the server itself moves. 

None of this impacts Blackberries, as they authenticate via the RIM network, and not directly to the Exchange servers.  If you're using Blackberries, wildcard certificates offer the ability for all other mobile systems (OWA, Outlook Anywhere, etc) to move with your servers in the event of a loss of a particular physical machine, while RIM will handle moving the Blackberry devices.

Long story short, if you're on WM 5 or earlier, be ready for a few support calls when you need to move the services or fail over between servers - even if you use wildcard certificates.  If your users are on the new iPhone systems, be sure to keep a close watch on Apple's forums, as new information is being discovered every day.

Exchange on DC's

Smaller shops have always been faced with a fundamental quandary when the time has come for them to move off Small Business Server and into the world of the independent version of Microsoft Exchange.  One of the biggest considerations is where to put the Domain Controllers for Active Directory.  The first thought most of my clients have is to put it on the same server as the Exchange system, but outside of SBS that's a very bad idea for a handful of important reasons.

For example, if you're ready to make the move from a single server to a cluster, you cannot install Exchange 2003 on a cluster node that is also a DC.  That is just simply not supported by Microsoft, and therefore isn't an issue up for debate.

It's much more likely that you'll be moving to a single-server, however, as if you're just moving off of SBS then the likelihood you need a cluster is reduced quite a bit.  In those cases, while you could install Exchange on the DC, it's still not a good idea.  The Exchange Best Practices Analyzer will even flag this condition as going against best practices, which should be a giant red flag for anyone installing an Exchange Server.

Here's a few reasons it isn't a good idea:

1 - Since Exchange runs many services under Local System accounts with elevated privileges, a successful attack via the Exchange system could easily expose your AD controller to outside influences.  With the potential to gain access to all domain resources if the AD controller is compromised, reducing attack surface is vital, and installing Exchange (or any app, this is not an Exchange-specific issue) will *increase* attack surface.  If you get hit on an Exchange Server that is not a DC, the attacker has access to only that server, not the entire Domain.

2 - 3rd Party integration can suffer.  Since many 3rd Party tools may expect to use Local Accounts and Groups (which wouldn't exist on a DC) or may otherwise need to control aspects of the Exchange System that may not be available or accessible on a DC, integration with Anti-Virus, Backup and Recovery solutions can suffer if the server hosts both roles.

3 - Performance impacts will be clearly visible.  Since the Exchange Server constantly uses the DC and vice-versa, placing both on one physical server can effectively double the amount of work that server must do.  Some visible effects of this are slower overall Exchange performance; longer response times for Domain commands (such as logging in, changing passwords, etc.; and much longer reboot times.

Putting the Active Directory and Exchange Server systems on one physical machine may seem like a way to save money, but in reality it will cost more in downtime and headaches than it will save on server hardware.

 

[Note: Small Business Server contains a highly modified version of Microsoft Exchange Server that is specifically designed to run on an SBS server.  It does not permit all 3rd Party applications, limits functionality and is also limited to only being able to run on SBS.  Therefore, running the SBS version of Exchange on an SBS DC is both a supported installation and a best practice. This article specifically applied to non-SBS versions of Exchange Server.]

To MTA or not to MTA?

Possibly the longest running debate in the history of Exchange 2003 is if you should disable the MTA Stacks service on an Exchange 2003 Cluster.  There are some very valid reasons to remove it, and one HUGE reason to leave it alone.

First, what is the MTA?  The Message Transfer Agent is a compatibility solution put in place by Microsoft to allow the movement of messages from Exchange 2000 and 2003 Servers (stand-alone and clustered) to other messaging platforms.  That could be an Exchange 5.5 Server, Lotus Notes, etc.  This is not the only method that could be used to move information between server platforms, but especially for 5.5 it is the preferred method.

You may be tempted to remove the MTA Stacks resource from a cluster, as it's one more resource that could go sideways on you, and it does offer an additional attack surface for those who would try to hack your system.  You may also try to remove it when you move to a pure Exchange 2003 environment, as such a configuration would seemingly not require it at all.

In theory, you'd be technically correct.  Reducing potential problems and removing avenues to attack are generally considered good things.  But an explicit ruling from Microsoft on the matter and the pain caused by the operations required to reinstall it if you need it in future can make you think twice.

The Microsoft Knowledge Base is pretty explicit that removing the MTA is not supported.  You can read about it here:

KB 810489 "MTA Stacks service supportability guidelines for Exchange 2000 Server and Exchange Server 2003"

Within that KB, along with the explicit note that removing the MTA is a bad idea, is the problem of reinstallation later on.  Most notably, if you ever want to re-install the MTA Resource, you have to delete the Exchange Virtual Server entirely (by deleting the SA Resource) and reinstall the whole EVA.  So, if you have to call into support, they'll tell you that you need the MTA resource in place. And in order to put it back in place, you have to first destroy the live clustered Exchange server.  That's a catch-22 you can avoid by not removing the MTA at all.

Long story short, keeping the MTA does offer a potential avenue for attack, but removing it creates an absolute headache if you need support later on.  For now, at least, leaving the MTA in place is a much better option - just make sure you have set up your firewall to block potential attacks!

I Luv Exchange.

Over the years that I've been working with Exchange Server, one of the more common questions I get asked is "Why Exchange?"  With all the other messaging systems out there, it's a valid inquiry.  The short answer is, "I am a masochist."

Yes, that was a joke, please do not send any more of those really weird emails my way...

So why do I stand by Exchange Server year after year, even as yet another version releases without a SQL back-end, forces hardware upgrades and has a difficult migration path?  Because - at its heart - it does what I and my clients need it to do.

Exchange Server is a complex animal, but one that other vendors have not yet been able to completely match in terms of what it does.  There are tools that do messaging and shared calendars well - some might even say better.  There are collaboration tools that are superb!  There are even Unified Messaging platforms that rival the best efforts of the guys in Redmond.  There are not, however, many total packages that can do it all, and do it well enough that you can rely on them.

The combination of Exchange Server, Office Communication Server, SharePoint Server and Outlook (well, all of Office) creates a solution set that does everything that most businesses would require for messaging and collaboration.  Add in the new Groove systems and now you're moving well past what most Enterprise Messaging solutions can provide.

Yes, it is true that Exchange is challenging. It is also true that the Messaging and Collaboration Platform is complex and overwhelming for inexperienced users.  However, I challenge a novice to take Lotus or SendMail and get it working without either a little help or a lot of reading.

So, enough with the Exchange bashing!  Reach out to others in the community, search knowledge bases and web forums, and generally let others who have been in your shoes before help you.  In most cases you can find someone who has experienced the same problem you have, and who found a way to fix it.  And in any case you'll learn a lot more about how the platform works.

No, it's not perfect - not by a long shot - but Exchange is the best Messaging and Collaboration platform (with its eco-system software tools) that I have used in my time as a Systems Consultant and Engineer.  So yea, I do enjoy working with it, and there might even be some luv in there too.

More than zero?

There are a ton of interesting settings and switches in Exchange 2000/2003, but one of the more unusual is the "Zero Out Deleted Database Pages."

Exchange 2000 and higher all have several levels of potential access points for un-delete operations.  When anything is deleted from a mailbox, the data isn't truly removed, but rather marked as scheduled for delete at some later date.  The default settings keep mail items for 15 days and mailboxes for 30, after which point they're removed from the server.

Data pages within the ESE database are another story; however.  They get deleted all the time, but not removed from disk.  They're simply marked as deleted, ready to be overwritten as needed.  Normally, this doesn't pose any specific security threat, as File servers and most other Windows servers do the same thing when data is deleted.  In higher-security environments; however, this could be a policy violation, as the data would be stored "in the clear" and vulnerable  to anyone able to get their hands on the physical disk devices.

Microsoft has taken steps to natively remove the potential for security breaches, but at a cost.  Built into every version of Exchange supported today is the ability to literally overwrite each deleted ESE page with zeros.  The processes is therefore referred to as "zeroing out" the pages, and the "Zero Out Deleted Database Pages" setting controls this behavior.  By default, the setting is not turned on, but can be enabled via the Exchange System Manager GUI, Powershell or via Active Directory.

The problem is that the benefits rarely outweigh the costs in the real world.  Zeroing out pages takes up a good chunk of processor time, and Exchange deletes pages - quite literally - all the time.  Zeroing out will also have a huge impact on any form of replication, just because a large amount of data change is required for the process to work.  In addition, this method of secure wipe meets only very low-level security concerns.  Most regulatory requirements require multiple overwrite passes (7, 15, sometimes even more) and mandate random data be used, not just zeroes. 

So, while you might feel like you'll gain some security by using this setting, the extra overhead and falling short of regulatory compliance should make you think twice before you turn it on.

Get wild with wildcard certificates

Most organizations that are considering securing their email systems are also looking at SSL certificates to aid in that effort.  Security certificates allow for encrypted communications using Outlook Web Access, Outlook Anywhere (RPC over HTTPS) and ActiveSync devices like Windows Mobile Smartphones. The issue is that a normal SSL certificate is bound to a single server, which means you'd have to obtain a signed certificate from a provider for each server you want to put into the solution.

You could use self-signed certificates to get around the issue of having to purchase individual certificates, but only at a cost.  Many applications will not accept self-signed certificates, and even Microsoft-controlled systems like OWA will kick up several layers of security errors - which will increase support headaches for you and your staff.

Alternately, you can obtain a Wildcard Certificate from a provider, which will allow you to use one certificate for multiple servers in your organization.  These certificates are more expensive than single-server options, but can give you much more flexibility with things like server rebuilds, migrations, and failover.  The only restriction is that the servers will have to be part of the same organization. That means that all the servers will have to be part of the same domain and/or forest in the sense of Active Directory.

Wildcard certificates can be defined at either the organizational level - such as *.wildcarddomain.com - or masked at another level, www*.wildcarddomain.com for example.  With the first example, all server identities that are part of wilddcarddomain.com will be able to share the certificate.  In the second, www1, www2 and even wwweb.wildcarddomain.com will be able to share the masked certificate.  Which you use will depend on the depth of security you are looking to implement, as more specific masks allow for less chance of someone spoofing the certificate for a foreign machine.

Once you install the wildcard certificates, you can allow services to move between servers within the same organization without breaking security. This can be extremely helpful for both migrations and disaster recovery, as you can have both of the servers shielded under the same certificate, and therefore you will not get naming errors when the users move or fail over. 

In a worst-case scenario, you can use single-server certificates on a migration or failover target, but your end users will get errors about the certificate being valid, but assigned to a different server name than the one they are connecting to.  You can then purchase another certificate after the fact and assign it to the target server to remove the error if/when needed.  However, wildcard certificates give a much better option for flexibility to you and your team.

No x64 for you! (on Exchange 2003 that is)

A whole slew of clients has been asking me about this one, so let me set the record straight.  Exchange 2003 will *not* run on x64 OS's.  For those who do not believe me:

MSFT TechNet article on Supported OS for Exchange 2003

Which spells out:

"Note: Exchange Server 2003 does not run on 64-bit editions of Windows Server 2003."

So what does this mean?

1 - You CAN install Exchange 2003 on x64 hardware, just not on the x64 editions of Windows

2 - You CANNOT do an in-place upgrade to 2007, since that needs x64 to run in production environments

So, while you can buy new hardware safely and continue to install and run Exchange 2003 properly with an x86 OS, you can't leverage the 64-bit framework. 

Also, you can't run 2003 on Server 2008 - at least not yet.  This means that the advanced clustering and other 2008 features are not available for Exchange 2003.  It also means that if you buy a new server and get an OEM version of Windows, you will have to specify that you need Windows 2003 x86, otherwise they'll send it with 2008.

Sorry to deflate everyone's balloons at once, but hopefully this will help a few bets pay out between Exchange techies.

Dial-tone revisited

The theory of Dial-Tone Recovery (DTR) is one that has often been overlooked in the world of Disaster Recovery (DR) for Exchange Server.  However, even in Exchange 2007, DTR can provide a great method for immediate restoration of email services, though with a few things to keep in mind.

For those who haven't heard of DTR before, here's a primer:

If a primary Exchange 2000, 2003 or 2007 server fails, you can attempt to restore services by deleting the corrupted databases and re-starting Exchange services.  This will create blank databases and allow users to send and receive new mail, access new calendar items and access all shared contact information.  Running a /disasterrecovery install of Exchange on a rebuilt box with no data will do the same thing.  Though end-users can access their email systems again, there will be no historical data, so this isn't a total solution set for true DR, but gives you some options for immediate availability.

In an emergency, this can give you time to perform restoration steps - which could take quite a while to finish - without making everyone wait to get back basic send/receive capability. You can restore a copy of the historical data via several methods, taking the time you need to do it right.

If you used a brick-level tape or disk backup solution, you can restore mailboxes via that tool's recovery system. Archiving solutions and Continuous Data Recovery systems (like TimeData from Double-Take - see disclaimer below), can let you move mailbox, folder and other data back over time as well.  If neither of those tools are at your disposal, but you to have a backup of the database and logs, you can restore those to a Recovery Storage Group, and use ExMerge or Exchange 2007 tools to bring back mailbox data and merge it with the new information on the DTR-recovered server.

If no backup is available at all, you can still provide Exchange services from the point of DTR onward.  While no historical info will be available to them, the end-users will be able to send and receive new email, calendar entries and Public Folder data.

Hyper-V-active Exchange

As most folks have already seen, Hyper-V - the newest Microsoft Virtual Server Solution - released to market yesterday.  This is a pretty big step forward for MSFT, adding a datacenter-ready virtualization solution to the Server 2008 platform.

I'm doing a little research now on the Exchange front with Hyper-V, but I was able to get an un-named source (ooooh spooooky) from MSFT to confirm that some MSFT Exchange 2007 servers at their facilities are running on Hyper-V, so support on that platform will most likely be forthcoming.  More info on this front as I can get my hands on it.

From the limited contact I've had with the betas, I can safely say that this product is a different animal than Virtual Server 2005.  The overall performance in both the Host and Guest components has ratcheted up by a factor of multiples, The management interface is also re-designed to allow for more fine-tuned control over the vitalized systems, and cluster support means the ability to move VM's between hosts is built-in.

We're still in the very early stages of this technology, so expect more reports as I run into it more and more in the field.  Within a short period of time, I'm also going to get to put Exchange 2007 through its paces on Hyper-V, and I'll be happy to report on that too!

It goes without saying that Double-Take is going to be supporting Hyper-V in the very near future with some amazing tool sets.  If course we can already replicate and fail over Exchange from Physical or Virtual (of any flavor) into a Hyper-V VM, but keep your eyes peeled to find out what new stuff we're working on over here especially for the Hyper-V platform.

Legally important

Providing Disaster Recovery (DR) for Exchange servers has always had strong arguments in its favor.  As an overall requirement, being able to get the email systems, calendars and contacts back up and running ranks pretty high up the list.  But there are other reasons to look at Dynamic Infrastructure solutions for Exchange that go beyond the convenience of Exchange end-users.

When email is stored only on the production Exchange server, it can be altered or destroyed by anyone with access to that server, which means anyone with an email account that allows them to see that particular user's mailbox.  This leaves you with a whopping legal liability (check local listings on exactly what that is for you), but one that can be avoided in most cases.

Using an Operational Recovery tool, like Double-Take TimeData (see disclaimer below), will allow you to ensure that another copy of the data is not only held off-site, but held in a repository that tracks all changes to the email, calendars, contacts and all other information.  This way if a critical change is accidentally applied, or if someone maliciously attacks the data on the production server, you have the ability to revert either individual items or the entire Store or Storage Group as required.

This means that if the data is required for a legal requirement (like court-ordered discovery), you can be sure that not only will the system be available, but also that any information that could be lost without impacting the server can also be quickly and efficiently restored.  Of course, being able to get that information back to some other server or even to a desktop or laptop is best, so look for tools that give you that flexibility.  After all, if you don't happen to have the original server running to perform the restore of information, you'll need to be able to designate someplace else to receive the data instead.

This doesn't change anything you might be doing with DR and Dynamic Infrastructure in your Exchange Environment, but gives you a deeper level of protection and flexibility that most standard DR tool-sets just can't natively offer.