Take Care With Exchange 2010 Throttling Policies

Many of my clients are currently evaluating Exchange 2010 for their production environments. While they probably won’t be implementing for a little time yet (average target dates are in Q4 for most clients I talk to), they are beginning to see different switches and settings that need to be tweaked from their defaults.  One of those settings is the native Throttling Policies that Exchange 2010 puts into effect to limit how much of various Exchange resources other applications are permitted to use.

Brian Desmond wrote a great article (you can read it here) that goes into detail about one potential problem that these limits impose.  Since the Throttling Policies are designed to stop individual client applications (among other things) from using up all the resources of an Exchange Server, they can impact 3rd-Party tools just as much as POP, IMAP and other mail client software.  In Brian’s article, he brings up the difficulties that Blackberry Enterprise Server (BES) runs into with the default policies, as they would normally hamstring BES’s ability to use a single account to ferry messages back and forth between Blackberry devices and the Exchange Server itself. 

In this case, what applies to the BES server can also impact other 3rd-Party applications that need to access an Exchange database via a client connection.  For example, archiving systems not using the new native tools may be working via a Journaling Mailbox or other single-account connection system. This means that the policies, as set up by default, would severely hamper the ability of the archiving system to get the data from the Exchange Server and into the archive solution.  Using the native archiving and compliance tools will help fix this, but many clients have invested a huge amount of time and money in setting up existing solutions. This means they face the choice between paying and implementing for an upgrade to the 3rd-Party system, versus completely re-engineering their archiving and compliance systems to switch to the native tools. 

Also, consider periodic mass-connectivity.  Anti-virus and other scanning tools may periodically connect to the exchange system using client connectivity, as would some 3-rd Party search and indexing systems.  Both of these will most likely not be hit by the policy limits, but could be if they do a mass-crawl. Careful evaluation of all systems that use client connections is necessary to eliminate any issues before they start generating client complaints.

Luckily, it is not difficult to change the Throttling Polices in Exchange 2010.  Brian’s article offers a great explanation of it, and of course Microsoft TechNET has a full set of details.  Once you determine which applications may need extended client access to the Exchange Server, you can set up custom policies for those applications and the accounts they use.

Time Keeps on Slippin…Slippin…Slippin into the…past?

Once again, I grabbed the title of the post from a play on words on lyrics from “Fly Like an Eagle” by the Steve Miller Band.  For those of you who have no idea what I’m talking about, see this YouTube video, and know that you have made me feel very, very old.

The point to the title was that some of you may have noticed that calendar appointments, email time stamps and many other things seem to be off by 1 hour starting yesterday.  The reason is that unless you have all your updates and hot fixes from Microsoft, Windows 2000 and 2003 would believe that Daylight Savings Time changes occurred on Sunday, October 25th at 2am, and changed the clocks on any non-updated servers.  If that’s an Exchange Server, the incorrect change will flow over into all Exchange functions as well, causing quite a few problems.

Normally, we here in the US change our clocks twice per year.  The latter one used to happen on the last Sunday of October, when we all set the clocks back by one hour at 2am on that day.  The problem is that the US Government changed the rules late last year, changing the dates that these one-hour shifts take place on.  This year, it will be November 1st, but Windows wasn’t originally programmed to deal with that.

Most of us update Windows and Exchange regularly, so we got all the appropriate patches and ran the required updaters on the systems in question.  You’ll know you did it right if the clock did not change Sunday, and do change on November 1 at 2am.  You know you missed at least one if either your server time, or your email time stamps are all an hour off today.

Yet another reason to patch regularly, but everyone can miss one now and then, so visit this Microsoft Support site on DST changes if things are acting odd time-wise.  If thing are acting odd in other ways, throw me an email, I might do a column on it =)

Friday Poll – DNS types.

Why use AD DNS?

My recent article on the need for (and use of) PTR Records in DNS have sparked quite a few questions on using DNS with Exchange Server in general.  The biggest one I get is “Do I need to use Active Directory DNS in order for Exchange Server to work?”  The answer to that one is a bit complicated, but in its simplest form, it boils down to, “No, but you really, truly should.”

Exchange Server 2000 and up required some form of DNS in order to function correctly.  This is mainly because the Windows Internet Naming System (WINS) was “depreciated” starting with that version of Exchange.  What that means is that MSFT officially asked the community to stop using it whenever possible, because it could be removed completely soon.  As it turns out, WINS was phased out in Exchange 2007, though it may still be required for certain Outlook functions.  That’s a topic for a whole different series of blog posts though.

As for DNS integration, it’s quite possible to install Exchange 2000-2007 without having Active Directory DNS configured in your domain, though it isn’t a best practice.  As long as your DNS system can handle Server Name Records (SRV type records), you can successfully use a 3rd-party DNS for your Exchange environment.  There are, however; some good reasons to go with the native Windows Active Directory Integrated DNS solutions:

1 – Exchange can natively talk to Active Directory DNS, and therefore can do some interesting tricks with that DNS platform that it can’t do with 3rd-party DNS.  Things like AutoDiscovery when you move a user to different mailbox servers, or after a recovery operation with Database Portability just don’t work the same way if you’re not using Active Directory DNS.

2 – Many 3rd-Party tools leverage AD DNS to figure out where Exchange resources are.  Note, I’m far from unbiased on this topic, so please see the disclaimer at the end of the blog.  Since many Windows-based tools will natively use AD DNS API calls (like DNSCMD and the newer variants in PowerShell), you may need to make manual updates to your 3rd-Party DNS, or may have to give up functionality.

3 – Many other non-mailbox objects are stored in AD DNS, and must be mapped manually in other DNS systems in order for Exchange to work properly.  You will have to track your Global Catalog servers, Domain Controllers and other resources in order for Exchange to function.

So, as you can see, there are some very good reasons to use Active Directory DNS if you plan on using Exchange Server.  While you may have external DNS records hosted with an ISP or other provider; internally you will be better off with the native DNS solutions in Windows unless you are ready and willing to fine tune your DNS systems and stay on top of it. 

If you are in doubt, you can use the Exchange Best Practices Analyzer to test your environment before you begin to install Exchange.  This tool will test for many things that Exchange needs, including properly configured AD or 3rd-Party DNS systems.

What is a PTR record and why should I care?

True story, I heard this exact question from a client not too long ago.  Through my trials and travails as an Exchange Engineer, DNS information is one of the very most confusing aspects of email systems that my clients have to deal with.  Just figuring out how to use normal DNS records tends to lead to a strong desire to give up on the whole project, so attempting to discuss reverse records can cause many to throw up their hands and run screaming from the server room.

Alright, that only happened once, but wow was it fun to watch.

PTR, or reverse lookup, records are used to allow external servers and systems a way to find out what identity a server has based on its IP address.  So, for example, we can look at the IP address information for Bing.com, Microsoft’s replacement for Live Search.

If I do an nslookup on bing.com, I get the following:

Name:    bing.com
Address:  64.4.8.147

So it appears that 64.4.8.147 is the IP address for that URL.  Now if I put the IP address into nslookup, my DNS server will attempt to backtrack to see what that server is identified as:

Name:    origin.bay.ux.search.live.com
Address:  64.4.8.147

Not a perfect return, as I was looking for it to reply that the IP was assigned to a bing.com address, but knowing that Bing Search replaced Live Search, the results are clearly traceable.

You can read up on what PTR records do, and how to create them at this Wikipedia page.

Now that you have some idea of what PTR records are used for, we can discuss why you will want to make sure that all mail domains you have authority over are properly configured to use them.

These days, you can do a quick Bing (or Google) search and find dozens of software packages designed to allow you to send out email as if you were someone else.  Sometimes there’s a legitimate reason to do this, such as if you manage email lists for multiple organizations through one mail domain.  In many cases, these tools are used toward nefarious ends, allowing a hacker to send email that appears to be from a domain in order to scam or infect the recipient.  Famous examples of this are phishing emails (see this Wikipedia page) where a scammer will send an email that appears to be from – say – a bank or other institution.  Your end user receives the mail, and since it appears to be from a trusted source, they’ll click the link and possibly enter in private information, leaving your organization open to attack.

To combat this, many SMTP servers either natively support, or can be configured to support, PTR lookups before accepting email.  This way, the header information can be examined to ensure that the email isn’t coming from a suspect source.  The end users don’t see much difference, but email that is from unknown domains, or domains known to be fraudulent, can be rejected before ever getting to their mailboxes. Exchange 2003 and 2007 do not natively reject email based on a bad reverse DNS lookup, so left to their own devices; you don’t have to worry about blocking incoming mail by accident.  This doesn’t mean you can ignore PTR records though.  Since many other mail server systems can be configured to reject non-verifiable mail, and since there are a host of 3rd-party systems that work with Exchange Server that can do the same, failure to properly configure a PTR record can cause your outgoing email to get bounced because you cannot verify your identity.  This means that you need to set up a PTR record for your Mail eXchanger (MX) records and your domain in general, or you risk having email returned as non-deliverable.  Having no PTR record is just as bad as an incorrect configuration here, as a lack of a reverse DNS lookup will cause the same results as an incorrect reverse DNS lookup.

There is a great debate over if this is a good or bad thing.  PTR responses could be forged, and so this is not a foolproof method of confirming identity.  Also, if multiple organizations all use the same SMTP server (thing about that multi-list email server from before) then which one gets the PTR record assigned to it?  If you have an email domain, then for now it is a good idea to ensure that PTR records are properly configured so you don’t run into the problem, but hopefully there will be better, more secure systems to rely on in future to confirm identity (such as the SMTP-AUTH proposed specification as part of E-SMTP).

To avoid blacklists, blocks and email black-holes, configure correct PTR records for your SMTP servers and domains.  It is by no means a perfect system, but it is one that you will run into, so an ounce of prevention is definitely worth a pound of cure here.

Get back to where you once belonged (Failover Cluster version)

In honor of the re-release of the Beatles stuff all over the world (games, CD’s, maybe iTunes at some point), I took the title of today’s post from their song “Get Back” on the album Let It Be (Remastered).

I am, of course, going to tie this to something in Exchange; specifically Exchange 2007 Standby Clustering. Standby clustering refers to the theory of using a replication engine (like the native CCR or a 3rd-party system like Double-Take Availability – see disclaimer below) to place a copy of the data for the Storage Groups of the production cluster onto a secondary cluster.  Once the data is replicated, you can use the /RecoverCMS commands to recreate the production Exchange Cluster Mailbox Servers (CMS’s) on that secondary cluster.

The solution set for bringing up the Storage Groups and CMS’s on another physical cluster setup in the same or another location is fairly well established.  If a single node fails on a production cluster, other nodes take over the failed Storage Groups and work resumes in a very automated fashion.  If multiple nodes, or the entire cluster, fail you use /RecoverCMS and the associated protocols to manually get everything working on another system – so long as a copy of the data exists to work from.

The problem has traditionally been best expressed by the phrase, “And then what?”

If the original cluster failed completely, the answer was simple.  Rebuild the systems with the same node names, but prepare the systems as though they would be a new /RecoverCMS target system.  However, if you have not lost the production systems, and they’re stable enough to be used again, you would still have to reinstall them without some additional help.  The most common reasons for this kind of outage are routine testing of the failover systems and extended power failures that generators and UPS systems can’t handle.

Microsoft does offer a command set to fix this particular problem, but it is not well known or publicized.  As a matter of fact, during a recent client troubleshooting session, we had a couple or techs from Microsoft on the phone (Premier Support in this case) and they were not aware of this particular method for cluster restoration.

Once you have fixed whatever went wrong, if your production cluster is still viable (and is suitably stable for continued use), you can use a command set called /ClearLocalCMS to remove the original CMS entries from the original production cluster.  Doing so is not without risks, and you should familiarize yourself with this KB article on the subject before you try it. 

/ClearLocalCMS will remove the CMS components off the original production nodes, clean up AD, and disable the virtual computer object for the original cluster CMS.  This ensures that Exchange doesn’t accidentally address the original cluster system, even after the restore process begins.  Once the CMS is cleaned, you can go about restoration of the data using the same tools as you used to get it over to the standby cluster in the first place.

To get back to your original servers, use the /RecoverCMS command in the opposite direction (from DR back to production) and then use /ClearLocalCMS commands to re-prepare your DR cluster for use in the next emergency.

Jumping between clusters is not an automated or easy process, but it does work correctly if you follow all the steps in both directions.  This set of command suites (/RecoverCMS and /ClearLocalCMS) can allow you to get back to where you once belonged, every time.

Storage Groups, or, Keep Them Moving and Spread Them Out.

In a recent conversation with a colleague (James from VA in this case), he mentioned that there were a lot of folks who simply jammed all their users into one Storage Group in Exchange, even when they had the option to use multiple groups, and really should have.

Storage Groups are databases in Exchange Server.  They contain Information Stores and Log Files, and logically group together users and Public Folders that are related in some way.  There are a lot of different ways that you can choose to use allocate users to Storage Groups, but here are some of the more common methods:

1) Geography – Many organizations are centralizing Exchange servers to one or two datacenters, instead of putting a server at each location.  This saves money overall, allows for better message hygiene and, with the use of technologies like Outlook Anywhere, doesn’t alter the end-user experience to a any large degree.  Creating a Storage Group for each physical location allows you to better administer the Exchange system as a whole. If a new policy or procedure impacts only one office, you won’t be stuck combing through all the mailboxes and folders trying to find the users who are impacted by that policy. Just apply it to the Storage Group for that office and move on.

2) Department/Employment Level – This is a very popular method for splitting up users into Storage Groups, especially where there are a relatively small number of locations or other physical defining factors.  Many companies will define their Storage Groups by the departments that exist within the firm, mirroring their political structure into the messaging systems.  Sales, Support, Administration, Management and other logical groups can be translated into their own Storage Groups.  This has the dual benefit of not only allowing you to better control policies and procedures for each group, but also making it easy to put all managers into their own server which gets clustered, while the rest of the employees are on servers that will be restored from tape.  That’s just one example of the benefits of this segregation system for Storage Groups from the real world, so please don’t send hate mail. Of course, you can use a tool like Double-Take to give you even more options, but see the disclaimer at the bottom of the blog, as I’m far from unbiased on that topic.

4) Alphabetical – Where there may be only one or two locations for the business, or where everyone’s email is equal, then using something as simple as the alphabet can help figure out where to split out your Storage Groups.  A-M and N-Z or even 3 or 4 different breakouts can allow you to allocate your users across multiple Storage Groups without worrying about which users should go in which groups by other factors.

The reasons for splitting up Storage Groups go well beyond simple administrative tasks like policy and procedure enforcement. Storage groups can take an unmanageable situation, like 3000 users all in one gigantic glom, and make life easier to deal with by allowing you to manage smaller groups as required.  Database maintenance on a 500 user, 50GB Storage Group is a lot easier than database maintenance on a 3000 user, 1.2TB set of Stores.  It also means that the other 2500 users aren’t offline while you do that maintenance on the one Storage Group in question.

Physical limitations of Exchange could also force you to break things up into groups.  Granted, the built-in limit of 16TB per Storage Group (in Exchange 2007) is one most folks won’t ever hit without already breaking things out into multiple Storage Groups, but there are other size limitations.  Storage Groups must have their database files (.edb in 2003 and 2007, .stm in 2003 only) exist on a single logical volume.  This means that if you have 1TB of disk space, your Storage Group can’t grow past that point.  Multiple Storage Groups allows you to allocate your user data across multiple volumes, allowing easier growth and better storage management overall. This also leads to more spindles and read/write heads being used at once, which increases overall performance.

Even the Standard version of Exchange 2007 allows for up to 5 Storage Groups, so even smaller organizations can now split their users out across multiple Groups.  Exchange 2003 and earlier only supported one Storage Group, so if you’re on those versions, you will have to keep everyone together.  Otherwise, live by the simple mantra that more is better when it comes to the allocation of users to Storage Groups.

Getting Smart (Smarthosts that is)

No matter what version of Exchange Server you’re using – or even if you don’t use Exchange at all – you need to perform message hygiene on all mail in your organization.  Virus files, spam, phishing schemes and tons of other attacks are thrown at every email domain, every day.  If you’re not ready to deal with them, you’re dead in the water.  Smaller companies often just make due with hygiene software on the Exchange Server itself, but all sized firms need solution sets, and even smaller firms can take advantage of Smarthosts in a few different ways.

Smarthosts are servers or hosted solutions that are dedicated to performing message hygiene on all mail coming and going from your organization.  They come in many flavors, but tend to center around three types:

1 – Fully hosted solutions.  This method has no Smarthost hardware on-site, but instead contains everything at a hosting provider’s location.  Lowest up front cost, but highest ongoing (monthly, quarterly, etc.) costs.

2 – Hybrid solutions.  Here, you host a small appliance on-site to handle some of the workload, with the rest of the message hygiene functions handled by the service provider in their facility.  This isn’t quite as popular as the other two solutions, as the costs can quickly outweigh the benefits.

3 – Fully owned solutions.  Appliances and servers are in your datacenter, and totally controlled by you.  Start up costs are high, but ongoing costs are very low, as you’re only paying for updates to the virus/spam filter engines.

For smaller organizations, hosted solutions allow you to assign your Mail eXchanger (MX) record  to point to your Smarthost partner’s datacenter.  There, all incoming mail will go through the filters and checkers before it ever reaches your internal server set.  Most hosted providers allow you to update a list of “known good” recipients, so that any mail not destined for a valid employee is rejected immediately.  Virus scans are performed, and optionally spam filtration happens as well.  The remaining mail goes off to your internal mail servers for delivery to the end-users.

When mail leaves your organization, your internal servers are configured to send it first to the Smarthost provider, instead of directly to its destination SMTP servers.  The service provider performs all the same message hygiene on those messages, then acts as the SMTP gateway for your domain out to the rest of the world.

Hybrid solutions do everything that the fully hosted solutions do, but typically place a small appliance on-site to do periodic scans of the mail server itself.  They can also help speed up the whole process by making the “first hop” for email flow out of the organization be local instead of across the WAN.  It’s rare to see a hybrid Smarthost system that only does email hygiene, as the fully hosted or fully owned methods are much better for this kind of single-task solution.  Instead, the appliances on-site will typically handle all anti-malware processes including updating desktop protection and server scanning.

Fully owned solutions place appliances and/or other equipment at your production facility.  They act the same as the hosted solution, but you own all of the hardware and pay only for periodic updates to virus definitions, malware profiles and spam allow/deny lists.  This solution offers much more flexibility than a hosted solution, as you have complete and immediate control over any changes that might need to be put into effect.  The drawback is that if the hardware needs to be upgraded, you’ll be responsible for buying the new boxes.

Smarthosts are available from a large number of vendors.  Postini, Barracuda, and Microsoft Exchange Edge Services are just three that can be found with a Bing Search.  Most perform the same tasks, but each has their own selling points.  Your best bet is to talk with multiple vendors and see which one has the solution set closest to what you need.  Ask about things like ease of administration, web or application interfaces, ability to create custom allow/deny lists, and integration with your own Domain services to get updates to users and allowed accounts without compromising security on your AD servers.

No matter which vendor you choose, or which solution, using a Smarthost removes the message hygiene load off the Exchange Servers, which is always a good thing to do.  By dedicating either a service or server to handling message filtering and security, you can free up resources to provide a better end-user experience and a safer messaging environment.

Exchange 2003 only wins with WINS.

Since Microsoft released the Release Candidate for Exchange 2010, I figured that a look back at some things in Exchange 2003 are well in order.  After all, a lot of folks are currently on that platform, and waiting for 2010 to RTM before they upgrade.

So today, let’s take a look one of the components of Windows that Exchange 2003 needs to have in place, but that causes confusion and doubt every time it comes up in conversation.

The Windows Internet Naming Service (WINS) is a component of Windows Server NT4, 2000 and 2003 that – as its name implies – translates NetBIOS names into internet names and addresses.  And to be absolutely clear on the subject, Microsoft *requires* the installation of WINS for full functionality of Exchange 2003.  Here’s the proof.

The reasons for this insistence on having WINS present in Exchange 2003 are varied, but the controversy over the requirement has raged ever since Exchange 2003 was released.  The confusion stems from the apparent dichotomy between the fact that 2003 is supposed to be totally DNS integrated, and the view that NetBIOS dependencies make it look as though it is not.  Another reason for confusion is that even though Microsoft says that the installer will not run properly without WINS in place, many clients have managed to install Exchange 2003 without the WINS services running at all, or worse yet; with the WINS system improperly configured and malfunctioning.

Exchange 2003 leverages Active Directory (AD) DNS for nearly all name resolution functions, from finding other servers in the local environment to finding SMTP hosts for external mail destinations.  For internal resolution, Exchange can leverage AD and Global Catalog servers to find things, as long as everything in the domain in question is using Windows 2003 or higher, and configured appropriately.  MAPI systems in 2003 also can use DNS to find Exchange servers (Outlook 2003 and up) or to locate other resources.  So if you are in a pure Windows/Exchange/Outlook 2003 or higher environment, and you have only one Active Directory site, then you’re all set without WINS.

That last sentence is key to figuring out where the requirements for WINS come from, and where most of the confusion stems from as well.  Most environments that use Exchange 2003 still have mixed-mode domains, possibly even some Windows 2000 servers around.  They’ve also brought legacy systems up to Windows 2003 over the years, meaning that older names and objects may still be preserved in AD.  So if an Exchange 2003 server needs to find a resource that AD only has a “short name” (non-fully qualified domain name) for, the traditional method to find that resource would be WINS, not DNS.

This also comes into play if you have multiple AD sites across multiple physical locations.  Since short names don’t translate properly from site to site, or domain to domain in the same Forest, WINS is necessary to translate the resources name into a location for the resource itself. 

Finally, any legacy Outlook clients (XP and earlier) rely on WINS to find all resources, as the MAPI clients they contain don’t understand DNS lookups at all.  So if you have any older Outlook clients running around, you’ll need to ensure WINS is configured properly.  Similarly, ExMerge relies on WINS since it was a hold-over application from earlier Exchange versions, and therefore never designed to leverage DNS.

So, until your environment moves up to Exchange 2007 or 2010, and a Native 2003 or Native 2008 Domain architecture, you’re stuck with WINS.  The good news is that WINS is just another component of Windows in 2003, and therefore not a bear to implement at all.  Go to Add/Remove Programs and choose to install Windows Components.  You’ll need your OS CD’s to finish the install, and as always; be sure to Service Pack after you are done and run Windows Update.  There have been many changes and patches to WINS over the years, especially in light of recent attacks against it.

WINS is a legacy system that is – thankfully – going away as we move toward Exchange 2007 and 2010 native environments, and as legacy Outlook clients are phased out.  Until then, make sure you keep up with WINS patches and ensure it is installed and configured. Not doing so can cause resolution problems for Exchange 2003, and installing it is a requirement from Microsoft, so take the time now to make sure your WINS systems are operating full steam.

Exchange 2007 Installation – continued (finally)

Some avid readers brought to my attention that I had promised an article on the minimum required for installation of Exchange 2007 in a default server config last week, and didn’t deliver.  Sorry about that, and here we go!

Once you’ve stepped through the pre-requisites for Exchange 2007 (see two articles back), you’ll be ready to run the installer proper for the Exchange system.  Today, we’ll focus on the setup for a single Exchange 2007 server holding all required roles.

Before you can install Exchange 2007, we’ll have to get AD ready to roll.  Previous versions of the Exchange installer had the /forestprep and /domainprep switches that could be run by Domain Admins who didn’t have Exchange permissions and didn’t want to give those domain privileges to the Exchange Admin. Exchange 2007 doesn’t have those switches, but instead segments out the different tasks into a set of 5 switches, each doing a specific prep job.  You have two choices here.  First you can get Enterprise Admin rights and just run the setup wizard.  Second, you can have someone with Enterprise Admin rights at the root domain and Domain Admin rights for each sub-domain to run the individual commands.  They can be found, and are explained, in this TechNet article. 

Once the domain and forest have been prepped, you can run the Exchange installer on the server where you want Exchange installed directly.  While there are various command-line and silent install methods, I’m going to focus on the wizard-based installation.

After you step through the welcome screens, you’ll be asked a few critical questions. From here on out, we’re going under the assumption that you’re running as an Enterprise Admin and that you’re doing everything (domain prep and all) at once.

You’ll need to define the Organization Name.  This is the Exchange Org, and not the company name or AD domain name.  Though the three names (Domain, Company and Exchange Org) may be related, the Domain Name and Exchange Org Name can’t be identical.  Choose something that makes sense, and doesn’t use any special characters – stick to numbers, letters and underscores.

You’ll also have to allow or deny permission for Microsoft to be informed about errors that Exchange sees, and you’ll tell Exchange if there are users on legacy Outlook clients (Outlook XP and 2003) and if there are 3rd-party MAPI clients (like Entourage) in the client-base.  Be careful here, if there is any chance that you’ll have non-Outlook 2007/2010 clients, use the legacy setting.  This creates a Public Folder hierarchy to handle administrative Public Folder tasks like the Offline Address Book distribution; even if you do not use Public Folders for anything else or you are setting up different Public-Folder Only servers.  Without the administrative folders, Outlook before 2007 will not be able to function correctly.

You will also need to choose what Roles to install.  The default is to create Mailbox (MBX), Hub/Transport (HT) and Client Access Services (CAS) roles, which are the three mandatory roles you must have in place for Exchange 2007 to run.  While you do not have to have these all on one server, each site has to have at least one of each of these roles running somewhere.  The default selection in the wizard will put these three roles onto the server you’re installing to, which is fine for smaller organizations.  If you have heavy MAPI users or a lot of users, you will want to install the MBX role only and put CAS and HT on one or two independent servers.  If that’s the case, select to install MBX role only here, and run the installer on the machine(s) that will host the HT and CAS roles and re-run the installer there, choosing the appropriate options as you go to install just the roles you need on each box.

The remainder of the installation is pretty much automated.  You’ll be able to watch the progress of each installation task, and as Exchange moves forward you will see a status report (with green check, yellow bang or red stop sign) as each is completed.  Hopefully, you’ll only see the green checks across the board.

While it is sometimes not required to reboot after the install, it’s not a bad idea to reboot anyway.  The reboot should be quick, and will ensure that resources used by the installers are freed up, and that all the services start properly.  Neither of those things is bad, and no one is yet using the server, so reboot, please.

For a default installation, that’s about it!  Next week, we’ll start talking about what the individual roles do, so you can decide if you want them on independent servers (or at all, for the non-required roles).  I’ll also endeavor to actually write up what I promise next week  And finally, for my non-Exchange-2007 users, I promise to do some articles on Exchange 2003 again in the very near future!

Back to basics: Prepping a 2008 Server for Exchange 2007

Got a basic Exchange 2003 or 2007 topic you’d like covered in this series?  Email MikeTalonNYC@gmail.com and you might see your topic covered here!

A lot of my clients have been asking me to write about some more basic topics dealing with Exchange Server 2007 here on the blog, and so this week I’d like to talk about pre-requisites for installation of Exchange Server 2007 on Windows Server 2008.  These will change as Server 2008 R2 goes live later this year, so stay tuned for updates.

To start, run through the basic install of a Windows Server 2008 x64 box in either Standard or Enterprise Edition.  For a stand-alone Exchange server, you can use either version of Windows.  If, however, you need to cluster your Exchange 2007 systems with SCC or CCR clusters, then you must install Windows 2008 Server Enterprise Edition.  This article focuses on a stand-alone Exchange server, and therefore which version you install depends more on cost versus performance.  Check out this article for the details on how many processor cores, how much RAM and other factors that change based on versions of Windows Server 2008:

Windows Server 2008 Editions on Microsoft.com

Note: Since Exchange requires .NET, right now you cannot install Exchange to a Server Core installation.

This is a good time to install any tools your organization uses which are not Exchange-specific.  Monitoring tools, storage management, etc.

Once you have the basic Windows Server installed, make sure you run Windows Update to get the latest Service Pack and all the assorted hotfixes and patches.  Some of these are required for Exchange, so do not skip this step.  The most critical of these (beyond the Service Pack) is the .NET platform.  While you must have at least .NET 2 installed, the current version is 3.5.1, so you may wish to install that version as well.  No doubt future tools in Exchange 2007 will need some of those components.  Since most of the .NET platform shows up as Optional, you may need to manually check the boxes for these downloads before they will be installed by Windows Update.

Next up, install PowerShell to the machine if you haven’t already.  This shows up in the Server Manager Wizard as a Feature, and is a wizard-driven installation, so you do not need to download anything from Microsoft to do the install anymore.  The default parameters are fine unless you have a preferred setup for PowerShell.  (for those who will inevitably email me on this, according to MSFT, it is spelled “PowerShell,” not “Powershell.” Go yell at them if you don’t like odd capitalization.)

Exchange 2007 also requires that the IIS system be present on the server.  You can install this by bringing up the Server Manager and adding the Web Services (IIS) Role to the box. When you select this role, you will be asked if you wish to also add dependency features/roles.  Select Yes here to automatically add in quite a few feature sets that both IIS and Exchange need. 

A couple of notes: Two items that are not selected by default or by auto-selection are Dynamic Compression and the IIS 6 Admin Tools.  You should select both of those manually, as Exchange will need them.   Also, you have the option of selecting different authentication types for the Web Services in this wizard as well.  Be sure to select any and all that your organization will use.  As with most things, you can go back and add them later, but since you’re doing the install anyway, might as well get them on the box now.  Aside from these items, you can safely use the defaults for this wizard and allow it to perform the installation.

After you run through the IIS Role setup, you’ll need to install the remote AD configuration tool kit on the server.  The easiest way to perform this step is to open a command prompt and run the following command:

ServerManagerCmd -i RSAT-ADDS

When you run that command, the system will appear to be doing not much of anything for quite a few minutes.  If your experience is like mine, it will appear to be stuck at 10% for most of that time.  In my lab, with virtual machines that were resource starved, it took about 10 minutes.  Wait until you are returned to the command prompt (C:\>) before you move forward or the tools will not be installed and the Exchange validation tool will error out.

Next week, I’ll walk you through a basic installation of the Exchange 2007 server will the 3 main roles on one box.  Until then, have fun getting prepped.

OAB – Oh Boy! Yet another annoying error conquered

Exasperation no more!  OAB 0x80190194 error solved - finally

As many of you already know, I do a lot of Exchange configuration in my day to day job (see disclaimer below).  From time to time, I run into errors that seem to occur over and over, regardless of the environment, until someone can finally figure it out.  Outlook attempting to download the Offline Address Book (OAB) has usually been one of those odd errors, especially when a server is first created.

Starting in Exchange Server 2003, the OAB was only generated for the first time when the server entered the first maintenance window after installation.  So if you built a brand new Exchange environment, you would need to wait until somewhere between 1 and 3 AM before you had an OAB.  This meant that Outlook clients would exhibit a strange error about "File Not Found" during every synchronization.  Outlook went to look for an OAB to download, there was none there, and the sync would abnormally end.

This was easy to fix, however.  You simply located the OAB in the Exchange Management tools and forced a rebuild, which immediately cleared up the problem.  In rare circumstances you followed the TechNet articles and removed the OAB and created a new one, then forced a rebuild.  All in all, not a very difficult fix, but one that eludes many Exchange Engineers and Administrators for hours and hours until they either figure out the fix from reading about it on forums; or the give up in frustration and see the miracle of the error going away the next day (since the OAB auto-rebuilt during the night).

When Exchange Server 2007 came along, I had hoped that this would be a thing of the past.  After all, the least that Microsoft could do would be to put a message in the installer about having to rebuild the OAB. The best would be to have it auto-generate immediately on install.  Alas, as most of us in the Exchange world figured out, neither is the case.  On a new Exchange build, you still have to manually force an update of the Offline Address Book.

It is a lot easier now, though.  One Powershell command does it all:

Update-OfflineAddressBook -id "Default Offline Address Book"

Since the first OAB is always "Default Offline Address Book", the command is universal and can just be done immediately on finalizing your installation of the Exchange software.

All was well until I started getting a stream of calls from colleagues across the country about a new error cropping up in Exchange 2007 and Outlook 2007.  These folks had already regenerated the OAB manually or waited until after the first maintenance window, and now - unlike the File not Found errors before - they were getting a very odd error indeed:

Task 'Microsoft Exchange' reported error (0x80190194) : 'The operation failed.'

Google searches turned up suggestions for Microsoft Update, the Windows Update Site, and a few other red herrings.  Further investigation led only to the usual recommendations to regenerate the OAB.  This was driving many people, including me, nuts!  Remarkably, for several months none of us found an answer, but in all cases the problem went away on its own. Sometimes in a few days, sometimes in a few weeks, but always resolved itself in time. 

Now, my readers know me, that's not quite good enough for my palate.  So when time finally permitted I did some deeper investigation.  I found several things out that had nothing really to do with the error, but are fun:

- Avira Antivirus considers Experts-Exchange.com a phishing site.

- People tended to suggest everything short of lighting candles, standing on your head and typing mysterious incantations into Powershell (though frankly some of the stuff was pretty close)

- Many folks just let it go away on its own and never though about it again - until they had to build a new Exchange system.

Finally, I tripped over the MSServerAdmin site and to this article:

http://www.msserveradmin.com/how-to-overcome-the-exchange-2007-oab-issues/

Which got me to look at the Microsoft Exchange File Distribution service on my Client Access Role server (which in my case was on the same box as the Mailbox Role).  Once I stopped and started that service, Outlook immediately began to download the OAB and all was well once again.  That's really all there is to it.  This only seems to happen on Server 2008, which explains why I wasn't seeing it back when most folks were installing Exchange 2007 to Windows 2003.  I do not have an answer as to why this problem is so common with Server 2008, but if any of the Microsoft folks who read this can tell me, I'll happily post an update.

So, why did it go away mysteriously by itself over time?  That was a little more of a quandry, but I got that answer too.  These days, it's not usual to have to reboot an Exchange 2007 server after installation is finalized.  So folks were not restarting services regularly by rebooting the boxes regularly.  However, eventually, Windows Update Services would download a critical OS patch, and - left to its own defaults - automatically reboot the box.  This might happen the next day, or might be several weeks away, but eventually we'd reach Patch Tuesday and the boxes would reboot - restarting the service and clearing the problem.

Good things come to those who wait - this time.  Hopefully the answer will always be as simple as "Just wait and it'll fix itself," but until then I shall keep searching for answers and reporting them here.

Anyway, Anyhow, (Outlook) Anywhere

For some strange reason on my a classic rock kick these last few weeks, so today’s posting title comes from another song by The Who.

No clue?  Watch this YouTube video.

And listen for this lyric:

Nothing gets in my way / Not even locked doors / Don't follow the lines
That been laid before / I get along anyway I dare / Anyway, anyhow, anywhere

Outlook 2003 and 2007 offered a new approach to Exchange Server connectivity that did not exist in the wildest dreams of earlier versions of Exchange and Outlook.  These two versions of Outlook (and the upcoming 2010 version as well) have a feature set called Outlook Anywhere, previously known as RPC over HTTPS.

Since the dawn of distributed email systems, there has been a schism between the need for connectivity and the need for security.  Most servers can be isolated from the outside world in some way.  Many can be completely locked down and held within the corporate network.  Others can have limited web-based connectivity via tightly controlled front-end servers, allowing the sensitive back-end solutions to be protected inside the network.  The major issue for email was that the systems are inherently designed to communicate with the outside world.  In fact, it’s the whole point of the exercise for the majority of Exchange users.

Microsoft, to their credit, has created a series of systems that allow Exchange 2007 server to only expose the bare necessities to the outside world with the new Edge Server Role on that platform.  But not every company can afford that type of infrastructure, and it wasn’t available before the Exchange 2007 Server platform.  So most companies these days cannot take advantage of that role, and even when they do, the limitations of the Edge Server may not make it the best chose for remote users.

Before Exchange 2003, the alternative was to limit connectivity between the Exchange system and the world at large by using port control on the server side, and VPN connectivity on the user side.  This allowed for Exchange to see only what it needed to on the net, but also required administrators to set up and manage VPN software on all their mobile client laptops, home-office desktops, etc.  It was definitely a trade-off if all the users really needed to use the VPN for was Outlook.

The came RPC over HTTPS.  This technology lets MAPI RPC calls (the communications protocol Outlook uses natively until the 2010 version) to go over the public network in a secure manner.  Essentially, the Outlook 2003 or 2007 client will use HTTPS security – much like you would in a web browser to communicate with a bank or other secure website – but in this case it is Outlook talking to Exchange.  It is secure, reliable, and requires no other software on either server or client side.  It does require enterprise CAL’s in Exchange 2007, so keep that in mind, but it doesn’t need to have anything outside of Exchange Server 2003 or 2007 installed on the servers themselves.

With the advent of Exchange 2007, RPC over HTTPS became a more accepted part of Exchange and gained a formal name – Outlook Anywhere.  The system is simple. Outlook is configured to make a secure HTTPS connection to the Exchange Server or (in 2007) the Client Access Server/Hub Transport Server.  This lets the administrator strictly control access both via security certificates and limiting of what ports have to be exposed, but let end users function with just Outlook and no 3rd party or other VPN client.

To configure, you alter your Outlook Profile, under Account Settings|Microsoft Exchange account|Advanced Settings in Outlook 2007.  Once there, open the Connection tab and click the check box near the bottom to enable HTTP connections.  You will also need to click the button under the checkbox and supply the connection URL for your server. Typically, this is your Outlook Web Access URL, though you can configure a specific record if you prefer.

There are several settings that can help improve network function over slower links, but the defaults should be fine for most networks.  Note that you MUST have the ability to connect to webmail via SSL (in other words https:\\mail.yourdomain.com and not just http:\\mail.yourdomain.com).  Since the data needs to be encrypted, you have to have security set up beforehand.  Though, you can use a self-signed certificate, that will cause your end-users to have to click through a security warning when they log in, so obtaining a public SSL certificate is a good idea.

Once configured, Outlook Anywhere will allow normal Outlook functions to occur as if the users were on your corporate network, but with all information between them and your system secured and limited to avoid potential attack.  It is a great solution where setting up a VPN communication tunnel for each remote user just isn’t feasible or where the only reason to do so is for Outlook communication.

Microsoft has made a lot of good progress in helping secure an application platform that – by its nature – must communicate with the outside world in order to function properly.  Outlook Anywhere is one way that this type of communication can occur, and is a great way to allow your users freedom to get their work down without requiring specialized network technology to be put into play.  Anyhow, anyway, anywhere – a thought never more true for Exchange and Outlook.

Avoid bumpy rides with Exchange Online

I’m posting from a hotel room in Hartford, CT, getting ready for a meeting with a client tomorrow.  Getting here reminded me of why you might want to choose to not try to go it alone when you set up Exchange Server for your organization.  More details on the trip later.

One of the biggest problems I see clients facing when they deploy Exchange Server is the simple fact that it’s a huge software package.  In reality, it’s more of a suite of tools than one application, and therefore requires either pre-existing expertise, or a very fast ramp-up to be able to properly install, manage and maintain the solution set.  Luckily, there are options today that were simply not available in the past, namely Hosted Exchange Services (a.k.a.. Exchange Online).

Hosted Exchange is – at its heart – renting space on a shared or dedicated Exchange Server that is managed and maintained by a 3rd party.  These are hosting providers vetted by Microsoft (or at least they should be) and skilled in creating, deploying and managing Exchange Server solutions. No hardware on-site, and no major management needed by your own tech people.  So what are the pros and cons of using a solution like this.

The major con is that you have to keep your data on another server system that doesn’t reside within your firewall and security network.  Though the reality is that anything which requires connectivity to the outside world is technically open to attempted attack, some companies may not be able to tolerate hosted data.  Either corporate or external compliance policies and laws may prohibit this, so be sure to check with your compliance staff to make sure it’s ok before you ship the email off-site.

Barring compliance issues, security may still be a concern.  Make sure you completely check out your hosting provider, make sure they are a Microsoft Partner and make sure to get references before you sign over your digital life.  It’s totally fair to put the hard questions to your hosting provider, as you’re trusting them with a tremendous amount of sensitive data.

Beyond those two potential cons, the pros are pretty deep.  Many small to mid-sized organizations may attempt to host an Exchange Server in their own network.  Due to inexperience or just plain lack of time, patches are missed, settings are mis-configured, security concerns are overlooked.  This leads to either a poorly functioning Exchange system, or worse, an attack against the organization using the messaging platform as the attack vector.  Larger organizations who have dedicated messaging staff can afford to have people who look after these details, but smaller shops can’t, and often the company infrastructure suffers because of it.  I’ve discussed a lot of potential pitfalls to running an Exchange environment – this is a well-rounded way beyond most of them.

As a matter of fact, when talking with Jon Orton – one of the TPM’s that works with Exchange Online for Microsoft – he related to me that “we’re seeing more and more people, especially in the “small Exchange deployment, not much expertise” category, signing up for Exchange Online and partner-hosted Exchange.”  By placing the infrastructure in the hands of a team of experienced Exchange engineers, you remove that burden from your IT staffers.  Don’t get me wrong, I believe anyone can indeed learn to manage an Exchange Server messaging and collaboration environment However, if you do not need to, then this is one thing you can get off your plate and off your mind – safely.

So, even though it might be very tempting to set up Exchange Server within your network, the facts that it is readily available for purchase and that server hardware is getting cheaper should not be the major factors in decision.  Hosted Exchange Services with a trusted provider are easier to work with and just as readily available, and with a little legwork ahead of time, will be a better solution for many small and mid-sized organizations.

Which leads me to my harrowing journey today.  When considering how to get to Hartford from my native New York City (Queens, to be exact),  I had a few choices.  All were pretty easy.  I could take a train to a mid-way city and hitch a ride with a co-worker going to the same meeting from there.  I could take a different train and then a quick shuttle here.  But, because it was readily available and easy to set up, I chose to take a bus directly out to Hartford.  Much as with the decision to leap before you look and set up an Exchange Infrastructure on-site, I nabbed the tickets and got on the bus without a lot of pre-existing experience with the bus route.  After getting bounced around the bus for about 3 hours, and nearly losing the sandwich I had grabbed at the terminal, I finally got off the bus from hell and collapsed into the hotel room.  Had I researched my options a little better – and sought out some experience from others – I would have realized that a little more legwork would have led to a much better experience for me.

So do a little legwork before you put in your Exchange Server.  Know what you’re getting into and know that there are great options for getting out of the situation.  Exchange Online with a trusted provider can offer a convenient, safe and workable option for those who are just not quite ready to roll their own.

Exchange gets tired.

I’m not talking about getting old here, but rather about exhaustion, which can happen in even newer versions of Exchange Server. Checkpoint exhaustion is a function of the ESE database system set used in Exchange, and designed to stop the system from corrupting itself if too many database transactions get held up in a log file by anything.

The normal sequence of events is well known for most relational databases.  Information added to a database is sent to a so-called Lazy Page Writer to minimize disk access by aggregating I/O operations.  When this process happens, a write-ahead logging system tracks the operations, so that if they need to be replayed, they can be safely re-generated if the Lazy Writer data is lost.  As the transactions are committed to the database itself, the log systems note this by checkpointing those transactions, essentially noting that they’ve been safely committed to the database.  For most versions of Exchange Server this happens every 20MB or so (that’s 20MB of log data, not database writes).

When things do not go according to schedule, the logging systems can recover data that was in the Lazy Writer but not yet committed to the database with emergency systems.  Some, like the normal startup process for an Exchange database, are done automatically.  Others, like ESEUtil rescue operations are done manually, and hopefully avoided entirely.  But, the point at which the logs logjam up so much that the system forces emergency procedures can be more difficult to explain.

The short answer is that Exchange can log up to 1,008 transactions before Checkpoint Exhaustion hits.  This is the point where Exchange can no longer just keep writing to the logs and still know that those transactions can be safely recovered even if the Lazy Writer doesn’t have the transactional data anymore.  If the system sees that the logs are tracking more than 1,008 transactions that have not yet been checkpointed, it will forcibly dismount that database – in this case Storage Group and related Stores.

Now, when does Checkpoint Exhaustion happen?  That’s a bit trickier to explain, but there are a few definite times you might see it.

1 – The system is horrifically under-powered in terms of how fast it can get data out of the Lazy Writer and onto disk.  This is rare, but can happen if there is a failing RAID controller or other disk-device.

2 – Something has placed a lock on log files or databases.  In this case, the offending application is stopping Exchange from writing to either a log or database, or at the very least stopping the checkpointing process from happening regularly. Since this means that Exchange cannot successfully checkpoint the database, logs continue to build and build without being “caught up.” This is also not common, but can occur with management tools that go awry.

3 – Backup systems don’t work as expected.  With most Exchange-aware backup tools, the backup system prevents Exchange from checkpointing during the backup operation.  Normally, this is a good thing.  Checkpoint prevention means that no new data will be officially committed while the backup is in the process of…well…backing up.  This means that a point-in-time tape or disk backup can be transactionally intact, even if it cannot maintain strict write integrity native to the backup solution.  Since the Lazy Writer and the log files both have copies of the information, as long as the block is lifted at the end of the backup, all is safe and all is well.

However, if the backup barfs and never lifts the lock, you could have a major issue.  Since no new checkpoints can be set, unchecked log data starts piling up, and eventually you’ll hit the 1,008 transaction limit and see a forced-dismount happen on that Storage Group.  Luckily, restarting the Exchange services nearly always clears this problem, and since the logs still have a copy of all non-checked data, you’re not going to lose anything but time.

Checkpoint Exhaustion was built to solve a particularly thorny problem.  The earlier versions of Exchange Server that are still supported had a hard-coded limit to 1,024 transactions before the database would fault (not just go offline).  That meant that if the Checkpoint Depth hit that level, you could lose data.  So the exhaustion systems allow the database to safely shut down without losing track of any data – which is a not-so-great but definitely better than data loss type of solution set.

So, to avoid exhaustion, there are several things you can do.

1 – Make sure you’re not overloading your Exchange Servers.  Stay within best-practice guidelines for Microsoft Exchange versions, which can be found on TechNET. and be sure you periodically check for hardware issues.

2 – Try to minimize the use of applications that could lock the Exchange Server file sets.  3rd-party non-Exchange indexing for example

3 – Make sure your backup runs are completing properly. One bad run that never got caught can snowball into an exhaustion situation easily.  If you do have a bad backup run, declare a maintenance window of 20-30 minutes and restart the Exchange services, or even gracefully bounce the box.

 

Don’t forget, if you have an Exchange related topic you’d like to ask about, drop an email to miketalonnyc@gmail.com .  Next update should be on Monday or Tuesday of next week, back to my usual schedule. 

A special hello to Larry Meister at Mercury Networks in Rochester, NY, thanks for inviting me to speak at the White Hat Security Day conference.  It was a great event, as usual!  If you’re looking for a partner for your IT needs in Western NY State, Larry is your guy.

When your cluster goes “oops,” Using RecoverCMS

First, a quick note:  I’m posting this one from Windows Live Writer on Windows 7 RC1, which I’m happy to say is remarkably stable and much faster overall than Vista.  I’d recommend it wholeheartedly!

Funny story, I once had a client who swore that clustering was enough protection for their messaging environment, until an outage took out their entire cluster at once – causing them to be down for about week.  Now, that’s not the funny part, but what caused the outage is somewhat hilarious, more on that later.

Exchange 2003 and earlier had a pretty straight-forward method for recovering an entire MSCS cluster if one had failed on you.  You built one or more nodes of a brand new cluster, created an Exchange Virtual Server (EVS) Resource Group with the same parameters (names, IP’s etc) as the production system had, and Exchange would do the rest.

With Exchange 2007, the rules changed significantly, leaving many cluster users confused as to how the system now works if they suffer a cataclysmic failure of the production cluster.  Adding both Single Copy Cluster (SCC) and Continuous Cluster Replication (CCR) to the mix just makes things more confusing, so Microsoft created a new recovery method for Exchange 2007 clusters.  Called RecoverCMS, the system is really a setup task rather than a true failover system, but since your failover system just went belly-up, that’s not a bad thing.

If your Recovery Time Objectives are flexible enough to handle some downtime if an entire cluster fails, then you can leverage this system to get back up and running, either at the original production site, or at a new location.  There are some definite limits to what you can do with it which I’ll explain later, but he basics of how it works are pretty simple.

Step one is rebuild, repair or replace the original cluster hardware. If the repair works then you’re done, just restore any missing data from tape or other backup (due disclaimer, see below, I am biased on backup tools) and then resume normal operations. If you rebuild or replace completely, bring up a new server that is configured with Exchange 2007 in the Passive Cluster Node configuration.  You can find out how to do that:

Here for CCR clustering or,

Here for SCC Clustering

During that process you will also have installed the Exchange 2007 binaries on at least one node of the cluster system, so go to the directory that has the Exchange setup files and execute the following command:

Setup.com /recoverCMS /CMSName:<name> /CMSIPaddress:<ip>

Where <name> is the name of the EVS you’re restoring from, and <IP> is the IP address you want the recovered system to have – in theory the same IP as the original EVS had.

The rest of the procedure is pretty automated, and when finished, you will have a new EVS running on your new cluster node(s) that matches the original EVS and has all the users already assigned to it.  From there, you can restore your data if it was also lost to the disaster.

There are a few things that are extremely important to be aware of before you begin:

1 – Keep in mind that /recoverCMS is designed to restore a failed cluster only.  Attempting to use it for migration or for any other purpose will result in unpredictable behavior and is not supported by MSFT.

2 – You will need to manually create the volumes that existed on the failed cluster before you run /recoverCMS.  If volumes are missing then the recovery will fail.  They don’t have to be the same physical disk or size, just large enough to hold the data and with the same drive letters as the original cluster held.

3 – The System Attendant service will start and then immediately stop after you recover, this is normal, just bring the resource back online when you’re ready.

4 – Your databases are not mounted after a recovery, you must do this manually through PowerShell or the Exchange Management Console after you’re done with the restore.

5 – Do NOT try to use this across OS’s. If you started on Windows Server 2003, you must recover to Windows Server 2003, and 2008 to 2008.   It will not work if you try to go from one to the other.

6 – While you can pre-configure many portions of this system, it will still take some time to run through a /recoverCMS procedure from start to finish, so if you need a second-stage failover, /recoverCMS isn’t the best bet.  I’m quite biased on this (see disclaimer below), but unless you can be down for a few hours if both cluster nodes fail, you might want to go with another tool to provide remote site failover in addition to SCC or CCR clustering.

7 – Finally, SCR and CCR will not automatically work with /recoverCMS.  You will need to stop SCR if it’s running before you recover, and neither will resume automatically after the recovery is done.  Once you’re set up in the new node configuration, re-enable CCR and SCR manually as required.

/RecoverCMS is a great way to restore a failed cluster system to new hardware or rebuilt hardware after a fault.  You still need to back up your data to some device outside the cluster itself, but once you have that backup /recoverCMS can get your cluster back up and running much faster than the manual methodologies used in previous versions of Exchange.

As to the funny story I mentioned at the top of the blog, this particular client was in a hardened datacenter with UPS systems, 24/7 staff and a backup generator.  They were convinced that clustering was going to be more than enough for them.  After trying to explain that a shared-disk cluster (the only option at the time) had weak points, I finally gave up and let them be.  A few months later I got a great phone call.  Apparently – unbeknownst to the client – the datacenter crew had run all power connections through the UPS – including the generator.  The UPS was rated to handle the full power load of the datacenter on 1 of its 2 redundant circuit loops.  So far so good.  Well, this particular datacenter was in the middle of the dot-com boom (this was some time ago) and had grown exponentially in a short period of time.  What they had was well over half the full expected load on each of the two circuits, and one was failing.  So they diligently got replacement parts and moved the load over to the good circuit.  Since was over half the expected load, and circuit 2 was already under over half the load, they immediately overloaded the UPS, shorting it out.  The way it was explained to me, a solenoid shot through the casing of the UPS…and there was indeed a nice hole in the unit to back that up. No one was hurt, but needless to say, the whole datacenter was offline until they replaced the UPS, 4 days later, so they lost about one business week, without anything happening to the physical cluster at all.  Just goes to show you that anything that can go wrong, will.

Public airing of Exchange laundry...

Last week we finished up talking about what happens to Private Mail Stores during Online Maintenance (OM) on your Exchange servers.  While most people really tend to focus on what happens in those Private Stores the Public Stores are just as important, especially if you're not on Exchange 2007 yet.

For all versions of Exchange, Public Folders experience most of the same processes that their Private Mailbox cousins do, with the Exchange system scavenging items that are marked for deletion and have passed deleted item retention to start.  The system will also get rid of expired folders - a similar process to deleted mailbox systems in the Private Mailbox Stores - and perform an online defragmentation of the databases.  General maintenance and cleanup for Public Folders should be considered to be the same priority as OM on your Private Mailbox Stores, even if you're on Exchange 2007.

With Public Folders in Exchange 2000 and 2003, and even in 2007 in some circumstances, there is another vital process that goes on each day. It is not technically part of OM, but should be considered on par with the OM process.

As many people notice, the first time you boot up a newly-installed Exchange server, Outlook will complain about not being able to find "a required file" when you do a send/receive operation.  This is because - unless you're using a native Exchange 2007/Outlook2007 environment - Outlook cannot find the Offline Address Book (OAB) which doesn't get created in the Administrative Public Folders until around 5am the morning after the Exchange server goes online.  While not officially part of the OM process, the creation and subsequent updates of the OAB do happen overnight - 5am by default, just after the OM process is typically completed. 

OABgen (the process that - wait for it - generates the OAB) should be considered part of your OM systems. If you do not allow this to occur regularly, your OAB doesn't get updated and therefore your end users don't have the latest information - causing no end of frustration for you and your technical staff.  You could manually create/update the OAB using tools inside of Exchange, but the automated process is still your best bet for making sure these critical systems get updated regularly.

And one final note.  For those who believe they can get away with no Public Folders and/or PF maintenance because you are on Exchange 2007, be aware of one very important caveat.  All Outlook clients must be on Outlook 2007, and no Entourage or other MAPI clients can be used. That's because all MAPI systems before Outlook 2007 required the use of Public Folders for administrative information, like the OAB and other components.  In Outlook 2007, that's handled by Active Directory and Exchange itself, but if there is even one legacy client in your environment, you'll still need at least one PF tree, and therefore need to perform regular Online Maintenance on those Public Folder Stores.

Next week, back to regular blog topics - and speaking of which, if you have a topic you'd like me to cover, go ahead and drop me a line at MikeTalonNYC@yahoo.com

Mailbox Separation Anxiety

As we have been discussing over the last few weeks, Exchange 2000-2007 runs through quite a few operations during the Online Maintenance (OM) process.  We've talked about how databases get cleaned up, and how messages get deleted (temporarily and then permanently).  This week, let's look at Deleted Mailbox Retention (DMR) and what it can do for you.

DMR is the process that Exchange uses to remove mailboxes that have aged past their expiry date in the Private Database.  These expired mailboxes can be created by different operations within Exchange, either directly by the administrator of Exchange, or by operations in Active Directory that inadvertently cause a mailbox disconnect.  Disconnected mailboxes are simply mailboxes that not longer have an associated user or object in Active Directory. The simplest way to do this is to delete a user account in AD, which will disconnect the mailbox assigned to that user as part of the process.  However, this isn't the only operation that can cause mailboxes to become marked for deletion.  There is an odd side-effect to moving mailboxes that can happen if you jump between AD sites and don't finish fast enough (You can read about it here), and of course; if you get hit with AD corruption it could disconnect mailboxes as well.

Once a mailbox is disconnected, the first OM run that happens after the disconnect will mark the mailbox for expiry in a user-defined number of days. The default is 30 days, so if you run maintenance daily, the mailbox will stay in the database, but disconnected, for 30 days after it is disconnected from the mail-enabled object (like a user) in AD.  It also means you have to take when you do OM into account for your deleted item retention numbers.  For example, if you only do OM once per week, then you will need to add up to 7 days to your overall retention time for planning purposes.  Since the 30 day clock doesn't start ticking until the mailbox is marked during OM, you will extend the amount of time data remains in the database.  This is critical for regulatory compliance, as holding data for an extra week could put you over the line in terms of meeting your regulatory guidelines.

The other part of this OM process happens when Exchange scavenges the database for expired mailboxes.  If it finds one, and the mailbox hasn't been re-assigned to another AD object, Exchange will delete the data from the database permanently and create white-space in the database that gets cleaned up by OM to be reused by Exchange.

So, here again, it becomes vital that OM is permitted to run on a regular basis.  If not, deleted mailboxes will not be removed from the database unless you manually launch the cleanup wizards.  This is more than just a regulatory problem, as without removing the old information and freeing up white space, your databases will grow much more rapidly than if you are regularly cleaning up your mail stores.

Next week... Public Folder operations during Online Maintenance, so stay tuned!

If I cannot bring you comfort, then at least I bring you hope

As we get ready to celebrate the beginning of a new year (in many countries), now is a good time to plan for your Exchange Servers and what you'll be doing in the year ahead.  While this is not the best time to plan out new infrastructure or massive overhauls immediately, you can start planning what you'll do later in the year on that scale.  You can also plan for what you'll do right off the bat in January for things like maintenance and updates/upgrades.

I plan to dedicate entire columns to these topics in the next few weeks, but here's the overview to get you thinking (once the champagne wears off...):

Offline Maintenance is a good thing to do regularly for Microsoft Exchange.  About once or twice a year you will probably want to take the time and walk through the process.  Microsoft says that this is not a solid requirement, but it can help identify long-trending problems deep within a database, so doing it once or twice per year is time-consuming, but worth it.  Plan now for downtime, so that you don't have to worry about when you'll have the ability to do it later. 

Online Maintenance is automatic, but you may want to look at your Exchange Server profile and see when the best time to have it done really is.  By default, the online maintenance system will run every night, but if you're also performing backups overnight, the two systems can overlap, causing the maintenance to be halted and never finish.  Take a look at your event logs to see if both systems are operating normally, or if the online maintenance run is terminating early.  If it is, you may want to change the times it runs, or even only run it on certain days.  Unlike with Exchange 2000, later versions of Exchange can hold up admirably well with even weekly maintenance runs, but be aware of what won't get done until maintenance by checking out Microsoft's Knowledge Base and/or waiting until my upcoming column on the topic.

Test, test, test, test, test......then test again! Now is the perfect time to set out your goals for testing of recovery/availability solutions (see disclaimer at the bottom of the page).  Backup systems, failover systems and other recovery solutions MUST be tested, otherwise how do you know they're doing what they say they're doing.  Best bets: do a test restore of Exchange data (at all the levels available from your tool like database, mailbox, mail item) to a different directory on the Exchange server and mount the DB in a Recovery Storage Group to ensure it is working properly.  This should most likely be done once per month, but at least once per quarter.  It doesn't require downtime, and doesn't require end-users to do anything.  Perform a failover test (if you use a failover tool) or a total restore test at *least* once per year.  2x per year or once per quarter is better, but of course since this type of testing will impact end-users, scheduling can be an issue.

Housekeeping is never a bad thing to look at after the clock strikes midnight on January 1.  Are you reaching your database limits in Exchange Standard edition?  If so it's time to either start ratcheting down mailbox limits or else starting to plan your upgrade strategy.  If you're already on Enterprise edition, are you pushing the limits of the free disk space you have on your drives?  Plan now to move the databases to another partition with more space.  Have your Stores grown to alarming proportions?  Might be time to start planning out new Stores and even Storage Groups to distribute the load.

Finally, give a look through the event logs and such to see if there are repetitive errors or other trends that can indicate problems brewing.  One more common example of this is recurring event ID's 1018, 1019, and 1022.  These indicate that there could be problems brewing in the database itself, though it is very possible that the error events are the only indication you see until the DB crashes.  If you're looking for a tool to help wade through the log information, check out Splunk, an IT search solution set that can make your life a lot easier.  [Due Disclosure: Splunk is currently working on partnerships with Double-Take Software.]

Have a healthy, happy and safe New Year's celebration.  Here's hoping 2009 exceeds everyone's wildest dreams in terms of greatness, and that Exchange never ceases to help us all live in these very interesting times.

Until next year.....

Mike Talon, Chief Columnist, BeingExchanged.com

PS: The title of this posting is taken from the song "Closing of the Year" from the movie soundtrack to "Toys."  The lyrics of the first verse are incredibly relevant to our combined experience this year, and I offer them here:

If I cannot bring you comfort,
then at least I bring you hope
For nothing is more precious than the time we have, and so
We all must learn from small misfortunes
Count the blessings that are real
Let the bells ring out for Christmas
At the closing of the year

- The Musical Cast of Toys Featuring Wendy and Lisa

Under Pressure (da da dum diddy dum dum)...

First off, welcome to the new subscribers who hopped on-board with BeingExchanged!  Also, congratulations to everyone who recognized the riff I used in my Subject line, and for those who don't, it's referring to This Song, and don't we all feel old?

Back Pressure is a phenomenon in Exchange 2007 that can grind your email flow to a halt with little or no warning, so it's a great thing to begin to understand now, before you run into it in the real world.  The basic idea is that Exchange Servers through the years were famous (sorry in advance to the guys from Microsoft) for collapsing under heavy I/O load and/or lack of disk space.  Back Pressure allows the Exchange 2007 system to halt some of the processing of data temporarily to allow the system to keep running without the danger of I/O overload.

Specifically, Back Pressure allows Exchange Servers to stop accepting new messages into transport queues, thereby allowing system resources to free up over time and for those queues to empty out.  This translates into no mail coming or going from your organization while Back Pressure is in effect, which will have quite an impact on your operations.

While the Back Pressure system could impact your mail organization as a whole, it's good to point out that only Hub/Transport and Edge Services servers are impacted.  Messaging queues and the disks they sit on are what the Back Pressure systems monitor, not database locations.  Pure Mailbox Role (MBX) servers won't experience Back Pressure, but they will be impacted by it if mail cannot flow in or out of the MBX servers due to mail being unable to transverse the other roles.

When the Back Pressure system kicks into gear, the most noticeable impact is that Outlook clients will have all their outgoing mail stuck in either the Outbox or Drafts folders.  OWA will show everything in Drafts and other client software will show that mail isn't being sent.  Of course, no mail is coming in either, but that's a little more difficult to "see" unless the user was expected a message.  On further examination, you will see that no mail is coming into or going out of mail queues on the Hub Transport or Edge servers, a dead giveaway that Back Pressure kicked in if your connectivity is working just fine but no mail is moving.

There are three ways Back Pressure can be brought online on an H/T or Edge server:

First off, if the logical disk that contains logging and other information for either role runs below a set level of available space, Back Pressure will be invoked.  This is done to keep Exchange from "crashing" a disk, or filling up all space available and causing problems within the Exchange system or (if it's also the system drive) Windows itself.

Next, if the amount of physical memory or other system resources run low; Back Pressure comes online.  Since mail isn't coming in or going out, the resources will return to a normal level once whatever process is spiking finishes or is terminated.

Third, the number of pending operations in the memory queue (as opposed to the overall amount of memory in use) can build up if something is slowing down Exchange like a virus, DDOS attack or just a sudden influx of mail transfer.  If the depth of these transactions exceeds a limit set within Exchange, the result is Back Pressure stopping mail flow, allowing the existing transactions in memory to commit through to disk without new transactions coming in.

Each of these metrics has a different number of resources assigned as the "break point" where Back Pressure kicks in.

For disks, Exchange uses a formula to determine what space must be present and free before Back Pressure is invoked.  Specifically the formula is 100*(hard disk drive size - fixed constant) / hard disk drive size. The "fixed constant" number is 4GB for the release version of Exchange 2007, 500MB for the SP1 version. If the total available free space on the drives which contain queuing systems drops below that number, new mail flow stops and Exchange system spools out the queued data until the disk systems free up enough space to get above the threshold.

Finding the amount of memory used to reach the "high" utilization number, or the depth of the queues is a bit trickier.  There are several calculations and settings used to figure those out, and a complete description can be found in this article about Back Pressure technology in Exchange 2007.

The end result is the same.  Build an Exchange 2007 server with the wrong sized resources (disks, memory or server load), and the system will effectively halt itself to make sure that those limited resources don't crash.  So it is vital to properly size out both your expected user load, and the resources (physical and otherwise) that you're planning on running that load on.  You can change the thresholds for the Back Pressure triggers (see this link) but that's designed to be more of a temporary fix than a permanent change.  Instead, work with the publicly available sizing guides from Microsoft and other sources to make sure your hardware can handle the mail transfer without feeling the pressure.

More than zero?

There are a ton of interesting settings and switches in Exchange 2000/2003, but one of the more unusual is the "Zero Out Deleted Database Pages."

Exchange 2000 and higher all have several levels of potential access points for un-delete operations.  When anything is deleted from a mailbox, the data isn't truly removed, but rather marked as scheduled for delete at some later date.  The default settings keep mail items for 15 days and mailboxes for 30, after which point they're removed from the server.

Data pages within the ESE database are another story; however.  They get deleted all the time, but not removed from disk.  They're simply marked as deleted, ready to be overwritten as needed.  Normally, this doesn't pose any specific security threat, as File servers and most other Windows servers do the same thing when data is deleted.  In higher-security environments; however, this could be a policy violation, as the data would be stored "in the clear" and vulnerable  to anyone able to get their hands on the physical disk devices.

Microsoft has taken steps to natively remove the potential for security breaches, but at a cost.  Built into every version of Exchange supported today is the ability to literally overwrite each deleted ESE page with zeros.  The processes is therefore referred to as "zeroing out" the pages, and the "Zero Out Deleted Database Pages" setting controls this behavior.  By default, the setting is not turned on, but can be enabled via the Exchange System Manager GUI, Powershell or via Active Directory.

The problem is that the benefits rarely outweigh the costs in the real world.  Zeroing out pages takes up a good chunk of processor time, and Exchange deletes pages - quite literally - all the time.  Zeroing out will also have a huge impact on any form of replication, just because a large amount of data change is required for the process to work.  In addition, this method of secure wipe meets only very low-level security concerns.  Most regulatory requirements require multiple overwrite passes (7, 15, sometimes even more) and mandate random data be used, not just zeroes. 

So, while you might feel like you'll gain some security by using this setting, the extra overhead and falling short of regulatory compliance should make you think twice before you turn it on.