Bury the Berries? Not yet.

Research in Motion (RIM) changed the game when they introduced their Blackberry devices to provide mobile email solutions for Exchange and other email platforms.  Over the last few years, many other companies have stepped up to produce handheld messaging devices, mostly centered around mobile phones and carriers.  There are so many good competitors these days, that some companies are considering if they want to ditch the Blackberry Enterprise Server (BES) platform in exchange for… well… Exchange.

Exchange 2003 and up allowed for Exchange Active Sync, a series of mobile technologies that allow Exchange servers to communicate directly with mobile devices using push synchronization to deliver email, contacts, calendar items and other information.  Windows Mobile phones are natively equipped with the Active Sync technology; and the iPhone, PalmOS and WebOS, and various versions of software for Android phones have also licensed the Active Sync technology for those platforms. This means that these devices can communicate directly with an Exchange server and send/receive mail, sync contacts and tasks, and answer calendar requests without the need for an intervening server platform like BES.

However, there are two key reasons that I am (at least for now) sticking with my Blackberry device.  First, there is battery life. Talk time is a pretty standard 4-6 hours no matter what OS runs on the phone you’re holding.  But standby/messaging time is another story all together.  With push email active, I have yet to see a Windows Mobile phone go more than 6-8 hours without needing a charge.  If I turn off Bluetooth, push email and just about everything else that uses a transmitter, I can stretch that number to 12-14 hours, but then what’s the point of carrying the device?  Blackberry devices routinely go 24-36 hours of messaging, even with Bluetooth active (but not transmitting).  Android devices can go a bit longer than Windows Mobile, as do iPhones and WebOS systems. They typically run the middle-of-the-road between those two extremes.

Secondly, there is security.  BES allows for a remote device to be wiped natively.  Windows Mobile devices can also be remote-wiped if you’re on Exchange 2007 and up, but iPhone and other OS devices must use 3rd-party software to accomplish this goal.  Remote-wiping is used to provide security if a device is lost or stolen. Once activated, all personal/corporate data on the phone is deleted, rendering it both useless and safe.  Good Technologies has reinvented itself over the last few years as a great provider of remote-wipe and encryption systems for Android and iPhone platforms, but it is one more thing you have to manage in an environment that (based on who would use it) already has a BES server running.

Based on these two factors, I cannot envision organizations ditching BES in favor of an Active Sync solution set exclusively.  Battery life makes these devices difficult for mobile personnel to use successfully, and the need for additional security systems only adds to that concern. For non-mobile users, battery life isn’t as much of an issue (as they’re going to be near a power outlet in many cases).  Security, though, is still a huge concern.  Unless they are on Windows Mobile devices exclusively (no iPhones, no Droids, etc), they’re going to need additional server solutions to be safe.

My prediction is that BES will stay in the Enterprise space for quite some time.  We may see Windows Mobile, Android an WebOS take over more of the marketplace, but until these two issues are significantly improved, they will remain the minority for corporate mobile messaging.

Manage your migrations to minimize co-existence

Exchange 2010 is beginning to get traction out in the real-world, which is by no means a bad thing.  As folks begin to upgrade, though, there is a chance that you’ll end up with extended co-existence between different software versions and packages, and that can make life a living hell for technical staff. Multiple versions of multiple platforms can be difficult to manage at best, and if that management must continue for months (or longer) you could be setting yourself up to fail right from the start.

Co-existence of platforms is a necessary thing. Especially in larger organizations, the likelihood that you will be able to move all your users and systems from one software package to another in a weekend is slim at best. So, no matter if you are moving from a totally foreign system like Notes, or just upgrading between 2003 and 2010 on the Exchange platform, you will almost definitely need to have both systems running for a period of time as you move users and 3rd-Party tools.  The key is to keep that time period as short as you can, and here’s why.

When multiple versions of a messaging platform (or multiple platforms) you must make sure you’re patching both independently, staying up to date on multiple security threats and dealing with end users who have to have changes made across both system sets.  Since it is highly unlikely that you will have the benefit of extra staff during the migration, that means that the existing staff suddenly find themselves doing twice the work.

If the same number of people are forced to do twice the work, things fall through the cracks.  Patches get applied incorrectly or not at all, threats are left unaddressed and shortcuts abound.  It’s easier to open a hole in the firewall than try to deal with two sets of rules to manage connectivity. It’s easier to only focus on news and updates for the newer systems than to try to keep up with the influx of information for both platforms. The upshot of this is; that the longer you have both systems co-existing, the more opportunity you’ll find for something to break on one platform or the other.  This endangers both systems, and possibly everything else in your environment.

So, when planning for your migrations, work as much as you can to ensure that you do not require extended periods of co-existence if they can be avoided.  Sometimes you will need to have extended migration timelines – it’s unavoidable – but wherever you can do migrations quickly, you should.

Why use AD DNS?

My recent article on the need for (and use of) PTR Records in DNS have sparked quite a few questions on using DNS with Exchange Server in general.  The biggest one I get is “Do I need to use Active Directory DNS in order for Exchange Server to work?”  The answer to that one is a bit complicated, but in its simplest form, it boils down to, “No, but you really, truly should.”

Exchange Server 2000 and up required some form of DNS in order to function correctly.  This is mainly because the Windows Internet Naming System (WINS) was “depreciated” starting with that version of Exchange.  What that means is that MSFT officially asked the community to stop using it whenever possible, because it could be removed completely soon.  As it turns out, WINS was phased out in Exchange 2007, though it may still be required for certain Outlook functions.  That’s a topic for a whole different series of blog posts though.

As for DNS integration, it’s quite possible to install Exchange 2000-2007 without having Active Directory DNS configured in your domain, though it isn’t a best practice.  As long as your DNS system can handle Server Name Records (SRV type records), you can successfully use a 3rd-party DNS for your Exchange environment.  There are, however; some good reasons to go with the native Windows Active Directory Integrated DNS solutions:

1 – Exchange can natively talk to Active Directory DNS, and therefore can do some interesting tricks with that DNS platform that it can’t do with 3rd-party DNS.  Things like AutoDiscovery when you move a user to different mailbox servers, or after a recovery operation with Database Portability just don’t work the same way if you’re not using Active Directory DNS.

2 – Many 3rd-Party tools leverage AD DNS to figure out where Exchange resources are.  Note, I’m far from unbiased on this topic, so please see the disclaimer at the end of the blog.  Since many Windows-based tools will natively use AD DNS API calls (like DNSCMD and the newer variants in PowerShell), you may need to make manual updates to your 3rd-Party DNS, or may have to give up functionality.

3 – Many other non-mailbox objects are stored in AD DNS, and must be mapped manually in other DNS systems in order for Exchange to work properly.  You will have to track your Global Catalog servers, Domain Controllers and other resources in order for Exchange to function.

So, as you can see, there are some very good reasons to use Active Directory DNS if you plan on using Exchange Server.  While you may have external DNS records hosted with an ISP or other provider; internally you will be better off with the native DNS solutions in Windows unless you are ready and willing to fine tune your DNS systems and stay on top of it. 

If you are in doubt, you can use the Exchange Best Practices Analyzer to test your environment before you begin to install Exchange.  This tool will test for many things that Exchange needs, including properly configured AD or 3rd-Party DNS systems.

New Poll: Exchange Versions

Net Neutrality and EAS

Mobile messaging and collaboration makes for a big, bright world in business today. What was once the domain of a single service provider (Research in Motion - RIM) has evolved into a robust set of platforms to convey email, appointments and contacts from one device to another.  Blackberries, Windows Mobile devices, iPhones, Android phones and so many other systems can communicate either directly or through a proxy to a Microsoft Exchange platform.  This unleashes the workforce and allows for your people to be where they need to be in order to work, instead of where they have to be just to talk to each other through the email system.

Net Neutrality is the idea that no matter what the network provider offers in terms of services and software, you should be able to use the devices of your choice and the platforms of your choice on those networks.  It’s a great theory, but putting it into practice is causing some issues along the way.  The FCC set forth a set of basic rules that they wanted carriers to follow, and in the greater sphere of comments, they were well received.  They recently added in two more proposed rules that directly impact cellular networks (digital broadband) and the services that run across it – including Exchange Active Sync (EAS).

You can get a rundown of the entire proposed rule set in this article, but the two that directly impact EAS most are:

5. Broadband providers cannot block or degrade lawful traffic over their networks, favor certain content or applications over others and cannot "disfavor an Internet service just because it competes with a similar service offered by that broadband provider."

On the surface, this looks like a standard anti-competitive rule.  In reality, however, many service providers in the cellular world are viciously blocking competing technologies, and their claim is that forcing neutrality will destroy their business.  EAS is a great example of this phenomenon, as not that long ago many providers didn’t allow that traffic on their mobile networks.  Mostly, this was due to the fact that they wanted to pus their own version of enterprise email synchronization (such as Sprint’s ill-fated attempt on the earlier Palm Treo devices).  Eventually, the need to allow this traffic or lose business to other devices and networks overrode the desire to use and sell their own platform, but that took a great deal of time, and lead to quite a bit of bad press and back-end attempts to circumvent the blocks.  By forcing mobile providers to allow all valid and legal traffic, the atmosphere for open communication standards will grow and more people will be able to take advantage of more technologies.

6.Broadband providers must be transparent about the service they are providing and how they are running their networks.

Proprietary networks are nothing new, but trying to create an EAS client for a phone on a network that actively blocks your ability to figure out how it sends and receives data makes this close to impossible.  Some providers have blocked all traffic they do not wish to have on their networks by simply making it very difficult – or nearly impossible – to figure out how a 3rd-party tool can possibly communicate with it.

I’m of two minds on these proposed rules.  On one side, EAS and other technologies require open, transparent communications platforms to work. Exchange can communicate with a whole world of different vendors’ mobile applications, but only if those apps can talk to the Exchange Server.  On the other side, competition drives better software and platforms.  If it wasn’t for all the things you can only do (or could only do) on an iPhone, RIM and Google would never have had the impetus to push their own platforms to new heights, and we’d still be staring at plain-text emails on black and white Blackberry devices.

It’s going to be a very loud fall season as the mobile providers and the FCC battle out these proposed rules.  The end result will have a huge effect, either good or bad, on how flexible and feasible your mobile Exchange platform plans will be. Competition is a good thing, but it cannot be forced on the market at the expense of profits.  There must be a way to balance these scales, and it will need to be found before Net Neutrality can be forged in the mobile marketplace.

 

 

Big Brother is Watc…[Recalled]

The big kerfuffle over Amazon’s Kindle Team removing several books without warning from users’ devices (story and link at the end of the blog) has lead me to think about the various time clients have used the Recall Message feature. Typically, they have found that it behaved just slightly better than opening their office door and screaming “NOBODY READ THAT!”

There are a lot of great reasons to use Recall Message in Outlook and Exchange.  Perhaps something was sent before it was finished, accidentally clicking Send without typing in that last bit of information.  Maybe you meant to add an attachment but did not – in which case may I suggest some of the great tools at Sperry Software.  You may also have accidentally hit Reply All instead of Reply, or added in a group email list via AutoComplete instead of the one person you wanted to send email to.  All of these are legitimate reasons to really want to get the email out of everyone’s Inbox.

Years ago, Microsoft attempted to build the functionality of yanking back an errant email on command.  It required (and still requires) that you be using Outlook in Exchange Mode, and a Microsoft Exchange Server as your email platform.  When first introduced, this worked great.  There was no such thing as a preview pane, you were always using an Exchange Server and MAPI for communication, and folks were probably online and connected since there was no Cached Mode.

The issues arise when you look at what is required for a Recall attempt to succeed:

• The user must be online and connected to the Exchange Server
• They must not be in Cached or Offline Modes
• The message must be unread in the recipient’s Inbox (including unread by the Preview Pane)
• The recipient cannot have moved the message to another folder (unread or otherwise)
• They must be using MAPI. Even POP3 or IMAP on an Exchange Server won’t count.

So, as you can see, since the majority of email users these days do not meet one or many of these requirements, Recall Message will fail for the majority of your users.  What they will see is the original email (unchanged) and a notice saying you would like to recall the original email.  This can often be just as embarrassing as the original gaff, as it acts as a glaring exclamation point that someone made a goof.

One more note, Recall Message will not impact the copy of the information on a Blackberry or other 3rd-party Smartphones and mobile devices.  So even if the user’s desktop has all the requirements met, they’ll still have a copy of the message (and the recall request) sitting on their mobile device.

In the days of Exchange and Outlook with MAPI connections being the only conduit to email, Recall Message was a great way to manage errant information.  Today, with so many ways to connect to any mail system, including Exchange Server, this feature has lost its usefulness and created a false sense of security.  Remember, if there is a chance that someone’s using a non-Outlook client or not meeting all of the requirements above, you will not recall the mail.  An ounce of prevention is your only cure these days.

As for the Amazon gaff, for those who did not see the news:  Amazon recently pulled all the books by a major author and deleted them off all Kindle devices that held them.  This was due to someone posting the books on the Kindle store, without the publishing rights needed to earn money from them.  Once Amazon realized what was going on, they of course stopped selling all the books from that vendor.  Unfortunately they also removed all the digital copies that were stored on users’ Kindle devices without mentioning it to the Kindle owners.  They did refund everyone’s money, and they did send out an explanatory email, but not until several days later after the blogosphere detonated with Amazon hate postings.

Frankly, Amazon’s user agreement does say that they reserve the right to do exactly what they did.  I can’t fault them for heading off trouble (in the form of a copyright litigation) at the pass. However, it could have been handled a lot better, with more information supplied immediately to end users about what was going on and why.  Amazon has promised to make this process much more transparent so that next time there is no confusion.

The reason there was such an outcry about the process this time (as it has happened in the past, quietly and without fanfare or hate mail)?  The author in question was George Orwell, and the books were the likes of 1984 and Animal Farm.  Both of these books deal with subjects like retroactively removing information that didn’t meet the criteria of an overarching body of government or law.

From Dictionary.com:

Irony: an outcome of events contrary to what was, or might have been, expected.

Diversity takes another one for the team

Belts are tightening everywhere – and though it appears that budget freezes might be thawing out slowly over the next 6 months or so, there are cost-cutting measures all over that are still being brought to bear.  One of the more interesting ones (that wasn’t in my company, that is) was the report that Microsoft was going to be ending its policy of letting employees expense fees for any Smartphone besides Windows Mobile devices.  So no more Pres, no more BlackBerries and no more iPhones.

You can read a blurb about it in this Network World article

These are tough times, and I will not fault any company for tightening the expense account reins.  Standardizing on one mobile technology is a fairly common practice for most companies, and not paying for other technologies is also normal in the business world. So, I’m not quite upset that MSFT has decided to only reimburse for WinMo devices.  Even so, part of me is a bit miffed on reading this one.

MSFT isn’t like other companies – in so many ways.  Specifically here, they make a product line (Exchange Server) that interacts and interoperates with iPhones, BlackBerries and other mobile devices either directly or indirectly.  The diversity of devices that their previous policies made possible was a boon to developers internally who needed to make sure that the latest changes to the Exchange line allowed the more popular devices to continue working correctly.  It’s much easier to see that Exchange Active Sync broke when the guy next to your cube suddenly finds his iPhone stops getting email.

Since real-world businesses are adopting devices like the iPhone and new new Palm Pre (at least if you listen to Palm they are), knowledge of how those devices interop with Exchange Server is critical.  MSFT would be technologically better off having folks using those devices every day right there in Redmond.  The easiest way to ensure that happens is to make them fiscally available to employees.

Granted, when push comes to shove, cost cutting is cost cutting.  Sometimes it’s just necessary to hold back on expenses in order to keep everything else running smooth.  So I do not fault MSFT for making this policy and putting it into effect.  I think that it may have a larger impact than just bottom-line savings, but it is a responsible, viable method for holding back costs while not resorting to more layoffs.

As we put notch after notch in our belts, and as we shakily get back to our financial feet again, there will be more of these cost cutting measures.  A lot of them will be much worse than this, but most will not have such a direct impact on the development of a product platform.  So, hopefully the Exchange team will continue to have Pres and iPhones and BlackBerries running around, even if the company doesn’t officially subsidize them anymore.

Exchange gets tired.

I’m not talking about getting old here, but rather about exhaustion, which can happen in even newer versions of Exchange Server. Checkpoint exhaustion is a function of the ESE database system set used in Exchange, and designed to stop the system from corrupting itself if too many database transactions get held up in a log file by anything.

The normal sequence of events is well known for most relational databases.  Information added to a database is sent to a so-called Lazy Page Writer to minimize disk access by aggregating I/O operations.  When this process happens, a write-ahead logging system tracks the operations, so that if they need to be replayed, they can be safely re-generated if the Lazy Writer data is lost.  As the transactions are committed to the database itself, the log systems note this by checkpointing those transactions, essentially noting that they’ve been safely committed to the database.  For most versions of Exchange Server this happens every 20MB or so (that’s 20MB of log data, not database writes).

When things do not go according to schedule, the logging systems can recover data that was in the Lazy Writer but not yet committed to the database with emergency systems.  Some, like the normal startup process for an Exchange database, are done automatically.  Others, like ESEUtil rescue operations are done manually, and hopefully avoided entirely.  But, the point at which the logs logjam up so much that the system forces emergency procedures can be more difficult to explain.

The short answer is that Exchange can log up to 1,008 transactions before Checkpoint Exhaustion hits.  This is the point where Exchange can no longer just keep writing to the logs and still know that those transactions can be safely recovered even if the Lazy Writer doesn’t have the transactional data anymore.  If the system sees that the logs are tracking more than 1,008 transactions that have not yet been checkpointed, it will forcibly dismount that database – in this case Storage Group and related Stores.

Now, when does Checkpoint Exhaustion happen?  That’s a bit trickier to explain, but there are a few definite times you might see it.

1 – The system is horrifically under-powered in terms of how fast it can get data out of the Lazy Writer and onto disk.  This is rare, but can happen if there is a failing RAID controller or other disk-device.

2 – Something has placed a lock on log files or databases.  In this case, the offending application is stopping Exchange from writing to either a log or database, or at the very least stopping the checkpointing process from happening regularly. Since this means that Exchange cannot successfully checkpoint the database, logs continue to build and build without being “caught up.” This is also not common, but can occur with management tools that go awry.

3 – Backup systems don’t work as expected.  With most Exchange-aware backup tools, the backup system prevents Exchange from checkpointing during the backup operation.  Normally, this is a good thing.  Checkpoint prevention means that no new data will be officially committed while the backup is in the process of…well…backing up.  This means that a point-in-time tape or disk backup can be transactionally intact, even if it cannot maintain strict write integrity native to the backup solution.  Since the Lazy Writer and the log files both have copies of the information, as long as the block is lifted at the end of the backup, all is safe and all is well.

However, if the backup barfs and never lifts the lock, you could have a major issue.  Since no new checkpoints can be set, unchecked log data starts piling up, and eventually you’ll hit the 1,008 transaction limit and see a forced-dismount happen on that Storage Group.  Luckily, restarting the Exchange services nearly always clears this problem, and since the logs still have a copy of all non-checked data, you’re not going to lose anything but time.

Checkpoint Exhaustion was built to solve a particularly thorny problem.  The earlier versions of Exchange Server that are still supported had a hard-coded limit to 1,024 transactions before the database would fault (not just go offline).  That meant that if the Checkpoint Depth hit that level, you could lose data.  So the exhaustion systems allow the database to safely shut down without losing track of any data – which is a not-so-great but definitely better than data loss type of solution set.

So, to avoid exhaustion, there are several things you can do.

1 – Make sure you’re not overloading your Exchange Servers.  Stay within best-practice guidelines for Microsoft Exchange versions, which can be found on TechNET. and be sure you periodically check for hardware issues.

2 – Try to minimize the use of applications that could lock the Exchange Server file sets.  3rd-party non-Exchange indexing for example

3 – Make sure your backup runs are completing properly. One bad run that never got caught can snowball into an exhaustion situation easily.  If you do have a bad backup run, declare a maintenance window of 20-30 minutes and restart the Exchange services, or even gracefully bounce the box.

 

Don’t forget, if you have an Exchange related topic you’d like to ask about, drop an email to miketalonnyc@gmail.com .  Next update should be on Monday or Tuesday of next week, back to my usual schedule. 

A special hello to Larry Meister at Mercury Networks in Rochester, NY, thanks for inviting me to speak at the White Hat Security Day conference.  It was a great event, as usual!  If you’re looking for a partner for your IT needs in Western NY State, Larry is your guy.

Outlook, can you hear me? Can you feel me near you?

Might be showing my age and/or taste in music with that particular title (and if you’re totally confused by it, check out This YouTube video), but I think that it’s a great way to describe an annoyance that can happen if you’re using versions of Outlook before Outlook 2007.  Since a large portion of the users of Office are on the 2003 version (and many even earlier than that), resolution to a new server in the event of a disaster recovery event is a subject that is just as confusing as the famous rock opera I’m making use of in my title today.

When Outlook 2007 was introduced to the world with Exchange 2007, a lot was made (and rightfully so) of the new AutoDiscover features that this platform brought into the Enterprise Email marketplace.  The long and short of the AutoDiscover solution set is this:

When an Outlook 2007 client cannot find its home server – either because it is a brand new install of Outlook or because the home server has moved or been replaced – the Exchange 2007 AutoDiscover system can help Outlook 2007 find its home.  If the Outlook client can see an Exchange Server (or be directed to one by Active Directory), the Server can tell Outlook where the mailbox information for the user’s profile exists, and direct Outlook to connect to the appropriate CAS or Mailbox systems and get connected.  All the user/Admin has to do is tell Outlook the user’s email address and password, and AD with Exchange 2007 will handle it from there.  So if you’re installing Outlook for the first time, you don’t have to manually configure the Profile anymore – a great boon to Admins everywhere.

This system also kicks in if you perform Database Portability during a disaster, and have replicated the database with SCR; or have used a 3rd party disaster recovery/availability solution (see disclaimer below for all my bias information on that one =).  Once the Exchange system is responding again, AD can ferry the Outlook 2007 client to the new home for that mailbox, requiring only that the end-user close and reopen Outlook to complete the process.

However, what many folks do not realize right off the bat is that this solution set is ONLY available if you have both Outlook 2007 and Exchange 2007 as your messaging platform.  All users who need to take advantage of AutoDiscover must be using that combination of tools, and no other.  As you might expect, POP3 and IMAP systems do not AutoDiscover, but the majority of my clients were unaware that Outlook 2003 and earlier also cannot take advantage of this system, even if you have upgraded to Exchange 2007 as the messaging platform of choice.  It’s also worth noting that AutoDiscover doesn’t officially work in Exchange 2003 – no matter what Outlook version you are on.  Before I get blasted by mail on this one, I know some folks have sometimes seen it to work on Outlook 2003 with Exchange 2007, but it bombs more than it works, and officially it’s not supported.  For proof, I direct you to this article by the MS Exchange Team.

Since the code to perform AutoDiscover wasn’t in Outlook 2003, users on that client software will not be able to dynamically re-link to the new Exchange server unless the original mailbox server is still responding.  If it is, then Outlook can find the new server via the original server and re-home itself.  If not, Outlook must be manually re-directed to the new server.

Of course, there are ways around this.  You could update DNS to re-direct anyone calling for “Server 1” to the IP Address of “Server 2” – effectively re-routing all client software including POP and IMAP.  Outlook 2003 will still need to be re-profiled unless you take over the Service Principle Name (SPN) of “Server 1” on “Server 2,” but it will be a smoother transition.  Using a 3rd party tool (see disclaimer below) you may have the option of automated DNS and SPN updates, which will allow even legacy Outlook clients to jump to the new server with no more intervention than is required on Outlook 2007 with Exchange 2007 – even if you’re whole system in on the 2003 versions of those software platforms or earlier.

So, you are not without lots of options if you have any legacy servers and/or clients – or non MAPI clients – in your environment. You just need to be aware that the Exchange 2007/Outlook 2007 solutions for AutoDiscover services are not backward compatible, and plan accordingly.  Right now it looks like Exchange 2010 will have AutoDiscover that is backward compatible to Outlook 2007 only, so this soon-to-be-released platform is not going to solve this particular problem unless you’re planning on upgrading everything else in the environment to at least Outlook 2007 first.

Since I like to let folks continue to discover on their own, here’s a link to the White Paper from MSFT on AutoDiscover.

Finally, Update Roll-Up 8 for Exchange 2007 is out there, which can make life easier if you’re doing a fresh install of Exchange 2007 and want to get up to date with patches and fixes post SP1 quickly.  You can get it at this link.

The Dread Pirate Re-Seed – Part 2

Last week, we talked about re-seeding operations between two nodes of an Active/Passive CCR Cluster.  Since both nodes will most likely be inside the same logical network (though are not required to be in Server 2008), even unexpected re-seed operations on a CCR Cluster should be relatively painless.  Though it’s a full copy to target, it is happening over a LAN, and therefore faster and less resource-intensive than a WAN copy would be.

This week, let’s look at how the game changes when you work with Server (or Standby) Continuous Replication (SCR) when you have re-seed operations.  Just to be clear, re-seeds are not the normal method for data protection with SCR. Normally, when a log file is closed and a new prime log (usually E00 for the first Storage Group) is created, the closed log is shipped over to a target Exchange 2007 if SCR is enabled.  Once on the target, the log is held until the target reaches the number of logs specified in the log replay caching system – 50 by default.  After that, Exchange commits the logs to the database on the target, effectively providing SQL-like Log Shipping (or Continuous Data Mirroring, Asynchronous) between two Exchange 2007 Servers.

Whenever the two servers are not in communication with each other, there is the potential for operations to occur on the source that are not seen by the target, and vice versa.  Since this could easily result in a corrupted database, Exchange will check to ensure it knows what state the data on both servers is in before resuming normal SCR operations.  If the state is known, then SCR continues – transmitting all logs not yet on the target, committing all but the last 50 logs, and continuing on its way as normal.

If the state is unknown, a re-seed operation will need to occur.  More specifically if there is a gap in log file enumeration for some reason, you will require a re-seed.  You can see the reasons why re-seeds are required at the TechNet website listed here.

The one that is of most concern to Exchange Admins is that if the two servers are not in communication during a backup window, you will have to re-seed the database before normal SCR operations can continue.  When speaking of a local SCR pair this is not a big issue, as connectivity will be much more solid than WAN performance, and even if a re-seed is required, it will be relatively fast.  But across a WAN, it is more likely that you can occasionally suffer WAN outages that do not interrupt business operations.  If these outages extend past the time of the backup window, the backup tool will most likely truncate logs committed to tape without SCR being able to transmit those changes to the target Exchange Server.

Since minor network outages are a common (though hopefully not very common) issue with modern networks, the likelihood of requiring a re-seed due to this sequence of events is something that could be considered a normal part of Exchange SCR operations.  If your databases are small, then it won’t be an issue.  For larger databases, remember that a re-seed operation is a full copy of all data files to the target device from source, which could be problematic depending on your WAN throughput.

So with SCR, re-seeds do become a definite issue to contend with.  Since the occurrence of re-seed operations should be limited, you may be able to keep the systems successfully in sync without too much trouble. However, if you have larger databases or smaller WAN pipes, re-seeds can create problems for your network, especially if you use a backup tool that truncates your logs, or use circular logging for any reason.

Of course, there are alternatives to SCR for WAN protection and availability (see disclaimer below, I’m biased here), so you are not out of luck in terms of WAN operations with Exchange 2007.  Locally, CCR is a spectacular choice for nearly any sized Exchange system, with the exception of very large (over 1TB) datasets that may just take to long to re-seed even over a LAN.  Remotely, if you have properly planned for re-seeds then you will also be able to successfully utilize SCR, but if you do not – or cannot – plan for these required operations, then you’ll risk a failure in your failsafe system, which is not a great situation to be in.

In short, re-seeds are a normal part of SCR operations. They should not happen every day (or even every week) but you will most likely experience them over the course of your SCR lifetime, so plan for them now.

 

Next week is TechEd 2009!  I’ll be out in Los Angeles with the Double-Take Software crew, so please stop by and say hi. I’ll also be blogging from the event, so keep an eye on this column and follow me on Twitter at http://twitter.com/talonnyc.

Public airing of Exchange laundry...

Last week we finished up talking about what happens to Private Mail Stores during Online Maintenance (OM) on your Exchange servers.  While most people really tend to focus on what happens in those Private Stores the Public Stores are just as important, especially if you're not on Exchange 2007 yet.

For all versions of Exchange, Public Folders experience most of the same processes that their Private Mailbox cousins do, with the Exchange system scavenging items that are marked for deletion and have passed deleted item retention to start.  The system will also get rid of expired folders - a similar process to deleted mailbox systems in the Private Mailbox Stores - and perform an online defragmentation of the databases.  General maintenance and cleanup for Public Folders should be considered to be the same priority as OM on your Private Mailbox Stores, even if you're on Exchange 2007.

With Public Folders in Exchange 2000 and 2003, and even in 2007 in some circumstances, there is another vital process that goes on each day. It is not technically part of OM, but should be considered on par with the OM process.

As many people notice, the first time you boot up a newly-installed Exchange server, Outlook will complain about not being able to find "a required file" when you do a send/receive operation.  This is because - unless you're using a native Exchange 2007/Outlook2007 environment - Outlook cannot find the Offline Address Book (OAB) which doesn't get created in the Administrative Public Folders until around 5am the morning after the Exchange server goes online.  While not officially part of the OM process, the creation and subsequent updates of the OAB do happen overnight - 5am by default, just after the OM process is typically completed. 

OABgen (the process that - wait for it - generates the OAB) should be considered part of your OM systems. If you do not allow this to occur regularly, your OAB doesn't get updated and therefore your end users don't have the latest information - causing no end of frustration for you and your technical staff.  You could manually create/update the OAB using tools inside of Exchange, but the automated process is still your best bet for making sure these critical systems get updated regularly.

And one final note.  For those who believe they can get away with no Public Folders and/or PF maintenance because you are on Exchange 2007, be aware of one very important caveat.  All Outlook clients must be on Outlook 2007, and no Entourage or other MAPI clients can be used. That's because all MAPI systems before Outlook 2007 required the use of Public Folders for administrative information, like the OAB and other components.  In Outlook 2007, that's handled by Active Directory and Exchange itself, but if there is even one legacy client in your environment, you'll still need at least one PF tree, and therefore need to perform regular Online Maintenance on those Public Folder Stores.

Next week, back to regular blog topics - and speaking of which, if you have a topic you'd like me to cover, go ahead and drop me a line at MikeTalonNYC@yahoo.com

Mailbox Separation Anxiety

As we have been discussing over the last few weeks, Exchange 2000-2007 runs through quite a few operations during the Online Maintenance (OM) process.  We've talked about how databases get cleaned up, and how messages get deleted (temporarily and then permanently).  This week, let's look at Deleted Mailbox Retention (DMR) and what it can do for you.

DMR is the process that Exchange uses to remove mailboxes that have aged past their expiry date in the Private Database.  These expired mailboxes can be created by different operations within Exchange, either directly by the administrator of Exchange, or by operations in Active Directory that inadvertently cause a mailbox disconnect.  Disconnected mailboxes are simply mailboxes that not longer have an associated user or object in Active Directory. The simplest way to do this is to delete a user account in AD, which will disconnect the mailbox assigned to that user as part of the process.  However, this isn't the only operation that can cause mailboxes to become marked for deletion.  There is an odd side-effect to moving mailboxes that can happen if you jump between AD sites and don't finish fast enough (You can read about it here), and of course; if you get hit with AD corruption it could disconnect mailboxes as well.

Once a mailbox is disconnected, the first OM run that happens after the disconnect will mark the mailbox for expiry in a user-defined number of days. The default is 30 days, so if you run maintenance daily, the mailbox will stay in the database, but disconnected, for 30 days after it is disconnected from the mail-enabled object (like a user) in AD.  It also means you have to take when you do OM into account for your deleted item retention numbers.  For example, if you only do OM once per week, then you will need to add up to 7 days to your overall retention time for planning purposes.  Since the 30 day clock doesn't start ticking until the mailbox is marked during OM, you will extend the amount of time data remains in the database.  This is critical for regulatory compliance, as holding data for an extra week could put you over the line in terms of meeting your regulatory guidelines.

The other part of this OM process happens when Exchange scavenges the database for expired mailboxes.  If it finds one, and the mailbox hasn't been re-assigned to another AD object, Exchange will delete the data from the database permanently and create white-space in the database that gets cleaned up by OM to be reused by Exchange.

So, here again, it becomes vital that OM is permitted to run on a regular basis.  If not, deleted mailboxes will not be removed from the database unless you manually launch the cleanup wizards.  This is more than just a regulatory problem, as without removing the old information and freeing up white space, your databases will grow much more rapidly than if you are regularly cleaning up your mail stores.

Next week... Public Folder operations during Online Maintenance, so stay tuned!

Update now - no I mean it, NOW

Though I'm known for keeping a level head during these kinds of scenarios, there is an emerging vulnerability in both Exchange Server and Internet Explorer that you'll have to address immediately to keep your enterprise safe.  I wish I was kidding, but no, it's true.

The long and short of it is that specially crafted messages could be sent to an Exchange server that - even if only previewed by a user - can allow an attacker to take control of the Exchange server itself with full Exchange Admin privileges.  The details can be found at this link, but I'll summarize here.

If an attacker crafts a message in certain ways via TNEF (Transport Neutral Encapsulation Format), and that message is then either previewed or opened by an end-user (which is the point of email after all), then the attacker can gain elevated privileges on the Exchange server and wreak havoc. While that sounds like a major problem, there is good news.  The attack would require a sophisticated attacker creating just the right message and then finding a live mailbox to send it to.  Not only that, but it would need to get past the SPAM and CAPTCHA systems if you use them.  Finally, eventually the anti-virus vendors will jump in as well, scanning for messages crafted with the exploit.

The really bad news is that end-users do NOT have to open an attachment this time.  They can unwittingly infect the mail servers by previewing the message body or opening the message itself.  Since Outlook clients and now even Outlook Web Access both Auto-Preview by default, end-users can follow all best-practice safety protocols and still end up infecting the organization through no fault of their own.  Add to this that some anti-virus tools may use MAPI viewers to scan mail, which means that the system designed to shield you could accidentally infect you instead.  The same goes for backup agents that use MAPI clients to do brick-level backups.

There is one more bit of scariness to be told, though information on this last peice is more sketchy than the other exploit vectors. It would appear that the TNEF message may not need to be even previewed by an end user, but could in fact attack a server just be being received by the server itself.  Even until we know more for sure on this last point, I think the previous points - which are verified by this author and others - are bad enough that you should jump on this right now.

Microsoft has released a patch - also available via this same link, under "Security Update Deployment" and so you can protect yourself and your organization.  I strongly urge you all to do this immediately, and to have your end-users run Windows Update (Start|Windows Update on Vista, Microsoft Windows Update Online for everyone else) to get critical Internet Explorer patches as well.

Now, this is not only your Exchange server that you are protecting.  If an attacker gains privileges on your mail server, they can use it as a launching pad for attacking other servers with the same exploit or any other they'd care to try.  That means that if you don't patch your systems, you could inadvertently be responsible for attacks against other servers all over the net.

And so, I am asking my readers to protect the community at large as much as they can, by spending the time necessary to download and install the appropriate patches.  Yes, this will mean a maintenance window for a reboot.  Yes, it's annoying that we've gotten hit with yet another critical vulnerability.  But, yes, it is our responsibility to make sure that the attacks stop at our own borders, and that our own mail servers do not become the launching point for the next Code Red.

If I cannot bring you comfort, then at least I bring you hope

As we get ready to celebrate the beginning of a new year (in many countries), now is a good time to plan for your Exchange Servers and what you'll be doing in the year ahead.  While this is not the best time to plan out new infrastructure or massive overhauls immediately, you can start planning what you'll do later in the year on that scale.  You can also plan for what you'll do right off the bat in January for things like maintenance and updates/upgrades.

I plan to dedicate entire columns to these topics in the next few weeks, but here's the overview to get you thinking (once the champagne wears off...):

Offline Maintenance is a good thing to do regularly for Microsoft Exchange.  About once or twice a year you will probably want to take the time and walk through the process.  Microsoft says that this is not a solid requirement, but it can help identify long-trending problems deep within a database, so doing it once or twice per year is time-consuming, but worth it.  Plan now for downtime, so that you don't have to worry about when you'll have the ability to do it later. 

Online Maintenance is automatic, but you may want to look at your Exchange Server profile and see when the best time to have it done really is.  By default, the online maintenance system will run every night, but if you're also performing backups overnight, the two systems can overlap, causing the maintenance to be halted and never finish.  Take a look at your event logs to see if both systems are operating normally, or if the online maintenance run is terminating early.  If it is, you may want to change the times it runs, or even only run it on certain days.  Unlike with Exchange 2000, later versions of Exchange can hold up admirably well with even weekly maintenance runs, but be aware of what won't get done until maintenance by checking out Microsoft's Knowledge Base and/or waiting until my upcoming column on the topic.

Test, test, test, test, test......then test again! Now is the perfect time to set out your goals for testing of recovery/availability solutions (see disclaimer at the bottom of the page).  Backup systems, failover systems and other recovery solutions MUST be tested, otherwise how do you know they're doing what they say they're doing.  Best bets: do a test restore of Exchange data (at all the levels available from your tool like database, mailbox, mail item) to a different directory on the Exchange server and mount the DB in a Recovery Storage Group to ensure it is working properly.  This should most likely be done once per month, but at least once per quarter.  It doesn't require downtime, and doesn't require end-users to do anything.  Perform a failover test (if you use a failover tool) or a total restore test at *least* once per year.  2x per year or once per quarter is better, but of course since this type of testing will impact end-users, scheduling can be an issue.

Housekeeping is never a bad thing to look at after the clock strikes midnight on January 1.  Are you reaching your database limits in Exchange Standard edition?  If so it's time to either start ratcheting down mailbox limits or else starting to plan your upgrade strategy.  If you're already on Enterprise edition, are you pushing the limits of the free disk space you have on your drives?  Plan now to move the databases to another partition with more space.  Have your Stores grown to alarming proportions?  Might be time to start planning out new Stores and even Storage Groups to distribute the load.

Finally, give a look through the event logs and such to see if there are repetitive errors or other trends that can indicate problems brewing.  One more common example of this is recurring event ID's 1018, 1019, and 1022.  These indicate that there could be problems brewing in the database itself, though it is very possible that the error events are the only indication you see until the DB crashes.  If you're looking for a tool to help wade through the log information, check out Splunk, an IT search solution set that can make your life a lot easier.  [Due Disclosure: Splunk is currently working on partnerships with Double-Take Software.]

Have a healthy, happy and safe New Year's celebration.  Here's hoping 2009 exceeds everyone's wildest dreams in terms of greatness, and that Exchange never ceases to help us all live in these very interesting times.

Until next year.....

Mike Talon, Chief Columnist, BeingExchanged.com

PS: The title of this posting is taken from the song "Closing of the Year" from the movie soundtrack to "Toys."  The lyrics of the first verse are incredibly relevant to our combined experience this year, and I offer them here:

If I cannot bring you comfort,
then at least I bring you hope
For nothing is more precious than the time we have, and so
We all must learn from small misfortunes
Count the blessings that are real
Let the bells ring out for Christmas
At the closing of the year

- The Musical Cast of Toys Featuring Wendy and Lisa

Get the message? If not, it may not be your fault.

Microsoft Exchange engineers have always come under fire when a message that was sent to someone in their organization didn't get delivered, for whatever the reason may be.  The same goes for messages that bounce back for no apparent reason.  It could be a problem with the user's account (disabled, misspelled, etc) or it could be a problem with the server (offline, not connected to the net, etc).  It could also be a problem totally outside your organization, and that's what I'd like to talk about today.

In this ever-more interconnected world, routing of email can often depend more on external factors than internal ones.  Email messages can get mis-routed or even totally lost well before they reach your system, or can get purposely removed from the Internet somewhere along the way through no fault of your own.  If you're finding that more and more outgoing email is lost without any visible reason, the most common problem that is beyond your immediate control is getting accidentally blacklisted.

Blacklists are specific shared lists of either email servers or - in some cases - individual email addresses that spam control companies make available to end-users.  The most common of these is the DNS BlackList (DNSBL) which can either track IP addresses of servers known (or thought to be known) to send spam, or can track the domain names of such servers.  You can find out more about DNS BlackLists from Wikipedia at this link.  These systems collect information on servers and domains that send out spam and/or dangerous files (like viruses), and share that information with anti-malware companies (like McAfee and Kaspersky among others).  The anti-malware companies then use those lists to update their anti-spam and anti-virus solutions to protect consumers.

So if your server's IP address or domain accidentally ends up on these lists, suddenly external servers won't send mail to your servers anymore and won't accept incoming mail from you either.  It isn't easy to get on these lists by accident, but it could happen to you.  One or more internal users may get hit with a virus that sends tons of spam mails out from your email server, or someone totally outside your sphere of influence gets their computer taken over by a virus that forges headers to make it look like your servers are sending spam - and that's just two examples of things that could get you accidentally listed.

Once you're on these lists, it can be a very time-consuming process to get yourself un-listed.  Generally speaking, it will be a manual process of contacting each organization that has you listed on their blacklist system, and proving that you don't deserve to be there.  You can start your search at w3dt.net, which will assist you in figuring out which companies and filters have got your IP listed as up to no good.  Once you have that information, contact those organizations directly and either ask what caused you to get blacklisted or - if you already know that info - what you've done to stop the problem.

Be prepared to show that you identified who was sending the spam and why (and that you stopped it), or that you've isolated and removed the virus.  You will need to be patient and persistent, and will probably need to make several calls over a significant period of time (days to even weeks) to make sure that the problem gets solved.  Each organization that has you blacklisted will want their own proof that you're not a threat anymore, and in reality - provided they're not abusing their power - I can understand why they're being so careful.  Spam and viruses are a huge problem in the net today, and making sure they protect their clients is their business model.

Once you have the blacklist fixed to show you're not a spammer anymore, it can still take several days or longer for end-users of those services to get the updates to their spam and virus control systems.  So, if you do end up on a blacklist, you have quite a road ahead of you, but well worth it to clear your good name!

Getting blacklisted accidentally is a horrible thing.  Blacklists are a good thing overall for the benefit of email users all over the world, but when they suddenly stop you from emailing anyone (or getting email yourself), they can be a giant headache that you'll have to deal with effectively to get back in business.

Exchange on DC's

Smaller shops have always been faced with a fundamental quandary when the time has come for them to move off Small Business Server and into the world of the independent version of Microsoft Exchange.  One of the biggest considerations is where to put the Domain Controllers for Active Directory.  The first thought most of my clients have is to put it on the same server as the Exchange system, but outside of SBS that's a very bad idea for a handful of important reasons.

For example, if you're ready to make the move from a single server to a cluster, you cannot install Exchange 2003 on a cluster node that is also a DC.  That is just simply not supported by Microsoft, and therefore isn't an issue up for debate.

It's much more likely that you'll be moving to a single-server, however, as if you're just moving off of SBS then the likelihood you need a cluster is reduced quite a bit.  In those cases, while you could install Exchange on the DC, it's still not a good idea.  The Exchange Best Practices Analyzer will even flag this condition as going against best practices, which should be a giant red flag for anyone installing an Exchange Server.

Here's a few reasons it isn't a good idea:

1 - Since Exchange runs many services under Local System accounts with elevated privileges, a successful attack via the Exchange system could easily expose your AD controller to outside influences.  With the potential to gain access to all domain resources if the AD controller is compromised, reducing attack surface is vital, and installing Exchange (or any app, this is not an Exchange-specific issue) will *increase* attack surface.  If you get hit on an Exchange Server that is not a DC, the attacker has access to only that server, not the entire Domain.

2 - 3rd Party integration can suffer.  Since many 3rd Party tools may expect to use Local Accounts and Groups (which wouldn't exist on a DC) or may otherwise need to control aspects of the Exchange System that may not be available or accessible on a DC, integration with Anti-Virus, Backup and Recovery solutions can suffer if the server hosts both roles.

3 - Performance impacts will be clearly visible.  Since the Exchange Server constantly uses the DC and vice-versa, placing both on one physical server can effectively double the amount of work that server must do.  Some visible effects of this are slower overall Exchange performance; longer response times for Domain commands (such as logging in, changing passwords, etc.; and much longer reboot times.

Putting the Active Directory and Exchange Server systems on one physical machine may seem like a way to save money, but in reality it will cost more in downtime and headaches than it will save on server hardware.

 

[Note: Small Business Server contains a highly modified version of Microsoft Exchange Server that is specifically designed to run on an SBS server.  It does not permit all 3rd Party applications, limits functionality and is also limited to only being able to run on SBS.  Therefore, running the SBS version of Exchange on an SBS DC is both a supported installation and a best practice. This article specifically applied to non-SBS versions of Exchange Server.]

I Luv Exchange.

Over the years that I've been working with Exchange Server, one of the more common questions I get asked is "Why Exchange?"  With all the other messaging systems out there, it's a valid inquiry.  The short answer is, "I am a masochist."

Yes, that was a joke, please do not send any more of those really weird emails my way...

So why do I stand by Exchange Server year after year, even as yet another version releases without a SQL back-end, forces hardware upgrades and has a difficult migration path?  Because - at its heart - it does what I and my clients need it to do.

Exchange Server is a complex animal, but one that other vendors have not yet been able to completely match in terms of what it does.  There are tools that do messaging and shared calendars well - some might even say better.  There are collaboration tools that are superb!  There are even Unified Messaging platforms that rival the best efforts of the guys in Redmond.  There are not, however, many total packages that can do it all, and do it well enough that you can rely on them.

The combination of Exchange Server, Office Communication Server, SharePoint Server and Outlook (well, all of Office) creates a solution set that does everything that most businesses would require for messaging and collaboration.  Add in the new Groove systems and now you're moving well past what most Enterprise Messaging solutions can provide.

Yes, it is true that Exchange is challenging. It is also true that the Messaging and Collaboration Platform is complex and overwhelming for inexperienced users.  However, I challenge a novice to take Lotus or SendMail and get it working without either a little help or a lot of reading.

So, enough with the Exchange bashing!  Reach out to others in the community, search knowledge bases and web forums, and generally let others who have been in your shoes before help you.  In most cases you can find someone who has experienced the same problem you have, and who found a way to fix it.  And in any case you'll learn a lot more about how the platform works.

No, it's not perfect - not by a long shot - but Exchange is the best Messaging and Collaboration platform (with its eco-system software tools) that I have used in my time as a Systems Consultant and Engineer.  So yea, I do enjoy working with it, and there might even be some luv in there too.

From the field

Brace over at Double-Take Software had a great report from the field with info for those folks who are about to move their Exchange infrastructure to a new location:

"A good engineer that I worked with on a Dell project contacted me yesterday and brought up a good question. He is working with a client in New England and they are in the process of planning to move their Exchange Server to a different site subnet. Because Exchange is so tightly integrated with Active Directory there is much more to plan for with moving Exchange than there is Double-Take. However, the question is below and I have some answers from our of our own PS engineers.

"I'm working on a project where the customer needs to change the IP address of all their existing Exchange 2003 servers and move them to another subnet. I'm familiar with the process for Windows and Exchange, but how does that affect Doubletake? Are there any KB articles that might explain the process?"

If using the DTAM functionality within Double-Take it should be as easy as re-enabling protection once the server has been moved. "Just disable protection before doing the re-IP'ing, then go back through setting up the protection afterwards. Should probably not select the "use last configuration" or whatever that option is...since it might get the wrong IPs."

Pretty easy but as a best practice you should always contact our technical support department for any tpe of planning deployment just to verify potential changes being made. Double-Take Technical Support can be reached at 24/7 at 800-775-8674 or 317-598-2066.
For Technical Support in the European Union +44 (0)1905 330820 +33 (0)1 47 77 15 06 +49 69 6897776-66
Or find answers online at support.doubletake.com!"


Of course, that client would probably want to check out the latest version of the Full-server Failover Option too, as that could make life a lot easier during migrations of Exchange, or anything else running on Windows.