What is a PTR record and why should I care?

True story, I heard this exact question from a client not too long ago.  Through my trials and travails as an Exchange Engineer, DNS information is one of the very most confusing aspects of email systems that my clients have to deal with.  Just figuring out how to use normal DNS records tends to lead to a strong desire to give up on the whole project, so attempting to discuss reverse records can cause many to throw up their hands and run screaming from the server room.

Alright, that only happened once, but wow was it fun to watch.

PTR, or reverse lookup, records are used to allow external servers and systems a way to find out what identity a server has based on its IP address.  So, for example, we can look at the IP address information for Bing.com, Microsoft’s replacement for Live Search.

If I do an nslookup on bing.com, I get the following:

Name:    bing.com
Address:  64.4.8.147

So it appears that 64.4.8.147 is the IP address for that URL.  Now if I put the IP address into nslookup, my DNS server will attempt to backtrack to see what that server is identified as:

Name:    origin.bay.ux.search.live.com
Address:  64.4.8.147

Not a perfect return, as I was looking for it to reply that the IP was assigned to a bing.com address, but knowing that Bing Search replaced Live Search, the results are clearly traceable.

You can read up on what PTR records do, and how to create them at this Wikipedia page.

Now that you have some idea of what PTR records are used for, we can discuss why you will want to make sure that all mail domains you have authority over are properly configured to use them.

These days, you can do a quick Bing (or Google) search and find dozens of software packages designed to allow you to send out email as if you were someone else.  Sometimes there’s a legitimate reason to do this, such as if you manage email lists for multiple organizations through one mail domain.  In many cases, these tools are used toward nefarious ends, allowing a hacker to send email that appears to be from a domain in order to scam or infect the recipient.  Famous examples of this are phishing emails (see this Wikipedia page) where a scammer will send an email that appears to be from – say – a bank or other institution.  Your end user receives the mail, and since it appears to be from a trusted source, they’ll click the link and possibly enter in private information, leaving your organization open to attack.

To combat this, many SMTP servers either natively support, or can be configured to support, PTR lookups before accepting email.  This way, the header information can be examined to ensure that the email isn’t coming from a suspect source.  The end users don’t see much difference, but email that is from unknown domains, or domains known to be fraudulent, can be rejected before ever getting to their mailboxes. Exchange 2003 and 2007 do not natively reject email based on a bad reverse DNS lookup, so left to their own devices; you don’t have to worry about blocking incoming mail by accident.  This doesn’t mean you can ignore PTR records though.  Since many other mail server systems can be configured to reject non-verifiable mail, and since there are a host of 3rd-party systems that work with Exchange Server that can do the same, failure to properly configure a PTR record can cause your outgoing email to get bounced because you cannot verify your identity.  This means that you need to set up a PTR record for your Mail eXchanger (MX) records and your domain in general, or you risk having email returned as non-deliverable.  Having no PTR record is just as bad as an incorrect configuration here, as a lack of a reverse DNS lookup will cause the same results as an incorrect reverse DNS lookup.

There is a great debate over if this is a good or bad thing.  PTR responses could be forged, and so this is not a foolproof method of confirming identity.  Also, if multiple organizations all use the same SMTP server (thing about that multi-list email server from before) then which one gets the PTR record assigned to it?  If you have an email domain, then for now it is a good idea to ensure that PTR records are properly configured so you don’t run into the problem, but hopefully there will be better, more secure systems to rely on in future to confirm identity (such as the SMTP-AUTH proposed specification as part of E-SMTP).

To avoid blacklists, blocks and email black-holes, configure correct PTR records for your SMTP servers and domains.  It is by no means a perfect system, but it is one that you will run into, so an ounce of prevention is definitely worth a pound of cure here.

Getting Smart (Smarthosts that is)

No matter what version of Exchange Server you’re using – or even if you don’t use Exchange at all – you need to perform message hygiene on all mail in your organization.  Virus files, spam, phishing schemes and tons of other attacks are thrown at every email domain, every day.  If you’re not ready to deal with them, you’re dead in the water.  Smaller companies often just make due with hygiene software on the Exchange Server itself, but all sized firms need solution sets, and even smaller firms can take advantage of Smarthosts in a few different ways.

Smarthosts are servers or hosted solutions that are dedicated to performing message hygiene on all mail coming and going from your organization.  They come in many flavors, but tend to center around three types:

1 – Fully hosted solutions.  This method has no Smarthost hardware on-site, but instead contains everything at a hosting provider’s location.  Lowest up front cost, but highest ongoing (monthly, quarterly, etc.) costs.

2 – Hybrid solutions.  Here, you host a small appliance on-site to handle some of the workload, with the rest of the message hygiene functions handled by the service provider in their facility.  This isn’t quite as popular as the other two solutions, as the costs can quickly outweigh the benefits.

3 – Fully owned solutions.  Appliances and servers are in your datacenter, and totally controlled by you.  Start up costs are high, but ongoing costs are very low, as you’re only paying for updates to the virus/spam filter engines.

For smaller organizations, hosted solutions allow you to assign your Mail eXchanger (MX) record  to point to your Smarthost partner’s datacenter.  There, all incoming mail will go through the filters and checkers before it ever reaches your internal server set.  Most hosted providers allow you to update a list of “known good” recipients, so that any mail not destined for a valid employee is rejected immediately.  Virus scans are performed, and optionally spam filtration happens as well.  The remaining mail goes off to your internal mail servers for delivery to the end-users.

When mail leaves your organization, your internal servers are configured to send it first to the Smarthost provider, instead of directly to its destination SMTP servers.  The service provider performs all the same message hygiene on those messages, then acts as the SMTP gateway for your domain out to the rest of the world.

Hybrid solutions do everything that the fully hosted solutions do, but typically place a small appliance on-site to do periodic scans of the mail server itself.  They can also help speed up the whole process by making the “first hop” for email flow out of the organization be local instead of across the WAN.  It’s rare to see a hybrid Smarthost system that only does email hygiene, as the fully hosted or fully owned methods are much better for this kind of single-task solution.  Instead, the appliances on-site will typically handle all anti-malware processes including updating desktop protection and server scanning.

Fully owned solutions place appliances and/or other equipment at your production facility.  They act the same as the hosted solution, but you own all of the hardware and pay only for periodic updates to virus definitions, malware profiles and spam allow/deny lists.  This solution offers much more flexibility than a hosted solution, as you have complete and immediate control over any changes that might need to be put into effect.  The drawback is that if the hardware needs to be upgraded, you’ll be responsible for buying the new boxes.

Smarthosts are available from a large number of vendors.  Postini, Barracuda, and Microsoft Exchange Edge Services are just three that can be found with a Bing Search.  Most perform the same tasks, but each has their own selling points.  Your best bet is to talk with multiple vendors and see which one has the solution set closest to what you need.  Ask about things like ease of administration, web or application interfaces, ability to create custom allow/deny lists, and integration with your own Domain services to get updates to users and allowed accounts without compromising security on your AD servers.

No matter which vendor you choose, or which solution, using a Smarthost removes the message hygiene load off the Exchange Servers, which is always a good thing to do.  By dedicating either a service or server to handling message filtering and security, you can free up resources to provide a better end-user experience and a safer messaging environment.

Avoid bumpy rides with Exchange Online

I’m posting from a hotel room in Hartford, CT, getting ready for a meeting with a client tomorrow.  Getting here reminded me of why you might want to choose to not try to go it alone when you set up Exchange Server for your organization.  More details on the trip later.

One of the biggest problems I see clients facing when they deploy Exchange Server is the simple fact that it’s a huge software package.  In reality, it’s more of a suite of tools than one application, and therefore requires either pre-existing expertise, or a very fast ramp-up to be able to properly install, manage and maintain the solution set.  Luckily, there are options today that were simply not available in the past, namely Hosted Exchange Services (a.k.a.. Exchange Online).

Hosted Exchange is – at its heart – renting space on a shared or dedicated Exchange Server that is managed and maintained by a 3rd party.  These are hosting providers vetted by Microsoft (or at least they should be) and skilled in creating, deploying and managing Exchange Server solutions. No hardware on-site, and no major management needed by your own tech people.  So what are the pros and cons of using a solution like this.

The major con is that you have to keep your data on another server system that doesn’t reside within your firewall and security network.  Though the reality is that anything which requires connectivity to the outside world is technically open to attempted attack, some companies may not be able to tolerate hosted data.  Either corporate or external compliance policies and laws may prohibit this, so be sure to check with your compliance staff to make sure it’s ok before you ship the email off-site.

Barring compliance issues, security may still be a concern.  Make sure you completely check out your hosting provider, make sure they are a Microsoft Partner and make sure to get references before you sign over your digital life.  It’s totally fair to put the hard questions to your hosting provider, as you’re trusting them with a tremendous amount of sensitive data.

Beyond those two potential cons, the pros are pretty deep.  Many small to mid-sized organizations may attempt to host an Exchange Server in their own network.  Due to inexperience or just plain lack of time, patches are missed, settings are mis-configured, security concerns are overlooked.  This leads to either a poorly functioning Exchange system, or worse, an attack against the organization using the messaging platform as the attack vector.  Larger organizations who have dedicated messaging staff can afford to have people who look after these details, but smaller shops can’t, and often the company infrastructure suffers because of it.  I’ve discussed a lot of potential pitfalls to running an Exchange environment – this is a well-rounded way beyond most of them.

As a matter of fact, when talking with Jon Orton – one of the TPM’s that works with Exchange Online for Microsoft – he related to me that “we’re seeing more and more people, especially in the “small Exchange deployment, not much expertise” category, signing up for Exchange Online and partner-hosted Exchange.”  By placing the infrastructure in the hands of a team of experienced Exchange engineers, you remove that burden from your IT staffers.  Don’t get me wrong, I believe anyone can indeed learn to manage an Exchange Server messaging and collaboration environment However, if you do not need to, then this is one thing you can get off your plate and off your mind – safely.

So, even though it might be very tempting to set up Exchange Server within your network, the facts that it is readily available for purchase and that server hardware is getting cheaper should not be the major factors in decision.  Hosted Exchange Services with a trusted provider are easier to work with and just as readily available, and with a little legwork ahead of time, will be a better solution for many small and mid-sized organizations.

Which leads me to my harrowing journey today.  When considering how to get to Hartford from my native New York City (Queens, to be exact),  I had a few choices.  All were pretty easy.  I could take a train to a mid-way city and hitch a ride with a co-worker going to the same meeting from there.  I could take a different train and then a quick shuttle here.  But, because it was readily available and easy to set up, I chose to take a bus directly out to Hartford.  Much as with the decision to leap before you look and set up an Exchange Infrastructure on-site, I nabbed the tickets and got on the bus without a lot of pre-existing experience with the bus route.  After getting bounced around the bus for about 3 hours, and nearly losing the sandwich I had grabbed at the terminal, I finally got off the bus from hell and collapsed into the hotel room.  Had I researched my options a little better – and sought out some experience from others – I would have realized that a little more legwork would have led to a much better experience for me.

So do a little legwork before you put in your Exchange Server.  Know what you’re getting into and know that there are great options for getting out of the situation.  Exchange Online with a trusted provider can offer a convenient, safe and workable option for those who are just not quite ready to roll their own.

Update now - no I mean it, NOW

Though I'm known for keeping a level head during these kinds of scenarios, there is an emerging vulnerability in both Exchange Server and Internet Explorer that you'll have to address immediately to keep your enterprise safe.  I wish I was kidding, but no, it's true.

The long and short of it is that specially crafted messages could be sent to an Exchange server that - even if only previewed by a user - can allow an attacker to take control of the Exchange server itself with full Exchange Admin privileges.  The details can be found at this link, but I'll summarize here.

If an attacker crafts a message in certain ways via TNEF (Transport Neutral Encapsulation Format), and that message is then either previewed or opened by an end-user (which is the point of email after all), then the attacker can gain elevated privileges on the Exchange server and wreak havoc. While that sounds like a major problem, there is good news.  The attack would require a sophisticated attacker creating just the right message and then finding a live mailbox to send it to.  Not only that, but it would need to get past the SPAM and CAPTCHA systems if you use them.  Finally, eventually the anti-virus vendors will jump in as well, scanning for messages crafted with the exploit.

The really bad news is that end-users do NOT have to open an attachment this time.  They can unwittingly infect the mail servers by previewing the message body or opening the message itself.  Since Outlook clients and now even Outlook Web Access both Auto-Preview by default, end-users can follow all best-practice safety protocols and still end up infecting the organization through no fault of their own.  Add to this that some anti-virus tools may use MAPI viewers to scan mail, which means that the system designed to shield you could accidentally infect you instead.  The same goes for backup agents that use MAPI clients to do brick-level backups.

There is one more bit of scariness to be told, though information on this last peice is more sketchy than the other exploit vectors. It would appear that the TNEF message may not need to be even previewed by an end user, but could in fact attack a server just be being received by the server itself.  Even until we know more for sure on this last point, I think the previous points - which are verified by this author and others - are bad enough that you should jump on this right now.

Microsoft has released a patch - also available via this same link, under "Security Update Deployment" and so you can protect yourself and your organization.  I strongly urge you all to do this immediately, and to have your end-users run Windows Update (Start|Windows Update on Vista, Microsoft Windows Update Online for everyone else) to get critical Internet Explorer patches as well.

Now, this is not only your Exchange server that you are protecting.  If an attacker gains privileges on your mail server, they can use it as a launching pad for attacking other servers with the same exploit or any other they'd care to try.  That means that if you don't patch your systems, you could inadvertently be responsible for attacks against other servers all over the net.

And so, I am asking my readers to protect the community at large as much as they can, by spending the time necessary to download and install the appropriate patches.  Yes, this will mean a maintenance window for a reboot.  Yes, it's annoying that we've gotten hit with yet another critical vulnerability.  But, yes, it is our responsibility to make sure that the attacks stop at our own borders, and that our own mail servers do not become the launching point for the next Code Red.

Get the message? If not, it may not be your fault.

Microsoft Exchange engineers have always come under fire when a message that was sent to someone in their organization didn't get delivered, for whatever the reason may be.  The same goes for messages that bounce back for no apparent reason.  It could be a problem with the user's account (disabled, misspelled, etc) or it could be a problem with the server (offline, not connected to the net, etc).  It could also be a problem totally outside your organization, and that's what I'd like to talk about today.

In this ever-more interconnected world, routing of email can often depend more on external factors than internal ones.  Email messages can get mis-routed or even totally lost well before they reach your system, or can get purposely removed from the Internet somewhere along the way through no fault of your own.  If you're finding that more and more outgoing email is lost without any visible reason, the most common problem that is beyond your immediate control is getting accidentally blacklisted.

Blacklists are specific shared lists of either email servers or - in some cases - individual email addresses that spam control companies make available to end-users.  The most common of these is the DNS BlackList (DNSBL) which can either track IP addresses of servers known (or thought to be known) to send spam, or can track the domain names of such servers.  You can find out more about DNS BlackLists from Wikipedia at this link.  These systems collect information on servers and domains that send out spam and/or dangerous files (like viruses), and share that information with anti-malware companies (like McAfee and Kaspersky among others).  The anti-malware companies then use those lists to update their anti-spam and anti-virus solutions to protect consumers.

So if your server's IP address or domain accidentally ends up on these lists, suddenly external servers won't send mail to your servers anymore and won't accept incoming mail from you either.  It isn't easy to get on these lists by accident, but it could happen to you.  One or more internal users may get hit with a virus that sends tons of spam mails out from your email server, or someone totally outside your sphere of influence gets their computer taken over by a virus that forges headers to make it look like your servers are sending spam - and that's just two examples of things that could get you accidentally listed.

Once you're on these lists, it can be a very time-consuming process to get yourself un-listed.  Generally speaking, it will be a manual process of contacting each organization that has you listed on their blacklist system, and proving that you don't deserve to be there.  You can start your search at w3dt.net, which will assist you in figuring out which companies and filters have got your IP listed as up to no good.  Once you have that information, contact those organizations directly and either ask what caused you to get blacklisted or - if you already know that info - what you've done to stop the problem.

Be prepared to show that you identified who was sending the spam and why (and that you stopped it), or that you've isolated and removed the virus.  You will need to be patient and persistent, and will probably need to make several calls over a significant period of time (days to even weeks) to make sure that the problem gets solved.  Each organization that has you blacklisted will want their own proof that you're not a threat anymore, and in reality - provided they're not abusing their power - I can understand why they're being so careful.  Spam and viruses are a huge problem in the net today, and making sure they protect their clients is their business model.

Once you have the blacklist fixed to show you're not a spammer anymore, it can still take several days or longer for end-users of those services to get the updates to their spam and virus control systems.  So, if you do end up on a blacklist, you have quite a road ahead of you, but well worth it to clear your good name!

Getting blacklisted accidentally is a horrible thing.  Blacklists are a good thing overall for the benefit of email users all over the world, but when they suddenly stop you from emailing anyone (or getting email yourself), they can be a giant headache that you'll have to deal with effectively to get back in business.

Update on Wildcard certificates

Not that long ago I talked about using wildcard certificates to allow you to move OWA and ActiveSync services from one physical server to another.  Since single certificates are assigned to a single server, failing over or moving to another server would cause the clients to suddenly lose SSL connectivity, as the certificate would not match up, and ActiveSync devices cannot pop up the error about a non-secure connection. OWA can, but it can be troublesome with end-users who suddenly start seeing security warnings.

Following up on this theory of using a domain-assigned wildcard certificate, research has shown that older Windows Mobile devices (WM 5 or earlier) cannot leverage these types of certificates at all.  WM 6 and higher can leverage this technology, but earlier versions were not coded with the required information to recognize that a certificate could possibly be assigned to more than one physical server or networked device.

So, for those using WM 6, OWA and Outlook Anywhere, you can use wildcard certificates to allow for services to move from server to server as required.  For those using earlier versions of WM (or the 3G iPhone - though the jury is still out on that one), you must use server-specific certificates, and re-configure the devices' ActiveSync connection if the server itself moves. 

None of this impacts Blackberries, as they authenticate via the RIM network, and not directly to the Exchange servers.  If you're using Blackberries, wildcard certificates offer the ability for all other mobile systems (OWA, Outlook Anywhere, etc) to move with your servers in the event of a loss of a particular physical machine, while RIM will handle moving the Blackberry devices.

Long story short, if you're on WM 5 or earlier, be ready for a few support calls when you need to move the services or fail over between servers - even if you use wildcard certificates.  If your users are on the new iPhone systems, be sure to keep a close watch on Apple's forums, as new information is being discovered every day.

More than zero?

There are a ton of interesting settings and switches in Exchange 2000/2003, but one of the more unusual is the "Zero Out Deleted Database Pages."

Exchange 2000 and higher all have several levels of potential access points for un-delete operations.  When anything is deleted from a mailbox, the data isn't truly removed, but rather marked as scheduled for delete at some later date.  The default settings keep mail items for 15 days and mailboxes for 30, after which point they're removed from the server.

Data pages within the ESE database are another story; however.  They get deleted all the time, but not removed from disk.  They're simply marked as deleted, ready to be overwritten as needed.  Normally, this doesn't pose any specific security threat, as File servers and most other Windows servers do the same thing when data is deleted.  In higher-security environments; however, this could be a policy violation, as the data would be stored "in the clear" and vulnerable  to anyone able to get their hands on the physical disk devices.

Microsoft has taken steps to natively remove the potential for security breaches, but at a cost.  Built into every version of Exchange supported today is the ability to literally overwrite each deleted ESE page with zeros.  The processes is therefore referred to as "zeroing out" the pages, and the "Zero Out Deleted Database Pages" setting controls this behavior.  By default, the setting is not turned on, but can be enabled via the Exchange System Manager GUI, Powershell or via Active Directory.

The problem is that the benefits rarely outweigh the costs in the real world.  Zeroing out pages takes up a good chunk of processor time, and Exchange deletes pages - quite literally - all the time.  Zeroing out will also have a huge impact on any form of replication, just because a large amount of data change is required for the process to work.  In addition, this method of secure wipe meets only very low-level security concerns.  Most regulatory requirements require multiple overwrite passes (7, 15, sometimes even more) and mandate random data be used, not just zeroes. 

So, while you might feel like you'll gain some security by using this setting, the extra overhead and falling short of regulatory compliance should make you think twice before you turn it on.

Get wild with wildcard certificates

Most organizations that are considering securing their email systems are also looking at SSL certificates to aid in that effort.  Security certificates allow for encrypted communications using Outlook Web Access, Outlook Anywhere (RPC over HTTPS) and ActiveSync devices like Windows Mobile Smartphones. The issue is that a normal SSL certificate is bound to a single server, which means you'd have to obtain a signed certificate from a provider for each server you want to put into the solution.

You could use self-signed certificates to get around the issue of having to purchase individual certificates, but only at a cost.  Many applications will not accept self-signed certificates, and even Microsoft-controlled systems like OWA will kick up several layers of security errors - which will increase support headaches for you and your staff.

Alternately, you can obtain a Wildcard Certificate from a provider, which will allow you to use one certificate for multiple servers in your organization.  These certificates are more expensive than single-server options, but can give you much more flexibility with things like server rebuilds, migrations, and failover.  The only restriction is that the servers will have to be part of the same organization. That means that all the servers will have to be part of the same domain and/or forest in the sense of Active Directory.

Wildcard certificates can be defined at either the organizational level - such as *.wildcarddomain.com - or masked at another level, www*.wildcarddomain.com for example.  With the first example, all server identities that are part of wilddcarddomain.com will be able to share the certificate.  In the second, www1, www2 and even wwweb.wildcarddomain.com will be able to share the masked certificate.  Which you use will depend on the depth of security you are looking to implement, as more specific masks allow for less chance of someone spoofing the certificate for a foreign machine.

Once you install the wildcard certificates, you can allow services to move between servers within the same organization without breaking security. This can be extremely helpful for both migrations and disaster recovery, as you can have both of the servers shielded under the same certificate, and therefore you will not get naming errors when the users move or fail over. 

In a worst-case scenario, you can use single-server certificates on a migration or failover target, but your end users will get errors about the certificate being valid, but assigned to a different server name than the one they are connecting to.  You can then purchase another certificate after the fact and assign it to the target server to remove the error if/when needed.  However, wildcard certificates give a much better option for flexibility to you and your team.