Exchange 2003 Default Recipient Policy Problems with OMA/EAS

So, yes, I spend most of my time writing about Exchange 2007, but most of my clients are still on Exchange 2003.

I ran into an interesting problem today. It took me about an hour to figure out what was going on, but then it was a “duh, of course” moment on my part.

The real issue? Too many cooks in the kitchen. This problem was caused by a client administrator making a change I wasn’t aware of, just because he thought it made things “cleaner”. If I had been aware of the change before it happened, I would’ve said “no, that’s gonna hurt, don’t do it!” After the case, I would’ve said “let’s change it back.”

Anyway, the problem started with a report of a user who just got a Treo and couldn’t sync with EAS and couldn’t log into OMA. This is why:

When Exchange is first installed on a server, either Exchange 2000 or Exchange 2003, it creates something called ExIFS – the Exchange Installable File System. The purpose behind ExIFS is to allow an application to access Exchange mailboxes as if they were a file system – each folder representing a directory, each mail message or calendar item being a file, etc.

In Exchange 2000, this was visible by default, as the “M: Drive”. That caused huge numbers of heartaches. Many anti-virus solutions would scan M: and end up corrupting items. (Note that this was not necessarily the fault of the anti-virus vendors, as Microsoft guidance was quite clear about configuring the a/v solutions to NOT do this.)

In Exchange 2003, the M: drive was still present – but hidden. To access it, you must use a special syntax of NTFS, often called “the volume syntax”, as it can also be used to see your normal disk drives and mount-points, if you know their system names (which contain a 128-bit value known as a GUID). You access it by using this syntax \\.\BackOfficeStorage\. If you do a “dir \\.\BackOfficeStorage\” from a command prompt on an Exchange Server, another directory will be visible. This directory will be the DEFAULT SMTP domain in the Default Recipient Policy.

Now, when you first install Exchange, the default SMTP domain is the same as your Active Directory domain. So, if your Active Directory is named test.local, then your default SMTP domain is also test.local. As you create users with mailboxes, that Default Recipient Policy will stamp your users with test.local as their primary SMTP domain.

That is generally not what they want. J Most people have a different Internet domain than their AD domain. To solve this, you have two options:

1] Change the default policy, or
2] Create a newer, higher priority policy.

Option [2] is the correct choice.

If you choose option [1], then you will change the default SMTP domain. OMA and EAS depend on the default SMTP domain, and they were setup when you installed Exchange. Now, if you change the default policy, and leave it so that the default policy still generates the original email address, just not as the primary, OMA and EAS will still work. If you change the default policy and eliminate the original email address, then users created without that original email address will not be able to use OMA and EAS.

This is especially critical in SBS 2003 servers and in servers where KB 817379 has been applied (which has automatically been done on all SBS 2003 servers).

This is because the directory gets updated in the Exchange virtual directory of the Default Web Site automatically by the DS2MB process of Exchange’s System Attendant. However, the Exchange-OMA directory is not updated.

If you run into this problem, you can update the Exchange-OMA virtual directory, you can update the Default Policy, or you can update individual users with the other email address; whichever works out best for your organization or your security standards.

This problem disappears on Exchange 2007, since it no longer has OMA or the ExIFS. EAS on Exchange 2007 uses Exchange Web Services (EWS).

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

adminCount, adminSDHolder, SDProp and you!

Quick Synopsis

Ever had the permissions on a user object keep changing and you don't know why? Well, this article tells you why and how to fix it.

If you just want to know how to fix it, page down to the Resolution section, and skip the Technical Explanation.

Technical Explanation

One of the many protection mechanisms built into Active Directory is known by several names: adminCount, adminSDHolder, or SDProp. The first two, adminCount and adminSDHolder, are directly tied to Active Directory. adminCount is an attribute that is set on each object affected by this protection mechanism. adminSDHolder is the reference object that all other critical objects are compared to.

The adminSDHolder object, which stands for Administrative Security Descriptor Holder, exists in each domain. That is, in a multi-domain forest, there is an adminSDHolder object in each of the domains in the forest. The object is located at CN=adminSDHolder, CN=System, DC=example, DC=com. There are a number of access control entries (ACEs) on the adminSDHolder object. Those ACEs are the proper ACEs for any object that can be defined as a critical object.

A critical object is a security principal which is either a very specific principal, or a member of a critical group. Security principals are currently limited to users, groups, computers, or inetorgpersons.

Security Description Propagator (SDProp) is the task that runs within LSASS.EXE which actually enforces the protection mechanism. SDProp performs other tasks as well (including replicating security descriptors from one domain controller to another), but in this case we are interested only in its adminSDHolder functions.

The purpose behind the SDProp mechanism is to stop inappropriate changes from being made to critical objects.

For example, assume the user SubAdmin is delegated full-control privileges over an OU named SubOU. If, under whatever circumstance, the Administrator user for the domain were to be moved under that OU, and then if normal inheritance rules applied to the Administrator user, the SubAdmin user could make whatever changes they wished to that user – including changing the Administrator password. However, the SDProp process prevents that.

Permission inheritance is disabled for all critical objects, and they only have the ACEs applied to them that are present on the adminSDHolder object.

When a critical object is processed by SDProp, it not only checks and resets the ACEs on the object, it also sets another attribute: adminCount. If SDProp has ever processed a security principal, that object will have adminCount set equal to 1 AND inheritance will be disabled.

This can cause problems in a number of environments. This is primarily because critical objects in an OU will not receive delegated permissions for that OU. A number of applications depend on that functionality (such as Cisco Unity and Blackberry Enterprise Server).

The critical objects which are users are:

  • Administrator
  • krbtgt

The critical objects which are groups are:

  • Enterprise Admins
  • Schema Admins
  • Domain Admins
  • Administrators
  • Account Operators
  • Server Operators
  • Print Operators
  • Backup Operators
  • Cert Publishers

Only the first four were critical objects in Windows 2000. The last five were added with Windows 2003 (and Windows 2000 service pack 4). This explains why the issue was not seen for most companies with Windows 2000 and it only began appearing with the deployment of Windows 2003.

Resolution

The proper workaround is for high-privilege principals (i.e., critical objects) not be used for day-to-day activities. That is, you should not log into your workstations with your account which is a member of any critical group. The Administrator user shouldn't have a Blackberry or a voice-mail account. Etc.

In order to reverse the effect of SDProp on a security principal, you must do three things:

  1. Remove the security principal from the critical group.
  2. Set adminCount equal to zero using ADSIEdit or LDP.
  3. Re-enable inheritance on the object (this can be done from within ADUC, with View -> Advanced, on the security tab or from within ADSIEdit or LDP).

Note that once adminCount is set equal to one, removing the security principal from the critical group will not cause SDProp to reverse its changes. As long as adminCount is equal to one, SDProp will continue to reset permissions on an object. Therefore, the order of the steps listed above is important. Depending on timing, you may be successful at following an alternate order, but you may not always be so lucky.

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Accessing a Web site with PowerShell

This is just another quick sample of what you can do in PowerShell by leveraging the .Net framework.

Need to access a website? No problem. System.Net.WebRequest to the rescue!

Note that you can specify credentials, read all the data, follow the links, etc. Just depends on how far you want to go…

function start-http([string]$url)
{
	$routine  = "start-http:"

	trap
	{
		log $routine "Trap: access URL $url."
		log $routine "Error = $_"
		log $routine $_.InvocationInfo.PositionMessage
		continue
	}

	$request  = [System.Net.WebRequest]::Create($url);
	$request.UseDefaultCredentials = $true

	$response = $request.GetResponse()
	if ($response)
	{
		log $routine "response statuscode = " $response.StatusDescription
		log $routine "response characterset = " $response.CharacterSet
		log $routine "response length in bytes = " `
			$response.ContentLength.ToString()

		$response.Close()
	}
}

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

Script for From The Field

The authors of the upcoming “Windows Server 2008 Terminal Services Resource Kit” asked me if I wanted to write a script for them for a section named “From The Field”. I said sure! I based the script on one I wrote a long time ago, which you can access here. So the readers have the EXACT script that was published I’m putting this on my blog.

The purpose for this script is to tie the script to a particular event that might occur from Terminal Services. This capability is a new feature of the Windows Server 2008 Event Log Subsystem. If that event occurs, then this script is executed to send an email.

Now, the Resource Kit goes into detail of the event id that happens and how you actually do that tying together. Go buy it! 🙂

Of course, there are multiple ways to skin this cat, but here is one for you to use.

Option Explicit

'''----- script configuration area
Const strSMTPServer = "arvon.alpineskihouse.com"
Const strFrom       = "alerts@alpineskihouse.com"
Const strTo         = "adam.barr@alpineskihouse.com "
'''----- end configuration area

Dim objMail               ' the CDO object
Dim objWSHNetwork         ' windows-script-host network object
Dim strNetBIOSComputer    ' the netbios name of our computer

''' get the NetBIOS computer name
Set objWSHNetwork  = CreateObject ("WScript.Network")
strNetBIOSComputer = objWSHNetwork.ComputerName
Set objWSHNetwork  = Nothing 

''' do the real work to send the message 
Set objMail = CreateObject ("CDO.Message")
objMail.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/sendusing")      = 2
objMail.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/smtpserver")     = strSMTPServer
objMail.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25
objMail.Configuration.Fields.Update

objMail.From     = strFrom
objMail.To       = strTo
objMail.Subject  = "Critical error!! " & strNetBIOSComputer & " failed to reboot " & Now
objMail.Textbody = "Critical error!! " & strNetBIOSComputer & " failed to reboot " & Now & vbCRLF
objMail.Send

Set objMail = Nothing

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!

—–

Edit on September 10, 2010 – The Microsoft Collaboration Data Objects (CDO) are not installed by default on Windows Server 2008 or above. You can download them from here.


Follow me on twitter: @EssentialExch

It’s All About The IOPS, Silly!

In yesterday’s EMO (Exchange Messaging Outlook eZine, subscribe at http://www.slipstick.com/emo) I had an article named It’s All About The IOPS, Silly! which discussed the impact of RAID-1 and RAID-5 arrays on total IOPS calculations.

However, space didn’t allow me to talk about SANs (Storage Area Networks) in regards to IOPS.

Where Exchange is concerned (and SQL Server as well, although that isn’t our focus here), you do not want to be sharing your “disk group” or “volume group” or “virtual array” (or whatever your SAN software calls it) with any other application – unless you’ve performance tested it in the worst case.

SANs allocate storage in terms of LUNs – Logical Unit Numbers. For a given server, a LUN is allocated to that server (note that in the case of clusters, quorum disk may be visible to multiple servers at one time) and only one server may have write access to that LUN at a time. A LUN may, or may not, be tied to a particular disk array or disk set contained within the physical hardware of the SAN.

Many types of SAN software support a concept known as “LUN Stacking”. In this case, an array (be it RAID-1, RAID-5, RAID-10, whatever) has multiple logical stripes of the disk where multiple LUNs are allocated on the same array. Arguably, this is what SANs are all about – allowing you to control storage logically instead of drive-by-drive. And, for many (perhaps most) applications this makes sense.

However, while the BEST CASE SCENARIO says that the IOPS available for a LUN is the maximum IOPS available for that disk set, the WORST CASE scenario says that the IOPS available for a LUN is equal to the number of accessors (servers) that are using that physical array.

So, for example, if you have five servers hitting a RAID-1 array via different LUNS, the best case performance scenario says that you can have the maxiumum IOPS available to the hardware available to each server. The worst case is IOPS / 5.

For example, if your RAID-1 group has a calculated IOPS of 250, one server could potentially benefit from having a total IOPS of 250 available to it. However, if all the servers are hitting the disk, the performance would be more like 50 IOPS. While for many small and medium businesses, an IOPS of 250 is probably sufficient to meet their Exchange needs – only the smallest of companies can get adequate performance on 50 IOPS total.

Another reason you want to avoid sharing LUNs is due to the usage profiles of the various Exchange needs.

Database access (which includes queues, mailbox stores, and public folder stores) is completely random. The creation of log files (transaction log files, message tracking logfiles, and protocol log files) is completely sequential. If you share LUNs between different usage profiles, then your performance will suffer (especially for those needs which are optimized for sequential access).

Conclusion: Just Say No! Don’t let your SAN guys tell you what kind of storage performance requirements Exchange has. You need to be prepared to tell them!

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

Groups and their Membership

I’m working with a new client and this client is a “wholly owned subsidiary” of another company and is getting ready to be spun off as their own company.

One of the things we are doing is configuring their network to be standalone and not part of the corporate Active Directory. That involves creating users, group, moving computers, installing a new Exchange organization, etc. etc.

But sometimes it’s the simple things that bite you…

We’d gotten an export of the OU for the subsidiary out of corporate in a CSV file – every object, every attribute. That was a mess in-and-of itself, but at least all the data was there.

So we started importing the data. And it broke.

Huh? We’d done this before and no problem. What was going on here?

Like I said – the simple things…

This set of users and groups were using embedded groups. Sometimes up to five levels deep. (That is, a group within a group within a group within a group within a group.)

Well, you can’t assign a group membership to a group that doesn’t exist! Seems obvious, in retrospect.

The solution? Go through and harvest all groups FIRST – just create the groups in Active Directory. Then, go back through and assign membership to the groups.

Here is a somewhat sanitized version of the code we used. On the first pass, it creates groups. On the second pass, it will assign membership to the groups. This shows a number of PowerShell techniques in group management – including creating groups, checking for membership in a group, and adding members to a group.

Also note: there are several places where this script could be optimized. But I was more interested in clarity (since, as a consultant, I won’t be there forever!).

I hope you find this useful for your scripts!

set-variable ADS_GROUP_TYPE_DOMAIN_LOCALGROUP 1          
set-variable ADS_GROUP_TYPE_GLOBAL_GROUP      2          
set-variable ADS_GROUP_TYPE_LOCAL_GROUP       4          
set-variable ADS_GROUP_TYPE_UNIVERSAL_GROUP   8          
set-variable ADS_GROUP_TYPE_SECURITY_ENABLED -2147483648 

$groupType = $ADS_GROUP_TYPE_GLOBAL_GROUP –bor $ADS_GROUP_TYPE_SECURITY_ENABLED

$csv = import-csv GroupsOnly.csv
#
# Each line contains the dn, the name, the description if one exists, and the member list
# with the dn of each group member. The member list has each member separated by a semi-colon.
#
foreach ($line in $csv)
{
	$dn = $line.dn
	$adsPath = "LDAP://" + $dn
	$objGroup = [ADSI]$adsPath
	if ($objGroup.Name)
	{
		# the above verifies that the group exists. Note that 
		# ADSI is "smart". It doesn't actually access AD and 
		# fill the property cache until you read an attribute
		# from the object.
		"Exists: " + $line.Name
		if ($line.member.Length -gt 0)
		{
			$i = 0
			$members = $line.member.Split(";")
			$lower   = $dn.ToLower()
			foreach ($member in $members)
			{
				$objMember = [ADSI]("LDAP://" + $member)
				if ($objMember.Name)
				{
					# check to see if member is already in group
					[bool]$already = $false
					$userGroups = $objMember.memberOf.Value
					if ($userGroups)
					{
						foreach ($usergroup in $userGroups)
						{
							if ($userGroup.ToLower() -eq $lower)
							{
								$already = $true
								break
							}
						}
					}
					$objMember = $null
					if ($already)
					{
						"`tAlready in group: $member"
					}
					else
					{
						$objGroup.Add(("LDAP://" + $member))
						if ($?)
						{
							"`tAdded to group: $member"
							$i++
						}
						else
						{
							# should be an error displayed on the PowerShell window
							"`tCouldn't add to group: $member"
						}
					}
				}
				else
				{
					# member specified in CSV doesn't exist in Active Directory
					"`tObject doesn't exist: $member"
				}
			}
			"`tadded $i members"
		}
		else
		{
			# members element of the CSV was empty
			"`tno members specified for group"
		}
		$objGroup = $null
	}
	else 
	{
		"Creating " + $line.Name

		$parent = "LDAP://" + $dn.SubString(1 + $dn.IndexOf(","))
		$object = [ADSI]$parent
		$child  = $object.Create("group", "CN=" + $line.Name)

		$child.Put("groupType",      $groupType)
		$child.Put("sAMAccountName", $line.name)
		$child.Put("name",           $line.name)

		if ($line.description.length -gt 0)
		{
			$child.Put("description",    $line.description)
		}

		$child.SetInfo()
		$child = $null

	}

	sleep -milli 500
}

$csv = $null

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

I Have a Logo!

In case anyone besides me is counting, I’ve been at this (consulting, writing, speaking, etc.) for 10 months now. I’m having a blast. I wish I’d done this years ago.

Two months ago I took on a partner and we formed Smith Consulting d/b/a The Essential Exchange as a legal entity (as opposed to the sole proprietership I was previously).

After several weeks of back-and-forth, we finally came up with a logo today, and I’m very pleased with it. It will replace the generic icons you see on this blog fairly soon (image manipulation is not my strong suit). Here it is:


Follow me on twitter: @EssentialExch