Monitoring Exchange Server 2007 with System Center Operations Manager 2007

Just a few days ago, my new book (of the subject title!) started shipping from Amazon. If it hasn’t already, it should start appearing in your local bookstores this week.

If you want to order from Amazon, you can get it here.

Eight months in the making, this book was a labor of love. Including information on installing and configuring OpsMgr 2007, it also includes much information about operating a reliable Exchange environment. Even if you don’t have OpsMgr in your organization, you can learn from this book the key areas of Exchange that need monitoring and how to do so.

A strong PowerShell component is provided in several chapters to assist you in the generation of synthetic transactions for testing your Exchange environment.

While the book is written toward a key audience of Exchange Server 2007 administrators, much material is also provided for the Exchange Server 2003 administrator. The book uses a virtualized environment to describe a test roll-out of an OpsMgr 2007 and Exchange 2003/2007 mixed environment.

Since Exchange Server depends on the health of Windows Server, Active Directory, DNS, and IIS; tracking the health and well-being of these key services is also covered.

Go buy it. You’ll like it. 🙂

The chapter titles are:

  1. An Evolution of Server Management
  2. Monitoring Exchange Server 2007
  3. Installing and Configuring OpsMgr 2007
  4. Deplying OpsMgr 2007
  5. The First Management Pack: WIndows Server
  6. The Active Directory Management Pack
  7. The Domain Name System (DNS) Management Pack
  8. The Internet Information Services Management Pack
  9. SQL Server: An Ancillary Management Pack
  10. Exchange Server 2003
  11. Exchange Server 2007
  12. Exchange Server 2007 Redundancy
  13. Exchange Server Operations
  14. Tracking Mail Flow

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

A Small Patch: Online Exchange Backup

If you only had a single storage group, the script in Exchange 2007 and Windows 2008: Online Exchange Backup (part 6 of 7) would not find any storage groups. The primary reason for that I always have a RSG too! 🙂 The fix is easy. In getStoragegroups, change the beginning few lines:

function getStorageGroups
{
	$count = 0
	#
	# locate the storage groups and their log files and system files
	#
	$colSG = get-StorageGroup -server $computername
	if ($colSG.Count -lt 1)
	else
	{
		write-host "No storage groups found on server $computername"
		return 1
	} 

to look like this:

function getStorageGroups
{
	$count = 0
	#
	# locate the storage groups and their log files and system files
	#
	$colSG = get-StorageGroup -server $computername
###	if ($colSG.Count -lt 1)
	if (($colSG -is [Microsoft.Exchange.Data.Directory.SystemConfiguration.StorageGroup]) -or
	    ($colSG -is [System.Object[]]))
	{
		## everything is good
	}
	else
	{
		write-host "No storage groups found on server $computername"
		return 1
	}

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

A Small Patch: Online Exchange Backup

If you only had a single storage group, the script in Exchange 2007 and Windows 2008: Online Exchange Backup (part 6 of 7) would not find any storage groups. The primary reason for that I always have a RSG too! 🙂 The fix is easy. In getStoragegroups, change the beginning few lines:

function getStorageGroups
{
	$count = 0
	#
	# locate the storage groups and their log files and system files
	#
	$colSG = get-StorageGroup -server $computername
	if ($colSG.Count -lt 1)
	else
	{
		write-host "No storage groups found on server $computername"
		return 1
	} 

to look like this:

function getStorageGroups
{
	$count = 0
	#
	# locate the storage groups and their log files and system files
	#
	$colSG = get-StorageGroup -server $computername
###	if ($colSG.Count -lt 1)
	if (($colSG -is [Microsoft.Exchange.Data.Directory.SystemConfiguration.StorageGroup]) -or
	    ($colSG -is [System.Object[]]))
	{
		## everything is good
	}
	else
	{
		write-host "No storage groups found on server $computername"
		return 1
	}

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

Single-Label Domains (SLDs) and the Next Version of Exchange

I’ve written several times about the inadvisability of having an Active Directory domain name that has a single-label. My most recent article was Wrapping Up SLDs and Exchange Server 2007 in April of 2008.

At that time, Ed Beck of Microsoft wanted to assure me that if customers found bugs, they should report them and Microsoft would investigate the issues and address the issues based on the standard engineering review decision process within Microsoft. That is, Microsoft was’t closing the door on fixing problems with SLDs in Exchange 2007.

Now, the next version of Exchange, which is code-named E14, is in development. Yesterday, Ed emailed me and told me that Microsoft released a forward-looking statement indicating that they are investigating the SLD policy for E14 (Ed authored the article on the MS Exchange Team Blog: Next version of Exchange and Single Label Domain (SLD) policy under review).

I guess if I were you – and using an SLD – now would be the time to make yourself known! Feedback is accepted on the Exchange Team Blog site for quite a while after a posting. Let them know your opinion!

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

Exchange 2007 and Windows 2008: Online Exchange Backup (part 6 of 7)

In the first five parts of this series, I’ve given you the background to understand how Exchange backup works when using VSS and how to acquire the necessary information from your Exchange server to know what you should back up. Today, I present to you a full-blown working script that will generate a full-backup of your Exchange 2007 Server on Windows Server 2008, verify that the backup is good using ESEUTIL, and flush the transaction logs for the Exchange storage groups if the backup is good.

The first five parts of the series were:

Part 1: Getting a List of Storage Groups in a PowerShell Script

Part 2: Getting a List of Stores in a PowerShell Script

Part 3: Exchange 2007 and Windows 2008: Offline Exchange Backup

Part 4: Volume Shadow Copy Services (VSS) and Exchange – The Basics

Part 5: Exchange 2007 and Windows 2008: Using Diskshadow for Online Exchange Backup

Now, before you ask – is this a supported backup tool? The answer is yes and no. VSS backups are a supported way to back up an Exchange server’s databases. Diskshadow is a supported tool on Windows Server 2008. Is my script supported? No. Only so far as I find the time, energy, and effort to provide support for it. I can’t warrant that it will work in your environment. It’s worked everywhere I’ve tested it, that’s all I can tell you. If you find a problem, let me know and I’ll try to help, but there are no guarantees.

You won’t find anything new in this script (from the prior postings in this series), except that the Diskshadow script is generated within the PowerShell script. This makes it easier when you run into a situation that you are using multiple volumes in your Exchange environment (which is a best practice for performance reasons).

The script takes three parameters:

$backupLocation – This is the volume and directory (or mountpoint) where the backup should go. It defaults to C:\Backups. You will probably need to change this for your environment.

$startLetter – This is the first letter that should be used by the script for exposing shadow copies as drive letters for the backup scripts. This defaults to g.

$startScript – This is a switch parameter. When set, the PowerShell script will initiate the backup using diskshadow.exe as soon as the script is built. The switch defaults to unset.

The #1 limitation of this script is that it backs up all storage groups on an Exchange server. I have plans to address that in a future revision.

The #2 limitation of this script is that it’s “an ugly command line tool”. I have plans to address that in a future revision.

This script will create a metadata file named (join-path $backupLocation “online-backup.cab”) (That is C:\Backups\online-backup.cab by default). This file is used by diskshadow.exe for storing information required for restores. We will cover basic restores in part 7 of this series.

The cmd.exe script is stored as (join-path %TEMP% “online-backup.cmd”). The Diskshadow.exe script is stored as (join-path %TEMP% “online-backup.dsh”).

On my server, I store PowerShell scripts in the c:\scripts folder, and name this particular script MBS-online-backup.ps1. A typical invocation is:

join-path “F:\ExchangeBackups” (get-date -uFormat “%Y-%m-%d-%H-%M-%S”) |% { md $_ |% { ./MBS-online-backup.ps1 -backupLocation $_.FullName -startScript } }

This causes a unique directory to be created for each invocation of the backup script and for the script to be automatically run.

Granted, that’s a dense for a beginner to understand. You can separate that into multiple lines quite easily:

$backupSubDir = get-date -uFormat “%Y-%m-%d-%H-%M-%S”
## As specified, the uFormat string means: yyyy-mm-dd-hh-mm-ss
## where the first ‘mm’ is the month number, and
## the second ‘mm’ is the minute number.
$backupDir = join-path “F:\ExchangeBackups” $backupSubDir
md $backupDir
./MBS-online-backup.ps1 -backupLocation $backupDir -startScript

And that is much easier to understand. Without further ado, here is the script:

##
## MBS-online-backup.ps1
##
## Michael B. Smith
## January, 2009
##
## This program generates an online VSS-based backup of an Exchange server
## (Exchange related files only) to a specified remote disk location.
##
## No warranties, express or implied, are available. It works for me. If
## you find errors or have problems, please feel free to let me know, but
## I can't guarantee that I can fix them.
##
## Feel free to use this in your own scripts. I would appreciate attribution.
##
Param(
	[string]$backupLocation = "C:\Backups",
	[string]$startLetter    = "g",
	[switch]$startScript    = $false
)

## $backupLocation is where the files and metadata go.

## $startLetter will contain the first letter we use to remap
## the volume letters contained in $volumes.

## $nl is the DOS newline character string

$nl = "`r`n"

## $volumes will contain the volume letters used by all named
## files and directories.

$volumes = @{}

## any storage group will contain:
## a] a system file directory
## b] a log file directory
## c] a filename for each database within the SG
##
## $pathPattern contains the dos patterns of files in the storage group

$pathpattern = @{}		### Exx.chk, Exx*.log, *.edb

## $storeList contains the filenames of the Exchange databases that need
## to be checked.

$storeList = @{}

## $letters contains the mapping between the original drive letter
## and the exposed driver letter in the shadow copy.

$letters = @{}

## $computerName contains the local computer's name

$computerName = $env:ComputerName

function buildRobocopyString($collection)
{
	$str = ""

	foreach ($filepath in $collection)
	{
		$file = split-path $filepath -leaf
		$path = split-path $filepath -parent
		#
		# the destination path is the source path appended to
		# the backup folder location.
		#
		$destpath = join-path $backupLocation $path.SubString(3, $path.Length - 3)
		#
		# the source path is the true path modified by the
		# letter of the exposed shadow copy
		#
		$letter = $letters.($path.SubString(0, 1))
		$subpath = $path.SubString(1, $path.Length - 1)
		$srcpath = "$letter$subpath"
		$str += "echo Copying " + $filepath + "..." + $nl
		$str += "robocopy " + '"' + $srcpath + '" "' + $destpath + 
		        '" "' + $file + '" /copyall /ZB >nul' + $nl
		$str += "if not errorlevel 0 goto :abort" + $nl
	}

	return $str
}

function buildESEUTILString($collection)
{
	$str = ""

	foreach ($path in $collection)
	{
		#
		# the destination path is the source path appended to
		# the backup folder location.
		#
		$path = $path.ToString()
		$destpath = join-path $backupLocation $path.SubString(3, $path.Length - 3)

		$str += "echo Checking " + $destpath + "..." + $nl
		$str += "call :checkit " + '"' + $destpath, '"' + $nl
		$str += "if not errorlevel 0 goto :abort" + $nl
	}

	return $str
}

function buildCMD
{
	$script = "@echo off" + $nl

	$script += buildRobocopyString $pathPattern.keys
	$script += $nl
	$script += buildESEUTILString $storeList.keys
	$script += $nl
	$script += "exit 0" + $nl
	$script += ":abort" + $nl
	$script += "exit 1" + $nl
	$script += $nl
	$script += ":checkit" + $nl
##	$script += "echo Checking %1" + $nl
	$script += "eseutil /k %1 >nul" + $nl
	$script += "if not errorlevel 0 exit 1" + $nl
	$script += $nl

	$scriptFile = join-path $env:temp "online-backup.cmd"
	$script | out-file $scriptFile -encoding ascii

	return $scriptFile
}

function writerOptimizationGarbage
{
	$script  = ""

	$script += "# verify presence of Exchange Writer" + $nl
	$script += "writer verify {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}" + $nl
	$script += "# exclude system writer" + $nl
	$script += "writer exclude {e8132975-6f93-4464-a53e-1050253ae220}" + $nl
	$script += "# exclude IIS config writer" + $nl
	$script += "writer exclude {2a40fd15-dfca-4aa8-a654-1f8c654603f6}" + $nl
	$script += "# exclude ASR writer" + $nl
	$script += "writer exclude {be000cbe-11fe-4426-9c58-531aa6355fc4}" + $nl
	$script += "# exclude BITS writer" + $nl
	$script += "writer exclude {4969d978-be47-48b0-b100-f328f07ac1e0}" + $nl
	$script += "# exclude WMI writer" + $nl
	$script += "writer exclude {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}" + $nl
	$script += "# exclude registry writer" + $nl
	$script += "writer exclude {afbab4a2-367d-4d15-a586-71dbb18f8485}" + $nl
	$script += "# exclude iis metabase writer" + $nl
	$script += "writer exclude {59b1f0cf-90ef-465f-9609-6ca8b2938366}" + $nl
	$script += "# exclude com+ regdb writer" + $nl
	$script += "writer exclude {542da469-d3e1-473c-9f4f-7847f01fc64f}" + $nl
	$script += "# exclude shadow-copy optimization writer (does not apply to exchange)" + $nl
	$script += "writer exclude {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}" + $nl
	$script += $nl

	return $script
}

function buildDSH([string]$cmdfilename)
{
	write-host "Building backup script"

	$script  = ""
	$script += "# Diskshadow backup script." + $nl
##	$script += "set verbose on" + $nl
	$script += "set context persistent" + $nl
	$script += "set metadata " + (join-path $backupLocation "online-backup.cab") + $nl
	$script += $nl
	$script += writerOptimizationGarbage
	$script += "begin backup" + $nl
	$script += $nl

	foreach ($drive in $volumes.keys)
	{
		$script += "add volume " + $drive + ": alias shadow_" + $drive + $nl
	}

	$script += $nl + "create" + $nl + $nl

	foreach ($drive in $volumes.keys)
	{
		$script += "expose %shadow_" + $drive + "% " + $letters.$drive + ":" + $nl
	}

	$script += $nl
	$script += "exec " + $cmdfilename + $nl

	#
	# If the batch file from exec fails, diskshadow terminates without
	# executing any more commands.
	#
	$script += "end backup" + $nl

	foreach ($drive in $volumes.keys)
	{
		## remove the temporary shadow copy and unexpose the letter
		$script += "delete shadows exposed " + $letters.$drive + ":" + $nl
	}

	$script += $nl
	$Script += "exit" + $nl

	$scriptFile = join-path $env:temp "online-backup.dsh"

	$script | out-file $scriptFile -encoding ascii
	write-host "Diskshadow script file $scriptFile"
	return $scriptFile
}

function getStores
{
	## locate the databases, both mailbox and public folder

	$colMB = get-MailboxDatabase -server $computername
	$colPF = get-PublicFolderDatabase -server $computername

	## parse them for volumes too

	foreach ($mdb in $colMB)
	{
		if ($mdb.Recovery)
		{
			write-host ("Skipping RECOVERY MDB " + $mdb.Name)
			continue
		}
		write-host ($mdb.Name + "`t " + $mdb.Guid)
		write-host ("`t" + $mdb.EdbFilePath)
		write-host " "

		$pathPattern.($mdb.EdbFilePath) = 1
		$storeList.($mdb.EdbFilePath)   = 1

		$vol = $mdb.EdbFilePath.ToString().SubString(0, 1)
		$volumes.$vol += 1
	}

	foreach ($mdb in $colPF)
	{
		## a PF db can never be in a recovery storage group
		## which is why the Recovery check isn't done here

		write-host ($mdb.Name + "`t " + $mdb.Guid)
		write-host ("`t" + $mdb.EdbFilePath)
		write-host " "

		$pathPattern.($mdb.EdbFilePath) = 1
		$storeList.($mdb.EdbFilePath)   = 1

		$vol = $mdb.EdbFilePath.ToString().SubString(0, 1)
		$volumes.$vol += 1
	}

	return
}

function getStorageGroups
{
	$count = 0
	#
	# locate the storage groups and their log files and system files
	#
	$colSG = get-StorageGroup -server $computername
	if ($colSG.Count -lt 1)
	{
		write-host "No storage groups found on server $computername"
		return 1
	}

	## parse the pathnames for each SG to determine what
	## volumes it stores data upon and what directories are used

	foreach ($sg in $colSG)
	{
		if ($sg.Recovery)
		{
			write-host ("Skipping RECOVERY STORAGE GROUP " + $sg.Name)
			continue
		}

		$count++

		$prefix  = $sg.LogFilePrefix
		$logpath = $sg.LogFolderPath.ToString()
		$syspath = $sg.SystemFolderPath.ToString()

		write-host $sg.Name.ToString() "`t" $sg.Guid.ToString()
		write-host "`tLog prefix:      $prefix"
		write-host "`tLog file path:   $logpath"
		write-host "`tSystem path:     $syspath"

		## E00*.log
		$pathpattern.(join-path $logpath ($prefix + "*.log")) = 1

		$vol = $logpath.SubString(0, 1)
		$volumes.$vol += 1

		## E00.chk
		$pathpattern.(join-path $syspath ($prefix + ".chk")) = 1

		$vol = $syspath.SubString(0, 1)
		$volumes.$vol += 1

		write-host " "
	}

	if ($count -lt 1)
	{
		write-host "No storage groups found on server $computername"
		return 1
	}

	return 0
}

function validateArrays
{
	$drives = $volumes.keys
	if ($drives.Count -lt 1)
	{
		write-host "No disk volumes were found. Aborting."
		return 1
	}

	write-host ("There were " + $drives.Count.ToString() + " disk volumes for Exchange server $computername. They are:")
	foreach ($drive in $drives)
	{
		write-host "`t$drive"
	}

	write-host " "

	$paths = $pathPattern.keys
	if ($paths.Count -lt 1)
	{
		write-host "No paths were found. Aborting."
		return 1
	}

	write-host ("There are " + $pathPattern.Count.ToString() + " directories to be backed up. They are:")
	foreach ($directory in $pathPattern.keys)
	{
		write-host "`t$directory"
	}
	write-host " "

	$letter = $startLetter.Chars(0)

	foreach ($drive in $volumes.keys)
	{
		$letters.$drive = $letter
		$letter = [char]([int]$letter + 1)
	}

	return 0
}

	##
	## Main
	##

	if ((getStorageGroups) -eq 0)
	{
		getStores
		if ((validateArrays) -eq 0)
		{
			$scriptFile = buildCMD
			$scriptFile = buildDSH $scriptFile
			if ($startScript -and ($scriptFile.Length -gt 0))
			{
				diskshadow.exe -s $scriptFile
			}
		}
	}

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

Exchange 2007 and Windows 2008: Using Diskshadow for Online Exchange Backup (part 5 of 7)

If you hadn’t noticed, I’ve been following a theme the last couple of months with some of the entries on this blog:

Part 1: Getting a List of Storage Groups in a PowerShell Script

Part 2: Getting a List of Stores in a PowerShell Script

Part 3: Exchange 2007 and Windows 2008: Offline Exchange Backup

Part 4: Volume Shadow Copy Services (VSS) and Exchange – The Basics

In this posting, which is part 5 of a 7-part series, I’ll talk about a command-line tool which is new for Windows Server 2008: Diskshadow. The entire purpose behind Diskshadow is to allow a system administrator to harness the power of VSS from an easy-to-use utility. This is not the first utility from Microsoft that uses VSS (BETest and VShadow were both part of the VSS Software Development Kit), but it is the first supported utility for using VSS.

Let me begin by displaying a Diskshadow input file, and then I’ll discuss it, line by line. In case it isn’t obvious, you do not include line numbers inside a Diskshadow input file. They are shown here just for discussion. Also, the typical extension used for a Diskshadow input file is DSH. So, you might call this file offline-backup.dsh.

1 # set verbose on
2 set context persistent
3 writer verify {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
4 begin backup
5 add volume c: alias shadow_c
6 create
7 expose %shadow_c% g:
8 # exec offline-backup.cmd
9 end backup
10 delete shadows exposed g:
11 exit

Line 1 begins with a hash-mark. This is indicative of a comment to the Diskshadow utility. Anything that occurs after the hash-mark on this line is ignored. However, if the hash-mark were not present, the command “set verbose on” would cause Diskshadow to output additional information as it determines the writers and components that will be included within the shadow copy.

Line 2 indicates that the shadow copy which is created will be persistent – that is, the shadow copy will continue to exist after the “end backup” (line 9) statement is executed. Shadow copies must be persistent in order to expose them (line 7). It is much easier to work with a shadow copy exposed as a drive letter than to use native format shadow copy names. For example, a typical shadow copy may be named \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy25 with a GUID of {e18b18b2-c8dd-4429-9996-af8d582616d8}.

Line 3 causes Diskshadow to verify that the particular writer having the named GUID is present on this computer. This GUID is the specific ID for “Microsoft Exchange Writer”. This check effectively requires that this script is executed on an Exchange server. In order to see a list off all writers and their writer IDs, you can enter “vssadmin list writers” from a command prompt or within Diskshadow you can enter the “list writers” command. Note that you can also use the “writer exclude” command to ensure that a specific writer is not called as part of this backup.

Line 4 will start the actual VSS communication process by notifying all non-excluded writers to PrepareBackup. PrepareBackup was discussed in Volume Shadow Copy Services (VSS) and Exchange – The Basics.

Line 5 is used to identify a specific volume (could be a mount-point) that must be included in this VSS snapshot. You must identify all volumes that are involved in a backup. You learned how to do that for Exchange in Getting a List of Storage Groups in a PowerShell Script and Getting a List of Stores in a PowerShell Script.

Line 6 signals VSS to initiate Freeze and to create a snapshot. When the snapshot is complete, Diskshadow will signal VSS to Thaw. Freeze and Thaw were discussed in Volume Shadow Copy Services (VSS) and Exchange – The Basics.

Line 7 tells Diskshadow to expose the snapshot of a particular volume as a different drive letter (which may also be a mount point). This is primarily for ease-of-access, as I discussed for Line 2.

Line 8 is another comment. If it were not a comment, the EXEC command would cause Diskshadow to execute an external script. THAT SCRIPT is where a copy from a snapshot is actually created. You learned about how to generate that type of script in Exchange 2007 and Windows 2008: Offline Exchange Backup. If the script returns a non-zero value, then Diskshadow aborts.

Line 9 causes Diskshadow to signal VSS to send PostBackup to all involved writers. PostBackup was discussed in Volume Shadow Copy Services (VSS) and Exchange – The Basics.

Line 10 removes the snapshot associated with the drive letter G:, and deletes the drive mapping for G:. The storage space used by the snapshot is returned to the system.

Finally, line 11 terminates Diskshadow.

It is important to realize that after the CREATE statement finishes, what you have is a snapshot. It is not a backup, just a copy of the MFT and in-use bitmap for the disk drives that were included in the snapshot. You learned about the MFT and in-use bitmaps in Volume Shadow Copy Services (VSS) and Exchange – The Basics.

The script executed by line 8 is what creates copies of the files that are your actual backup.

You should also be aware that for any writer that was excluded from the backup, the files protected by that writer are still present in the snapshot; however they are CRASH CONSISTENT (i.e., in the same condition that would’ve happened if the power plug had been pulled on a server) not APPLICATION CONSISTENT (which is what you want for maximum recoverability).

In part 6 of this series, I will put all of the pieces together, and come up with a single PowerShell script that does “everything” to create an Exchange backup for you.

In part 7 of this series, I will cover doing simple restores of Exchange databases.

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

Volume Shadow Copy Services (VSS) and Exchange – The Basics

Introduction

The Volume Shadow Copy Service (VSS) was originally added to Windows Server 2003. The first version of Exchange Server to support VSS was Exchange Server 2003. The primary reason for the existence of VSS is to improve the performance of backup operations. VSS is only available for NTFS volumes.

I will not cover specific implementation details here, but concepts; some of those concepts will be specific to NTFS and Exchange specific and some will be general to VSS.

VSS backups (as implemented by Windows, not including the implementations by SAN/NAS/and other third-party vendors), are based on two primary concepts:

  1. A disk volume’s in-use bitmap, and
  2. The Master File Table (MFT) for the volume.

A bitmap is a way of identifying the blocks of a disk volume which are being used vs. those blocks which are unused. In terms of implementation, it is usually an array (an ordered list of locations in a computer’s memory), where each entry in that array represents the status of 32 blocks of that disk. In C-language terms:

    #define block_in_use(x) (drive_map_array [Floor((int)x / 32)] & (1 << ((int)x % 32)))

This assumes that the "element size" of the array is 32 bits per element. In the case of differing element sizes, adjust the value of 32 in the define to match the word size in bits per element.

Using this define, block_in_use() will return a non-zero value if the bit-field for a given block is set to 1. This means that the block is in use by some disk-related construct. If that bit-field is zero, block_in_use() will return a value of zero. This means that the block is not in use.

The in-use bitmap doesn't care about operating systems or file systems. It simply defines what blocks are available on a volume and what blocks are NOT available on a volume.

The MFT is both an operating system and file system specific construct. For a given disk volume, the MFT identifies what files/directories/etc. exist within that operating system/file system and what block(s) of the disk volume map to those file system constructs.

Every block contained within the MFT should be marked as in-use by the in-use bitmap.

Note that the in-use bitmap and the MFT work together. Neither provides a complete picture of a disk volume without the other. The MFT works per file system. Therefore, you can have multiple MFTs per disk volume. However, the in-use bitmap is per disk volume. You can only have a single in-use bitmap per disk volume.

When a particular file system needs to allocate a block on a disk volume, it calls a disk Application Programming Interface (API) that allows a block to be added to the MFT for the file system.

VSS Components

VSS has three main components:

  • the requester - the backup application
  • the provider - an operating system component that sits "on top of" the file system and mediates access to a VSS copy, while that VSS copy is active
  • the writer - an application component that has registered with the provider to enable support for an application that needs extra processing prior to a VSS backup (typically to flush buffers to disk)

A VSS Backup Overview

A VSS backup consists of five separate phases. Errors can occur and cause a backup to fail in any phase. The phase names are:

  1. PrepareBackup
  2. Freeze
  3. ...Backup
  4. Thaw
  5. PostBackup

When the requester wants to make a backup, it makes a PrepareBackup call to the VSS API. Part of this call is identifying the writers that need to be used to make the backup (e.g., if you are making an Exchange-only backup, you can exclude the COM+DB writer and the Registry writer; in fact you can exclude all writers except for the Exchange writer).

During PrepareBackup, VSS initializes each involved writer and indicates to the writer that a backup will be starting soon. After the return from PrepareBackup, the requester will make a Freeze call to the VSS API.

During the Freeze process, all involved writers will cause their subsystems to be application consistent on disk. That is, if buffers need to be flushed; they will be. If transaction logs need to be switched, they will be. Once a writer returns success to its Freeze call, it cannot make any modifications to disk until notified by VSS. This is enforced by the writer.

Once all writers are frozen, VSS creates a snapshot. A snapshot consists of, for all practical purposes, a copy of the MFT and the in-use bitmap of the disk. The entire file system is not processed, only a copy is made. That is what provides the speed in creating snapshots. Also note that only the involved writers have application consistency at this point. All other files are crash-consistent. A snapshot is exposed to the application as a mount-point. This mount point may be persistent or it may be ephemeral. A persistent mount point continues to exist after a VSS backup is complete.

After the snapshot is created, VSS will inform each involved writer to Thaw. At this time, applications are again allowed to make updates to the disk volume, using copy-on-write (which is done by the writer - the complexity of this is hidden from the application). That is, if a block in a file contained within the snapshot changes, then that block is not over-written on disk. Instead a new block is allocated and the MFT for that file will be updated to contain the new block. This allows a snapshot to maintain an unchanged image of the disk from when the snapshot occurred.

Now, the requester can take its backup from the snapshot. The requester may be as simple as cmd.exe's copy command or it may be a third-party backup application or it may be a SAN's special disk duplicating software. The requester is required to return a status back to VSS to indicate whether the backup was successful.

When the backup phase is complete, VSS will call each involved writer indicating a status of PostBackup. At this point each writer can do whatever clean up (if any) may be necessary. The writer is also informed of the status of the requester's backup. In some cases, this may affect PostBackup handling by the writer. For example, if a successful backup of a storage group occurred, the Exchange writer will flush the transaction logs for that storage group. If the backup failed, the transaction logs will not be flushed.

If the the snapshot was persistent, it will remain after PostBackup. If it was ephemeral, it will be deleted.

So, consider the example of backing up an Exchange server and only the Exchange portion of the server. The process will go like this:

  1. A backup application calls VSS and says that it wants to back up Exchange and only Exchange
  2. VSS calls the Exchange Writer with PrepareBackup.
  3. Asynchronously, Exchange initiates a transaction log switch and pauses the Lazy Writer process (which is responsible for flushing updates from the memory cache to the Exchange database) and the Log Writer process (which writes updates to a log file).
  4. VSS calls the Exchange Writer with Freeze
  5. Exchange waits for the transaction log switch to be complete and for the Lazy Writer and Log Writer processes to be paused
  6. VSS creates the snapshot
  7. VSS calls the Exchange Writer with Thaw
  8. Exchange unpauses the Lazy Writer and the Log Writer processes
  9. The backup application will back up all system files (Exx.CHK for each storage group), all log files (Exx*.log for each storage group), and all database files (*.EDB for each database) and return success or failure back to VSS
  10. VSS calls the Exchange Writer with PostBackup with the status of the backup
  11. If the backup was successful, transaction log files will be flushed
  12. The backup application disconnects from VSS and the backup is complete

As you can see, while there are a number of steps involved in the process it really isn't that complicated. Given a little help (in terms of a programmatic interface to VSS), it shouldn't be that hard to back up an Exchange server.

We'll talk about that in our next article in this series.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

Exchange 2007 and Windows 2008: Offline Exchange Backup

In my article Getting a List of Stores in a PowerShell Script you learned how to obtain a list of all the files involved for the Exchange database stores on a particular Exchange server. In the preceding article, Getting a List of Storage Groups in a PowerShell Script, you learned how to obtain a list of all the files unique to the Exchange storage groups on a particular Exchange server.

As a part of both of those articles, you learned how to create a list of the volumes used by the files in the storage groups and in the database stores.

Now that we have that information, what can we do with it?

Easy! We can generate a script that can create an offline backup of our Exchange databases. In future articles, you’ll learn how to turn this offline backup into an online backup, using VSS (the Volume Shadow Copy Service).

As a quick reminder, the following global objects are important and were introduced in the earlier articles of this series:

$volumes – a hash array containing the disk volumes used by the Exchange storage groups and database stores

$pathPattern – a hash array containing a list of all the regular-expression patterns required to back up all the files involved for all Exchange storage groups and database stores

getStores – a function populating $volumes and $pathPattern for the files used by Exchange database stores

getStorageGroups – a function populating $volumes and $pathPattern for the files used by Exchange storage groups

validateArrays – a function verifying that the $volumes and $pathPattern arrays are not empty; the function returns zero if the main program should proceed.

A utility that we have not previously discussed is robocopy. Robocopy was introduced as a part of the Windows 2000 Server Resource Kit. Among other features, it copies large files as quickly as possible, much more quickly than the cmd.exe copy and xcopy. Robocopy is a standard utility in Windows Vista and Windows Server 2008.

Two additional global variables need to be introduced:

$destination – the directory below which backups will be stored; in the case of Exchange backups, the directory structure is reproduced identically. For example, if $destination is “C:\Backups” and the file “C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\E00.CHK” is a file to be backed up, then the destination file name will be “C:\Backups\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\E00.CHK”.

$nl – a DOS newline

So, after all the preparation we’ve already done, an offline backup script actually turns out to be quite simple. The PowerShell script below will generate a DOS script to be executed by cmd.exe. Robocopy will copy all relevant files to the backup location specified by $destination. If all copies succeed then the script succeeds. If any copy fails, the script aborts.

A couple of things to be careful of – this script actually executes the backup! To do that, in an offline mode, the Microsoft Exchange Information Store service must be stopped. When that service is stopped, Exchange is basically down. So…don’t run this on your production system without commenting out the line that starts with cmd.exe (unless you are actually doing the offline backup).

Secondly, offline backups do not purge transaction logfiles. We’ll need to learn how to do an online backup before we can make that happen.

Finally, the out-file cmdlet uses an unusual parameter: “-encoding ascii”. This is because cmd.exe does not understand Unicode files (which is the default for out-file). Something to remember for your own scripts!

  
    $destination = "C:\backups"

    $nl = "`r`n"

    function buildRobocopyString($collection)
    {

        $str = ""

        foreach ($filepath in $collection)
        {
            $file = split-path $filepath -leaf
            $path = split-path $filepath -parent
            #
            # the destination path is the source path appended to
            # the backup folder location.
            #
            $destpath = join-path $destination $path.SubString(3, $path.Length - 3)

            $str += "echo Copying " + $file + "..." + $nl
            $str += "robocopy " + '"' + $path + '" "' + $destpath + 
                '" "' + $file + '" /copyall /ZB >nul' + $nl
            $str += "if not errorlevel 0 goto :abort" + $nl
        }

        return $str
    }

    function buildCMD
    {
        $script = "@echo off" + $nl

        $script += 'net stop "Microsoft Exchange Information Store" /y' + $nl

        $script += buildRobocopyString $pathPattern.keys
	$script += $nl
        $script += 'net start "Microsoft Exchange Information Store"' + $nl
        $script += "exit 0" + $nl
        $script += ":abort" + $nl
        $script += "exit 1" + $nl

        $script | out-file (join-path (gc env:temp) "offline-backup.cmd") -encoding ascii
        cmd.exe /c (join-path (gc env:temp) "offline-backup.cmd")
    }

    #
    # Main
    #

    if ((getStorageGroups) -eq 0)
    {
        getStores
        if ((validateArrays) -eq 0)
        {
            buildCMD
        }
    }

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

Getting a List of Stores in a PowerShell Script

In my last post, Getting a List of Storage Groups in a PowerShell Script, you saw how to use the information from the Get-StorageGroup cmdlet to discover the particular disk volumes used by a storage group and to build a list of the files that were used by the storage group.

Now, that list of files did not contain the databases contained within the storage group. Instead, it simply contained the system file (Exx.CHK) for each storage group and the log files (Exx*.LOG) for each storage group. This is because the Get-StorageGroup cmdlet does not return us that information. Instead, we use the Get-MailboxDatabase and Get-PublicFolderDatabase cmdlets to find out the name of our databases. It’s unfortunate, but there is not a single Get-ExchangeDatabase cmdlet that combines the functionality of both,

Note that in Exchange Server 2007, each database consists of a single file, with an extension of EDB. In Exchange Server 2003, there was also a second file per database called the streaming file with an extension of STM.

To remind you of the global variables being used:

  • $computername is a string that contains the name of the computer on which the script is being executed.
  • $volumes is a hash array that contains a list of all the disk volumes so far detected.
  • $pathpattern is a hash array that contains a list of all the fully-qualified paths to all files so far discovered.

A store is always a simple filename. However, it’s important to remember that all of the file names that are used for naming stores and storage groups in Exchange Server may contain spaces and parentheses. That can lead to a requirement for special handling of file names.

When interrogating the list of stores on a particular server, any stores present in the Recovery Storage Group are also listed. These should be ignored.

Each cmdlet we use will return a collection. Get-MailboxDatabase will return a collection of all mailbox databases present on the given server. This collection could be empty even if storage groups are present. Get-PublicFolderDatabase will return a collection of all public folder databases present on the given server. This collection could also be empty (and, in fact, is more likely to be empty). The PowerShell code needs to be prepared to handle those eventualities.

In the PowerShell code below, the getStores function obtains all the stores on the server and adds the store filenames to the $pathpattern array and adds the disk volumes used by the stores to the $volumes array. Following getStores is the validateArrays function. If either the $volumes or the $pathpattern array is empty, it returns a value of 1 and displays a message on the PowerShell host. If both have contents, validateArrays returns a value of 0 and displays the contents of those arrays.

Without further ado:

function getStores
{
	## locate the databases, both mailbox and public folder

	$colMB = get-MailboxDatabase -server $computername
	$colPF = get-PublicFolderDatabase -server $computername

	## parse them for volumes too

	foreach ($mdb in $colMB)
	{
		if ($mdb.Recovery)
		{
			write-host ("Skipping RECOVERY MDB " + $mdb.Name)
			continue
		}
		write-host ($mdb.Name + "`t " + $mdb.Guid)
		write-host ("`t" + $mdb.EdbFilePath)
		write-host " "

		$pathPattern.($mdb.EdbFilePath) = 1

		$vol = $mdb.EdbFilePath.ToString().SubString(0, 1)
		$volumes.$vol += 1
	}

	foreach ($mdb in $colPF)
	{
		## a PF db can never be in a recovery storage group
		## which is why the Recovery check isn't done here

		write-host ($mdb.Name + "`t " + $mdb.Guid)
		write-host ("`t" + $mdb.EdbFilePath)
		write-host " "

		$pathPattern.($mdb.EdbFilePath) = 1

		$vol = $mdb.EdbFilePath.ToString().SubString(0, 1)
		$volumes.$vol += 1
	}

	return
}

function validateArrays
{
	$drives = $volumes.keys
	if ($drives.Count -lt 1)
	{
		write-host "No disk volumes were found. Aborting."
		return 1
	}

	write-host ("There were " + $drives.Count.ToString() + " disk volumes for Exchange server $computername. They are:")
	foreach ($drive in $drives)
	{
		write-host "`t$drive"
	}

	write-host " "

        $paths = $pathPattern.keys
        if ($paths.Count -lt 1)
        {
                write-host "No paths were found. Aborting."
                return 1
        }

	write-host ("There are " + $pathPattern.Count.ToString() + " directories to be backed up. They are:")
	foreach ($directory in $pathPattern.keys)
	{
		write-host "`t$directory"
	}
	write-host " "

	return 0
}

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch

Getting a List of Storagegroups in a PowerShell Script

In Getting Our Computername in a PowerShell Script, you learned how to do just that – store the running computer name (the short NetBIOS name) into a PowerShell variable. At the same time, you learned WHY and HOW it worked. From this point forward, I’ll presume that you’ve executed this particular PowerShell statement:

$global:computername = (gc env:computername)

Note that this causes the storage scope of the $computername variable to be global. It can be interrogated from any function as just $computername (as long as there are no variables that have the same name in a closer scope [such as local scope or function scope]), but for the global value to be updated requires the inclusion of the “global:” specifier.

Thankfully, when we install the Exchange Tools on a workstation or server, one of the Exchange cmdlets is get-storagegroup. To see a list of all the storage groups on all the (Exchange 2007+) servers in your Exchange organization, you just enter get-storagegroup at the prompt in an Exchange Management Shell (EMS).

So, are we done? Nope.

While that list is probably all we need if we are working from a command prompt, if we are writing a script, we are probably interested in more than just the displayed information. For example, consider these requirements of the things we may want to know:

  • where the system files for the storage groups are placed
  • where the log files for the storage groups are placed
  • the log file prefix for the storage groups
  • a list of all the disk volumes used by the storage groups

Using get-storagegroup plus a feature of PowerShell makes all of this pretty easy. The feature we will use is called an associative array. Also known as a hash array, for those of you with a Perl background. An associative array is basically a two-dimensional array that is mapped to a single dimensional array based on the index element. Unlike a normal array, an associative array can be indexed by almost anything – a string, a regular expression, an integer – anything you may want to use. In order to pull off this “magic”, an associative array actually consists of two parallel arrays – the keys array and the values array. Now, these arrays get indexed as you are used to – by integers, starting at zero. However, their contents may consist of any object.

Note: in Perl, associative arrays aren’t quite as flexible. In PowerShell, you can literally have an object as a key and an object as a value. This allows you the flexibility of storing arbitrarily complex values into an associative array. However, the search through the keys array is linear. Thus, you should probably keep the arrays fairly small.

While in the EMS, you may get the impression that get-storagegroup returns text, that would be incorrect. Instead, get-storagegroup returns a collection (which is a fancy word for an enumerable array) of objects of type Microsoft.Exchange.Data.Directory.SystemConfiguration.StorageGroup. That’s a mouthful. Let’s just say that it returns an array containing all of the storage group objects.

So, to process that array, you would do something like this:

$collection = get-storagegroup
foreach ($entry in $collection)
{
        ... do something with $entry ...
}

However, you are more likely to want to look at the storage groups on a particular server. So, that leads you to a construct like this:

$collection = get-storagegroup -server $computername
foreach ($entry in $collection)
{
        ... do something with $entry ...
}

Note the tie-in back to $computername! This will access the global $computername variable which has the name of the running computer stored within it.

Now, what does a storage group consist of? It really has only two things:

  • log files
  • system files

And each of those two things also has a prefix associated with them (which allows multiple storage groups to share a single directory). All log files have a particular format:

.LOG

All system files (there is only a single system file per storage group) have a particular format:

.CHK

A prefix follows this format:

[E ] | [R00]

In words, a prefix is either R00, which is exclusively for the Recovery Storage Group, of which there may be only one per server; or the letter ‘E’ followed by a two-digit number. The number represents the index of the storage group on the server, where the first storage group created is ’00’ the second is ’01’, etc. So the first storage group created on a server will have a prefix of E00.

For that same first storage group, the log files have a format of E00*.log and the system file is named E00.chk.

Note: the number of digits contained in went from five hexidecimal digits in Exchange 2003 (and all earlier versions) to eight hexidecimal digits in Exchange 2007+. Even with the change in log file size from 5 MB to 1 MB, this means that you get 80+ times as many log files before rollover in Exchange 2007+ as you did in earlier versions of Exchange Server.

Let’s see…what else? Oh yes – you want to ignore recovery storage groups. Except for recoveries from backup, you are not supposed to touch them.

Now, given all those above details, what can we do with them? This!

## $volumes will contain the volume letters used by all named
## files and directories.

$global:volumes = @{}

## any storage group will contain:
## a] a system file directory
## b] a log file directory
## c] a filename for each database within the SG
##
## $pathPattern contains the dos patterns of files in the storage group

$global:pathpattern = @{}		### Exx.chk, Exx*.log, *.edb

function getStorageGroups
{
        $count = 0
	#
	# locate the storage groups and their log files and system files
	#
	$colSG = get-StorageGroup -server $computername
	if ($colSG.Count -lt 1)
	{
		write-host "No storage groups found on server $computername"
		return 1
	}

	## parse the pathnames for each SG to determine what
	## volumes it stores data upon and what directories are used

	foreach ($sg in $colSG)
	{
		if ($sg.Recovery)
		{
			write-host ("Skipping RECOVERY STORAGE GROUP " + $sg.Name)
			continue
		}

                $count++

		$prefix  = $sg.LogFilePrefix
		$logpath = $sg.LogFolderPath.ToString()
		$syspath = $sg.SystemFolderPath.ToString()

		write-host $sg.Name.ToString() "`t" $sg.Guid.ToString()
		write-host "`tLog prefix:      $prefix"
		write-host "`tLog file path:   $logpath"
		write-host "`tSystem path:     $syspath"

		## E00*.log
		$pathpattern.(join-path $logpath ($prefix + "*.log")) = 1

		$vol = $logpath.SubString(0, 1)
		$volumes.$vol += 1

		## E00.chk
		$pathpattern.(join-path $syspath ($prefix + ".chk")) = 1

		$vol = $syspath.SubString(0, 1)
		$volumes.$vol += 1

		write-host " "
	}

	if ($count -lt 1)
	{
		write-host "No storage groups found on server $computername"
		return 1
	}

	return 0
}

This routine stores, for each storage group, the files that are contained within that storage group. It also stores away the disk volumes used by that storage group. For the write-host output of the function, you could surround the blocks by $debug conditional statements to minimize the output of the routine (or just remove them entirely).

So, what is contained within storage groups? Databases! In our next post in this series, you’ll learn how to deal with them programatically too.

Until next time…

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Follow me on twitter: @EssentialExch