Problem Removing Recovery Storage Group After a Restore

Probably just a minor bug in the Exchange Console GUI (SP1 w/ hotfix rollup 7), but I found myself unable to remove the recovery storage group after a restore. I dismounted the database (which reported a successful completion), but when I attempted to remove the storage group, the console complained that the database must first be dismounted. Switching to the shell, I ran the following command…

Remove-MailboxDatabase -Identity "mailbox-server\recovery storage group\restoreddatabase" -verbose

… which produced the following output:

VERBOSE: Remove-MailboxDatabase : Beginning processing.
VERBOSE: Remove-MailboxDatabase : Administrator Active Directory session settings are:  View Entire Forest:
'False', Default Scope: 'mydomain.pvt', Configuration Domain Controller: ‘mydc.mydomain.pvt',
VERBOSE: Remove-MailboxDatabase : Searching objects "mailbox-server\recovery storage group\restoreddatabase" of type
"MailboxDatabase" under the root "$null".
VERBOSE: Remove-MailboxDatabase : Previous operation run on domain controller 'mydc.mydomain.pvt'.
VERBOSE: Remove-MailboxDatabase : Establishing the Admin RPC connection with Server
"mailbox-server.mydomain.pvt".
VERBOSE: Remove-MailboxDatabase : Checking the status of database "mailbox-server\Recovery Storage
Group\restoreddatabase".

Confirm
Are you sure you want to perform this action?
Removing Mailbox Database "mailbox-server\recovery storage group\restoreddatabase".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

Pressing enter to accept the default [Y] response yields this:

VERBOSE: Remove-MailboxDatabase : Establishing the Admin RPC connection with Server
"mailbox-server.mydomain.pvt".
VERBOSE: Remove-MailboxDatabase : Checking the status of database "mailbox-server\Recovery Storage Group\restoreddatabase".
VERBOSE: Remove-MailboxDatabase : Taking cluster resource of database "mailbox-server\Recovery Storage Group\restoreddatabase" offline on server "mailbox-server".
VERBOSE: Remove-MailboxDatabase : Checking the status of database "mailbox-server\Recovery Storage Group\restoreddatabase".
VERBOSE: Remove-MailboxDatabase : Searching objects "mailbox-server\Recovery Storage Group" of type "StorageGroup" under the root "$null".
VERBOSE: Remove-MailboxDatabase : Previous operation run on domain controller 'mydc.mydomain.pvt'.
VERBOSE: Remove-MailboxDatabase : Dismounting the database, "mailbox-server\recovery storage group\restoreddatabase".
VERBOSE: Remove-MailboxDatabase : Searching objects "mydomain.pvt/Microsoft Exchange System
Objects/SystemMailbox{021114f7-3b45-464c-bffd-c5c9821aabbb}" of type "ADRecipient" under the root "$null".
VERBOSE: Remove-MailboxDatabase : Previous operation run on domain controller 'mydc.mydomain.pvt'.
VERBOSE: Remove-MailboxDatabase : Deleting "mailbox-server\Recovery Storage Group\restoreddatabase" of type
"MailboxDatabase".
VERBOSE: Remove-MailboxDatabase : Previous operation run on domain controller 'mydc.mydomain.pvt'.
WARNING: The specified database has been removed. You must remove the database file located in H:\MyStorageGroup\RSG20091202084953\restoreddatabase.edb from your computer manually if it exists. Specified database:
restoreddatabase
VERBOSE: Remove-MailboxDatabase : Searching objects of type "MailboxDatabase" with filter "(OriginalDatabase Equal
021114f7-3b45-464c-bffd-c5c9821aabbb)", scope "SubTree" under the root "$null".
VERBOSE: Remove-MailboxDatabase : Previous operation run on domain controller 'mydc.mydomain.pvt'.
VERBOSE: Remove-MailboxDatabase : Updating cluster database resource on server "mailbox-server".
VERBOSE: Remove-MailboxDatabase : Ending processing.

Now the restored database has been both dismounted and removed from Exchange. All that’s left is to go into the file system and delete the files themselves.

To finish cleaning up, we just need to remove the Recovery Storage Group. With the database dismounted and removed, we would be able to do this from the Exchange Console now, but since we already have a shell window open, we’ll just do it from there.

Remove-StorageGroup -Identity "mailbox-server\recovery storage group" -verbose

VERBOSE: Remove-StorageGroup : Beginning processing.
VERBOSE: Remove-StorageGroup : Administrator Active Directory session settings are:  View Entire Forest: 'False',
Default Scope: 'mydomain.pvt', Configuration Domain Controller: 'mydc.mydomain.pvt',
VERBOSE: Remove-StorageGroup : Searching objects "mailbox-server\recovery storage group" of type "StorageGroup" under the root "$null".
VERBOSE: Remove-StorageGroup : Previous operation run on domain controller 'mydc.mydomain.pvt'.

Confirm
Are you sure you want to perform this action?
Removing Storage Group "mailbox-server\recovery storage group".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):
VERBOSE: Remove-StorageGroup : Deleting "mailbox-server\Recovery Storage Group" of type "StorageGroup".
VERBOSE: Remove-StorageGroup : Previous operation run on domain controller 'mydc.mydomain.pvt'.
WARNING: The specified storage group has been removed. You must remove the log file located in J:\MyStorageGroup\RSG20091202084953 from your computer manually if it exists. Specified storage group: mailbox-server\recovery storage group.
VERBOSE: Remove-StorageGroup : Ending processing.

That’s all there is. Just have to delete the RSG20091202084953 folders under the MyStorageGroup folders on the H: and J: drives to remove the restored database and transaction logs. Then on to the next restore!

Exchange TechCenters

Exchange 2010 TechCenter - http://technet.microsoft.com/en-us/exchange/2010/default.aspx
Exchange 2007 TechCenter - http://technet.microsoft.com/en-us/exchange/bb330841.aspx
Exchange 2003 TechCenter - http://technet.microsoft.com/en-us/exchange/bb288525.aspx

Free Windows PowerShell ebook

Sapien (makers of the PrimalScript script editor) has released their Windows PowerShell v.1 TFM eBook as a FREE download to the public. This is the one book I constantly have open on my desk, and it contains loads of valuable information about PowerShell's core workings and cmdlets. Download the eBook today at http://www.primaltools.com/downloads/communitytools/. Just look for the entry entitled "Powershell TFM V1 eBook".

No Exchange 2010 in 2010

Found out today that we have no plans to implement Exchange 2010 next year. Bummer.

List Calendar Delegates for One or More Mailboxes

A not-so-uncommon problem, at one time or another in most Exchange organizations, in when a user sends out a meeting request and receives a non-delivery report for a former employee. In most of these cases, this is caused by the former employee being listed as a delegate on another user’s calendar. Tracking down the source of this problem can be time-consuming.

Prior to Exchange 2007, this type of problem was difficult to track down, especially if the meeting request had been sent to a large group of people, because you would have to open each user’s mailbox, choose Options from the Tools menu, and then check the Delegates tab. With Exchange 2007, you can just use a powershell command to retrieve the delegates list for one or more mailboxes.

To view the delegates list for a single mailbox, use this command:

Get-MailboxCalendarSettings -Identity <mailbox> | format-list identity,resourcedelegates

To view the delegates list for all mailboxes in a distribution list, use this command:

Get-DistributionGroupMember <distribution group name> | Get-MailboxCalendarSettings | fl identity,resourcedelegates

Since most users do not have delegates, it might be useful to filter out those users with empty delegates lists, leaving only those with delegates, so let’s add a filter to the previous command:

Get-DistributionGroupMember <distribution group name> | Get-MailboxCalendarSettings | where {$_.resourcedelegates -notlike “”} | fl identity,resourcedelegates

To view the delegates list for all mailboxes in a particular database, use this command:

Get-Mailbox -Database <server\database> | Get-MailboxCalendarsettings | where {$_.resourcedelegates -notlike “”} | fl identity,resourcedelegates

To view the delegates list for all mailboxes on a particular server, use this command:

Get-Mailbox -Server <server name> -ResultSize unlimited | Get-MailboxCalendarsettings | where {$_.resourcedelegates -notlike “”} | fl identity,resourcedelegates

Finally, to view the delegates for all mailboxes in the organization, use this command:

Get-Mailbox -ResultSize unlimited | Get-MailboxCalendarsettings | where {$_.resourcedelegates -notlike “”} | fl identity,resourcedelegates

Depending on the size of your organization, these commands may take a bit of time to run.

Microsoft Exchange 2010 Virtual Labs

Configuring Exchange Server 2010 Unified Messaging and OCS 2007 R2 Integration Virtual Lab:
http://go.microsoft.com/?linkid=9672532

Transitioning from Exchange Server 2007 to Exchange Server 2010 (Beta) Virtual Lab:
http://go.microsoft.com/?linkid=9670971

Exchange Server 2010 (Beta) HA and Storage Scenarios Virtual Lab:
http://go.microsoft.com/?linkid=9670970

Configuring Exchange Server 2010 Unified Messaging Virtual Lab:
http://go.microsoft.com/?linkid=9670969

Exchange Server 2010 (Beta) Setup and Deployment Virtual Lab:
http://go.microsoft.com/?linkid=9670799

Transaction Logs and Checkpoint Files

When creating a new storage group, if you’re not careful, you could end up putting your transaction logs in a different location than the checkpoint file. The installation wizard is a little vague on this topic, calling the locations, “log file path” and “system file path”. Transaction logs are written to the log file path, but the checkpoint file is written to the system file path.

Although using different locations for these paths performs just fine, the Exchange Best Practices Analyzer (ExBPA) flags it, since it could affect your ability to recover messages in the event of a drive failure.

Rectifying this situation is fairly simple - dismount the databases in the storage group, then use the Move-StorageGroupPath command to specify the same “system files” path as the “log files” path. Exchange will automatically move the checkpoint file to the new location for you, so all you have to do is re-mount the database and you’re done. Unless your mailbox server is a CCR cluster.

Since CCR clusters use separate storage for each node, Exchange won’t move the transaction logs or checkpoint files automatically. In a CCR environment, the Move-StorageGroupPath command only updates Active Directory with the new configuration, but YOU are responsible for moving the files to the new locations manually, on both nodes. So the process goes like this:

1. run Suspend-StorageGroupCopy to suspend the copying of transaction logs to the other node
2. run Dismount-Database to dismount the mailbox database in this storage group
3. run Move-StorageGroupPath to specify the new location for the system and/or log files
4. manually move the files to the locations specified in the previous command, on both nodes
5. run Mount-Database to mount the mailbox database
6. run Resume-StorageGroupCopy to resume the copying of transaction logs to the other node again

Repeat the above steps for each storage group, then run Get-StorageGroupCopyStatus to verify the status of the transaction log copy process. Depending on how long the copy process was suspended, you may have to run this several times until everything catches up and the SummaryCopyStatus is healthy for all storage groups.

Exchange services won’t start after patching

After our monthly OS patch and reboot, most of the Exchange services would not restart on one of our Exchange 2007 servers. The server in question is a special-purpose machine, holding the CAS, HUB, and Mailbox roles, but it’s for internal use only, and thus has no internet access. The event log showed only that the services would, “not start in a timely fashion”, and that each timed out after 30000 ms.

Since the services worked just fine prior to patching, our first thought was to remove the patches. They uninstalled with no problem, but the services still wouldn’t start. Rebooted again. Same problem, same errors in the event log.

Since this was affecting a production system, I opened a support incident with Microsoft, then set about researching the problem myself while waiting for the callback from an Exchange engineer. Since all I had to go on were the rather generic-sounding event log entries, I tried googling a few phrases. I quickly found several links that proved useful.

KB944752 – this article describes almost exactly what we were encountering, except that it concerns service failures that occur after installing a hotfix rollup, which we had not done.

This article describes a problem with services not starting in a secure Exchange environment. It also mentions post hotfix rollup installation.

This MS Exchange Team blog entry is very similar to the two above, and concerns service failures related an inability to verify the certificate that Microsoft used to sign the code in the .Net framework common runtime assembly.

The basic problem seems to be that whenever a module in the .Net assembly is updated, the first time the service is run, it attempts to validate the code signature by verifying the certificate against a list maintained by Microsoft at crl.microsoft.com, and since these secure servers can’t get to that site, the call times out and the services refuse to start.

Microsoft hadn’t called back in three hours, so I figured why not try the suggestions in the articles, and it worked. I added the following line to the config files for each of the affected service executables:

<generatePublisherEvidence enabled="false"/>

Once entered, the services started without delay, and the problem was solved. For a detailed explanation of the fix, see the articles above, as they explain it better than I ever could.

Now that that problem was solved, the whole reason I was on site that day was to install the latest hotfix rollup on the Exchange servers. But even that took extraordinarily long on that same server, when the installer seemed to hang while updating the .Net binaries. I did some more research and discovered a known problem concerning the installation of hotfix rollups on secure Exchange servers, for the same reason – the installer was attempting to verify the new modules against the certificate revocation list at Microsoft. The install will still work, but it will take very long since it tries to verify every module as it is being installed.

After 45 minutes, I started looking for ways to speed up the process, and once I found it, I was a bit embarassed that I hadn’t thought of it myself. I opened the hosts file on the secure Exchange server, and added an entry for crl.microsoft.com, then pointed it at 127.0.0.1. It won’t find the certificate revocation list there, but it will fail immediately instead of waiting for the site lookup to time out. Within 5 minutes, the hotfix rollup installation completed with no errors and everything is running smoothly now.

Oh, and by the time Microsoft finally called back, I had solved the original problem and was installing the hotfix rollup, so I just thanked him for calling and closed the case. The engineer congratulated me on solving the problem myself, which was nice of him. :-)