Understanding Single Item Recovery for Exchange (4034387)

This article describes how single items are deleted and recovered in Exchange 2003, 2007, and 2010.

Users often delete data from their mailboxes that they later want recovered. Recovery of these deleted items is the most common reason for IT admins to recover data from traditional point-in-time backups today. With previous versions of Exchange Server, administrators implemented two solutions to enable single item recovery, dumpster and restores. Both had their issues, unfortunately. Exchange 2010 aims to reduce the complexity and administrative costs associated with single item recovery.

The following definitions may be useful for understanding the content within this article:

1. Store Soft Delete – this is when an item has been deleted from the Deleted Items folder and placed within dumpster.
2. Store Hard Delete – This is when an item has been marked for purging out of the store.
3. Hard Delete via Outlook (Shift-Delete) – This is when the end user performs a shift-delete on an item. This results in the item bypassing the deleted items folder and being directly placed in dumpster.

Single Item Recovery in Previous Versions of Exchange

The end user single item recovery functionality was enabled through the store via the store dumpster. Administrators configured the dumpster setting on a per database or per mailbox basis. The default deleted item retention window in Exchange 2003 is 7 days, while with Exchange 2007 the default is 14 days.

The end user recovery process worked typically as follows:

1. User receives message.
2. User deletes the message. The message is moved to the Deleted Items folder.
3. The user empties the deleted items folder (or an automated process like Managed Folders or Records Management deletes items out of the deleted items folder on a scheduled basis).
4. The user then realizes he requires the previously deleted item. Using Outlook the user accesses Recover Deleted Items.
5. Assuming the deletion timestamp of the deleted item is still within the deleted item retention period, the user has two choices:The user can recover the item. The recovered item is placed back in the Deleted Items folder.
   The user can purge the item from recover deleted items. This results in the deletion of the message permanently from the user’s mailbox.

If the item’s deletion timestamp is beyond the deleted item retention period, then the item is not available for end user recovery. Instead, the user must call the help desk and request recovery of the item. This involves:

• The user knowing when he deleted the item.
• The Exchange administrator locates a backup that contains the item in question.
• The Exchange administrator successfully restores the database to a recovery storage group.
• The Exchange Administrator extracts the deleted item and provides it back to the user. This of course assumes that there is a policy that allows for single item restoration and that this process isn’t a really low priority for the Exchange administrator.

If the backup is invalid, or if there is no backup for the time period in question, then the deleted data is unrecoverable. The other issue that needs highlighting is the fact that in previous versions of Exchange there is no way to prevent the end user from purging the data from the Recover Deleted Items view. This poses a significant legal risk for organizations that must ensure compliance requirements are met.

Dumpster Changes

In previous releases of Exchange Server, dumpster is essentially a view stored per folder. Items in the dumpster (henceforth known as Dumpster 1.0) stay in the folder where they were soft-deleted (shift-delete or delete from Deleted Items) and are stamped with the ptagDeletedOnFlag flag. These items are special-cased in the store to be excluded from normal Outlook views and quotas. In addition, data with this flag cannot be searched or indexed.

Note: Users can perform a shift-delete and cause a message to bypass the Deleted Items folder and go straight to dumpster. Prior to Outlook 2007, the Recover Deleted Items tool, by default, only exposed the dumpster view for the Deleted Items folder.

By setting the DumpsterAlwaysOn registry key (http://support.microsoft.com/kb/886205) the Recover Deleted Items tool can enable all folders in the mailbox and thus expose dumpster data for any folder in the Exchange 2003 or Exchange 2007 mailbox.

One of the key architectural changes in Exchange 2010 is to enable a litigation hold experience for customers that have legal compliance requirements. As a result, Exchange 2010 must meet these requirements:

• Exchange has to ensure that dumpster data moves with the mailbox.
• Dumpster data must be indexed and discoverable.
• Dumpster must have a quota.
• Exchange has to prevent purging of data from dumpster.
• Exchange has to track edits of certain content.
• Dumpster should be per mailbox and not per folder.

In order to facilitate these requirements, Dumpster was re-architected. Unlike Dumpster 1.0, Dumpster 2.0 is no longer simply a view. Dumpster in Exchange 2010 is implemented as a folder called the Recoverable Items and is located within the Non-IPM subtree of the user’s mailbox (note that this is a hidden section of the mailbox and is not exposed to the end user through any client interface).

The folder has three sub-folders:
1. Deletions
2. Versions
3. Purges

The Deletions folder replaces the ptagDeletedOnFlag view that was displayed when a user accessed the Recover Deleted Items tool. When a user soft deletes or performs an Outlook hard delete against an item, the item is moved to the Recoverable Items\Deletions folder. When the user accesses Outlook/OWA Recover Deleted Items, the RPC Client Access service translates the request and returns the Recoverable Items\Deletions folder view.

The Versions and Purges folders will be covered in the Single Item Recovery in Exchange 2010 section.
By architecting Dumpster to be a folder, three of the requirements are immediately met:

• Dumpster data is now indexed and discoverable.
• Dumpster data can now be moved with the mailbox.
• Dumpster data is now stored on a per-mailbox basis rather than a per folder basis.

From an end-user perspective, this means that deleted data is now easier to recover because Recover Deleted Items tool will now expose deleted data across the entire mailbox.

To ensure that Denial of Service attacks by placing large quantities of data into dumpster are prevented, Dumpster 2.0 has configurable quota settings. These settings can be configured per database and per mailbox:

• RecoverableItemsWarningQuota – Sets the soft limit that defaults to 20 GB. When the Recoverable Items folder reaches that size, the Exchange administrator will be notified via an event log alert. This alert should fire at the time the mailbox reaches the limit and once daily afterward. By default, the mailbox is not configured with this property, meaning that database-level limits are used. At this limit items will begin to be deleted from the dumpster using the first-in / first-out (FIFO) principle – essentially, the oldest items in the dumpster are deleted first. For example, consider a mailbox that has a dumpster that is 20GB in size and the user deletes an additional 50MB worth of data. In this case, the oldest 50MB worth of data is deleted to make room for the new 50MB worth of data.
• RecoverableItemsQuota – Sets the hard limit that defaults to 30 GB. At that limit, soft deletes will fail. The Exchange administrator will be notified via an event log alert. This alert should fire at the time the mailbox reaches the limit and once daily afterward. By default, the mailbox is not configured with this property, meaning that database-level limits are used.
  Note: Exchange 2010 includes capability for each mailbox to also maintain an archive mailbox as well. There is a dumpster for both the primary mailbox and the archive mailbox. Data deleted in the primary mailbox is placed in the primary mailbox dumpster, while data deleted in the archive mailbox is placed in the archive mailbox dumpster.

Single Item Recovery in Exchange 2010

But how does Exchange 2010 meet the other two requirements, ensuring data is not either accidentally or maliciously purged and that versions are tracked? Exchange 2010 now includes two mechanisms to meet those requirements:

1. Short-term preservation of data
2. Long-term preservation of data

Short-Term Preservation of Data

Exchange 2010 includes the ability to ensure that data within the mailbox is preserved for a period of time. This feature can be enabled enabled on a per mailbox basis by running the following cmdlet:

Set-Mailbox -SingleItemRecoveryEnabled $true

Note: When enabling this feature, you will be notified that it could take up to 60 minutes for single item recovery to take effect. This is an approximation due to Active Directory replication. Be sure to evaluate the Active Directory replication topology with respect to administering recipients to understanding how Active Directory replication may impact changes in environment.

The time period by which the deleted data is maintained is based on the deleted item retention window. The default time period is 14 days in Exchange 2010 and is configurable per database or per mailbox. The following cmdlets can be altered:

Note: Regardless, of whether Single Item Recovery is enabled, calendar items are maintained in the Recoverable Items folder structure for 120 days. Long-term data preservation via litigation hold will disable the expiration of the items.

Now, how is this process any different than in previous versions of Exchange? Short-term preservation deleted items will still be moved into the Recoverable Items folder structure. However, the data cannot be purged until deletion timestamp is past the deleted item retention window. Even if the end user attempts to purge the data, the data is retained. Consider this example by a malicious user:

• User sends or receives a message that is legally incriminating.
• User deletes the message. The message is moved to the Deleted Items folder.
• The user empties the deleted items folder.
• The user accesses the Recover Deleted Items functionality in Outlook or OWA.

The user then selects the item and deletes the item. At this point the user believes he has removed the incriminating evidence. And this is a key strength in this work flow as the end user’s actions are not interrupted or prevented; in other words, the end user’s work flow is not impaired with single item recovery enabled.

However, the message was not purged from the mailbox store. Instead the message was moved from the Recoverable Items\Deletions folder to the Recoverable Items\Purges folder. All store hard-deleted items end up in this folder when single item recovery is enabled. The Recoverable Items\Purges folder is not visible to the end user, meaning that they do not see data retained in this folder in the Recover Deleted Items tool.

When the message deletion timestamp has exceeded the deleted item retention window, Records Management will purge the item. See Figure 2 for a visual representation of this behavior.

Not only does short term preservation prevent purging of data before the deleted item retention window has expired, but it also enables versioning functionality. Essentially when an item is changed, a copy-on-write is performed to preserve the original version of the item. The original item is placed in the Recoverable Items\Versions folder. This folder is not exposed to the end user. What triggers a copy-on-write?

• For messages and posts (IPM.Note* and IPM.Post*), copy-on-write will capture changes in the subject, body, attachments, senders/recipients, and sent/received dates.
• For other types of items, copy-on-write will occur for any change to the item except for moves between folders and read/unread status changes.
• Drafts will be exempt from copy-on-write to prevent excessive copies when drafts are auto-saved.
  The data stored in the Recoverable Items\Versions folder is indexed and discoverable by compliance officers.

Long-Term Preservation of Data

Customers sometimes require mechanisms by which data is maintained for longer periods of time, say indefinitely. This is typically due to litigation hold that occurs when the organization is undergoing a lawsuit. With Exchange 2010, litigation hold can be enabled via the Exchange Control Panel or by setting the property LitigationHoldEnabled via the Set-Mailbox cmdlet.

Note: When enabling this feature, a message will notify that it could take up to 60 minutes for single item recovery to take effect. This is an approximation due to Active Directory replication. Be sure to evaluate the Active Directory replication topology with respect to administering recipients to understanding how Active Directory replication may impact changes in environment.

When litigation hold is enabled, records management purging of dumpster data ceases. Consider the following four cases:

When the end user deletes an item from the “Deleted Items” folder or shift-deletes Outlook/OWA from any folder (soft delete), the item is moved to Recoverable Items\Deletions folder, where it can be recovered in the Outlook /OWA “Recover Deleted Items” view.

1. If the end user purges data from the “Recover Deleted Items” view (hard delete from the Recoverable Items\Deletions folder), the item will be moved to the Recoverable Items\Purges folder.
2. When records management purges data from the Recoverable Items\Deletions folder (due to the deleted item age limit being reached), the item will be moved to the Recoverable Items\Purges folder.
3. When an end user modifies an item, a copy of the original item is placed in the Recoverable Items\Versions folder based on the criteria discussed in the Short-Term Preservation of Data section.

Also, when litigation hold is enabled, the properties RecoverableItemsWarningQuota and RecoverableItemsQuota (defined either on the mailbox database or mailbox) are ignored. In other words, since litigation hold is enabled, the dumpster quota is not enforced, thereby ensuring that data is not purged or data is prevented from being stored within the dumpster.

Recovering Purged Data

Data that is stored in the Recoverable Items\Purges folder is not accessible or discoverable by the end user. However the data is indexed and discoverable for those that have the proper access role in the Exchange organization role. Role Based Access Control (RBAC) provides the Discovery Management role to allow secure search access to non-technical personnel, without providing elevated privileges to make any operational changes to Exchange Server configuration. Compliance officers or other Exchange administrators with the Discovery Management role can leverage the Exchange Control Panel (ECP) to perform discovery searches using an easy-to-use search interface.

When a single item recovery request is received by help desk, the following actions can be taken:

1. Either Help Desk or a member of the Discovery Management role utilizes the Exchange Control Panel to target a search against the user’s mailbox to locate the data in question. The framework to perform the search allows the administrator to use Advanced Query Syntax. The recovered data is then extracted and placed in the Discovery Mailbox in a folder that bears the user’s name and the date/time that the search was performed.
2. The administrator then opens the Discovery Mailbox via OWA or Outlook and verifies that the content recovered is the content requested by the end user.
3. The administrator then leverages the Exchange Management Shell to perform an export-mailbox against the discovery mailbox targeting the end user’s mailbox. The data is then placed back into the end user’s mailbox.
4. Help Desk can close the ticket.

Backups and Single Item Recovery

Naturally after understanding the features included in Exchange 2010, a logical follow up question is “Do I still need backups for single item recovery?” The answer depends on the backup requirements and capacity planning. Today many customers minimize the deleted item retention window, yet they maintain long backup retention time periods (from 14 days to several months to years). Let’s consider a customer that currently maintains backups for 90 days and only retains deleted items within Exchange for 5 days. This customer is performing backup restores on a weekly basis to recover deleted items for end users. If the customer moved to Exchange 2010 they could move that process into Exchange by simply increasing their mailboxes capacity for dumpster:

Users send/receive 100 messages per work day and have an average message size of 50KB

• Single Item Recovery is enabled and the deleted retention window is configured to be 90 days
• 10% of items are edited

Mailbox capacity calculations
5 work days * 100 emails = 500 emails / week

For Purges:
§ 500 emails / week * 13 weeks = 6500 emails / retention period
§ 6500 emails * 50KB ? 318MB

For Versions:
§ 500 emails / week * 13 weeks = 6500 emails / retention period
§ 6500 emails * .1 = 650 emails
§ 650 emails * 50KB? 32MB
Total Space Required per mailbox: 350MB

By increasing each mailbox’s capacity by a minimum of 350MB, backups are no longer needed for single item recovery. Single item recovery can be maintained and performed within Exchange. But let’s not stop there. What if the requirement is that items must be recoverable for 1 year? Assuming the same assumptions used in the previous example with the exception that deleted item retention is now configured for 365 days, each mailbox needs an additional minimum 1.4GB of space. Ultimately, if the storage subsystem is planned and designed appropriately and mailbox resiliency features are leveraged, traditional point-in-time backups can be relegated to a disaster recovery mechanism, if they are even needed at all.

Conclusion

Exchange 2010 introduces the concept of single item recovery. Single Item Recovery prevents purging of data and provides versioning capability (the ability to retain the unaltered form of the item). By default this data is retained until the age of the deleted item has exceeded the deleted item retention window. In addition, Exchange 2010 enables long-term preservation of data for litigation hold scenarios by preventing the purging of data all together.