How to manage deduplication in Windows 2012/2012R2 when working with AppAssure (4032913)

In many situations customers need to tweak or manage Data Deduplication Services native to Windows 2012/2012R2. Examples are restoring a deduplicated volume or attempting to add a repository extent on a deduplicated volume belonging to an AppAssure core. Since this is a relatively new feature, a guide as of how to manage deduplication in AppAssure context is needed. This KB attempts to show how to take control over this feature and put together solutions for the most common situations.

The requirements for Data Deduplication Services are:

* Windows Server 2012/2012R2/Windows 8/8.1 Operating system

* Minimum 4GB of RAM

* 1 CPU core and 350MB of RAM for EVERY 1.5TB worth of data

* Cannot be enable on the system volume/boot volume (SRP)

* Mapped drives are not supported. 

* Only local volumes are supported

* Only NTFS with MBR or GPT partitions are supported

* The ReFS file system is NOT supported

Please note that Cores with accidentally enabled deduplication on Repository drives suffer considerable performace loss.

Deduplication is a Windows feature and it can be managed as such, either from the Windows GUI (specifically Server Manager) or from Powershell.

The easiest way to interact with the Data Deduplication Feature is to combine both. All the powershell commns below need to be run from an elevated Powershell console.

All the powershell commns below need to be run from an elevated Powershell console.

To determine if DataDeduplication is enabled, run:

get-windowsfeature -name *deduplication*

To determine the status of deduplication, run

get-dedupstatus

The result is a table detailing the volumes on which deduplication is present and the general status

To disable Deduplication on a volume. run 

disable-dedupvolume -volume <volumeletter>: 

(i.e. disable-dedupvolume -volume F:)

Disabling data deduplication only stops further deduplication from occuring – but data that has already been deduplicated will remain deduplicated

To "re-hydrate" the data back to the original file and out of the deduplication store, run 

start-dedupjob -Volume <VolumeLetter>: -Type Unoptimization

(i.e. start-dedupjob -volume F: -type unoptimization)

To check the status of the job, run

get-dedupjob

Please note that you may notice that the size of the the deduplication store does not decrease after hydration. To "clean it" run

start-dedupjob -Volume <VolumeLetter>: -Type GarbageCollection

(i.e. start-dedupjob -Volume F: -Type GarbageCollection)

Please note that the garbage collection works only if deduplication is enabled. If deduplication was disabled already, please ree-able it for the duration of the garbage collection job by running:

enable-dedupvolume -volume <VolumeLetter>:

(i.e. enable-dedupvolume -volume F:)

Data scrubbing checks integratiy and validate the checksum data. Run it, 

Start-DedupJob –Type Scrubbing

Installing data deduplication can be done from the Server manager (Server Manager > Server Roles > File and storage services > File services > Data Deduplication.)

Deduplication information for each volume can be obtained from server manager as well by navigating to 

Server Manager > File and Storage Services > Volumes

Highlight the volume of interest and right click on it.

Choose "Configure Data Deduplication" from the context menu.

In the Deduplication Settings you can enable or disable deduplication, exclude file extensions types, whole folders and you can set a deduplication schedule. 

Alternatively, if desired, you can get in depth information about the deduplication (i.e. store & volume ids, DataChucks & containers counts etc) on each volume be running

get-dedupmetadata

(Please note that executing this commandlet in the format above includes all volumes and may take a long time; running get-dedupmetadata -volume <volumeletter>: will yield the information only for the desired volume).

Please note that installing the “Data Deduplication” feature installs a tool called DDPEVAL.exe as well. This is a tool is meant to calculate how efficient deduplication could be on a volume without deduplication enabled. 

To disable deduplication temporarily you can stop the following services:

ddpsvc (Data Deduplication Service) and ddpvssvc (Data Deduplication Volume Shadow Copy Service)

Please note that, to be on the safe side, it is recommended to disable ddpsvc first by running

set-service ddpsvc -startuptype disabled

To reenable it from a command prompt run

set-service ddpsvc -startuptype manual

(Please note that, by default, the deduplication service startup type is manual and depends on the Remote Procedure Call Service).