-
Title
Sizing and Optimizing the DVM Deduplication Cache -
Description
What is Data Deduplication?
AppAssure and Rapid Recovery (RR) performs Global Deduplication across the backup data from all protected machines. The deduplication process begins when an agent sends data from a protected machine to the Repository. The Core compresses, encrypts and deduplicates incoming data before sending it to the Core’s storage. The Core calculates a checksum for each incoming block after compression and optional encryption. The checksum determines if the Core has “seen” the block before from any of the multiple servers AppAssure/RR is protecting on the network, hence the name Global Deduplication.
If the block is unique, it’s written to the storage, but if the block already exists in the Core’s repository, the Core then replaces the discarded block of data with a pointer to its identical block in storage. This process of deduplication in conjunction with compression would result in more than an 80% reduction in storage requirements.
-
Cause
Global Deduplication Benefits:
- Greatly reduces the size of AppAssure/RR’s image-level disk snapshots as they committed in the repository which greatly reduce your backup storage requirement.
- AppAssure/RR globally deduplicates across all protected servers on the network, yielding several times the efficiency than is possible with individual machine data deduplication.
- Performs simultaneous deduplication, compression and encryption at line speeds, allowing backup speeds of several gigabytes per second.
- Moves data faster. AppAssure/RR combines inline deduplication and compression with incremental-forever and change-block tracking technologies to reduce the amount of backup data that would otherwise have to be transmitted over the LAN or WAN.
- Keeps data secure. AppAssure/RR Global Deduplication also performs source side encryption of data during the deduplication process without introducing time-consuming processing cycles. This speeds transmission of highly secure backup data offsite for cloud or offsite recoveries.
- With Global Deduplication, the AppAssure/RR Replication process is self-optimizing with a unique Read-Match-Write (RMW) algorithm that is tightly coupled with deduplication. With RMW replication the source and target replication service matches keys before transferring data and then replicates only the compressed- encrypted- deduplicated data across the WAN resulting in a 10x reduction in bandwidth requirements.
-
Resolution
For more information on this topic, please see:
- Rapid Recovery Basic Administration Course - Web Based Training (WBT)
- Rapid Recovery Administration: Module 3 – Core Installation and Configuration
Optimization Considerations:
Many factors impact data deduplication. The following table provides factors to consider that may yield higher/lower compression ratios based on the type of data:
Sizing your Deduplication Cache:
To ensure achieving the best Deduplication ratio which will optimize the repository storage and disk I/O it is recommended to size the Deduplication Cache properly when you first deploy the AppAssure Core Server. The Deduplication Cache (Dedupe Cache) is a table where all hashes are stored during deduplicating new incoming data, if you decide to resize the Dedupe Cache after taking backups the benefits of the added size to the Dedupe Cache will only be useful for new incoming data by making more space to store new hashes, so it is recommended to set a sufficient size during the deployment phase.
The default Dedupe Cache size of 1.5 GB is enough to cover 800 GB worth of Deduplicated data (unique data). Assuming that the total deduplication/compression ratio is 50%, the 1.5 GB is estimated to be sufficient for 1.5 TB of raw protected data. If your total protected data is more than 1.5 TB use the following calculations to determine the amount of Dedupe Cache size requirements:
- Dedupe Cache size in (X) = Y / 1000, where Y is the total protected data in GB. X is the Dedupe Cache size in GB and is limited to 50% of the installed physical memory (RAM) on the AppAssure Core Server
- Min Core Server RAM size = 2 * X
- Min Dedupe Cache Storage Space (for storing both the Primary and Secondary Dedupe Cache files) = 2 * X
Example:- Total Protected Data from all agents = 20 TB. The recommended requirements for Deduplication:
- Dedupe Cache recommended size is: 20000 / 1000 = 20 GB
- The Core Server should have a minimum of 40 GB of RAM
The storage space for Dedupe Cache files should be a minimum of 40 GB
NOTES:
- The above is an approximate calculation based on expected Deduplication/Compression ratio of 50%, if the type of protected data is expected to yield a lower ratio please adjusts the Dedupe Cache size accordingly.
- The Secondary Dedupe Cache does not require additional RAM so the above calculation for memory requirement does not need to be doubled .
- The Dedupe cache is shared by all repositories on a Core Server, however; deduplication does not happen across repositories, so if the same block appears once in two repositories, it will not be deduped.
For more details regarding changing the Dedupe Cache size and settings please refer to Configuring Deduplication Cache Settings in the User Guide.
Best Practices:
- Use the formulas above to sizing the Dedupe Cache properly during the AppAssure/RR Core deployment before starting to transfer the first backups. That will give an idea about the resources requirement and if adding more memory or storage is required at an earlier stage. Also, configuring a sufficient Dedupe Cache size at that time will result in higher deduplication ratio and less used space on the Repository in the future.
- Start taking backups for agents with similar OS to optimize the Dedupe Cache. For example, if you have 10 agents with Windows Server 2012 R2 and 5 others with Windows XP SP3, let the backups complete for one of those groups before starting the backups for the other. Please note that this protection order is only relevant in the case where the dedupe cache size is insufficient for the amount of protected data.
- Protect Server with similar data in batches. For example, if you have a 5-node Exchange DAG, it is recommended to backup these agents together one after the other. Please note that this protection order is only relevant in the case where the dedupe cache size is insufficient for the amount of protected data.
- If you plan to enable Encryption on your agents, it is recommended to enable that before taking the first backups as encrypting backups at a later point will require a new Base Image backup and that would affect the deduplication ratio.
- The Dedupe Cache size will be loaded in the RAM every time the Core Service is started and flushed from the RAM to the disk when the service stopped so increasing the size would incur in added delay for starting/stopping the Core Service. In addition to that, the cache contents will be flushed to the disk on hourly bases, and after every time a Base Image completes transferring. During that time, all Repository I/O activities will be paused, thus it is highly recommended to store the Dedupe Cache files on a dedicated spindle or even better on a Solid State Drive (SSD).
- Avoid shrinking the dedupe cache once it has been expanded. Shrinking the dedupe cache results in the cache being deleted and recreated at the new size. This will reduce the cores deduplication capability and may result in more repository space being consumed than would have been used otherwise.
- When possible, strive to keep the DC below 20GB in size. A Dedupe Cache of 20GB or below cantypically be written to disk very quickly.
- In environments where the Dedupe Cache must be larger than 20GB, it is highly recommended to:a. Move the Dedupe Cache to a dedicated disk separate from the OS and repository volumes.b. Use faster, dedicated SSD storage for the primary and secondary Dedupe Cache. The use ofSSD drives will ensure that the DC can be written in a timely manner and preventdisruption to normal Core activities.
Is your AppAssure or Rapid Recovery solution working as efficiently as possible? Learn more about our Rapid Recovery Health Check.