vRanger 7.8 - Integration Guide for NetVault SmartDisk

Processes

•	Storage Service: The NetVault SmartDisk Storage Service listens on a user-defined port for data flowing to and from NetVault SmartDisk. There is one Storage Server process per data stream.

•

Deduplicator: The NetVault SmartDisk Deduplicator performs byte-level, variable-block software-based deduplication by breaking the data streams into unique Chunks. The Deduplicator uses the Chunk Store Daemon (CSD) to query the Chunk Index to determine if the Chunk is unique. When a Chunk is found in the Chunk Index, the Chunk Store Daemon stores the unique Chunk in the Chunk Store. There is one Deduplicator process per data stream.

•	Chunk Store Daemon: The NetVault SmartDisk Chunk Store Daemon is the only process that writes to the Chunk Index and Chunk Store. There is only one CSD per NetVault SmartDisk Instance. The CSD also manages the Garbage Collection process.

•	percolatormonitor: The percolatormonitor is a privileged process that launches and monitors other NetVault SmartDisk Processes.

•	percolatorslave: The percolatorslave is responsible for communication routing. It routes messages between NetVault SmartDisk Processes on the local machine and between NetVault SmartDisk and other remote servers.

•	logd: The logd is the logging daemon that creates the NetVault SmartDisk logs and stores them in an internal database. It also sends them to the applicable system location, such as the Application log in the Windows® Event Viewer.

•	resourcemanager: The resourcemanager manages the combined number and size of all trace files by periodically deleting inactive files, that is, those files that are not associated with any running processes, starting with the oldest files first.

•	superserver: The superserver is the high-level manager of all other processes. It coordinates operations, balances the load between file systems, and schedules deduplication and Garbage Collection.

•	winservicerunner: On Windows, the winservicerunner runs the Windows Service and starts the percolatormonitor.

Deduplicated backup process

Figure 2. Back up data and deduplicate it

1	After the user initiates a backup job targeted to the NetVault SmartDisk repository, vRanger streams data on the user-defined port to the Storage Service.

2	The Storage Service stores the backup stream from vRanger in the Staging Store.

3	The Storage Service updates the Content Index to indicate where the backup is stored in the Staging Store.

4	The Storage Service updates vRanger to indicate that the backup stream has been successfully stored.

 

	A	During the user-defined deduplication window, backup data is retrieved from the Staging Store and sent to the Deduplicator.
	B	The Deduplicator performs byte-level, variable-block software-based deduplication by breaking data streams into unique Chunks and sending them to the Chunk Store Daemon.
	C	The Deduplicator creates a backup-specific Manifest, which is an ordered list of Chunks that make up the backup.
	D	The Chunk Store Daemon sends the unique Chunks to the Chunk Store for storage and updates the Chunk Index to indicate where each Chunk is stored in the Chunk Store.
	E	The Content Index is updated to indicate that the backup is stored in the Chunk Store.
	F	The backup data stream is removed from the Staging Store.

Deduplicated restore process

Figure 3. Restore data that was deduplicated

1	After the user initiates a restore of a deduplicated backup that is stored on the NetVault SmartDisk repository, vRanger requests the data on the user-defined port from the Storage Service.

2	The Storage Service queries the Content Index to determine whether the backup is stored in the Staging Store for nondeduplicated backups or the Chunk Store for deduplicated backups.

3	The Storage Service queries the backup’s Manifest for the first or next batch of Chunk Names in the backup.

4	The Storage Service queries the Chunk Store Daemon for the location of the Chunks in the Chunk Store.

5	Using Chunk Names from the Manifest, the Chunk Store Daemon queries the Chunk Index for pointers to the Chunks in the Chunk Store.

6	The Chunk Store Daemon provides the Storage Service with the location of the Chunks in the Chunk Store.

7	The Storage Service retrieves the Chunks from the Chunk Store and streams the Chunk to vRanger.

Step 4 through Step 7 repeat for each batch of Chunk Names in the Manifest.

On completion of restoring all Chunk Names in Manifest, vRanger returns a Restore Completed job status.

Garbage Collection process

When backups targeted to NetVault SmartDisk are retired, the data is removed from the backup database; however, the unique Chunks are not automatically removed from the Chunk Store. The NetVault SmartDisk Garbage Collection process is designed to remove orphaned Chunks from the Chunk Store and reclaim disk space. To do so, Garbage Collection maintains reference counts of added and deleted Chunks, and it deletes or rewrites partially used Chunk pages.

Garbage Collection cannot run at the same time as deduplication, because deduplication needs to write to the Chunk Store. During the configured Garbage Collection window, data waiting for deduplication is queued until Garbage Collection has completed. By default, the Garbage Collection window is from 18:00 through 06:00 on every night; however, you can revise this setting — for more information, see Configuring NetVault SmartDisk. Outside the Garbage Collection window, Garbage Collection is only started if no data is being deduplicated or is queued for deduplication. If data becomes available for deduplication outside the Garbage Collection window and Garbage Collection is active, Garbage Collection is stopped and deduplication proceeds.

Quest also recommends scheduling regular Garbage Collection to avoid delays associated with collecting large amounts of data.

 

NOTE: If a backup job that uses deduplication is canceled, the Chunk Store space used by segments that have already been deduplicated is not reclaimed until the next Garbage Collection occurs. Space used by the Staging Store and license enforcement is reclaimed immediately.