SmartDisk crashes while attempting to reallocate memory for deduplication (4047674)

The SmartDisk service stops unexpectedly several times a day.
Once restarted it carries on working properly but stops few hours later.

The logd.log file shows the service stops right after failing a deduplication job:

diskmanager:dedupe_queue:resource Queueing for dedupe /NVBU/server/nvserver/segments/6f6417ac5c63104f677300000423000003000000
diskmanager:resource_dedupe_failed:resource Dedupe failed /NVBU/server/nvserver/segments/6f6417ac5c63104f677300000423000003000000
diskmanager:resource_operation_ok:resource write complete /NVBU/server/nvserver/segments/6f6417ac9d63104f313700002016000003000000
diskmanager:process_exit:process Process exiting superserver

Looking at the chunkstored.*.buffer trace file we see the process fails to allocate 1.6GB of memory before SmartDisk stopped:
0 MEMORY ??? 9 184037438808 Failed to reallocate total of 1765242984 bytes (user requested 1765242960).
0 SYS ??? 1 184037458622 /home/buildbot/buildslaves/k6linux32bot/smartdisk-linux-x86_Hybrid/build/k6/foundation/source/libraries/basics/memory.c:726 - Failed assertion: NULL != pNewBlock
0 MEMORY ??? 10 184037454534 Currently have 2048190856 bytes in 23925 allocations.

Looking at the trace extract above we can see the SmartDisk build used in this instance is a Hybrid build.
A hybrid build uses mostly 32-bit executables that will have a limited address space (usually a max of ~3.5GB for both kernel and user space),
which explains why chunkstored is having trouble allocating the 1.6GB of contiguous space.
Upgrading the SmartDisk installation to a pure 64-bit build in this case will ease the restriction enough to allow the deduplications to run.

SmartDisk should be installed on a 64 bit OS and as such the Pure 64 bit SmartDisk build should also be used with it. This is because the chunk index needs to be able to address large amounts of memory.
Although installing SmartDisk on a 32 bit OS and using the 32 bit Hybrid build is possible on both 32 and 64bit platforms, it should only be done for testing purposes or would only suit very small backup sites that have less than 2TB of Unique Data protected and don't have a 64 bit machine at hand.
 

1) First determine whether the system has enough RAM to cope with the amount of data protected.
The SmartDisk rule of thumb is a minimum of 2GB of RAM + 400MB per TB of deduplicated data.

2) If the SmartDisk build installed is a Hybrid one, and the OS is a 64 bit one, the executables used will have limited memory address space, so upgrade your SmartDisk instance to a Pure 64 bit build:
a) As a precautionary measure, take a copy or backup of the current Smartdisk installation directory only (excluding the stores which should not be affected by an upgrade).
b) Download the Pure 64 bit SmartDisk build from our website
c) Perform a normal upgrade which will replace the Hybrid executables with 64 bit ones.

3) Finally, please note that as mentioned in the "Quest NetVault SmartDisk Installation/Upgrade Guide", depending on the size of the unique data protected by SmartDisk and the OS bit the following measures are required:
"The Unique Data Size is the amount of unique data that is stored inside the NVSD Instance.
An additional NVSD Instance must be deployed for every 15TB of unique data on a 64-bit OS and for every 1.8TB through 2.5TB of unique data on a 32-bit OS (depending on the OS)".