Netvault deduplication process stops working and the ...\smartdisk\diskmanger\trace directory is filling up with deduplicator trace
deduplicator trace files have the following entry and it can be seen that there is a SQL database error from this extract from the trace file "sql error: database disk image is malformed"
0 DEDUPE ??? 44 083057514892 Dedupe Failed While in DedupeWriteCookie
0 LIBERROR ??? 1 083057514892 ErrorStack start
0 LIBERROR ??? 2 083057514892 dom="libdedupe", func="LibDedupeFindCookiePath", line=4048, err=1, desc="error from below"
0 LIBERROR ??? 2 083057514892 dom="libdb", func="exec_internal", line=199, err=8, desc="sql error: database disk image is malformed"
0 LIBERROR ??? 3 083057514892 ErrorStack stop
0 DEDUPTOR ??? 10 083057514892 Failed to deduplicate URI '/NVBU/server/nvserver01/segments/5f02040a17272350e6430000141b000004000000' Cookie '91050000213223503a5f71d14c06000000a60300' : 'unknown error'
From the buffer file we see that the last SmartDisk is trying to access \\?\E:\Store\dedupe\catalog\database' at the time of the error
6 FILEMANP ??? 11 193521681441 Calling FindFirstFileW '\\?\E:\Store\dedupe\catalog\database'
5 LIST ??? 9 193521681441 Finished walk of list 0x000000000053cef0 with 4 items
0 DEDUPE ??? 44 193521759441 Dedupe Failed While in DedupeWriteCookie
0 LIBERROR ??? 1
SQL catalog database is malformed
This procedure will repair the catalog database before giving it to customer confirm with engineering you have found the correct malformed database and that it is ok to supply this procedure
Ensure that NVSD is shut-down
Disable NVSD Deduplication
To check and As NVSD is shut-down you will need to do this by editing ...\NetVault SmartDisk\diskmanager\etc\smatdisk.cfg in WordPad and setting
[Deduplication:Enable]
Value = FALSE
Create a temporary directory somewhere convenient on the file system, and copy (don't move) the following files into it:
C:\Program Files\Quest Software\NetVault SmartDisk\foundation\bin\sqlite3.exe
C:\Program Files\Quest Software\NetVault SmartDisk\foundation\bin\sqlite3.exe.manifest
C:\Program Files\Quest Software\NetVault SmartDisk\foundation\lib\libsqlite3.dll
C:\Program Files\Quest Software\NetVault SmartDisk\foundation\lib\libsqlite3.dll.manifest
E:\Store\dedupe\catalog\database
Open a command window and cd to the temporary directory created in the previous step, and then enter the following commands:
'sqlite3.exe database'
'PRAGMA integrity_check' - This will show that the database is malformed, which we already know.
'.mode insert'
'.output dump.sql'
'.dump'
Quit sqlite, which for me was 'ctl c', but is sometimes 'ctl z', and then enter the following commands;
'sqlite3.exe reformed.db'
'.read dump.sql'
'PRAGMA integrity_check' - This should now show 'ok' if the reformed database is no longer malformed.
Quit sqlite
Copy/rename reformed.db to E:\Store\dedupe\catalog\database (i.e. replace the malformed database with the repaired version). You may want to take another copy of to E:\Store\dedupe\catalog\database before replacing it.
Before restarting NVSD ensure that it's not targeted for any backups. We don't want any fresh data coming in until we're happy the issue has been resolved. You could remove the SmartDisk from netvault
Restart NVSD.
Wait for GC to successfully complete (you can check this by looking at 'csd.last_successful_gc' in the NVSD statistics) using following command
...\SmartDisk\foundation\bin\Smartdisk.bat stats
Assuming GC completed without error, enable Deduplication by editing ...\NetVault SmartDisk\diskmanager\etc\smatdisk.cfg in wordpad and setting
[Deduplication:Enable]
Value = TRUE
Monitor the NVSD statistics and log to ensure that Deduplication is completing without error.
Once the Deduplication queue is empty and GC collection has once again completed it is safe to start using the NVSD Instance for backups.
This procedure will repair the catalog database before giving it to customer confirm with engineering you have found the correct malformed database and that it is ok to supply this procedure
Ensure that NVSD is shut-down
Disable NVSD Deduplication
To check and As NVSD is shut-down you will need to do this by editing ...\NetVault SmartDisk\diskmanager\etc\smatdisk.cfg in WordPad and setting
[Deduplication:Enable]
Value = FALSE
Create a temporary directory somewhere convenient on the file system, and copy (don't move) the following files into it:
C:\Program Files\Quest Software\NetVault SmartDisk\foundation\bin\sqlite3.exe
C:\Program Files\Quest Software\NetVault SmartDisk\foundation\bin\sqlite3.exe.manifest
C:\Program Files\Quest Software\NetVault SmartDisk\foundation\lib\libsqlite3.dll
C:\Program Files\Quest Software\NetVault SmartDisk\foundation\lib\libsqlite3.dll.manifest
E:\Store\dedupe\catalog\database
Open a command window and cd to the temporary directory created in the previous step, and then enter the following commands:
'sqlite3.exe database'
'PRAGMA integrity_check' - This will show that the database is malformed, which we already know.
'.mode insert'
'.output dump.sql'
'.dump'
Quit sqlite, which for me was 'ctl c', but is sometimes 'ctl z', and then enter the following commands;
'sqlite3.exe reformed.db'
'.read dump.sql'
'PRAGMA integrity_check' - This should now show 'ok' if the reformed database is no longer malformed.
Quit sqlite
Copy/rename reformed.db to E:\Store\dedupe\catalog\database (i.e. replace the malformed database with the repaired version). You may want to take another copy of to E:\Store\dedupe\catalog\database before replacing it.
Before restarting NVSD ensure that it's not targeted for any backups. We don't want any fresh data coming in until we're happy the issue has been resolved. You could remove the SmartDisk from netvault
Restart NVSD.
Wait for GC to successfully complete (you can check this by looking at 'csd.last_successful_gc' in the NVSD statistics) using following command
...\SmartDisk\foundation\bin\Smartdisk.bat stats
Assuming GC completed without error, enable Deduplication by editing ...\NetVault SmartDisk\diskmanager\etc\smatdisk.cfg in wordpad and setting
[Deduplication:Enable]
Value = TRUE
Monitor the NVSD statistics and log to ensure that Deduplication is completing without error.
Once the Deduplication queue is empty and GC collection has once again completed it is safe to start using the NVSD Instance for backups.
