-
Title
NVBU Index compression is not working -
Description
Index compression does not appear to function, however it is correctly configured under nvconfigurator > Media Manager >
"Days of inactivity before an index is compressed (0 = never)“ = 7
“Hours between scans for indexes to be compressed (0 = never)” = 6
Daily MediaDatabase growth is observed with no space being reclaimed.
"grep -i compress messages | wc -l" doesn't return any number of compressed indexes for the period covered in the messages file, showing compression is not running.
Examining the index files under netvault/db/MediaDatabase/Files/Backup/0000/00, shows that some have consequent sizes thus do not appear to be compressed.
When an index is compressed, running the ‘head’ command on the file should show the following string: "NetVault - Compressed Index" at the beginning of the file. However a large number of files older than the set number of days of inactivity before compression are not showing the string.
Finally, another symptom that could point to an index compression issue is a sudden and noticeable MediaDatabase decrease in size, implying index compression had stopped working for some time and suddenly restarted compressing all the indexes it had missed before.
-
Cause
Under the current NVBU architecture, background tasks such as scanning for indexes to be compressed, only run when the MediaManager database is not locked.
If there is a busy backup schedule, scans may not happen exactly every x hours as configured, and will only run when the MediaManager process is able to accept the task.
With "unlucky" timing the scan could never have a chance to be executed.
Furthermore, in the event MediaManager was able to accept a scan task, a reason for the failure to compress could be a hung compressor process:
Compression of an index is kicked off by sending a message from MediaManager to the compressor process.
Currently, it would appear the return value from that send command isn't checked, in other words, if the compressor process was really hung we would never know it is, explaining why failures to compress can go by unnoticed.
-
Resolution
In such context, there are steps one can take to ensure compression has the maximum chances to run:
- Decrease the value for “Hours between scans for indexes to be compressed (0 = never)” , making the time interval between scan shorter and running scans more often.
- Restart the NVBU service. Upon restarts, the background tasks are triggered to run immediately instead of waiting for their schedules.
In both instances, if there was a compression backlog, it will be processed in one go.