-
Title
Smartdisk crashing -
Description
SmartDisk crashing and going offline in netvault
After restarting the SmartDisk services SmartDisk can appear online but shows spaced used as 0KB and deduplication ratio as infinity: 1
SmartDisk dump shows failure when deleting segments in the store during expiration of backups or during Garbage collection as seen below in stracestorageservice trace extract.
We also see file system error in the Linux messages file found in dump folder \var\log. This points to the SmartDisk store having FileSystem problem.
from smartdisk-trace-diagnostics\var\log\messages file
Mar 9 08:29:54 SmartDisk kernel: EXT4-fs error (device dm-7): ext4_journal_check_start:56: Detected aborted journal
Mar 9 08:29:54 SmartDisk kernel: EXT4-fs error (device dm-7): ext4_journal_check_start:56: Detected aborted journal
Mar 9 08:29:54 SmartDisk kernel: EXT4-fs (dm-7): Remounting filesystem read-only
From stracestorageservice trace
0 LIBERROR ??? 2 171446982223 dom="libdedupe", func="libdedupe_delete_cookie", line=3315, err=18, desc="error calling LibSwitchGCBlock(): subsystem not initialised"
0 LIBERROR ??? 3 171446982234 ErrorStack stop
0 DISKIO ??? 29 171446982256 Failed to delete '/NVBU/server/TEL-NETVAULT-01/segments/fa82887435c57ec04f951b23a1ff010043000000' from storage
from superserver trace
0 RASCONTR ??? 238 170740526644 RemoveItemsContext timeout has expired
0 RASCONTR ??? 238 171003796553 RemoveItemsContext timeout has expired
-
Cause
Corrupt Linux file system
-
Resolution
AS this is problem with the underlying file system the only course of action is to try to repair the file system using “FSCK” repair utility.
Note: This should resolve the FileSystem and allow us to use the SmartDisk store again on completion. It should be noted that if the corruption is too bad running the FSCK could corrupt the SmartDisk data on the store and all will be lost. But considering the problem is with the underlying Linux file system there is little other choice but to attempt this repair.
The decision to run the FSCK can only be made by the person responsible for this data.
To run FSCK issue the following command. For more info on FSCK please refer to the relevant Linux documentation.
fsck -y <device path>