Smartdisk crashing (4041976)

SmartDisk crashing and going offline in netvault

After restarting the SmartDisk services SmartDisk can appear online but shows spaced used as 0KB and deduplication ratio as infinity: 1

SmartDisk dump shows failure when deleting segments in the store during expiration of backups or during Garbage collection as seen below in stracestorageservice trace extract.

We also see file system error in the Linux messages file found in dump folder \var\log. This points to the SmartDisk store having FileSystem problem.

from  smartdisk-trace-diagnostics\var\log\messages file

Mar  9 08:29:54 SmartDisk kernel: EXT4-fs error (device dm-7): ext4_journal_check_start:56: Detected aborted journal

Mar  9 08:29:54 SmartDisk kernel: EXT4-fs error (device dm-7): ext4_journal_check_start:56: Detected aborted journal

Mar  9 08:29:54 SmartDisk kernel: EXT4-fs (dm-7): Remounting filesystem read-only

From stracestorageservice trace

0             LIBERROR            ???         2               171446982223  dom="libdedupe", func="libdedupe_delete_cookie", line=3315, err=18, desc="error calling LibSwitchGCBlock(): subsystem not initialised"

0             LIBERROR            ???         3               171446982234  ErrorStack stop

0             DISKIO  ???         29          171446982256               Failed to delete '/NVBU/server/TEL-NETVAULT-01/segments/fa82887435c57ec04f951b23a1ff010043000000' from storage

from superserver trace

0             RASCONTR         ???         238               170740526644  RemoveItemsContext timeout has expired

0             RASCONTR         ???         238               171003796553  RemoveItemsContext timeout has expired

Corrupt Linux file system

AS this is problem with the underlying file system the only course of action is to try to repair the file system using “FSCK” repair utility.

Note: This should resolve the FileSystem and allow us to use the SmartDisk store again on completion. It should be noted that if the corruption is too bad running the FSCK could corrupt the SmartDisk data on the store and all will be lost.  But considering the problem is with the underlying Linux file system there is little other choice but to attempt this repair.

The decision to run the FSCK can only be made by the person responsible for this data.

To run FSCK issue the following command. For more info on FSCK please refer to the relevant Linux documentation.

fsck -y  <device path>