Backup of Virtual client fails with 'Failed to write backup index to media' (4038798)

Large backup of virtual client with over 60 million files/folders fails during creation and writing of backup index.

Sometimes the backup will appear to hang and can still be seen as active job in the Web UI, however, the logs reveal the job failed.

The backup in this case will need to be aborted.

Splitting the backup into 4 smaller backups does not resolve the problem

Trace shows “Connection down” while writing the index to tape device on NetVault server.

From binary log:

Job Message    2019/03/20 11:12:29  1574             Jobs     NV_Server       Clustered Application SPIPOPIMFS is running on real machine Node2

Job Message    2019/03/27 22:02:11  1574             Media  NV_Server       (NV_Server: 4-0.5.1 (IBM 3573-TL)) Media in 'DRIVE 2:NV_Server' assigned to job ready for data transfer

Job Message    2019/03/28 09:28:01  1574             Data Plugin      Node2 Building Master backup index ...

Job Message    2019/03/28 09:28:01  1574             Data Plugin      Node2 Storing Master backup index

Job Message    2019/03/28 13:44:15  1574             Jobs     NV_Server       Saveset 1234 (instance 1)

Error    2019/03/28 13:48:01  1574    Data Plugin  Node2 Failed to write backup index to media

SevereError     2019/03/28 13:48:01  1574             Data Plugin      Node2 Failed when finalizing the backup

Error    2019/03/28 13:48:01  1574    Jobs             NV_Server       Fatal error: Backup failed

From filesystem trace file:

2           HBBACKUP       ???       11388  35             0           134738726812             Index sent to job mgr successfully

2           PLGLIB ???       11388  9344    0             134742628201             Transfer Sample for Stream :period = 15588705 (1553765274042,1553780862747); rate = 16, 3 samples

5           PLGLIB ???       11388  56         0             134742632202             Read 65536 bytes index

0           NET      ???       11388  81         0             134801520091             Connection down

0           PLGLIB ???       11388  558       0             134801520091             Failed to send buffer

0           PLGLIB ???       11388  93         0             134801520091             Backup Stream Down

0           PLGLIB ???       11388  177       0             134801520091             Failed to output index

0           HBBACKUP       ???       11388  181             0           134801529092             Failed to write index to backup stream

5           LOGGING         ???       11388  21             0           134801529092             MsgFormat='Failed to write backup index to media'

0           HBBACKUP       ???       11388  159             0           134801562095             Failed when ending the backup

0           PLGLIB ???       11388  9           0             134801567095             Attempted to disconnect from device on a downed stream

0           PLGLIB ???       11388  9081    0             134801567095             Ignoring tape device disconnect on a downed stream

2           PLGLIB ???       11388  9069    0             134801567095             TRUE = disconnection result when stream down

0           HBBACKUP       ???       11388  345             0           134801581096             A Fatal Error occurred