Large backup of virtual client with over 60 million files/folders fails during creation and writing of backup index.
Sometimes the backup will appear to hang and can still be seen as active job in the Web UI, however, the logs reveal the job failed.
The backup in this case will need to be aborted.
Splitting the backup into 4 smaller backups does not resolve the problem
Trace shows “Connection down” while writing the index to tape device on NetVault server.
From binary log:
Job Message 2019/03/20 11:12:29 1574 Jobs NV_Server Clustered Application SPIPOPIMFS is running on real machine Node2
Job Message 2019/03/27 22:02:11 1574 Media NV_Server (NV_Server: 4-0.5.1 (IBM 3573-TL)) Media in 'DRIVE 2:NV_Server' assigned to job ready for data transfer
Job Message 2019/03/28 09:28:01 1574 Data Plugin Node2 Building Master backup index ...
Job Message 2019/03/28 09:28:01 1574 Data Plugin Node2 Storing Master backup index
Job Message 2019/03/28 13:44:15 1574 Jobs NV_Server Saveset 1234 (instance 1)
Error 2019/03/28 13:48:01 1574 Data Plugin Node2 Failed to write backup index to media
SevereError 2019/03/28 13:48:01 1574 Data Plugin Node2 Failed when finalizing the backup
Error 2019/03/28 13:48:01 1574 Jobs NV_Server Fatal error: Backup failed
From filesystem trace file:
2 HBBACKUP ??? 11388 35 0 134738726812 Index sent to job mgr successfully
2 PLGLIB ??? 11388 9344 0 134742628201 Transfer Sample for Stream :period = 15588705 (1553765274042,1553780862747); rate = 16, 3 samples
5 PLGLIB ??? 11388 56 0 134742632202 Read 65536 bytes index
0 NET ??? 11388 81 0 134801520091 Connection down
0 PLGLIB ??? 11388 558 0 134801520091 Failed to send buffer
0 PLGLIB ??? 11388 93 0 134801520091 Backup Stream Down
0 PLGLIB ??? 11388 177 0 134801520091 Failed to output index
0 HBBACKUP ??? 11388 181 0 134801529092 Failed to write index to backup stream
5 LOGGING ??? 11388 21 0 134801529092 MsgFormat='Failed to write backup index to media'
0 HBBACKUP ??? 11388 159 0 134801562095 Failed when ending the backup
0 PLGLIB ??? 11388 9 0 134801567095 Attempted to disconnect from device on a downed stream
0 PLGLIB ??? 11388 9081 0 134801567095 Ignoring tape device disconnect on a downed stream
2 PLGLIB ??? 11388 9069 0 134801567095 TRUE = disconnection result when stream down
0 HBBACKUP ??? 11388 345 0 134801581096 A Fatal Error occurred
Network connection breakdown between the NetVault Server and the real client node of the cluster.
Configure preferred addressing on all the client nodes added to the Virtual Client.
Note: When configuring preferred addressing for clients, the preferred address should be the IP address used for the FQDN only, all other IP addresses seen on the node should be configured as barred, but the Cluster IP address should not be configured anywhere, NetVault will use this as required.
To configure preferred addressing select Change Settings > Client Settings > select the Client you wish to configure > Network Manager > edit “Preferred network address” and “Comma separated list of barred address(es)” as required.
Verify that all client preferred IP addresses and cluster address have forward and reverse lookup configured.
Use nslookup on netvault server to check this
nslookup <IPaddress>
nslookup <client name>
Note1:
This resolution could also apply to any non-clustered backup client
Note2:
Similar or the same error can be seen when the client runs out of disk space when creating the index in the …/netvault/tmp folder because the volume has run out of space. With large backups such as this with millions of files it is advised to have at least 30GB free space as the index file alone can be up to 15GB in size.
If the NetVault installation volume does not have enough free space it is advised to move the …/netvault/tmp folder to another volume that does.
Similar error or same error can happen if the …/netvault/tmp folder on the NetVault Server also does not have enough space as the index file will be written here while database in being updated with new index.
As there can be more that 1 index being updated at the same time it is would recommended to have 50 to 100GB free space for the netvault install volume.
To move the …/netvault/tmp folder to different volume on client or server use the following procedure
1/ Stop the NetVault service
2/ Delete all files in …/netvault/tmp folder
3/ Copy or create new '…/NetVault/tmp' directory on the new volume
4/ Select Change Settings > Select the Client or NetVault Server as required > general > modify option “Temporary Directory” to new location.
5/ Start the NetVault service.
