Backup jobs to NDMP attached devices fail with a channel error and context "Failed to receive rep (4043575)

Date: May 2007
NV Version: All
OS Version: N/A
Application version: N/A
Plugin version: All

Symptoms:
Tape library and drives are attached to the filer via Fibre Channel or SCSI. Backup jobs fail with a channel error.

The Binary Log shows:
Error 2007/05/06 15:21:44 31 Media S00SBK002 S00SBK002 Tape S00NETAPP03:nrst0a (HP Ultrium 3-SCSI): had channel error

With Log context:
DEVMGR :5560 740 0 142144 Channel at 00979948 active TRUE
DEVMGR :5560 591 0 142144 Defer 'DATA_CHANNEL_NDMP_CONNECT_REQUEST_MSG' until mover has completed its command
NDMP :5560 1602 0 142144 Failed to receive reply a00

This message indicates an invalid state of the NDMP sessions on the filer for the devices. This is commonly caused by a network problem that caused these sessions to be dropped, but not cleaned up correctly. NetVault will try to reconnect, but depending on the state the filer is in, these connections can sometimes not be established correctly.

In order to resolve this, stop the NetVault service, and on the filer check whether there are any left over NDMP sessions (ndmpd status). If there are, kill them (ndmpd killall). The stop and start the NDMP daemon to reinitialize it (ndmpd off, ndmpd on). After starting the NetVault service again, you should see new NDMP sessions for the devices (ndmpd status), and the next backup runs should complete successfully.

If you see this happening very frequently, you will need to address the underlying connectivity issues that lead to these unclean terminations of the NDMP sessions.