An 'RMAN-10038: database session for channel nvch1 terminated unexpectedly' error message happens when running an RMAN restore from the NetVault Console.
The following errors were entered in the Binary log:
1] Information 2011/11/21 18:46:30 115 Data Plugin cipsstandby RMAN-03002: failure of recover command at 11/21/2011 18:46:30
2] Information 2011/11/21 18:46:30 115 Data Plugin cipsstandby RMAN-10038: database session for channel nvch1 terminated unexpectedly
3] Error 2011/11/21 18:46:30 115 Data Plugin cipsstandby RMAN returned exit code 1
Here is the script generated by NetVault and if this script was manually given to RMAN, then the restoration ran OK:
run {
allocate channel nvch1 type 'SBT_TAPE';
send 'NV_BACKUP_SERVER=cipsbackup';
sql "alter tablespace TS_SYMPO offline immediate";
restore datafile 24 ;
recover datafile 24 ;
recover tablespace 'TS_SYMPO';
sql "alter tablespace TS_SYMPO online";
release channel nvch1;
}
Further investigation showed the following items in the NetVault Server and Client Trace files:
NetVault Server Job Manager Trace (nvjobmgr1303.log):
2 JOBMGR ??? 19889 351 0 92350286953 MSG_CHANNEL_FLUSHED_MSG received for Channel 0x96bf7d0
2 JOBMGR ??? 19889 100 0 92406765633 Report job status 'Job Failed' to schedule manager
2 JOBMGR ??? 19889 62 0 92406765948 Plugin is now in state Job Failed
2 JOBMGR ??? 19889 338 0 92406766724 Removed MsgAddressO (null):3 from channel list 0x96b57f0
2 JOBMGR ??? 19889 210 1 92406766747 Plugin exited: "Restore failed" status 2 flags 0x0
2 JOBMGR ??? 19889 211 0 92406766751 HAVE EXIT STATUS
2 JOBMGR ??? 19889 168 1 92406766776 Fatal error: Restore failed
NetVault Client Oracle APM Trace (oracleapm1668.log):
2 ORAOPS ??? 12997 38 0 92406765950 Failed to successfully sys exec
2 ORAOPS ??? 12997 4 0 9240
The restore was running fine, but is pointing to the Plug-in trying to spawn a new GUI restore sub-task, but it just ended. The NVBU core did not allow to the Plugin to spawn one more new restore sub-task, either because no more tasks were allowed, or because no more tasks were allowed specifically for the Oracle Plug-in.
The difference between the GUI restore and the CLI restore, is that the GUI is one task that spawns multiple sub-tasks (one per each backup piece restored), while the CLI performs one different task for each backup piece being restored.
The above errors occurred due to NetVault having a lack of progress buffers available.
Please change the following settings on the NetVault Oracle Client Configurator - Process Manager:
Shared memory for Process Table = 128000
Shared memory (Kb) allocated to a progress buffer = 4
Then stop and start the NetVault Services on the NetVault Oracle Client Configurator - Services tab.
