EMC Celerra NDMP restore fails with Data operation halted due to unrecoverable internal NDMP serv (79991)

Date: MAR 2010
Product & Version: NVBU 8.5
OS Version: ALL
Module & Version: NDMP V7.1.11 onwards
Application version: DARTOS 5.6



Symptoms:


Netvault binary log lists the following
Error 2010/03/04 12:24:52 39 Data Plugin axvc01 Data operation halted due to unrecoverable internal NDMP server error


establish whether this is a VBB dump by checking the ndmp restore request within the binary log

Job Message 2010/03/04 12:24:47 39 Data Plugin axvc01 Restore Request

expand the contents by double clicking on the ! and check for the value VBB = Y

i.e.
Restore Environment:

FILESYSTEM = /fs_at_cifs01
LEVEL = 0
UPDATE = Y
HIST = Y
SNAPSURE = Y
VBB = Y
MD_SIZE = 110632960
MD_SIZE_HIGH = 0
TYPE = dump
RECURSIVE = Y
DIRECT = Y

check whether the file system/volume that was backed up was deduplicated.

if so the emc filer syslog will list the following:


2010-03-15 16:07:12: VBB: 6: Recursively creating files and dirs
2010-03-15 16:07:12: VBB: 3: Cannot set offline data for inode 38 (UknownProtocol)
2010-03-15 16:07:12: NDMP: 3: < LOG type: 2, msg_id: 0, entry: Unable to perform a single file restore on a deduplicated file. Perform a full VBB

restore on a separate file system and copy the deduplicated files that you want., hasAssociatedMsg: 0, associatedMsgSeq: 0 >
2010-03-15 16:07:12: VBB: 3: vbbCreateFiles aborted 1
2010-03-15 16:07:12: VBB: 3: Error restoring offline inode for inode 38
Total Time elapsed : 1 sec
2010-03-15 16:07:12: VBB: 3: vbbFileRestore aborted
2010-03-15 16:07:12: VBB: 3: Restore aborted


Explanation:

This is an EMC limitation with De-duplicated file systems - a standard restore from a VBB dump is not possible. A full destructive restore must be performed.

Note R

EMC celerra VBB dump restore procedure -


how to perform a full destructive restore to recover VBB dump contents
## On the EMC filer

1- create a new raw file system (needs to be samesize as original FS)

i.e.

nas_fs -name rawtestfs -type rawfs -create samesize=zentafs pool=cmata_archive -o slice=y


2 - verify that target file system is raw by running on the emc filer :

nas_fs -i <restore_filesystem_name> | grep type

if it not raw then change it raw with the command :

nas_fs -T rawfs <filesystem_name> -Force

3 - create mountpoint

i.e.

server_mountpoint server_2 -create /rawtestfs_mountpoint

4 - mount the rawfs to the DM or VDM

i.e.

server_mount server_2 rawtestfs /rawtestfs_mountpoint

5 - Determine the volume id (number) of the target file system by running command

nas_fs -i <restore_filesystem_name>


i.e.

[nasadmin@ce18cs0 ~]$ nas_fs -i rawtestfs

id = 188
name = rawtestfs
acl = 0
in_use = True
type = rawfs
worm = off
volume = v376 <<<<<<
pool = cmata_archive
member_of = root_avm_fs_group_15
rw_servers= server_2
ro_servers=
rw_vdms =
ro_vdms =
auto_ext = no,virtual_provision=no
deduplication = unavailable
stor_devs = APM00073700706-0006
disks = d9
disk=d9 stor_dev=APM00073700706-0006 addr=c16t1l2 server=server_2
disk=d9 stor_dev=APM00073700706-0006 addr=c0t1l2 server=server_2

using what is shown in the volume field, run nas_volume -info <Volume.

nas_volume -info v386


## On the netvault server

create an NDMP restore job that


1 - selects the top level file system node

2 - select 'right click' rename method

3 - specify a rename path of /.celerra_vol_<ID>
where <ID> = the target volume id determined earlier
e.g. /.celerra_vol_376

5 -restore option set to 'Never Do Direct Access Restore"


Alternatively to avoid this issue use TAR or Dump backups with Dedupe file systems.
To do this de-select the VBB option within the NDMP backup options within the NetVault NDMP plugin.

BakBone SFDC Solution Number: 00005580