Troubleshooting VMware Backups in hotadd (4041103)

Troubleshooting failures with the VMware Plugin and hotadd mode

Appliance mode (hotadd) fails or configuration assistance is needed.

The following is shown in the statistics window while processing a VM:

Hot add is not supported for this disk, failing over to network mode...

The following can be found within the logs:

“output: --wn:Hot add is not supported for this disk, failing over to network mode...\n”

If the NetVault Proxy being used to process the source VM is on a VMFS 3 datastore it must be formatted with proper block size to be able to mount the largest virtual disk of the hot added VMs.

   1MB block size – 256GB maximum file size

   2MB block size – 512GB maximum file size

   4MB block size – 1024GB maximum file size

   8MB block size – 2048GB maximum file size

The Proxy being used to back up the VM must be a Virtual Machine on a host that is able to access the datastore where the source VM’s disks are stored.

• In versions 5.1 and older of vSphere, the maximum supported VMDK size is 1.98 TB.

•  The disks that are to be hotadd must be SCSI. IDE drives are not compatible with hotadd.

• VMware Tools must be installed and up-to-date on the NetVault server and any NetVault proxies
• Paravirtual SCSI controllers are NOT supported, only LSI controllers are supported.

•  The datastore needs sufficient space for a VM snapshot. Check that the datastore the Proxy server is on has sufficient space to contain the biggest VMDK (disk file) that you intend to backup. 

Note If you are backing up multiple VM’s there will be more than one VMDK file mounted at a time so this also needs to be taken into account.

Further Known Issues:

•     Hotadd may fail if any disk was created with a newer hardware version than the VM being backed up. For example, if a disk was moved from a hardware version 8 VM to a hardware version 7 VM. To resolve, upgrade the hardware version of the VM.

•     A single SCSI controller can have a maximum of 15 disks attached. To run multiple concurrent jobs with more than 15 disks, you need to add more SCSI controllers to your NetVault Proxy server that is responsible for hot adding the disks.

Note: It is advised to add the maximum number of SCSI controllers (4) with a small 10MB thin partitioned disk on each controller.

• In case of standalone host connection (no vCenter added to the console), you can only hot add disks of VMs which are located on the same host as the proxy server.  This may require additional proxy servers to be configured on each host in the environment.

•  Hotadd may fail if you are trying to back up the VM through the host added as a standalone server into NetVault Backup Infrastructure but actually being managed by vCenter. The following error message can be found in the source agent log:

                 Reconfigure attempt failed for VM "" Exception ("vim.fault.HostAccessRestrictedToManagementServer")

• When creating backups schedules it is advised NOT to backup Proxy servers and VCenter server VM’s at the same time you backup other VM’s.  If you wish to backup these VM’s create a separate backup for each of them and  schedule them during the day while other backups are not running. 

Troubleshooting for the above:

    Test to see if the disk of the VM to be backed up can be manually hotadded to the proxy. See below:

Performing the following steps will help determine if the infrastructure is present to allow the Proxy to attach the disks of the VM to be processed in hotadd. It is advised to do this with a VM during off-peak hours as it requires a snapshot on the production server.
 
1.  Create a snapshot on the VM to be processed by NetVault Backup

2.  Within a vSphere client, attach the base disk from the VM in Step 1 to the Netvault Proxy.

1. Edit the NetVault Proxy that will be processing the VM from Step 1.
2. Click Add…
3. Choose “Hard Disk”
4. Chose “Use an existing virtual disk”
5. Navigate to the location of the base disk from the VM in step 1.
6. Specify that the disk should be attached Independent - Nonpersistent
7. Click Finish

3.  If this task completes this means the infrastructure is present to allow the Proxy to hotadd the VMs base disk.

4.  Detach the disk that was attached to the NetVault Proxy.

a.       Edit the NetVault Proxy from Step 2
b.      Highlight the attached disk.
c.       Click Remove (DO NOT change the radio option)
d.      Click Finish

 Check all NetVault Proxies for old hotadded disks and release them manually.

Objective:

While using Virtual Appliance Mode, a VMDK is not removed from the NetVault Proxy.

Cause:

The VMware API call to remove the VMDK was not received or did not complete properly.

Solution:

Through vSphere, you should be able to right-click the NetVault Proxy VM and select edit settings. Look for the Hard disks attached to the VM. Normally the stuck disks will be listed at the bottom of hardware list.

You should also be able to identify the disks by the name and the datastore they are in under the disk file box (top right).
 
To remove them, select the disk by clicking on it and select remove at the top right.  You will see the disk crossed out.

Under removal options choose “remove from virtual machine” DO NOT choose the option that deletes it from disk (the second option).
 
We add the disk as Independent Nonpersistent, so we do not need to worry about making changes in Windows or removing it from disk management.
 
After successfully removing the disk, you will need to remove the snapshot the VM being backed up is running on.
 
To do this, right-click the VM in vSphere and select snapshot then snapshot manager. A box will appear listing current snapshots. Select the snapshot named BKB_SNAP and select “delete” and confirm the removal. This will consolidate all the changes back into the base VMDK of the VM.

  Disable automount on the backup proxy, as documented Below:

Objective:

Attempting to connect via RDP to a NetVault Proxy VM in VMware during a Virtual Appliance (hot-add) backup fails.

Cause

This occurs due to hung backup job utilizing that specific proxy, and that proxy is not configured to have automount disabled.

Solution:

In an Administrator Command Prompt run the following commands:

> diskpart

DISKPART> automount disable

DISKPART>  automount scrub