Incomplete recovery points are taken after Windows Agent is upgraded from 6.1.x to 6.2.0, 6.2.1, or 6.3.0 (4043099)

Incomplete recovery points are taken after Windows Agent is upgraded from 6.1.x to 6.2.0, 6.2.1, or 6.3.0

In Rapid Recovery 6.2.0 Windows Agent, a new changelog Bitmap was introduced. Upon upgrading an existing Rapid Recovery 6.1.x Windows Agent to 6.2.0 or newer, the changelog would be converted into a Bitmap. 

Only the first 1 MB of changes tracked in the changelog are converted. The remaining (more recent) changes are truncated, causing missing data in new incremental backups.  This results in incomplete volume file systems and a damaged recovery point chain.

This issue can result in file-level restore failures; the inability to browse a mount point; or virtual export machines failing to boot after a restore, with errors such as "INACCESSIBLE BOOT DEVICE" or restored machines booting straight into Windows Repair screen.

 

Likewise performing a Chkdsk on the filesystem of an affected mounted Recovery Point would indicate errors on the file system structure. 

This issue can only affect Agent protected Windows machines, that have had the Agent upgraded from version 6.1.3 and older to 6.2.0, 6.2.1, and 6.3.0.

• Agentlessly protected Hyper-V or VMware virtual machines.
• Machines protected with the Linux agent.
• Machines protected with a fresh install of the 6.2.0 or newer Windows agent.

This issue has been identified as defect RR-109147. It occurs when the agent change log file on a volume is larger than 1 MB and the Rapid Recovery agent is upgraded from a version 6.1.3 or older to 6.2.0 or newer. Most agents that take regular backups and are rebooted immediately after upgrade will not experience this issue. 

This defect was resolved in Rapid Recovery 6.4.0 Agent. If upgrading Agent protected Windows machines, running Agent 6.1.x. We recommend upgrading straight to Rapid Recovery 6.4.0 to avoid this defect. 

 

Below steps is applicable when upgrading 6.1.x to one of the affected Agent versions 6.2.0, 6.2.1 or 6.3:

Pre-Upgrade Workaround:

If you are considering upgrading to one of the affected versions, we advise the following steps to avoid this issue:

1. Ensure there are no outstanding windows updates or reboots pending on the system
2. Take an incremental backup immediately before upgrading the Rapid Recovery agent to version 6.2.0 or higher.
3. Upgrade the agent, rebooting immediately when the upgrade has complete.

 

If you follow these steps specifically we still recommend following the post-upgrade workaround steps to ensure the upgrade was successful and your recovery point chain is healthy.

Post-Upgrade Workaround:

If you have upgraded some agents to one of the affected versions, we recommend running the PowerShell script that we have attached to this KB on your Rapid Recovery Core server to test the validity of your recovery points and ensure there is no corruption in your recovery point chains.

This script has been designed to detect damaged recovery points caused by the agent upgrade defect RR-109147. Only NTFS partition types are supported by this script so REFS, EISA, and UEFI partitions will need to be checked manually. This script does not check:

• Agentless protected machines
• Recovery Point Only machines
• Recovery Points in attached archives

 

To detect damage from this defect, the script will scan your core server for machines protected as an agent. It will then scan the recovery points for the version of the agent used during creation. If an agent has a recovery point taken with an agent version prior to 6.2.0 AND a recovery point taken with an agent version post 6.2.0, then a check will be run. If an agent does not meet those requirements, then the agent will be skipped. ***Remember that this issue can only occur if you upgrade from an agent build 6.1.3 or older to an agent build 6.2.0 or newer.***

 

Once the script has determined an agent needs to be checked it will mount up to 3 recovery points and run chkdsk on each volume within that mounted recovery point. The recovery points the script will mount in order are:

1. The last recovery point taken with the 6.1.3 agent
2. The first recovery point taken with a 6.2.0 or newer agent
3. The most recent recovery point

 

After the script completes it will output the results in the PowerShell window, but will also create file called DiskHealthReport.txt inside the report path. This report file contains entries for each agent, the results of the check for each volume within each recovery point that was checked. Please review the output and follow these recommendations based on the results:

• If none of the recovery points are damaged, you have not experienced the defect and do not need to proceed with any further steps.
• If any recovery points display an error, identify the damaged recovery points through further manual testing and delete those recovery points.
• If the most recent recovery point is damaged, take a new base image.

 

To run the script:

1. Extract the script to a folder on the desktop of your core server.
2. Open an administrative PowerShell prompt.
3. Change directories to the folder with the script.
4. Set the execution policy for your system to unrestricted to prevent the script from being blocked (you can set the execution policy back after the script has finished)
   • Set-ExecutionPolicy Unrestricted
5. Run the script.
   • .\CheckKB301910.ps1

 

Every time the script is executed, a new sub directory with a current timestamp as the name will be created next to the script. Each agent has its own folder in that sub directory. Reports are grouped by volume (each volume has its own folder for reports). The DiskHealthReport.txt report will also be located in this sub directory.

The following optional parameters are available to allow you to modify the behavior of the script:

• - listening port of Core service (default is 8006) (usage example: -port 8007)
• - user name used to access the Core service (usage example: -coreAccessUserName "Administrator")
• - user password used to access the Core service, in plaintext (usage example: -coreAccessUserPass "secret")
  ** NOTE - if and are both omitted then the default credentials will be used
• - dir where the reports will be placed. (usage example: -reportsBasePath "C:\Reports")
• - agents to check the latest recovery points for. (usage example: -agentDisplayNames "agent"). If this parameter is not specified, all of the agents are going to be checked
• - specifies a temporary folder where the mount points are placed. After a check is performed it will be dismounted (usage example:- -mountPointsBasePath "C:\TmpPath")
• - mounts recovery point as writable (usage example: -mountWritable $true). Use this together with -forcedCheck parameter when you want to see if damage to the recovery points (if there is any) is repairable
• - calls chkdsk utility with /f key. Use this together with -mountWriatable parameter when you want to see if damage to the recovery points (if there is any) is repairable
• -- specifies path to a folder where RR Core is installed. Use it if RR Core is not installed in a default folder (usage example: -rrCoreInstallPath "C:\Program Files\AppRecovery\Core")

 

RR-109147

CheckKB301910