NVFR Agent repeatedly syncs, then connection is lost to FastRecover machine. (4043318)

Title: NVFR Agent repeatedly syncs, connection is lost to FastRecover
Date: 11/11/2011
FastRecover Version: 4.1.6
NVFR Agent Version 4.1.6
NVFR Agent OS: Windows 2003, 2008, HyperV and Physical machines.
Application version: SQL, Exchange and File System Protection


Description: NVFR Agent repeatedly syncs with the FastRecover machine and then connection from Agent to FastRecover machine is lost. Protection is stopped.

Symptoms:
From the NVFR Virtual Server: An attempt to connect from NVFR Agent was blocked by the NVFR Virtual Server; reason: DISS_BMNA_DS_HR_TIME_OUT_OF_SYNC: Protect bind mode is not allowed when DS max reported host reference time is greater than current system time.
Errors from the NVFR Virtual Server show that the system keeps trying to resync itself. ;

Tuesday, January 04, 2011 11:38:21 AM Resync running (97%) Resync system In Progress
Tuesday, January 04, 2011 11:32:07 AM Tuesday, January 04, 2011 11:36:49 AM Resync system Warning
Tuesday, January 04, 2011 11:16:48 AM Tuesday, January 04, 2011 11:24:28 AM Upload system Warning
Tuesday, January 04, 2011 11:16:16 AM Performing resynchronization (97%)

When the block occurs the service on the NVFR Agent Server stops;

HYPERV;
The problem is related to NVFR use of QueryPerformanceCounter() on a Windows Server 2003 Multi-Processor Hyper-V guest. This can cause the counter value to appear to go backwards, which causes an ASSERT:

2011/01/07-00:37:36.109 SEQ (0x0000108D) (f) TID (0x000006EC) ELOG (0xFFFFFFFF) 00155647
ASSERT 'getBcsTimeEndCounter >= getBcsTimeStartCounter && getBcsTimeStartCounter > syncCounter_' failed, file .\host_bcs_ref_clock.cpp, line 358,
arg1 = 3522130149651, <<=== this is the End Counter, expected to be larger than the Start Counter
arg2 = 3522130200984, <<=== this is the Start Counter
arg3 = 3352094206646

PHYSICAL MA

An edit of the boot.ini file was found to fix this problem upon Windows Server 2003 Multi-Processor Physical machines and Hyper-V guest environments.

Below is an excerpt from the following web page with the configuration change that is supposed to avoid the problem:

http://blogs.msdn.com/b/mikekol/archive/2008/10/15/problems-with-queryperformancecounter-on-windows-server-2003-multi-processor-hyper-v-guests-here-s-why.aspx


The root issue comes about from the Win32 QueryPerformanceCounter function. By default it uses a time source called the TSC. This is a CPU time source that essentially counts CPU cycles. The TSC for each (virtual) processor can be different so there is no guarantee that reading TSC on one processor has anything to do with reading TSC on another processor. This means back to back reads of TSC on different VP's can actually go backwards. Hyper-V guarantees that TSC will not go backwards on a single VP.

The fix for this is to modify the guest's BOOT.INI file by adding the /USEPMTIMER switch to your operating system's boot entry. This tells the system to use a different timer for QueryPerformanceCounter-related tasks, and should alleviate the problem.