Restore to CleanOS hangs at "Restart domain controller in normal mode" (4372403)

Restore to CleanOS hangs at "Restart domain controller in normal mode"

The log files show that there is an issue with the credentials when attempting to access the restore domain controller.

When the Forest Recovery Console connects to the Forest Recovery Agent, it first attempts to use the "Normal Mode" credentials, and if it fails, the DSRM credentials. And if neither of the two credentials work, it ends up in an infinite loop.

The "Normal Mode" credentials are the "Local Admin" credentials for the InstallAD, InstallFromMedia and RestoreOnCleanOS methods and the "Domain Admin" credentials for any other method.

Reviewing the DC logs from within the FR Log folder will show a pattern similar to below whereby the credentials are constantly switched:

Jul 25    10:49:48.734    [2072:0013]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:51:01.660    [2072:0014]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:52:27.593    [2072:0014]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:52:48.722    [2072:0022]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:54:14.874    [2072:0004]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:54:35.991    [2072:0014]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:56:07.648    [2072:0016]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:56:28.781    [2072:0016]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:58:00.261    [2072:0016]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:58:20.865    [2072:0022]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    10:59:46.852    [2072:0022]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:00:07.968    [2072:0016]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:01:39.523    [2072:0022]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:02:00.669    [2072:0022]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:03:32.271    [2072:0019]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:03:53.413    [2072:0004]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:05:25.151    [2072:0015]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:05:46.274    [2072:0014]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:07:17.935    [2072:0004]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:07:39.056    [2072:0018]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:09:10.712    [2072:0015]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:09:31.346    [2072:0015]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:10:57.340    [2072:0015]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:11:17.474    [2072:0018]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.
Jul 25    11:12:49.224    [2072:0015]    VERBOSE:     RestoreAgent.CallRpcClient(): UnauthorizedAccessException is handled. Switching credentials.

The logs will also show an auth-type as below:
VERBOSE:     AgentInstaller.GetInstalledAgentInformation(): Agent version: '10.3.0.39344', used auth type: RPC_C_AUTHN_GSS_NEGOTIATE, is up to date True.

Normally, the console-to-agent communication should use SCHANNEL authentication which is based on certificates and doesn't require any credentials. Whilst it is possible to install the FR agent on the CleanOS machine WITHOUT a certificate, this will cause the communications to fail at the "Restart domain controller in normal mode".

Install the FR agent on CleanOS machines using either the command line install or from the FR Console itself.

To manually install the Forest Recovery agent with Schannel Authentication, please do the following:

1. If using 10.0.1 or earlier, the communication keys can be exported from either the Certificate Management MMC or from the FR console:
    - Open MMC on the RMAD server
    - Click File | Add/Remove Snap-in
    - Select Certificates and click Add
    - Select Computer account
    - Click Next, Finish, and then OK
    - Expand Certificates | Personal
    - Select Certificates
    - Hold Ctrl and select both the 'Recovery Manager for Active Directory Agent' and the 'Recovery Manager for Active Directory Console' communication keys
    - Right click and select All Tasks | Export
    - Click Next
    - Select 'Yes, export the private key' and click Next
    - Ensure 'Personal Information Exchange is selected and click Next
    - Check Password and enter RMAD as the password
    - Click Next
    - Specify a path to save the file, click Next, and Finish
2. If using 10.1 or later, the communication keys can only be exported from the FR Console
    - Open the FR Console
    - Click Tools | Fault Tolerance | Export secure communication keys
    - Click Browse to select a path to store the exported keys
    - For the password type RMAD
3. Copy the pfx file to the DC
4. Copy the RecoveryAgent64.exe from the RMAD installation directory to the DC
5. Run the following command from an elevated PowerShell prompt from the DC where you copied the files in steps 3 and 4:
   .\RecoveryAgent64.exe /qn MANAGE_SERVICE=TRUE FRS_CUSTOM_PORT=xxxxx CERTIFICATE_PATH="C:\Temp\Cert.pfx" FRS_FIREWALL_SETTINGS_CONFIGURE=1 /l*!+v C:\Temp\Manual.Install.log
6. Once the agent has installed successfully, delete the cert export created in step 1.

NOTE: If you want the agent to use a specific port, substitute xxxxx with the port you want to use or set it to 0 to have the agent use the RPC Dynamic Port range. Ensure the CERTIFICATE_PATH= is set to the full path where the pfx file was copied (eg, C:\Temp\Cert.pfx)