NM: Migration Best Practices - Error Handling (4316980)

Check for errors on the Migration Control Center (MCC)
Prior to submitting users to the migration queue, check the Application Event log for any current errors where the source is “CMT Migration Server”. If errors are found, resolve the issues and issue an IISRESET from a CMD prompt. Failing to perform this may result in submitted migrations remaining in “Pending” status for an indefinite period of time.

Submitting small batches of migrations will increase the lag time seen on workers and extend the migration window.

When a user is submitted to the MCC server for migration, an update is made to the translation table for that user. At the beginning of each migration job, the worker requests from the server any changes which have been made to the name translation table. If any change is found in the translation table, the worker sets its status to “Offline” to process this change. Since the name translation table is over 1.1M rows this change process could potentially take up to 45 minutes to complete. As an example, if 20 users are submitted, each worker now must process these changes before beginning a migration job, which may cause each worker to appear to pause for 30 minutes before starting to work. If another 20 users is then submitted, at any point, when a worker gets assigned a new job, that worker now must pause for another 30 minutes to process these changes. If the full migration is 200 users and all submitted at the same time, then the process is a single 30 minute pause. This is expected behavior.

For very large migrations consider stopping all workers, submit the users to be migrated to the MCC, then start the workers with a slight pause between each worker to balance the requests to the MCC. This will allow all workers to process the bulk change one time with no additional updates required for name translation changes.

Never assume the CMT_MigrationWorker.exe is not running or stopped cleanly.

Before starting the CMT_MigrationWorker task on a machine check via Task Manager that the process, along with CMTProxy, is no longer running. The CMT_MigrationWorker task manages communication between the MCC and the worker, while the CMTProxy is launched by the CMT_MigrationWorker task to perform the data migrations.

Use Task Manager as a means to identify what is happening on a worker. If the CMT_MigrationWorker task is running at 40%+ of CPU and memory over 100MB then it is most likely processing name translation entries. When a migration job starts, you will see CMTProxy running and growing in both CPU and memory well before the worker splash screen appears for the migration job.

The Migration Control Center database

The database is located at D:\Program Files (x86)\BinaryTree\CMT for Exchange\CMT_XMLServer\App_Data\CMT_Univeral.MDB. Access to the database is accomplished via:

For 3.8 and newer versions, open SQL server to the CMT for Exchange database.

Notes application CMT for Exchange (CMTe)

CMT application on the MCC

Microsoft Access

If the database is opened with:

CMT, no migrations are to be submitted from CMTe and the access database must not be opened with Microsoft Access.

Microsoft Access, the CMT application must not be running and no migrations may be submitted via the CMTe application.

No changes should be made directly to the database via Microsoft Access without specific directions from Binary Tree.

Common Issues

Users are submitted for migration and remain in the status of “Pending” even though migration workers show online and available. There are two potential issues, step one is to check the Application Event log on the MCC.

Application Event Log – CMT Migration Server error

CMT Migration Server : Illegal call to release work. FHID:24 WorkID: 587
CMT Migration Server : wq_IsWorkerAndRecordValid Exception 587 : 5Object reference not set to an instance of an object.

This error will prevent a worker from being able to be assigned new work. This means the worker task needs to be stopped and started. To identify the worker, open the Access database with Microsoft Access. The rules above still apply about other entry of data with the database open. In the above example the FHID:24 is the reference required. Open the table “Farmhands”. The ‘farm_hand’ column is the FHID reference, find the corresponding number and then check the value in the ‘name’ column. This is the migration worker as it appears in the CMT Monitor. Close the database.

In the example above, the worker ‘wtpcpavmem30’ needs to be restarted.

User stuck in “Pending” state – no CMT Migration Server errors in Application Event log

Open the Access database with Microsoft Access. The rules above still apply about other entry of data with the database open. Open the “MigrationQueueTable” table and note the values in ‘status’ and ‘settings_id’ columns.

Open the “Settings” table and verify that the settings found in the “MigrationQueueTable” exist in the “Settings” table.

Close the database. From Notes Migrator (CMTe), reset the user(s) that are in a ‘Pending’ status in CMTe that show in the “MigrationQueueTable” table as ‘Other’. Optionally, open the database again with Microsoft Access and verify the entries were removed from the “MigrationQueueTable” table. If the rows were not removed, delete them from within Microsoft Access. On the MCC, issue an IISRESET from a CMD prompt.

If the ‘settings_id’ is not found in this database, then in CMTe re-assign the profile to the user(s). It is possible that a profile was assigned from a different farm. This can happen when users are moved from one farm to another farm or the settings on the Notes client is inconsistent with the farm being targeted.

In CMTe select the user(s) and submit them for migration adding them to the queue. Monitor they are picked up by migration workers.

Final migration Status is not displayed correctly in CMTe and CMT (3.7.x and older versions)

User(s) show as ‘In Progress’ in both CMTe and CMT. This happens when the server is unable to release a worker from a migration job. The status may be updated manually in CMT. Launch CMT on the MCC. The rules above still apply about other entry of data with the database open from CMT. This error is expected on the MCC, click OK to continue.

Click on the “Users” button.

Click on the Status column to sort by the Status, the ‘In Progress’ show at the top of the list. Right click on the user and select properties.

This dialog will display the CMTUser master record, highlighted in red below, and all migration jobs. Note that what is returned to CMTe and CMT is the CMTUser master record ‘Migration Status’.

All migration jobs appear below the master record. Each begins with Migration_Details/MigrationRecord. The last migration will be the last in the list. Select the last job value, can press Cntrl+C to copy the value to the clipboard.

Now select the CMTUser master record field Migration_Status

Press Cntrl+V to paste the value into the field and then press Enter.

Exit the dialog.

If the status will not change, please contact the Binary Tree support person.

Proven method to avoid issues

1. Stop all workers
2. Reboot MCC
3. Right after a reboot, IIS will be idle until some request is made to the CMTe server, so we want to make sure it is awake and ready for migrations. There are two ways to verify the CMTe server on the control center (MCC) is running and responding:

a. Open the CMTMonitor page, and allow it's natural refresh to continue until the workstation page lists the available workstations (they should be all off-line at this point).
b. Open the actual service page from a browser (http://localhost/cmt_xmlserver/cmtxmlservice.asmx). This is essentially making the same SOAP-type calls the workers and CMTe use to communicate with the back-end database on the control center.

4. Check the Application Event log for any CMT Migration Server errors and correct.
5. Queue all users for migration.
6. Start migration workers with a 2 minute pause between starts