Chat now with support
Chat with Support

Metalogix Content Matrix 8.9 - SharePoint Edition User Guide

Introduction Entering the License Key Content Matrix Console End User Interface Enabling Advanced Mode Connecting to SharePoint Migrating to O365 Using the Azure Pipeline/SPO Containers Preparing for a Migration How Content Matrix Handles Various SharePoint Components Initiating a Migration Configuring Copying Options Saving or Running a Migration Action Copying SharePoint Objects as a Separate Action Self-Service Migration Incremental Migration Log Files Using PowerShell with Content Matrix
Configuring PowerShell for Use with Content Matrix
Registering Metalogix Command DLL Files Adding PowerShell Snap-Ins for the Application Framework Content Matrix PowerShell Commandlet List
Metalogix.System.Commands Metalogix.SharePoint.Commands Metalogix.Jobs.Reporting.Commands Metalogix.SharePoint.Migration.Commands
Modifying Content Matrix Configuration and Settings Configuring Content Matrix for Distributed and Self-Service Migration Frequently Asked Questions
DB_Owner Permission Migrating with the Fabulous 40 Templates Item Count Discrepancies Keyboard Shortcuts License Key Troubleshooting Determining the Status of a Migration Running in the Background MySite and User Profile Migration Issues Optimal Setup for Best Performance Using Proxies, Load Balancing or Alternate Access Mappings Troubleshooting the Extensions Web Service Installing the Extensions Web Service on Specific Systems Extensions Web Service Installation Files Using Older Versions of the Extensions Web Service Preserving IDs when Migrating a Custom List as a Folder Migrating JavaScript Migrating Site Variations Migrating with SharePoint Document IDs Changing a Job Configuration for Multiple Files SharePoint 2003 Version Migration Limitations SharePoint 2013 and Later Site Collection Creation Issue Job List Database fails to Load After Upgrade Customized Wiki Page Web Part Zones Not Being Migrated Preserving SharePoint List Item IDs with a CSOM Connection Type Retrying Failed Document Copies to O365 CSOM Connections Migrating Content When the Source Custom List Template is Missing Are SharePoint Online Migrations Throttled? What to Expect when Migrating with StoragePoint on the Target Migration Error Message 'There was an Error Reading from the Pipe: The Pipe has been Ended (109, 0xd6)' How do I Remove Items from My Azure Blob Storage Account? Azure Batches Getting Stuck "Error, the batch will be resubmitted" Message Displays When Using Azure Item Count at Target is Increased When Migrating Using the Azure Pipeline Custom Lists with Versioning Enabled Not Being Migrated When Using the Azure Pipeline Supported Nintex Actions for SPO Migration "Insufficient Credentials" Message Connecting to Modern Team Site Using Web Browser Authentication Error Making a Browser-Based Connection with PowerShell Console Open
About Us

Retrying Failed Document Copies to O365 CSOM Connections


When I try to migrate some of my content into O365 using a SharePoint CSOM or Office 365 Tenant connection type, and this content contains a large Document Library, I see that some of my documents fail to migrate. The logs for the failure indicate that a "HTTP 500 - Internal Server Error" was thrown. What is the cause of this? And how can I fix it?


There is a potential issue when migrating specifically to O365 where users can encounter an "HTTP 500" error. This is caused by a combination of the CSOM/Tenant connection adapter and SharePoint's document upload methods. To help resolve this issue there is a document retry feature that can be configured. This retry feature will attempt to upload the document(s) into the O365 target, and depending on the configured setting, will attempt to retry the upload process if it fails or times-out on the initial try.

The below steps will explain how to enable and configure this retry setting.

NOTE: This retry method is only meant for migrating to O365 targets. If you are migrating to an on-premise target, then this retry feature will not benefit you since it makes use of O365 specific methods.

1.Make sure that the Metalogix Content Matrix client application is closed.

2.In the file system that the Metalogix Content Matrix Console client application is installed on, navigate to the following folder location:

·Microsoft Windows Vista and Windows 7 and Windows 2010 - <Drive>:\ProgramData\Metalogix

·Windows 98, Windows Me, Windows 2000, Windows 2003, and Windows XP- <Drive>:\Documents and Settings\All Users\Application Data\Metalogix

3.In this location there should be an EnvironmentSettings.xml file. Open this file in an editor program. For example, Notepad, etc.

4.There are two variables that will need to be modified in order for the retry method to work. They are:

·CSOMDocumentRetriesNumber - This value determines the number of times that Metalogix Content Matrix will make another attempt to upload/migrate any document(s) that has failed the initial document upload try, when migrating to a O365 CSOM target.

·CSOMDocumentRetriesDelay - This value determines the amount of time, in seconds (s), that Metalogix Content Matrix will wait before starting a document upload retry, and is based on the above ("CSOMDocumentRetriesNumber") variable.

NOTE: The default values for these two variables will be set as "0". Users can set these values as desired, based on what works best for their environment.

5.After the desired values have been entered in the two variables, save and close the file.

6.Restart the Metalogix Content Matrix client application. The changes should now be in Metalogix Content Matrix Console, and used when running any migration to an O365 target environment.

After these values have been set and the client application has been restarted, any documents that fail on the initial migration attempt we be retried.

IMPORTANT:  If these steps do not resolve the issue and you are using the Azure Pipeline for the migration, the error may be Azure-related.  See "Error, the batch will be resubmitted" Message Displays When Using Azure for details.

Migrating Content When the Source Custom List Template is Missing


I have a list on my source that uses a custom list template (ex. a "Fab 40" list) and I want to migrate that list to my new target SharePoint environment. However, I am unable to install that template on my target before migration. How will Metalogix Content Matrix deal with this?


If at all possible, it is recommended that users install any custom list templates on the target environment before migration. That would allow Metalogix Content Matrix to use those templates when migrating, and trying to reproduce the same results on the target. But there are some customizations within templates that Metalogix Content Matrix is unable to migrate content or metadata into, such as the "Fab 40" templates.

In some cases users are unable to install the custom templates in the target environment, such as when migrating to O365. In these cases Metalogix Content Matrix will attempt to find the base template of the source customized list. Then it will create a list on the target using that same base template, and migrate the content into that list.

To help with this two variables have been added to the list XML attributes. They are:

·BaseTemplate - This is the value for the custom list template.

·BaseType - This is the value for the default template that any custom template is based on. If the custom template does not exist on the target, this template will be used for the list creation instead. Please see for more information on the SharePoint BaseType value.

These variables will help Metalogix Content Matrix determine which base template to use so that an "Issue Tracking" list won't be created in a case where it should be a "Custom List" template.

Let's look at an example where you have a custom list on the source, and that list has a BaseTemplate value of "1011", a BaseType of "0", and has not been installed on the target. In this case, Metalogix Content Matrix will look at the source list and find the BaseTemplate value of 1011. It will then look on the target to see if there is a template that has the same value. If it finds one, then Metalogix Content Matrix will create a list with that template and migrate the data over. But, if the BaseTemplate value is not found on the target, then Metalogix Content Matrix will look at the BaseType value instead, since this value should exist by default in the target SharePoint environment. In this example we also happen to know that a BaseType value of "0" is a "Custom List" or "Generic List" type. So Metalogix Content Matrix will create a list on the target, using the "Custom List" template, and will then migrate the source data into that custom list.

Are SharePoint Online Migrations Throttled?


I am migrating some content to SharePoint Online/Office 365 (SPO/O365). Are migrations to Office 365 throttled? And if so, how does Metalogix Content Matrix deal with this?


Before we can answer these questions, there is some information that we need to review first.

SharePoint Online (SPO) is on the open Internet, and is a multi-tenant environment in which multiple MS customers (up to 10,000) are hosted on a single SharePoint Server Farm. In light of that, Microsoft uses a number of mechanisms to protect SPO customers' environments, and the integrity of these Server Farms.

Based on internal performance benchmarking, there is an impact of between 33% and 40% on the performance of migrations to SPO because of some of these necessary protection mechanisms. Our discussions with the SPO product group, and our experience with numerous customers who have migrated large farms to SPO have shown that the following are among some of the known factors that impact this performance:

·User-based throttling - This ensures that any single user cannot perform so many simultaneous operations on their SPO environment, that these would cause performance issues for other users within the same or other tenants. The following article, while written for SP2010, still applies to SP2013: HTTP Request Throttling in SharePoint 2010 (Part1). Until recently, if a migration was performed using a single username/password combination, migrations were significantly slower than migrations that involve multiple username and password combinations. However, Microsoft is starting to reduce user-based throttling and increase tenant-based throttling. Given that the rollout of throttling changes occurs over a period of time rather than all farms at once, it is difficult to say if the farm you are migrating into has been impacted by these changes yet.

·Tenant-based throttling - This ensures that no one tenant on a multi-tenant farm can use so many resources, that other tenants' performance suffers significantly. Microsoft has started using this to protect tenants on multi-tenant environments. We fully expect this to be the most significant throttling mechanism in use by Microsoft moving forward.

·Farm-based throttling - This ensures that if tenants on the farm are seeing low performance across the farm, no single tenant can create significant additional workload on the farm. Microsoft is expected to introduce this throttling if needed before the end of 2014.

To help address this issue Metalogix Content Matrix has implemented a Health Score feature. This feature is used automatically when users make a connection to SharePoint Online. It will make checks against SPO to find the "health" of the server that is connected to. If the "health" score that it gets back is OK, then migrations will run as normal. However, if the health score that it gets back tells Metalogix Content Matrix that the server is under significant load (i.e. the "health" is not OK), then Metalogix Content Matrix will pause any currently running migrations. Once paused, we will periodically make checks back to the SPO server for the health score. As long as the SPO server tells Metalogix Content Matrix that it is under load, all migrations will continue to be paused. If the health value we get back is OK again, then any paused migrations will resume from where they were paused. This is done because if the server health score gets very poor, migrations will be cut off with no time to pause gracefully.

While it is not possible for users to disable this feature, it is possible to modify some of the values. Users can change some of these values in the UserSettings.xml file, located at: <Drive>:\Users\<User>\AppData\Roaming\Metalogix\UserSettings.xml. The available settings are:

·ServerHealthCheckEnabled - This value determines if the "health score" is visible in the migration logs. This is a True/False value, and when "true" the health score will be seen in the migration logs. And when the value is set to "false" the health score value will not be included in the logs. This value is set to "true" by default.

·ServerHealthCheckInterval - This value determines how long Metalogix Content Matrix will wait before checking the health score with the target SPO environment. This is only applicable when a migration has already been started and then "paused" because a bad health score was returned. This value is set in milliseconds (ms), and is set to "10000" by default.

·ServerHealthScorePauseValue - This value determines at what health score value Metalogix Content Matrix will pause a running migration to SPO. If we check the health score and revive a value that is equal to or higher than the set value, then the migration will be paused. The higher the value, the worse the score. The highest values that can be set is a server score of "8". By default this value is set as "6".

·ServerHealthScoreRestartValue - This value determines at what health score value Metalogix Content Matrix will resume a migration that was paused due to a bad health score. When Metalogix Content Matrix receives a health score that is higher than or equal to the ServerHealthScorePauseValue value, then that migration will be paused. If the ServerHealthScoreRestartKey value is set to "4" then the migration will not start again until Metalogix Content Matrix receives a health score of 4 from that same server. The highest values that can be set is "5". By default this value is set as "4".

For more information on how the SPO server health score is calculated, please see the following article (from a Microsoft MVP): SharePoint 2013: SharePoint Health Score and Throttling deep dive.

Quest is working with Microsoft to determine workarounds that will treat migrations differently than other SPO activity, thereby potentially significantly improving the performance of this portion of the migration workload.

NOTE: In August of 2014 it is expected that Microsoft will introduce some additional throttling of SPO at the Farm level. This farm level throttling will be in addition to the already existing health score functionality. Quest is working with Microsoft in order to find a solution for the issue this would cause for user migrations, and we expect to have a solution once the farm level throttling is enabled.

What to Expect when Migrating with StoragePoint on the Target


I am migrating content to my target SharePoint environment and I have Metalogix StoragePoint installed. What kind of results should I expect after migration?


There are a number of different behaviors to expect when migrating to a target SharePoint environment that has Metalogix StoragePoint installed. The different results you will see depend on which option is selected in the StoragePoint Options tab. For more information about these options, please see Copy Options Dialog.

NOTE:  For the StoragePoint options to be available:

·StoragePoint 5.1 must be installed on the target environment.

·The target environment cannot use a CSOM connection type and/or have SharePoint 2019 installed.

The following is a breakdown of expected results:

StoragePoint Options - Successful Outcome

·Add documents to StoragePoint if an endpoint is configured - This option is for when content from a non-externalized environment (a source environment that does not have StoragePoint installed) is being migrated into an externalized SharePoint target. In this case, any of the files or documents that are included in the migration will be externalized, meaning that those files and documents will call StoragePoint as they are being migrated, and will be externalized directly with the reference link left behind in the file or document's place (in the target SharePoint environment). Once the migration is complete, users can navigate to one of the migrated documents in Metalogix Content Matrix, switch to the Items View, right-click the document, and then select StoragePoint Details. Here, users should be able to see the StoragePoint endpoint folder location for the externalized content, as well as the externalized file name. If the document is opened through SharePoint directly, it will open as expected.

·Shallow Copy of Externalized Data - Shallow copying is no longer supported in Metalogix Content Matrix. For more information, please see Shallow Copying Externalized Content.

·None - This option is for cases where Metalogix StoragePoint is not installed on the target SharePoint environment, or for cases where StoragePoint is installed but users do not want to use it as part of the migration process. When this option is selected, content will be migrated directly into SharePoint with all StoragePoint-related features ignored. However, if the target environment is configured to be externalized with StoragePoint, it is possible that the content will be migrated into the SharePoint target as if the StoragePoint options are ignored, but the target SharePoint environment would then externalize the content separately after the migration completes. While the end result would be the externalization of the data, the migration process itself is only migrating content directly into the SharePoint target.

StoragePoint Options - Unsuccessful Outcome

·Add documents to StoragePoint if an endpoint is configured - If this option failed to migrate the documents/files into StoragePoint, then when users go to the StoragePoint Details, there will be nothing set. This indicates that the files were not placed into the StoragePoint EndPoint and were not externalized.

·Shallow Copy of Externalized Data - Shallow copying is no longer supported in Metalogix Content Matrix. For more information, please see Shallow Copying Externalized Content.

·None - If this option is selected but fails, then the content migration to the target will have failed. This means that the documents/files that are included in the migration action will not exist on the target (externalized or not).

Related Documents