Introduction
This guide contains information that will help you plan your Quadrotech Archive Shuttle migrations.
Key terms
The following table introduces the terminology that is used throughout Archive Shuttle documentation, videos, and the user interface.
|
TERM |
ITEM + DESCRIPTION |
|---|---|
|
Link |
Enterprise Vault A Link is a Vault Store Exchange A Link is an Exchange database Office 365 A connection to Office 365 PST A connection to a PST Output Area Proofpoint A connection to a Proofpoint output area EAS A connection to an EAS IIS Server Metalogix A connection to a Metalogix server |
|
Container |
Enterprise Vault A Container is a Vault / Archive Exchange A Container is an Exchange mailbox Office 365 A Container is an Office 365 mailbox or Personal Archive PST A container is a PST file Proofpoint A container currently has no applicable context EAS A container is an archive relating to a user. Metalogix A container is an archive |
|
Item |
Enterprise Vault An Item is an archived Item. Exchange An Item is an item in the mailbox Office 365 An Item is an item in the mailbox or personal archive PST An item is an item inside a PST file Proofpoint An item currently has no applicable context EAS An item is a message in the archive Metalogix An item is a message in the archive |
Architecture
The following diagram shows the Archive Shuttle components:
Archive Shuttle Core
The Archive Shuttle Core consists of three parts:
·Archive Shuttle User Interface
·Archive Shuttle Web Services
·Archive Shuttle Service
These are explained below.
Archive Shuttle user interface (UI)
The main point of interaction with the administrator is through the web-based user interface. It allows access from anywhere within the enterprise by just using a web browser. It is even possible for archive migrations to be performed remotely from outside of an organization; for example, partner-managed migrations.
Using the wide range of options in the Archive Shuttle User Interface, the administrator can configure and manage archive migrations within an enterprise. The interface also provides at-a-glance progress monitoring, as well as reporting.
Archive Shuttle Web Services
All interaction between Archive Shuttle and Archive Shuttle Modules is done through the Archive Shuttle Web Services. All modules communicate with the Web Services using HTTP(s). If HTTPS is to be used, there is additional configuration that needs to be performed.
Archive Shuttle service
The Archive Shuttle Service is a Windows Service which periodically run Archive Shuttle scheduled tasks. These can be database maintenance tasks or other tasks that need periodic execution in Archive Shuttle itself. It also execute database updates over the directory and item databases.
Enterprise Vault to Enterprise Vault
This section explains what the requirements are for an Enterprise Vault (EV) to Enterprise Vault migration, and then covers the basic steps that should be followed to start such a migration.
Stage 1 Synch
As soon as a user is enabled for migration, the synch process-flow shown above starts.
Archive Shuttle Core sends a command to collect all needed metadata information. The results are reported back to Archive Shuttle Core to allow item level tracking and auditing.
The Enterprise Vault Export and Enterprise Vault Ingest process is then started. This runs continuously in the background until the Stage 2 (Switch) phase is initiated for the user, or a specified external trigger is fired (e.g the mailbox has been moved from Exchange 2003 to Exchange 2010).
Stage 2 - Switch
After the Stage 2 ('Switch') phase to the target environment is initiated, the configured workflow starts, and by default determines the gap between the source archive and the last imported item. Archive Shuttle then synchronizes the difference one last time to the target mailbox. The archive then gets assigned to the user and enabled. The last step is to cleanup shortcuts in the target mailbox.
Requirements
The following is a list of requirements.
|
Requirement |
Description |
|---|---|
|
Modules installed on source |
The EV-type modules should be installed on each appropriate source Enterprise Vault server involved in the migration. The EV import module is not required on these machines. |
|
Modules installed on target |
The EV-type modules should be installed on each appropriate target Enterprise Vault server involved in the migration. The EV export module is not required on these machines. |
|
AD Collector module installed |
An AD Collector module should be installed so that it can collect user-level information for the migration project. |
|
Staging Area created |
One or more staging areas should be setup, of an appropriate size and should be excluded from virus scanners. |
|
Workflows reviewed |
The Stage 2 workflows should be reviewed, and if necessary, customized to meet the needs of the project. |
|
Apply appropriate failed item threshold |
It is best practice to find an appropriate failed item threshold for the project/customer and apply that at the links level. This way all mappings which are created will inherit this value. |
Basic steps
The following are the basic steps to start this type of migration.
|
Step |
Screen |
Description |
|---|---|---|
|
Enable Modules |
Modules |
All Archive Shuttle modules must be enabled, and optionally a schedule for them defined. It is important to verify that none of the modules have a red background (this indicates that Archive Shuttle Core has not had contact with the module) and that the module versions are as expected. |
|
Enable Domains |
Active Directory |
Select and enable one or more domains for synchronization. |
|
Add Source EV Environment |
EV Environment |
Add the Enterprise Vault source environment. |
|
Add Target EV Environment |
EV Environment |
Add the Enterprise Vault target environment. This is not necessary if both the source and target are the same. |
|
Add Retention Category Mappings |
Retention Category Mappings |
Map the source and target environment retention categories. If the migration is intrasite, they can be automatically created in the admin interface. |
|
Add Source Links |
Links |
Configure all appropriate source links. Select the links, and then click on Map Modules, create the item database(s) and enable archive gathering. Ensure there is no import module specified on the links as this may impact the process later on. |
|
Add Target Links |
Links |
Add all appropriate target links. An EV Collector module should be associated with one of the links so that shortcut building information can be gathered from the site settings in Enterprise Vault. |
|
Add Staging Area |
Links |
Ensure that the Default Staging Area is configured correctly. Depending on the migration, the free space, and the number of vault stores being used as the source for the migration, the high water mark can be adjusted upwards. |
|
Configure Folder-Less Item Handling |
System Configuration |
Many versions of Enterprise Vault allow archiving of items in a folder-less area called the Top Of Information Store. On the System Configuration, enter a folder name to be used for placing the items in the target archive. |
|
Stop the EV Provisioning task from auto-enabling users |
Enterprise Vault Administration Console |
Ensure that the source and target environments do not have any provisioning groups set to auto enable users throughout the duration of the migration. This prevents users being targeted by alternating environments depending on when and which provisioning task runs. |
|
Stop Storage Expiry running |
Enterprise Vault Administration Console |
Ensure that the source and target environments do not have storage expiry running. If it is enabled then it is possible that items which are collected can not be found when it is time to export them. |
|
Map one or more containers |
Manual Mapping/Bulk Mapping |
One or more containers can now be mapped and migration can begin. |
Determining Enterprise Vault archive owner(s)
Archive Shuttle must determine the owner of a mailbox archive. This section explains how this takes place..
Archive Shuttle takes the owner information of a mailbox archive from the Auto Security Descriptor field within the Enterprise Vault Directory database. The first entry for an archive will be the owner. Take the following three examples:
Simple 1
·Bill usage to: somedomain\simple1
·Permissions tab: somedomain\simple with inherited read, write and delete permissions
Archive Shuttle will see the owner of this archive as simple1.
Simple 2
·Bill usage to: somedomain\simple2
·Manually grant an additional user read, and delete permissions on the archive
·Permissions tab: somedomain\simple2 with inherited read, write and delete permissions. somedomain\someotheruser with manually set read, and delete permissions
Archive Shuttle will see the owner of this archive as simple2.
Simple 3
·Bill usage to: somedomain\simple3
·Grant full mailbox access via Exchange Management Console or Exchange Management Shell
·Permissions tab: somedomain\simple3 with inherited read, write and delete permissions. somedomain\someotheruser also with inherited read, write and delete permissions
Archive Shuttle will see the owner of this archive as simple3.
|
|
NOTE: The order which the archives are listed in the permissions tab in Enterprise Vault does not reflect the order that the accounts are described in the Auto Security Descriptor field. The permissions tab shows the archive permissions in alphabetical order. |
More detail
1.Archive Shuttle resolves the 'Owner' of an Archive as follows:
In the EnterpriseVaultDirectory database, the following SQL is used:
select ADMbxDN, LegacyMbxDN from ExchangeMailboxEntry where DefaultVaultId=@VAULTID
2.Archive Shuttle then does a lookup for the ADMbxDN in Active Directory to get the SID:
LDAP Query: (ADMbxDN=)
If Archive Shuttle does not find the AD object, the archive will be marked as Ownerless and move to the next option.
3.Archive Shuttle then does a lookup for the LegacyMbxDN in Active Directory to get the SID:
LDAP Query: (LegacyMbxDN=)
If Archive Shuttle does not find the AD object, the archive will be marked as Ownerless and move to the next option.
4.We then compare the SIDs we got from ADMbxDN and from LegacyMbxDN, and if both of them match, we have an Owner. If they do not match, they will be marked as 'Ownerless' and move to the next option.
5.If we do not find an entry in ExchangeMailboxEntry, we resolve using the BillingOwner or the AutoSecurityDesc:
select SID from Root inner join trustee on OwningTrusteeIdentity = TrusteeIdentity where VaultEntryId=@ARCHIVEID
|
|
NOTE: Regarding OwnerUserSid versus SID fields in the EVArchive Table on in the Archive Shuttle Directory Database: ·SID is always filled out with the SID which is retrieved from the above steps ·OwnerUserSid has a foreign key constraint on the [User] Table. So Archive Shuttle only fill this out if the SID obtained from EV is actually contained in the Archive Shuttle Directory Database (i.e. has been synced from AD) |
5.The UserSidHistory table is stored Sid history for each user when they were migrated from AD1 to AD2. This SidHistory attribute is part of Active Directory, and Archive Shuttle collects this data from Active Directory directly.
Example
User1 was migrated from AD1 to AD2, SidHistory attribute was populated to Active Directory and when the Active Directory sync is performed, table UserSidHistory is populated in parallel. User1 was migrated to another Active Directory, so it is supposed to be the same user and therefore in owner resolution logic is UserSidHistory as the last step. Sometimes, even the migrated user from AD1 to AD2 is not supposed to be the same user as samAccountNames are different in both Active Directories. In this case, owner resolution will take AD2 user as the owner which could be wrong in some special situations.
Workaround
As a workaround, you can:
§remove SidHistory parameter from Active Directory, and allow Archive Shuttle to synchronize Active Directory entries (recommended option), run Enterprise Vault collection, to assign correct users in ContainerToUser table
§use the SQL to assign HistoryUserSid instead of actual Sid, example of such script is below:
--update ContainerToUser, take UserHistorySid and assign to Container type 1 (EV containers only)
SET NOCOUNT OFF;
declare @sid varchar(128)
DECLARE ITEM_CURSOR CURSOR FOR
Select UserSid From ContainerToUser a
inner join [dbo].[Container] b on a.ContainerId = b.ContainerId and ContainerTypeId = 1
OPEN ITEM_CURSOR;
FETCH NEXT FROM ITEM_CURSOR INTO @sid
WHILE @@FETCH_STATUS = 0
BEGIN
update x set UserSid = (SELECT UserHistorySid FROM [dbo].[UserSidHistory] where UserSid = @sid)
from [dbo].[ContainerToUser] x
where UserSid = @sid
print 'Updated containerid ' + @sid;
FETCH NEXT FROM ITEM_CURSOR INTO @sid;
END;
CLOSE ITEM_CURSOR;
DEALLOCATE ITEM_CURSOR;
·remove entries from UserSidHistory table, disable Active Directory sync to avoid entries are coming back with next Active Directory sync, run Enterprise Vault collection to assign correct users in ContainerToUser table
|
|
NOTE: Re-using of sAMAccountsName/MbxNTUser or MbxDisplayName in one domain can cause an issue with Enterprise Vailt Archive assigning process. |
General considerations
The following general considerations should be taken into account for this type of migration:
·Ingest speed is likely to be reduced versus the speed of export since ingesting is also going to place a load on the indexing engine on the target environment. In general do not expect the export and ingest speeds to be comparable.
·When migrating from a pre EV 10.0.3 system extraction of data may be slower than expected if Enterprise Vault Collections have been used. Access to the data inside the CAB file is single threaded. This issue does not exist in Enterprise Vault 10.0.3 and later.
·An issue is currently being investigated by Symantec. Items extracted from EV 9, may not be ingested into EV 11.0.1. This affects only a small number of items. This may be addressed in the future by Symantec.
·EVPM may have difficulty in connecting to, and processing, mailboxes which are homed on Exchange 2013. This knowledge base article may help.
·EVPM may need to have the DS Server registry key set. This knowledge base article may help.
Migrating an archive from Enterprise Vault to Enterprise Vault
For this scenario, the migration takes place between two vault stores in the same Enterprise Vault environment.
Prerequisites
The following steps must be completed before the migration begins:
·Two or more vault stores must exist in the Enterprise Vault environment.
·All modules must be enabled.
·Appropriate Active Directory domains must be enabled for scanning.
·An Enterprise Vault environment has been added for migration.
·A link database has been created for the source vault store/archives.
·Module mappings are configured for the source and the target vault stores. The source vault store needs to be linked to an EV Export module, and the target needs to be linked to an EV import module. In addition, an EV Provisioning Module needs to be linked to the source vault store and to the target vault store.
·During switchover, both the archiving and storage tasks need to run on the same Enterprise Vault server.
Add retention category mappings
The final stage of the setup of the archive migration is to map retention categories between the source and target environments.
1.Go to Configuration > EV Retention Mappings.
2.Click Create Mapping.
3.Select the values from the drop-down lists, and click [Add] to add the mapping to the retention category grid.
Retention mappings are managed from this screen.
There is an option to use Add Intrasite Migration Mappings, which maps each retention category to itself. This can be used in a situation where archives are to be migrated in the same environment. It simply moves the archives to the new location.
Map containers for the test archive
In order to migrate a test archive from one vault store to another, the source and target containers must be mapped. This can be done as follows:
1.Go to Manage > Bulk Mapping.
2.Type the beginning of the archive name in the Container Name filter. Click the button in the far right of the filter row in the Apply Filter column.
|
|
NOTE: If the test archive is not displayed, go back to the Configuration > EV Environment page and click Sync all AD Users. Then, select the vault store where the source archive is located and issue a Run Now for Archive Gathering. |
3.Select the checkbox next to the test archive, and click Add Mappings.
|
|
NOTE: A wizard begins gathering information related to the mapping for the select archive. (Multiple archives can also be selected.) |
4.When youre prompted to select a target container type, select Enterprise Vault, and then click [Next].
5.When youre prompted to select a target user, select Same User, and then click [Next].
6.When youre prompted to select a container strategy, select Create new containers, and then click [Next].
7.Select the link that corresponds to where the archive is to be migrated, and then click Next.
|
|
NOTE: If the target link drop-down list is empty, then it is likely that an EV Import module has not been correctly associated with the link. Review the Links page. |
8.On the Workflow Policy screen, select Standard Workflow Within same EV Site (without Archive deletion), and then click [Next]. You dont need to select the ownerless workflow policy in this situation; its optional.
|
|
NOTE:If a group of users was selected containing a mixture of ownerless and normal containers, this screen gives an administrator the option to specify which workflow to use for the ownerless containers. |
9.For the filter policy, selecIt Default (No Filter), and then click [Next].
10.On the Container Mapping Settings screen, select Enabled for both Migration Status and Item Gathering Status, and then click [Next].
11.If desired, set the mappings priority, and then click [Next].
12.Review the summary screen. If everything looks as expected, click [Confirm].
Review Stage 1 status
A few minutes after the mapping is created, the software tells the appropriate modules to start the actions defined in the mapping. Review the progress for this stage of the migration using these steps:
1.Go to Manage & Operations > Stage 1 (Sync Data).
2.Type the beginning of the archive name in the Name filter, and then click the button in the Apply Filter column, found at the far right of the filter row.
3.Once the source archive displays, click the Refresh button to see the status of the export of the archive and the ingestion of data into the target archive. Continue to click Refresh until both export and import are complete.
|
|
NOTE: If the data progress bars reach 100% for export, but show no progress for import, the Retention Category mappings have likely not been configured. |
If more data is added to the source archive, its synchronized to the target archive every 24 hours using the connection made with the mapping you created in the previous section. In addition, part of the Stage 2 workflow is to perform a final sync. Therefore, its not necessary to ensure that export and import has reached 100% before moving on to the next steps.
Validate exported data
If theres a large amount of data to export and import, and you want to check the progress, click the Refresh button located on the navigation bar on the Stage 1 (Sync Data) page.
In addition, you can use Windows Explorer to browse the staging area data on the disk and view the folder structure.
Also, the Enterprise Vault Admin Console shows that an archive exists in the source Vault Store and that a new archive has been created in the target vault store. Its also possible to grant a service account access to the target archive and perform searches on it to make sure the data matches the source archive.
Enable Stage 2
Before you enable Stage 2, the switch-over for the test archive, check the Stage 1 (Sync Data) page for issues like failed item-export or failed item-import.
To enable Stage 2 for the test archive, perform these steps:
1.Open the Stage 1 (Sync Data) page.
2.Select the checkbox next to the test archive, and click Enable Stage 2 in the navigation bar.
3.When you refresh the Stage 1 status, a check mark should display in the Stage 2 Enabled column.
Stage 2 is the switch-over to the target environment. A final synchronization of archived items is performed from the source environment to the target environment, before several additional migration tasks are performed.
Review Stage 2 status
After a few minutes, the progress of the test archive migration displays on the Stage 2 page:
1.Go to the Stage 2 (Switch User) page.
2.Type the beginning of the archive name in the Container Name filter. Apply the filter by clicking the button in the Apply Filter column, located at the far right of the filter row.
|
|
NOTE: If the archive isnt displayed, wait 1-2 minutes, and then click Refresh in the navigation bar. |
3.Once the source archive displays, click Refresh to see the progress of the final stages of the archive migration. By default, since the workflow policy that was selected was Standard policy (without archive deletion), these steps are performed:
4.Disable source mailbox from Enterprise Vault archiving.
5.Rename the source archive.
6.Collect any remaining items for migration.
7.Import the remaining items into the target.
8.Zap the mailbox, removing EV settings.
9.Assign the new archive to the user.
10.Enable the mailbox for archiving again.
11.Rename the target archive.
Update all the existing shortcuts to point to the new archive.
Verify data has been migrated
After all Stage 2 operations are complete, the Stage 2 (Switch User) page for the test archive shows a check mark in the Finished column.
In addition, if Outlook or Outlook Web Access is used to access the test mailbox, then all of the archived items are accessible via the new archive. Shortcuts to archived items will work and open correctly.