One ADAM and many DSA servers, dsamain.exe high cpu load, migration sessions time out. (4254907)

There are many (10 or more) Directory Synchronization Agents installed in the environment, but only one ADAM instance is available, so all DSA servers use single ADAM. Tens of thousands of objects have been migrated and the ADAM database is several gigabytes in size. Even when synchronizations are stopped, ADAM process (dsamain.exe) shows very high CPU usage and migration sessions take several minutes to initialize or sometimes time out on the initializing step.

All DSA agents are checking ADAM for any new jobs to do every 5 second. The query used is: (|(aelita-Amm-StartOperation=TRUE)(&(aelita-Amm-StopOperation=TRUE)(objectCategory=aelita-Amm-MigrationJob)(|(aelita-Amm-LastOperationCode=6)(aelita-Amm-LastOperationCode=7))))

Attributes that query is ran against are not indexed in the ADAM database, so with a big number of migrated objects, query may take up to 60 seconds to return results. When several agents run these queries, they may overlap and response time are even worse. Searching against unindexed attribute also increases CPU load, causing further delays. As a result, an agent trying to do the migration can stall for over 5 minutes, causing the migration session to time out.

A workaround is to index the three attributes in question in the ADAM database:

1) on the ADAM machine start ADAM adsiedit from Programs menu

2) configure a connection to ADAM host and port, chose Schema for a context

3) expand the attributes list, locate aelita-Amm-LastOperationCode attribute, right-click it and select properties

4) set the searchflags attribute of aelita-Amm-LastOperationCode to 1

5) repeat steps 3 and 4 for aelita-Amm-StartOperation and aelita-Amm-StopOperation attributes

6) after about 15 minutes check the ADAM event log in event viewer and look for messages saying that attributes have been indexed (there will be 2 events per attribute), if they are still not there, give it some more time until events appear

After the attributes have been indexed, query above should return results within 2-3 seconds and CPU usage for dsamain.exe should decrease.

In one particular environment the above modification has changed average migration session time from 8-9 minutes down to 20 seconds and reduced dsamain.exe CPU usage on a dual CPU Xeon server from 85% to around 30%. While results totally depend on the hardware, number of migrated objects, number of DSA servers and jobs currently running, it is still recommended to make the above change in any envrionment where high CPU load by ADAM process is an issue.

The defect id for this issue is ST74663, to be addressed in future versions of Migration Manager.