Secure Copy 5 Performance Tuning (4263008)

This article contains frequently asked questions about the performance tuning for Secure Copy 5.x.  

Performance tuning of Quest Software's Secure Copy is dependent on the environment in which it is to run as there are a number of factors that can affect the performance of Secure Copy. Network bandwidth and disk throughput are the largest factors in determining the performance of data migrations. 

Can variables affect the performance of Secure Copy?

Variables that can affect the performance of Secure Copy are as follows, not necessarily in the order presented.

• Log on as a member of the Administrators and Backup Operators groups

• The Hard Disk read/write performance of the Source and Target machines

• The number and average size of the data files to be copied

• Available RAM on the console machine

• Location of the Secure Copy console; ideally the console should run on the Source or Target machine

• The number of threads configured in Secure Copy

• The number of files configured in a Secure Copy batch (Batch Count)

• The size of the batch configured in Secure Copy

• Options selected per Secure Copy job

The above variables work together to create optimal performance when moving data from one device to another. Secure Copy does not have any direct bandwidth throttling controls however; the consumption of network bandwidth can be controlled by modifying the combination of threads, batch size and number of files in a batch. In the following section, each of the variables will be examined and explained so that performance can be better tuned for the current task.

What group membership does the user need to have for full access to files and folders?

Log on as a member of the Administrators and Backup Operators groups

Secure Copy runs under the context of the logged on user. For full access to all files and folders on the source machine the user must be a member of the Administrators and Backup Operators groups on the Source machine.

Note that in a Windows to Windows data copy the user must only be a member of the Administrators group to effectively grant the user the Backup Operator privileges and enable the “Override Security on Access Denied” feature. However, when copying to or from certain network storage devices (such as a NetAPP Filer), the ‘Override security on Access Denied’ option does not function. In this case, it is necessary for the user to be made a member of the Backup Operators group on this machine to avoid being denied access to certain files. If there is any question as to whether a machine may or may not support this option, simply ensure the user is a member of the Backup Operators group to prevent being denied access to files or folders. 

What performance can be expected during the copy procedure?

The Hard Disk performance of the Source and Target machines

The Hard Disk performance of the machines where the data is currently stored and where the data will be copied to can greatly affect the performance of the copy procedure. Machines that have Disk I/O Bottlenecks will slow the copy procedure. Secure Copy’s default copy job settings are ideal for a copy job containing a mix of small, medium, and large files running from a multi-core CPU Member Server with a single 7200rpm SATA 2.0 disk to another multi-core CPU Member Server with a single 7200rpm SATA 2.0 disk over a 100Mb Ethernet network. If you have better performing hardware such as a RAID array, high performance RAID controller, high performance SCSI disks, or even Solid State Drives you may consider adjusting the default copy settings for your environment.   

What are the recommendations for the number of threads and batch settings?

The number and average size of the data files to be copied

Depending on the number and size of the data files to be copied, the settings for the number of threads, batch size, and batch count can be modified. Information and recommendations for the thread and batch settings will be detailed in this document. 

Will RAM availability affect the performance?

Available RAM on the console machine

RAM will be consumed at the source and target machines; however, the majority of the RAM will be consumed on the machine running the Secure Copy console. The RAM being used in the console is for the configuration of the jobs. In the process of the building the jobs, Secure Copy will generate lists of files in the job(s) that will be portioned out amongst the available threads and batches. The Secure Copy console uses available resources on the machine where it is installed to build and manage the jobs. Maximizing the available RAM on the console machine will allow Secure Copy to perform its tasks without caching to disk because of lack of memory. 

Where should the Secure Copy Console be installed? 

Location of the Secure Copy console

Secure Copy should be installed on either the Source or Target machine whenever possible. The reason for this is that the console will stream RPC commands to the source and target machines as part of the copy process. Having the Secure Copy console on a third machine will cause numerous network connections to be opened to both source and target machines and thus consume additional network resources and have a minor latency between the Secure Copy console and the Source and Target machines. 

What are the recommended Thread settings?

The number of threads configured in Secure Copy

The number of copy threads is equivalent to the number of copy operations that can be performed simultaneously. If the job is moving a large number of small files then you may consider increasing the thread count. The result will be more threads per job moving data files resulting in moving massive numbers of files in the shortest period of time (assuming the Disk I/O does not become a bottleneck). When moving primarily larger files, fewer threads in use will be more effective because the threads will be able to focus on moving few large files, switching control between few threads in a fluid process rather than passing control to multiple threads for the same files which is not optimal for the copy process. 

NOTE:  If you are running multiple jobs at the same time, make sure the total thread count for all jobs is not more than 20.  For instance, if you run 5 jobs at the same time, the thread count for each individual job could be 4 (since 4 thread count x 5 jobs = 20 maximum threads).  Also, stagger the start times of the jobs by at least 5-10 minutes. 

If more than 20 threads are needed, you can install Secure Copy on an additional machine and move some of the jobs there.  Secure Copy is licensed on a named server basis.  Only source and target machines used during copy jobs are licensed. 

Also, the job(s) are saved in the securecopy.ssd file. It’s recommended to make a back-up copy of the securecopy.ssd file, anytime changes are made to the job(s).  The default location of this file is C:\Program Files (x86)\ScriptLogic Corporation\Secure Copy 5. 

What is the recommended Batch Count setting?

The number of files configured in a Secure Copy batch (Batch Count)

Batch Count is a limiter based on the number of files an individual thread can process at a time. The range is 25-100 files. When a job is being processed, a single thread will copy either the maximum number of files in the batch count or the maximum batch size, whichever is reached first. When processing a large number of small files, it is preferable to set this value to the highest level for efficiency. When processing large files, this value can be set anywhere because the batch size limitation will be met prior to meeting the limit of the batch count. In most cases, the default values are optimal. Changing these values incorrectly can lead to performance degradation. 

What is the recommended Batch Size setting? 

The size of the batch configured in Secure Copy

Batch Size is a limiter based on the size of the files a thread can process at a time. The range is 1-10MB of data. When a job is being processed, a single thread will copy either the maximum size of files or the maximum number of files whichever is reached first. When processing a large number of small files, it is preferable to set this value to the highest level for efficiency. When processing large files, this value can be set anywhere because the batch size limitation will be met prior to meeting the limit of the batch count. In most cases, the default values are optimal. Changing these values incorrectly can lead to performance degradation. 

What options should be selected as part of the copy job?

Options selected per Secure Copy job

There are options that can be selected as part of the copy job. Each of these options will cause more overhead to be built into the job. Where the overhead is minimal in most cases, there are options that cause additional overhead or could cause errors in the jobs if not properly configured.

Generation of a Legacy (in Secure Copy 5.5x+) Log file

Generation of Legacy log files required additional I/O on the console machine plus the retrieval of the information on the success and failure of the individual files and folder. In testing the performance impact of generating the Legacy log file has been mostly negligible but for larger copy jobs we recommend relying only on the New Logging Functionality of SCY 5.5x and higher, and disabling the Legacy Log file.

Purge of files and directories

Purging is part of the synchronize options within Secure Copy. To purge files the Source and Target machines and paths as defined by the job must be compared then adjusted or purged accordingly. Because it requires the comparison of the Source and Target directories and files, this option will impact the duration of the copy process.

Verify the file copy (CRC32 checksum)

Verifying the file copy is a time, I/O, and processor intensive activity because it compares the CRC32 checksums of the Source and Target files. Because it requires the comparison of the Source and Target files, this option will impact the duration of the copy process.

Compress files and folder on Target machine

When utilizing this option, the compression is not performed by Secure Copy; it is performed by the Operating System. Because the compression function is performed at the time of the file write operation, this option will impact the duration of the copy process.

Migration of Groups and Users

When a Group or User is migrated, Secure Copy will send an RPC call to a Domain Controller to validate the Group and its members. In a standard single console migration, this option will impact the duration of the copy process.

In a non-standard multiple Secure Copy console migration with each console migrating data and local groups and a large number of groups and users being migrated that Secure Copy can potentially flood a Domain Controller with RPC requests generating RPC timeout errors. To avoid these types of conflicts the user can stagger the start of the migrations or limit the number of threads being used across all consoles.

The option settings are not global settings but are job specific. This enables the user to create jobs optimized for small and large files as separate jobs. Each job with its own specific configuration of the thread and batch settings can produce a completely optimized data migration through Secure Copy.