Benchmark Factory for Database 8.0 - User Guide

Creating a Project With Benchmark Factory

This topic outlines the Benchmark Factory workflow. Click the links in each step to drill down to more-detailed information.

Step 1. Set Up Benchmark Factory

Use the Settings dialog to specify default settings for job and workload options, such as benchmark scale, the number of virtual users per iteration, timing, and latency. Changes to these settings affect only new jobs, not existing jobs.

To specify Benchmark Factory settings

1. Select Edit | Settings. The Settings dialog opens.
2. Select a page from the left-side navigation and then specify the corresponding options. See About Settings for more information.

Step 2. Create a Connection

Before you can run a job against a database, the database connection must be defined in Benchmark Factory. You can define a new connection from within the New Job Wizard, or you can define the connection using the New Connection dialog or the My Connections tab prior to creating a new job.

To create and save a new connection

1. Click in the main toolbar. The New Connection dialog opens.
2. Select a connection type from the drop-down list. The Connection properties for that connection type display.
3. Enter the connection information and click Create Connection. For more information about creating connections, see About Benchmark Factory Connections.

Step 3. Create a New Job

Use the New Job Wizard to create a new job. The New Job Wizard guides you through the steps required to create a database load test (workload) and specify general options for the job.

When creating a new job in the wizard, you can specify settings that apply to this job and override the defaults in the Settings dialog. See Quickstart: Create a New Job for more information.

To create a new job

1. Select New from the Benchmark Factory main toolbar. The New Job Wizard opens.
2. Select the connection for the database-under-test. See About Benchmark Factory Connections for more information.
3. Click Workload in the left pane to begin specifying a workload.
   1. On the Workload page, select the type of test from the drop-down list (for example, Industry Standard Benchmark). Then select a test to include in the workload.
   2. After selecting a test, click the Select Test or Add Test button at the bottom of the wizard page to specify options for that test.
4. To add another test to the workload, click Add Another Test/Step.
5. When you finish specifying the workload for the job, click Agent in the left pane to specify an agent.
   • Click the checkbox to the left of the agent name to select it. A checkmark displays for the selected agent.
6. Click Save/Close to save the job and close the wizard.
7. The new job is added to the list of jobs in the Jobs View pane.
8. To rename the job, right-click the job in the Jobs View and select Rename.

Modify a Job

After creating a new job, you can modify the job. For example, you can change the database-under-test, add tests/steps, or change test options.

To modify an existing job

1. In the Jobs View pane, right-click the job and select Edit Job.
2. To modify the number of virtual users for a specific test in a job, do the following:
   1. Right-click the test in the Jobs View pane and select Edit Job. The Edit Job wizard opens to the selected test in the workload.
   2. Select the Test Options tab. Then select the User Load tab.
   3. Modify the User Load. Click Save/Close when finished.

Step 4. Run the Job

To run a job

• In the Jobs View, select the job to run and click in the jobs toolbar, or right-click the job and select Run Job.

Step 5. Analyze Test Results

Benchmark Factory provides a number of metrics to analyze a system-under-test. These include:

• Response, retrieval, and transactions times
• Transaction throughput in transactions per second (TPS)
• Amounts of data transferred in bites per second (BPS)
• Errors incurred
• Transaction Distribution
• Any user defined counters (Windows NT/2000 Performance Monitor) added using the Benchmark Factory Console.

Benchmark Factory provides the following testing results:

• Real time statistics (See View Test Results - Running Job for more information.)
• View results and compare results for completed tests (See View Test Results - Completed Job for more information.)
• Run Reports (See Run Reports for more information.)
• Testing results exported to Excel (See Export Benchmark Factory Test Results to Excel for more information.)