Benchmark Factory for Database 8.0.1 - User Guide

Jobs View Pane

The Jobs View displays a list of saved jobs, shows which jobs are currently running, and which jobs are scheduled to run.

From the Jobs View pane, you can run jobs, view job status, and view real-time testing results.

To run a job

• Select the job in the Jobs View pane and click .

To view test results

• To view a job's test results, select a job in the Jobs View pane. Test results display in the right pane. See Benchmark Factory Console for an overview of the Benchmark Factory console.
• To compare two or more run results for a test, select the Compare Results tab. Use Ctrl+click to select multiple test runs. A comparison of the results for the various runs displays.

To edit an existing job

• To edit an existing job (to modify the workload or job setup options), select the job in the Jobs View pane and click .

View Job Status

From the Jobs View pane, you can view job status.

The following job states are identified:

• Scheduled: All jobs currently waiting to run or scheduled to run at a future time.
• Running: Job currently running.
• Completed: All completed jobs.

Jobs View Toolbar

The Jobs View toolbar provides user functionality to save or delete jobs.

	Move the selected test up in the list.
	Move the selected test down in the list.
	Delete the selected item from the Jobs View.
	Toggles tree-view on and off.

Note: To save a job as a Benchmark Factory script, select the job and click Save in the Benchmark Factory toolbar or select File | Save.

Quickstart: Create a New Job

Use the New Job Wizard to create a new job. The New Job Wizard guides you through the process of creating a database load test (workload), as well as selecting a database to test, setting general job options, and specifying agents.

Navigating the New Job Wizard is easy. The three major steps are displayed in the left pane: Database Under Test, Workload, and Agent. The individual tests you add are displayed under the Workload step. Click on a step or test in the left pane to display the available options and properties in the right pane.

To allow the wizard to guide you through each step, after completing a page click the green arrow at the bottom of the page. This will advance you to the next step. For example, after selecting a connection on the first page of the wizard, click .

To create a new job

1. Click in the Benchmark Factory main toolbar. The New Job Wizard opens.
2. Select a connection. Select a connection from the drop-down list in the Database Under Test page.
   • Click New to define a new connection. See About Benchmark Factory Connections.
   • Click Edit to edit the selected connection. See Edit Connections.
   • Click Test to test the selected connection.
3. The Database Under Test page also provides database and host information for the selected connection.
   • To refresh this information, click Detect Environment Information.
   • You can edit this information or add custom properties. This information is also displayed in the Environment tab of the Connection dialog. See Environment Information.
4. Add Workload. After selecting a database connection, click at the bottom of the Database Under Test page, or click Workload in the left pane,.
5. Select a test. On the workload page, select the type of test to perform from the drop-down list. Then select a test to add to the workload. To learn how to create a specific test, select from the following:
   • Industry Standard Benchmark Test—These tests simulate real-world application workloads. Select from a number of standard benchmarks included with Benchmark Factory.
     • For benchmark test descriptions, see Overview of Benchmark Testing.
     • To create a benchmark test, see Create Industry Standard Benchmark Test.
   • Capture / Replay Test—Capture and replay a workload; replay a workload from a trace file; or import a workload for a database repository, text file, or export file from another Quest Software tool.
     • To create a capture/replay test, see Capture and Replay Tests.
   • Artificial Test for Desired Effect—Design a workload using a mix of benchmark tests, transaction types, SQL statement types, or resource loads.
     • To create an artificial test, see Artificial Test for Desired Effect.
   • Scalability Test—Use this type of test to load test and compare the performance of different SQL statement variations.
     • To create a scalability test for SQL, see Create SQL Scalability Test.
   • Custom Test—To create a custom Mix test, Replay test, Goal test, or SQL Scalability test, see Custom Tests.
   • Create/Delete Benchmark Objects Test—To add a step to create or delete benchmark objects, see Add a Create/Delete Benchmark Objects Test.
   • Execute External File—To add a step to execute a file, see Execute External File.
6. Click Add Test or Select Test at the bottom of the page to add the test to the workload.
7. Specify test options. Specify test options for the selected test. Select one of the links above for detailed information about test options for each test type.
8. Job Setup. (Optional) You can specify job-level options. Click Job Setup at the bottom of the Test Options tab, or select the Job Setup tab. See Job Setup Tab to learn more.
9. Agent Setup. After specifying options for the test you selected, set up the agents for this job.
   1. Click the Agent Setup button at the bottom of the page, or click Agent in the left pane.

   2. Click inside the checkbox to the left of the agent name to select it. A checkmark displays for each selected agent. See About Agents for more information.

      Note: When you run a job using one or more local agents, if Agent utilization of resources on the local machine is too high, errors could occur.

10. After specifying the test-level and job-level options, you can save the job, run the job, or schedule the job. Review the following:

    • To run the job immediately, click .
    • To schedule the job, click Workload (left pane) and select the Job Setup tab. Then select the Schedule tab. See Schedule a Job.
    • To save the job without running it immediately, click . Use this option after scheduling a job.
    • To save the job to an existing job, select the Job Setup tab. In the Save Job section, select the name of an existing job. Click .

Notes:

• After creating a job, you can save it as a Benchmark Factory script. Select the job in the Jobs View pane and click Save in the main toolbar or select File | Save.
• To modify an existing job, right-click the job and select Edit Job.
• After creating a job, you can add tests to the workload. Right-click the job and select Edit Job.

Edit a Job

Use the Edit Job Wizard to modify an existing job.

To edit a job

1. Select a job in the Jobs View and click .
2. See the topic on the test type you want to edit for more information on editing test and job setup options. See Quickstart: Create a New Job for more information.

Industry Standard Benchmark Tests

This test type allows you to create load scenarios from one of the following industry standard benchmark tests: TPC-C, TPC-E, TPC-H, AS3AP, Scalable Hardware, TPC-B, or TPC-D. Each of these tests includes the steps to create/load all the required benchmark objects needed to execute the standard benchmark workloads. These synthetic workloads can be used when no real-world data is available to import for database load testing.

This test type also allows you to create a workload using the Benchmark Factory Replication test.

For more information, including a description of each of the standard benchmark tests included in Benchmark Factory, see Overview of Benchmark Testing.

To create an Industry Standard Benchmark test

1. Select File | New to open the New Job Wizard.
2. Select a connection. Then click Workload in the left pane.
3. In the Workload page, select Industry Standard Benchmark Test from the test type drop-down list.

4. Then select the type of benchmark test you want to use. Select each test to display a description in the lower pane.

5. Number of streams to include in the Stream Test. (TPC-H only) Select to include a Stream Test in a TPC-H test. Then specify the number of streams to include.
6. Click to add the test to the workload. The Summary tab opens.

7. Summary tab. The Summary tab provides a summary of the job and the workload, as well as links to the commonly edited options for this test. Click each link to navigate to the applicable tab where you can edit that option. Options shown in red are required. Review the following for additional information:

   Scale	Click to change the scale factor for this test/step. In the Benchmark Scale field, specify a scale factor. See Benchmark Scale Factor for more information.
   Size	Displays the total size of all objects in this Create Objects step. Click to open the Scale tab where you can modify the database size or the scale factor. See Benchmark Scale Factor for more information.
   Number of Tables	(Replication test only) Displays the number of tables to create. Click to modify the number of tables, the number of columns in a table, and the data types to create. See Replication Table Options Tab for more information.
   Transactions	Displays the number of transactions. Click to modify the transaction mix for the transaction step. See Transactions Tab for more information.
   User Load	Displays the user load—the number of virtual users per test iteration. Click to review or modify the user load. See Specify User Load for more information.
   Length	Click to modify the timing for this test. See Timing Tab for more information.

8. Create Objects - More Options. To specify more options for the Create Objects step, select the Create Object for test step in the left pane. Then select the Test Options tab. Review the following for additional information:

   Scale tab	See Benchmark Scale Factor.
   Options tab	Use the Options tab to specify some general options for this step, such as specifying the number of agents to use to load data. See Options Tab (Create Objects Step).
   Custom Creation SQL tab	Use the Custom Creation SQL tab to customize the SQL used to create objects in this step. See Customize SQL for Creating Objects.

9. Transaction Mix - More Options. To specify more options for the Transaction Mix step, select the test Transaction Mix step in the left pane. Then select the Test Options tab. Review the following for additional information:

   Transactions tab	See Transactions Tab.
   User Load tab	See Specify User Load.
   Timing tab	See Timing Tab.
   Advanced tab	See Advanced Tab.

10. To add another test to the workload, click Add Another Test/Step. This takes you back to the selection page of the Workload section.
11. Job Setup. To configure job-level options, select the Job Setup tab. See Job Setup Tab to learn more about job-level options.
12. Agent. To set up the agents for this job, click Agent in the left pane. Select the agents/computers to use for this test. See About Agents for more information.

13. After specifying the test-level and job-level options, you can save the job, run the job, or schedule the job. Review the following:

    • To run the job immediately, click .
    • To schedule the job, click Workload (left pane) and select the Job Setup tab. Then select the Schedule tab. See Schedule a Job.
    • To save the job without running it immediately, click . Use this option after scheduling a job.
    • To save the job to an existing job, select the Job Setup tab. In the Save Job section, select the name of an existing job. Click .