Benchmark Factory for Database 9.0 - User Guide

Use Run Reports

Benchmark Factory Run Reports is a separate executable that provides a comprehensive and detailed collection of database load testing results. With Benchmark Factory you can drill down into a database to view a wide array of information and statistics that gives you accurate insight into database performance. Run Reports Viewer allows you to access Benchmark Factory load testing results. See Run Reports Viewer for more information.

Note: Three instances of Run Reports can be viewed at one time.

Cutting and Pasting of Run Reports

From Benchmark Factory run reports you can cut and paste test results into the Clipboard.

Testing Server Throughput

Server throughput is the measurement of database capacity. Benchmark Factory allows you to view a variety of server throughput statistics that include:

• Transaction/Second (TPS)
• Transaction Time
• Bytes/Second (BPS)
• Rows/Second (RPS)
• Total Bytes
• Total Errors
• Total Rows
• Response Time

Testing SQL Scalability

The Benchmark Factory transaction comparison graph provides users with a comparison of the individual transaction performance as they compare to each other. This is most useful when analyzing a SQL Scalability test.

Testing Real Application Clusters

Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determine system capacity. The cluster performance graph allows users to view the performance of individual nodes.

Create/Delete Benchmark Objects for Load Testing

The Create/Delete Benchmark Objects step creates or clears the database objects (tables/indexes, etc.) needed to run industry standard benchmarks. The step is included when you create an Industry Standard Benchmark Test. You can also add the step to an existing workload through the Edit Job Wizard.

Note: To create, delete, and modify database tables/indexes you must have the user login rights required.

To add a Create/Delete Benchmark Objects Test to a Workload

1. Use one of the following methods to open the Job Wizard:
   • Select File | New to open the New Job Wizard.
   • Select an existing job and click Edit Job.
2. Click Workload in the left pane. If this is an existing job, click Add Another Test/Step
3. On the Workload page, select Create/Delete Benchmark Objects Test from the drop-down list.
4. Then select the benchmark type from the Benchmark Type drop-down list.
5. Select one of the following options:
   • Create Benchmark Objects—Creates and initializes benchmark objects (tables, indexes, etc.).
   • Delete Benchmark Objects—Deletes existing benchmark objects (tables, indexes, etc.).
6. If you selected Create Benchmark Objects, select one of the following options:

   Create/Load objects if objects don't exist (no backup sets created)	If objects do not exist, Benchmark Factory creates the objects and loads data. If you select this option, Benchmark Factory does not create backup tables.
   Delete benchmark objects after completion of job	Benchmark Factory deletes objects after a job is finished.
   Refresh data using backup sets	When backup sets exist, Benchmark Factory reloads data and recreates indexes. If backup sets do not exist, Benchmark Factory creates objects and loads data. Then Benchmark Factory creates the backup tables.
   Recreate objects and reload data every execution	Benchmark Factory always deletes the existing objects and then recreates/reloads for each job execution. This allows you to ensure that the dataset is always in initialized status.
   Refresh data using inserts	If data does not exist, Benchmark Factory uses bulk insert to load data. If the following Benchmark Setting is selected, and the Benchmark Scale factor of the existing tables does not match the new requirement, the process of inserting additional data or deleting data my run long, depending on the size of the existing data and number of indexes. Note: If database does not support bulk insert, insert is used.

7. Click the Import Test button at the bottom of the page.
8. Summary. The test is added to the workload and the Summary page opens. Review the following editable options for this test/step.
   • Scale—Click the Scale link to modify the scale factor. The Scale page opens. In the Benchmark Scale field, select a new scale factor. See Benchmark Scale Factor for more information.

   Note: The test is added to the top of the workload list. To move the test down (or up) in the list, right-click the test and select Move Down (or Move Up).

9. Test Options. To modify test options for the Create Objects test/step, select the Test Options tab. Review the following for more information:
   • Scale—To modify the scale factor. See Benchmark Scale Factor for more information.
   • Options—To modify options for object creation and retention. See Options Tab (Create Objects Step) for more information.
   • Custom Creation SQL—To use custom SQL to create objects. See Customize SQL for Creating Objects for more information.

Tip: You can specify a default setting for Object/Data Retention in Edit | Settings | Benchmarks.

Change Scaling Factors

When you create an Industry Standard Benchmark test or add a Create Benchmark Objects step, Benchmark Factory allows you to adjust the Benchmark scale factor. The scale factor increases the size of a database during the testing process allowing realistic testing.

By increasing the scale factor, the number of rows added to the tables increases, allowing for the use of larger user loads. This places a greater stress on the system-under-test. During load testing, you can run various tests with different Benchmark Scale factors to ensure your database performs at varying user-load levels.

You can adjust the Benchmark Scale factor when creating a new Industry Standard Benchmark Test or when adding a Create Benchmark Objects step.

To change the Benchmark Scale factor

1. In the New Job Wizard or Edit Job Wizard, select the Create Objects step (in the Workload) for which you want to adjust the scale.
2. Select the Test Options tab and then the Scale tab.

3. In the Benchmark Scale field, specify a scale factor.

   Valid benchmark scales are dependent upon the benchmark selected. For example, the AS3AP benchmark scales by a factor of 10 with valid scales of "1, 10, 20, 30, etc." The TPC-C benchmark scales by a factory of 1, with valid scales of "1, 2, 3, 4, etc." Refer to the specific benchmark standard for more information.

   (Replication test only) For a Replication test, each increment in scale factor adds 1000 rows to a table.

4. (Optional) Instead of specifying a scale factor, you can specify a Benchmark Size. The Benchmark Scale factor will re-adjust accordingly.
5. Finish specifying options for your test, or click Save/Close to save your modifications and close the wizard.

Benchmark Scale Factor and User Load Relationship

The TPC-C and the Scalable Hardware standard benchmark tests have a scale factor / user load relationship.

• For the TPC-C benchmark, the ratio is 10 virtual users per scale factor.
• For the Scalable Hardware benchmark, the scale factor is also the number of users that will return data from their queries.

If you attempt to create a benchmark test using a scale factor / user load ratio outside of these specifications, Benchmark Factory warns you.

Note: You can set a default Benchmark Scale factor to use for each standard benchmark test. Go to Edit | Settings | Benchmarks | benchmark type.

Related Topics

Benchmark Settings - Specific Test Types

Specify the User Load

The User Load tab in the Job Wizard allows you to specify the number of virtual users to use for each iteration of a test.

Note: You can set default values for User Load in Edit | Settings | User Load. If you modify the default settings, the changes will apply to new jobs only, not to existing jobs.

To specify virtual user load

1. In the Job Wizard, select the Workload Test for which you want to specify the virtual user load. See The Job Wizards.
2. To open the User Load tab, do one of the following:
   • On the Summary tab, click the User Load link.

   • Select the Test Options tab and then the User Load tab.

   

3. Add user loads individually, or specify a range to allow Benchmark Factory to calculate the user load list automatically. Review the following for more information.

   Option	Description
   Add Single User Load	Specify a single user load, then click Add to add it to the Selected User Loads list. Repeat until your list is complete.
   Add a Range of User Loads	Specify a range, then click Add to add the range to the Selected User Loads list.