Benchmark Factory for Database 9.0 - User Guide

Copy Test to Replay, Mix, Goal, or Scalability Test

A test can be quickly converted from one test type to another by using the "copy" command.  A workload test is an assembled test comprised of user scenarios and/or transactions. These tests can be run with multiple virtual users. A workload test can be one of three types: mix test, replay test, goal test, or scalability test.

In Benchmark Factory you can copy the following tests:

• Replay test—A Replay Test runs multiple transactions with each one running independently on a specified number of users. The test will run until the defined number of executions for each transaction or a specified time limit is reached.
• Mixed test—A Mixed Workload test runs for a specified time at each predetermined user loads level. Each user will run a transaction mix based upon the weights defined on the transactions tab.  For example, if a test has two transactions, A and B, with A having a weight of one and B having a weight of four, on average B will run four times for every time A is run once. The run order will be randomly generated for each user so they are not all running the same transaction simultaneously. That run order is used for that user each time the test is performed to ensure reproducible results.
• Goal test—A goal test is used to find maximum throughput or response time goals. A transaction mix is executed at user load levels, determined by setting a beginning, ending, and interval value. When run, the specified goal criterion is evaluated at the end of each iteration and the test ends once the goal or maximum user load has been reached.
• Scalability—A SQL Scalability test executes each transaction individually for each userload and timing period.  For example,  a test has two transactions, A and B, and two userloads of 10 and 20, with an iteration length of one minute. Transaction A would execute continually for one minute at userload 10, then B would do the same. Next A will run at userload 20, followed again by test B, for a total time of 4 minutes.

Copying a workload test creates a new test containing all compatible settings, user scenarios, and transactions from the original test.

To copy a workload test to a Replay Test, Mix Test, Goal Test, or Scalability test

1. Right-click a test in the Jobs View pane or in the New/Edit Job Wizard. A drop-down displays.
2. Select the desired type of test you want to copy to.
3. The test is created and displays in the Jobs View or New/Edit Job Wizard.

Benchmarks

Overview

A benchmark is a performance test of hardware or software. A benchmark simulates real-world application workloads. These models of real applications can then be run against the system being evaluated. Models work better than the actual application and offer reproducible results. The real application, on the other hand, has too many variables that can change the results over several test sessions.

Realistic Expectations When Using Benchmarks

Industry standard benchmarks represent real application workloads, their run results depend on workload definition (user load, latency, etc.), as well as server configuration and tuning parameters.  During the running of a benchmark, deadlock errors can occur. This is not an issue with Benchmark Factory, but more of an issue with the tuning of the workload as well as the system-under-test. For example, it would be unrealistic to expect no errors from a workload that runs 1000 users updating a single row on a small table.  To further troubleshoot any tuning issues, we recommend the Quest Spotlight® products. Learn more at: http://www.quest.com/

What Benchmarks Measure

Benchmarks measure performance that analyze:

• Raw performance of a complete system
• Raw performance of a specific subsystem (disk, video, CPU, memory, etc.)
• Performance of a computer running a particular application
• Performance of applications running on a network
• Capacity of a system. This is often refereed to as capacity planning.

Provided Benchmarks

Benchmark Factory provides the following benchmark tests:

AS3AP Benchmark

The AS3AP benchmark is an American National Standards Institute (ANSI) Structured Query Language (SQL) relational database benchmark. The AS3AP benchmark provides the following features:

• Tests database processing power
• Built-in scalability and portability that tests a broad range of database systems
• Minimizes effort in implementing and running benchmark tests
• Provides a uniform metric and straight-forward interpretation of benchmark results

Systems tested with the AS3AP benchmark must support common data types and provide a complete relational interface with basic integrity, consistency, and recovery mechanisms. The AS3AP tests systems ranging from a single-user microcomputer Database Management System (DBMS) to a high-performance parallel or distributed database.

To learn how to create an AS3AP benchmark test in Benchmark Factory, see Create Industry Standard Benchmark Test.

Best Practices

Do not load-test against a production server if possible.  Load-testing and benchmarking on a production server significantly degrades performance.  In some cases, load-testing can cause a server to fail.  However, if testing against a production server, take the following precautions:

• Perform the testing when no other users are on the system and no automated processes are running.  Users and automated processes can adversely affect testing results
• Have a recovery plan and backup all data prior to testing
• Determine how long it will take to restore a production server if it went down during load-testing
• Perform manual testing.  Manual testing ensures that no unexpected outside activity takes place during the testing process

Reinitialize the Database

To reinitialize a testing database, run a job containing a Benchmark Object node.  

There are two ways to create a Benchmark Object node:

• Run the Benchmark Object Wizard to add the Delete Benchmark objects for AS3AP node to a job as needed.  See Add a Create/Delete Benchmark Objects Test for more information.
• Run the Load Scenario Wizard to create a new script containing the Create Objects for the Benchmark, Associated Load Scenarios for the Benchmark, and Delete Objects for the Benchmark. Running the delete objects for 'AS3AP' job will clean the environment.

Scaling Factor

The AS3AP benchmark scales by factor of 10.