Chat now with support
Chat with Support

Benchmark Factory for Database 9.0 - User Guide

Welcome to Benchmark Factory What's New in Benchmark Factory Additional Resources Overview of Benchmark Factory Benchmark Factory Components Licensing Getting Started - the Benchmark Factory Workflow Agents Create and Edit Connections Create and Edit Tests and Jobs
Jobs View Pane Job Wizards Quickstart: Create a New Job Edit a Job Industry Standard Benchmark Tests Capture and Replay a Workload Artificial Test for Desired Effect Scalability Tests Custom Tests Create/Delete Benchmark Objects Execute External File Test Options for Create Objects Test Options for Transactions Job Setup Options Use Global Search/Replace Save Job as BMF Script Copy Test to Replay, Mix, Goal, or Scalability Test
Benchmarks How Do I... Settings Test Results and Run Reports BFScripts Repository Troubleshooting The Benchmark Factory REST API Appendix About Us Adding Virtual Users System/Upgrade Requirements/Supported Databases Shortcut Keys

Scalable Hardware Benchmark

The Scalable Hardware benchmark measures relational database systems. This benchmark is a subset of the AS3AP benchmark and tests the following:

  • CPU
  • Disk
  • Network
  • Any combination of the above three entities

To learn how to create a Scalable Hardware test in Benchmark Factory, see Create Industry Standard Benchmark Test.

How the Scalable Hardware Benchmark Works

The scale factor determines the amount of information initially loaded into the benchmark tables.  For the Scalable Hardware benchmark, each scale factor represents one user accessing the system.  Two tables are created in the database, and they are loaded with a varying number of rows.  

For each virtual user, a separate set of data must be created.  Therefore the scale factor used when loading the database should be the size of the maximum user load.  For example, with user loads of 1, 5, and 10, a scale factor of 10 should be used.

 Scaling Factor

The Scalable Hardware benchmark has a scaling factor of one.

Best Practices

Load-testing against production servers

Do not load-test against a production server if possible.  Load-testing and benchmarking on a production server significantly degrades performance.  In some cases, load-testing can cause a server to fail.  However, if testing against a production server, take the following precautions:

  • Perform the testing when no other users are on the system and no automated processes are running.  Users and automated processes can adversely affect testing results
  • Have a recovery plan and backup all data prior to testing
  • Determine how long it will take to restore a production server if it went down during load-testing
  • Perform manual testing.  Manual testing ensures that no unexpected outside activity takes place during the testing process

Reinitialize the Database

To reinitialize a testing database, run a job containing a Benchmark Object node.  

There are two ways to create a Benchmark Object node:

  • Run the Benchmark Object Wizard to add the Delete Benchmark objects for the Scalable Hardware node to a job as needed.  
  • Run the Load Scenario Wizard to create a new script containing the Create Objects for the Benchmark, Associated Load Scenarios for the Benchmark, and Delete Objects for the Benchmark. Running the delete objects for 'Scalable Hardware" job will clean the environment.

 

TPC-B Benchmark

To learn how to create a TPC-B benchmark test in Benchmark Factory, see Create Industry Standard Benchmark Test.

Overview

The Transaction Processing Council is an organization that establishes transaction processing and database benchmark standards. Find a complete overview and detailed explanation of the TPC-B  Benchmark, at: http://www.tpc.org/tpcb/default.asp

Certification of Transaction Processing Council (TPC) Testing Results

Transaction Processing Council  testing results cannot be published as certified unless the testing procedure is audited and approved by the TPC organization. If not certified, the testing results can be published as a "TPC-B like" test.

Best Practices

The following provides best practices for the TPC-B Benchmark.

Load-testing against production servers

Do not load-test against a production server if possible.  Load-testing and benchmarking on a production server significantly degrades performance.  In some cases, load-testing can cause a server to fail.  However, if testing against a production server, take the following precautions:

  • Perform the testing when no other users are on the system and no automated processes are running.  Users and automated processes can adversely affect testing results
  • Have a recovery plan and backup all data prior to testing
  • Determine how long it will take to restore a production server if it went down during load-testing
  • Perform manual testing.  Manual testing ensures that no unexpected outside activity takes place during the testing process

Reinitialize the Database

To reinitialize a testing database, run a job containing a Benchmark Object node.  

There are two ways to create a Benchmark Object node:

  • Run the Benchmark Object Wizard to add the Delete Benchmark objects for TPC-B node to a job as needed.  
  • Run the Load Scenario Wizard to create a new script containing the Create Objects for the Benchmark, Associated Load Scenarios for the Benchmark, and Delete Objects for the Benchmark. Running the delete objects for 'TPC-B' job will clean the environment.

History Tables

The TPC-B benchmark is made up of only one transaction that updates three tables and inserts a record into a history table.  Inserting one record into one history table limits testing performance. The Benchmark Factory properties page allows the user to set the number of history tables to create during a test.  The best ratio of history tables to virtual users is based on database configuration and hardware.  The number of history tables to use is determined by the tester.

Scaling Factor

The TPC-B benchmark scales by a factor of one.

 

TPC-C Benchmark

To learn how to create a TPC-C benchmark test in Benchmark Factory, see Create Industry Standard Benchmark Test.

Overview

Find a detailed overview of the TPC-C Benchmark at: http://www.tpc.org/tpcc/default.asp.

The TPC-C benchmark is an online transaction processing benchmark that simulates environments that have a number of terminal operators that send transactions to a database. This benchmark is focused on the concept of an order-entry type environment with transaction that include orders, payment recording, order status, and stock level monitoring. This benchmark portrays the activities of a wholesale supplier. However, the TPC-C is not limited to one particular business segment. It can represent numerous categories of a business that sell or distribute products and services.  

The TPC-C benchmark simulates a wholesale parts dealer operating out of warehouses. This Benchmark scales as a company, in theory, expands their business or number of facilities. As the TPC-C benchmark scales, so do the number components for the benchmark, for example, the sales districts and customers.

TPC-C Tables

The scale factor determines the amount of information initially loaded into the benchmark tables. For the TPC-C benchmark, each scale factor represents one warehouse as per TPC-C specification. The TPC-C benchmark involves a mix of five concurrent transactions of different types and complexity. The database is comprised of nine tables with a wide range of records.

A maximum of 10 users should be run against each warehouse. For example, user loads of 1, 5, and 10, set the scale to 1. If using other user load values, change the scale factor accordingly.

The TPC-C database consists of the following tables:

 

 

 

 

 

 

  • Warehouse
  • District
  • Customer
  • History
  • New_Order
  • Order
  • Order_Line
  • Item
  • Stock

 

 

Certification of Transaction Processing Council (TPC) Testing Results

Transaction Processing Council  testing results cannot be published as certified unless the testing procedure is audited and approved by the TPC organization. If not certified, the testing results can be published as a "TPC-C like" test.

Best Practices

The following provides best practices for the TPC-C Benchmark.

Load-testing against production servers

Do not load-test against a production server if possible.  Load-testing and benchmarking on a production server significantly degrades performance.  In some cases, load-testing can cause a server to fail.  However, if testing against a production server, take the following precautions:

  • Perform the testing when no other users are on the system and no automated processes are running.  Users and automated processes can adversely affect testing results
  • Have a recovery plan and backup all data prior to testing
  • Determine how long it will take to restore a production server if it went down during load-testing
  • Perform manual testing. Manual testing ensures that no unexpected outside activity takes place during the testing process

Reinitialize the Database

To reinitialize a testing database, run a job containing a Benchmark Object node.  

There are two ways to create a Benchmark Object node:

  • Run the Benchmark Object Wizard to add the Delete Benchmark objects for TPC-C node to a job as needed.  
  • Run the Load Scenario Wizard to create a new script containing the Create Objects for the Benchmark, Associated Load Scenarios for the Benchmark, and Delete Objects for the Benchmark. Running the delete objects for 'TPC-C' job will clean the environment.

 

TPC-D Benchmark

To learn how to create a TPC-D benchmark test in Benchmark Factory, see Create Industry Standard Benchmark Test.

Overview

The Transaction Processing Council is an organization that establishes transaction processing and database benchmark standards. Find a complete overview and detailed explanation of the TPC-D Benchmark at: http://www.tpc.org/tpcd/default.asp

Certification of Transaction Processing Council (TPC) Testing Results

Transaction Processing Council  testing results cannot be published as certified unless the testing procedure is audited and approved by the TPC organization. If not certified, the testing results can be published as a "TPC-D like" test.

Best Practices

The following provides best practices for the TPC-D Benchmark.

Load-testing against production servers

Do not load-test against a production server if possible.  Load-testing and benchmarking on a production server significantly degrades performance.  In some cases, load-testing can cause a server to fail.  However, if testing against a production server, take the following precautions:

  • Perform the testing when no other users are on the system and no automated processes are running.  Users and automated processes can adversely affect testing results
  • Have a recovery plan and backup all data prior to testing
  • Determine how long it will take to restore a production server if it went down during load-testing
  • Perform manual testing.  Manual testing ensures that no unexpected outside activity takes place during the testing process

Reinitialize the Database

To reinitialize a testing database, run a job containing a Benchmark Object node.  

There are two ways to create a Benchmark Object node:

  • Run the Benchmark Object Wizard to add the Delete Benchmark objects for TPC-D node to a job as needed.  
  • Run the Load Scenario Wizard to create a new script containing the Create Objects for the Benchmark, Associated Load Scenarios for the Benchmark, and Delete Objects for the Benchmark. Running the delete objects for 'TPC-D' job will clean the environment.

Scaling Factor

The TPC-D benchmark scales by the following factors:

  • 0.10
  • 1.00
  • 10.00
  • 30.00
  • 100.00
  • 300.00

 

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating