Welcome to Benchmark Factory
What's New in Benchmark Factory
Additional Resources
Overview of Benchmark Factory
About Benchmark Factory
Benchmark Factory Components
Understanding Benchmark Testing
Understanding Benchmark Factory Terminology
How Benchmark Factory Works
Feature/Edition Comparison
Benchmark Factory Components
Licensing
Getting Started - the Benchmark Factory Workflow
Agents
About Agents
Set Up New User Agent
Specify Agent Settings
Install Remote Agent on Windows
Install Remote Agent on Linux
Run Benchmark Factory with Multiple Agents
Agents View
Use the Agent Component Console
BMFAgent.exe
Specify Max Virtual Users
Configure Firewall - Remote Agent Install
Create and Edit Connections
Connections Overview
Create IBM DB2 CLI Connections
Create MySQL Connections
Create ODBC Connections
Create Oracle Connections
Create PostgreSQL Connections
Create SAP Connections
Create SQL Server Connections
Edit and Manage Connections
Timeout and Reconnect Options
Environment Information
Performance Counters
Oracle Statistics Collection
Oracle Clustering Options
Configure RAC Load Testing
Create and Edit Tests and Jobs
Jobs View Pane
Job Wizards
Quickstart: Create a New Job
Edit a Job
Industry Standard Benchmark Tests
Capture and Replay a Workload
Benchmarks
About Capture/Replay Tests
Capture and Replay Oracle Workload
Capture and Replay SQL Server Workload
Run Capture Scenario Wizard
Run Capture Wizard from Cmd Line
Replay Load from an Oracle Trace File
Replay Load from SQL Server Trace Table
Replay Load from ODBC Trace File
Import from Delimited Text File
Import from Oracle Dynamic Performance View
Import from Dell Tools Export File
Capture Oracle Workload - Privileges
Artificial Test for Desired Effect
Scalability Tests
Custom Tests
Create/Delete Benchmark Objects
Execute External File
Test Options for Create Objects
Benchmark Scale Factor
Options (Create Objects)
Create Objects Using Custom SQL
Table Options (Create Objects for Replication)
Test Options for Transactions
Transactions
User Load
Timing
Advanced
Options (Capture/Replay)
Add SQL Transactions
Add User Scenario from XML
Latency
Use BFScript Wizard
Run SQL Preview
Stagger User Startup
Job Setup Options
Use Global Search/Replace
Save Job as BMF Script
Copy Test to Replay, Mix, Goal, or Scalability Test
Overview
AS3AP Benchmark
Scalable Hardware Benchmark
TPC-B Benchmark
TPC-C Benchmark
TPC-D Benchmark
TPC-E Benchmark
TPC-H benchmark
Replication Benchmark
How Do I...
Import Benchmark Factory Self Signed Certificate
Local Access to REST API
Allow external host access REST API
Create a Connection
Use Run Reports
Create/Delete Benchmark Objects for Load Testing
Change Scaling Factors
Specify the User Load
Execute a File
Use Benchmark Factory Scripting Capabilities
Copy a Test to a Replay, Mix, or Scalability Test
Run SQL Preview
Replace Child Latencies
Use Global Search and Replace
Create an ODBC Trace File
Schedule a Job
Set Latency
Set the Number of Users on an Agent
Stagger User Startup
Run Benchmark Factory with Multiple Agents
SQL Scalability Testing from Toad
Test Database Scalability
Settings
About
General Settings
Benchmarks Settings - General
Benchmark Settings - Specific
TPC-B Benchmark Settings
Replication Benchmark Settings
Timing Settings
User Load Settings
Latency Settings
Error Handling Settings
Repository Settings
Statistics Settings
Agent Settings
Oracle Settings
SQL Server Settings
REST API Settings
Execute File Settings
Test Results and Run Reports
Overview
View Real-Time Test Results
BFScripts
View Test Results - Running Job
Realtime Statistics Tab
Run vs Userload Tab
Transaction vs Load Tab
Changing Graph View
View Test Results - Completed Job
Benchmark Factory Run Reports
Overview
Changing Graph View
Run Reports Viewer
Using Benchmark Factory Run Reports
Viewing Multiple Run Results
Exporting Test Results to Excel
Exporting Test Results to Excel Using the Command Line
Benchmark Factory Test Results Terminology
About Scripts
BFScript Wizard
Using Scripts in Username and Password Fields
Dates and Numbers
File Access
Global Variable
Name and Address
Repository
Troubleshooting
$BFAddress
$BFAddress2
$BFCity
$BFCompany
$BFCountry
$BFEmail
$BFFirstName
$BFFullName
$BFLastName
$BFMiddleInitial
$BFPhone
$BFState
$BFZipCode
Numerical Manipulation
Random Numbers
Random String
String Manipulation
$BFAsc
$BFChr
$BFConcat
$BFLeft
$BFLen
$BFLower
$BFMid
$BFRight
$BFTrim
$BFTrimLeft
$BFTrimRight
$BFUpper
Test Info
Support Bundle
Agent Connection
Standard Benchmark Tests
Use Benchmark Factory with SQL Server 2005
MySQL Initialization Settings
The Benchmark Factory REST API
Appendix
Trace Files or Fine Grained Access Control: Pros and Cons
Change Graph Views
Customizing List Controls
Oracle Instant Client Installation
Creating a SQL 2005 Trace Table Using the SQL Server Profiler
Creating a SQL 2008_2008 R2 Trace Table Using the SQL Server Profiler
Migrating Data Using the Command Line Wizard
Store Procedure Examples
Oracle Trace File Activation
Create an ODBC Trace File
Configure Firewall for Remote Agents
About Us
Adding Virtual Users
System/Upgrade Requirements/Supported Databases
Shortcut Keys
How Benchmark Factory Works
The following steps provide an overview of how Benchmark Factory components interact during the load testing process.
- The Benchmark Factory Console implements the workload testing process and controls one or more distributed agent machines. Each agent machine can simulate thousands of users. Each simulated user executes transactions and records statistics.
- Benchmark Factory About Agents machines simulate virtual users. The Agents send transactions to the system-under-test. The Agents record statistics that includes how much data the resulting transaction contained and how long it took to get the results. At the end of an iteration, each agent machine reports its findings back to Benchmark Factory.
- A server (system-under-test) is the database Benchmark Factory connects to. Benchmark Factory is server neutral and network protocol independent. Benchmark Factory uses vendor client libraries of the system-under-test. Any system that the client software can support, Benchmark Factory can support. Its only requirement is that Agent machines must be able to connect to a server through an appropriate manner and it supports the test system.
- The Repository Manager stores all testing data.
- Results are viewed from Run Reports.
Feature/Edition Comparison
Review the following table for a comparison of features available in the commercial and freeware editions of Benchmark Factory.
| Feature | Freeware Edition | Commercial Edition |
|---|---|---|
| Default maximum virtual user load (without additional virtual users packs) | 20 | 100 |
| Additional virtual user packs |  |
 (unlimited) |
| SQL Optimizer Integration | |
|
| Oracle Capture and Replay | |
|
| SQL Server Capture and Replay | |
|
| Maximum number of transactions that can be run during a load test/user scenario | 20 | Unlimited |
| Scaling factors for standard benchmarks (AS3AP, TPC-B, TPC-C, TPC-D, and Scalable Hardware) | Lowest two scaling factors for each benchmark | All are available |
| Repository database types | Only the default SQLite repository is enabled | The following repository database types are supported: Oracle, Microsoft SQL Server, SAP SQL Anywhere (Adaptive Server Anywhere), MySQL, and SQLite |
| BMFServer.exe | |
|
| Support |
Technical support from Quest is not available for the Freeware edition. |
For the Commercial edition, support from Quest is available at: https://support.quest.com |
Get the latest product information and find helpful resources at the Benchmark Factory community at: https://www.toadworld.com/products/benchmark-factory-for-databases
Benchmark Factory Components
Benchmark Factory Console
The Benchmark Factory console implements the database workload testing process. This interface is where:
- Jobs are created and executed.
- Users can access test results. See Benchmark Factory Test Results for more information.
- Agents are managed. See About Agents for more information.
The Benchmark Factory Console contains the following views:
