Chat now with support
Chat with Support

Benchmark Factory for Database 9.0 - User Guide

Welcome to Benchmark Factory What's New in Benchmark Factory Additional Resources Overview of Benchmark Factory Benchmark Factory Components Licensing Getting Started - the Benchmark Factory Workflow Agents Create and Edit Connections Create and Edit Tests and Jobs
Jobs View Pane Job Wizards Quickstart: Create a New Job Edit a Job Industry Standard Benchmark Tests Capture and Replay a Workload Artificial Test for Desired Effect Scalability Tests Custom Tests Create/Delete Benchmark Objects Execute External File Test Options for Create Objects Test Options for Transactions Job Setup Options Use Global Search/Replace Save Job as BMF Script Copy Test to Replay, Mix, Goal, or Scalability Test
Benchmarks How Do I... Settings Test Results and Run Reports BFScripts Repository Troubleshooting The Benchmark Factory REST API Appendix About Us Adding Virtual Users System/Upgrade Requirements/Supported Databases Shortcut Keys

Agents

The Benchmark Factory Agent is a component used in Benchmark Factory to create virtual users which simulate real-world user activity by placing transactions against the database-under-test. The Benchmark Factory Agent is installed when the Benchmark Factory Console is installed. In addition, you can install additional agents on other remote machines. Each Benchmark Factory agent can spawn multiple virtual-user sessions and Benchmark Factory can control hundreds of Agent machines.

After installing additional agents, use your Benchmark Factory Console to define a connection to each agent machine. When you create a new benchmark test through the Console, you can select which of the defined agents to use to generate the user load.

Each virtual user is a separate thread, acting independently of the other virtual users, with its own connection to the system-under-test. Each virtual user tracks its own statistics, including transaction times and the number of times a transaction executes.

View Agent Information

The Benchmark Factory Console allows you to view information about all currently running agents, provides a way to connect to and manage remote agents, and allows you to specify some global agent settings. In addition, the primary Agent, Agent.exe, includes a GUI which displays transaction and virtual user statistics during test execution.

  • Select View | Agent to open the Agents pane where you can view information about agents that are currently running.
  • Select Edit | Settings | Agent to view a list of all the configured agents which are available to be used in testing, as well as platform information about each agent machine. Use this page to view the agent global settings, as well.
  • Double-click Agent.exe in the bin directory to open the Benchmark Factory Agent. The GUI displays virtual user statistics during test execution. Select Options | Settings to configure options for this agent.

Types of Agent Components

Benchmark Factory provides several types of agent components:

  • Agent.exe—(Windows) The primary agent. This agent is installed with the Benchmark Factory client application and can be found in the bin directory. This component includes a GUI.

  • BMFAgent.exe—(Windows or Linux) The non-GUI agent. You can use BMFAgent.exe to integrate Benchmark Factory into your continuous integration or continuous testing process. See BMFAgent.exe for more information.

    • (Windows) The Windows version of this agent is installed with the Benchmark Factory client application and can be found in the bin directory.
    • (Linux) The Linux version of this agent can be installed on a remote Linux machine through the Benchmark Factory Console. See Install Remote Agent on Linux for more information.

Using Agents

Review the following topics to learn how to use the Benchmark Factory Agent.

 

Run Reports

Benchmark Factory Run Reports is a separate executable that provides a comprehensive and detailed collection of database load testing results. With Benchmark Factory you can drill down into a database to view a wide array of information and statistics that gives you accurate insight into database performance. Run Reports Viewer allows you to access Benchmark Factory load testing results. See Run Reports Viewer for more information.

Note: Three instances of Run Reports can be viewed at one time.

Cutting and Pasting of Run Reports

From Benchmark Factory run reports you can cut and paste test results into the Clipboard.

Testing Server Throughput

Server throughput is the measurement of database capacity. Benchmark Factory allows you to view a variety of server throughput statistics that include:

  • Transaction/Second (TPS)
  • Transaction Time
  • Bytes/Second (BPS)
  • Rows/Second (RPS)
  • Total Bytes
  • Total Errors
  • Total Rows
  • Response Time

     

Testing SQL Scalability

The Benchmark Factory transaction comparison graph provides users with a comparison of the individual transaction performance as they compare to each other. This is most useful when analyzing a SQL Scalability test.

                

Testing Real Application Clusters

Benchmark Factory allows you to load test Oracle Real Application Clusters (RAC) to determine system capacity. The cluster performance graph allows users to view the performance of individual nodes.

                  

 

Repository Manager

Note: If you create a new Benchmark Factory 5.5 (or later) repository, earlier versions of Benchmark Factory will not work against this repository.

The Repository is a database where all of the test results are stored. Benchmark Factory inserts test results into the repository and provides an easy way to access the data. By default, the Repository is a SQLite database that resides on the same machine as Benchmark Factory. The Repository can reside on another database server if required.

Note: By default in Benchmark Factory 7.1.1 or earlier, a MySQL database is created and used as the Repository, unless you selected the SQLite option during installation. In Benchmark Factory 7.2 or later, by default a SQLite database is created and used as the Repository.

To change the database, select the Data Source Name of the ODBC connection for the new database. To migrate data from one database to another, click Data Migration to open the Data Migration Wizard.

Note: If the database structure does not exist on the selected database, a prompt to create the structure will appear when OK is clicked.

The supported database types are:

  • MySQL
  • SQL Server®
  • SQLite
  • SAP SQL Anywhere® (Adaptive Server® Anywhere)
  • Oracle®

To open the Repository Manager

  1. To open the Repository Manager from the application, select Tools | Repository Manager. Benchmark Factory prompts you to close the application. This is necessary before opening the Repository Manager.

  2. The Repository Manager opens. Modify connection parameters or perform maintenance functions.

    Review the following for additional information:

    Connection Parameters  

    Data Source Name

    Data Source name of the ODBC connection used to connect to the repository database.

    ODBC Driver

    Current ODBC driver

    User Name

    The User Name used to log into the selected database.

    Password

    The Password associated with the user name used to log into the database.  

    Edit DSN

    Displays the ODBC connection information dialog for the selected data source.

    ODBC Administrator

    Displays the ODBC Data Source Administrator dialog. Use this to add and edit ODBC connections.

    Test Connection

    Tests the connection of the currently selected ODBC Data Source.

    Maintenance  

    Create

    Creates the repository objects on the selected database.

    Delete

    Deletes the repository objects on the selected database.

    Warning: This will delete all test results stored in the Repository.

    Data Migration

    Launches the Data Migration Wizard. See Data Repository Migration Wizard for more information.

 

 

Jobs View

The Jobs View pane displays the list of jobs. After you create and save a job, the job is displayed in the Jobs View pane. You can also use the Jobs View pane to identify the jobs that are currently running and the jobs that are scheduled to run.

Use the Jobs View pane to do the following:

  • Run a job
  • View job status
  • View real-time test results
  • Edit a job

Run and Edit Jobs

To run a job

  • Select the job in the Jobs View pane and click .

To edit an existing job

  • To edit an existing job, select the job in the Jobs View pane and click . The Edit Job Wizard opens. To learn more about the Job Wizard, see The Job Wizards.

Test Results

To view test results

  • To view a job's test results, select a job in the Jobs View pane. Test results display in the right pane. See Benchmark Factory Console for an overview of the Benchmark Factory console.
  • To compare two or more run results for a test, select the Compare Results tab. Use Ctrl+click to select multiple test runs. A comparison of the results for the various runs displays.

Job Status

From the Jobs View pane, you can view job status.

The following job states are identified:

  • Scheduled: All jobs currently waiting to run or scheduled to run at a future time.
  • Running: Job currently running.
  • Completed: All completed jobs.

Jobs View Toolbar

The Jobs View toolbar provides user functionality to save or delete jobs.

Move the selected test up in the list.

Move the selected test down in the list.

Delete the selected item from the Jobs View.
Toggles tree-view on and off.

Note: To save a job as a Benchmark Factory script, select the job and click Save in the Benchmark Factory toolbar or select File | Save.

 

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating