Chat now with support
Chat with Support

Benchmark Factory for Database 8.4 - User Guide

Welcome to Benchmark Factory What's New in Benchmark Factory Additional Resources Overview of Benchmark Factory Benchmark Factory Components Licensing Getting Started - the Benchmark Factory Workflow Agents Create and Edit Connections Create and Edit Tests and Jobs
Jobs View Pane Job Wizards Quickstart: Create a New Job Edit a Job Industry Standard Benchmark Tests Capture and Replay a Workload Artificial Test for Desired Effect Scalability Tests Custom Tests Create/Delete Benchmark Objects Execute External File Test Options for Create Objects Test Options for Transactions Job Setup Options Use Global Search/Replace Save Job as BMF Script Copy Test to Replay, Mix, Goal, or Scalability Test
Benchmarks How Do I... Settings Test Results and Run Reports BFScripts Repository Troubleshooting The Benchmark Factory REST API Appendix About Us Adding Virtual Users System/Upgrade Requirements/Supported Databases Shortcut Keys

Getting Started - the Benchmark Factory Workflow

This topic outlines the Benchmark Factory workflow. Click the links in each step to drill down to more-detailed information.

The Benchmark Factory workflow includes the following steps:

Step 1. Set Up Benchmark Factory

Use the Settings dialog to specify default settings for job and workload options, such as benchmark scale, the number of virtual users per iteration, timing, and latency. Changes to these settings affect only new jobs, not existing jobs.

To specify Benchmark Factory settings

  1. Select Edit | Settings. The Settings dialog opens.
  2. Select a page from the left-side navigation and then specify the corresponding options. See About Settings for more information.

Step 2. Create a Connection

Before you can run a job against a database, the database connection must be defined in Benchmark Factory. You can define a new connection from within the New Job Wizard, or you can define the connection using the New Connection dialog or the My Connections tab prior to creating a new job.

To create and save a new connection

  1. Click in the main toolbar. The New Connection dialog opens.
  2. Select a connection type from the drop-down list. The Connection properties for that connection type display.
  3. Enter the connection information and click Create Connection. For more information about creating connections, see Benchmark Factory Connections.

Step 3. Create a New Job

Use the New Job Wizard to create a new job. The New Job Wizard guides you through the steps required to create a database load test (workload) and specify general options for the job.

When creating a new job in the wizard, you can specify settings that apply to this job and override the defaults in the Settings dialog. See Quickstart: Create a New Job for more information.

To create a new job

  1. Select New from the Benchmark Factory main toolbar. The New Job Wizard opens.
  2. Select the connection for the database-under-test. See Benchmark Factory Connections for more information.
  3. Click Workload in the left pane to begin specifying a workload.
    1. On the Workload page, select the type of test from the drop-down list (for example, Industry Standard Benchmark). Then select a test to include in the workload.
    2. After selecting a test, click the Select Test or Add Test button at the bottom of the wizard page to specify options for that test.
  4. To add another test to the workload, click Add Another Test/Step.
  5. When you finish specifying the workload for the job, click Agent in the left pane to specify an agent.
    • Click the checkbox to the left of the agent name to select it. A checkmark displays for the selected agent.
  6. Click Save/Close to save the job and close the wizard.
  7. The new job is added to the list of jobs in the Jobs View pane.
  8. To rename the job, right-click the job in the Jobs View and select Rename.

Modify a Job

After creating a new job, you can modify the job. For example, you can change the database-under-test, add tests/steps, or change test options.

To modify an existing job

  1. In the Jobs View pane, right-click the job and select Edit Job.
  2. To modify the number of virtual users for a specific test in a job, do the following:
    1. Right-click the test in the Jobs View pane and select Edit Job. The Edit Job wizard opens to the selected test in the workload.
    2. Select the Test Options tab. Then select the User Load tab.
    3. Modify the User Load. Click Save/Close when finished.

Step 4. Run the Job

To run a job

  • In the Jobs View, select the job to run and click in the jobs toolbar, or right-click the job and select Run Job.

Step 5. Analyze Test Results

Benchmark Factory provides a number of metrics to analyze a system-under-test. These include:

  • Response, retrieval, and transactions times
  • Transaction throughput in transactions per second (TPS)
  • Amounts of data transferred in bites per second (BPS)
  • Errors incurred
  • Transaction Distribution
  • Any user defined counters (Windows NT/2000 Performance Monitor) added using the Benchmark Factory Console.

Benchmark Factory provides the following testing results:

 

Agents

About Agents

The Benchmark Factory Agent is a component used in Benchmark Factory to create virtual users which simulate real-world user activity by placing transactions against the database-under-test. The Benchmark Factory Agent is installed when the Benchmark Factory Console is installed. In addition, you can install additional agents on other remote machines. Each Benchmark Factory agent can spawn multiple virtual-user sessions and Benchmark Factory can control hundreds of Agent machines.

After installing additional agents, use your Benchmark Factory Console to define a connection to each agent machine. When you create a new benchmark test through the Console, you can select which of the defined agents to use to generate the user load.

Each virtual user is a separate thread, acting independently of the other virtual users, with its own connection to the system-under-test. Each virtual user tracks its own statistics, including transaction times and the number of times a transaction executes.

View Agent Information

The Benchmark Factory Console allows you to view information about all currently running agents, provides a way to connect to and manage remote agents, and allows you to specify some global agent settings. In addition, the primary Agent, Agent.exe, includes a GUI which displays transaction and virtual user statistics during test execution.

  • Select View | Agent to open the Agents pane where you can view information about agents that are currently running.
  • Select Edit | Settings | Agent to view a list of all the configured agents which are available to be used in testing, as well as platform information about each agent machine. Use this page to view the agent global settings, as well.
  • Double-click Agent.exe in the bin directory to open the Benchmark Factory Agent. The GUI displays virtual user statistics during test execution. Select Options | Settings to configure options for this agent.

Types of Agent Components

Benchmark Factory provides several types of agent components:

  • Agent.exe—(Windows) The primary agent. This agent is installed with the Benchmark Factory client application and can be found in the bin directory. This component includes a GUI.

  • BMFAgent.exe—(Windows or Linux) The non-GUI agent. You can use BMFAgent.exe to integrate Benchmark Factory into your continuous integration or continuous testing process. See BMFAgent.exe for more information.

    • (Windows) The Windows version of this agent is installed with the Benchmark Factory client application and can be found in the bin directory.
    • (Linux) The Linux version of this agent can be installed on a remote Linux machine through the Benchmark Factory Console. See Install Remote Agent on Linux for more information.

Using Agents

Review the following topics to learn how to use the Benchmark Factory Agent.

 

Set Up New User Agent

You must define a connection to each agent machine before you can use that agent in a Benchmark Factory test. The local Benchmark Factory Agent is defined by default. After installing a remote agent however, use this procedure to define a connection to the remote agent machine from the console.

For information about installing remote agents, see Install Remote Agent on Windows and Install Remote Agent on Linux

To setup a new user agent

  1. Select Edit | Settings | Agent.
  2. Click Setup New User Agent.
  3. Select which type of agent to set up, Windows or Linux, based on the platform on which the agent is installed.

  4. In the Setup Windows/Linux Agent dialog, enter the name or IP address of the computer where the agent is installed.

    Tip: If you are using this dialog to install an agent on a Windows or Linux platform, enter the name or IP address of the computer on which you want to install an agent. Then see Install Remote Agent on Windows or Install Remote Agent on Linux for more instructions.

  5. Enter the user name and password.
  6. Specify the number of agents.
  7. Click OK.
  8. Benchmark Factory connects to the computer and then checks if the agent is installed.
  9. If agent installation is verified, the agent is added to the list. Select the agent to display platform information.
  10. To learn how to use multiple agents when running a test in Benchmark Factory, see Running Benchmark Factory with Multiple Agents.

 

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating