Benchmark Factory for Database 8.3 - User Guide

Getting Started

Getting Started With Benchmark Factory

Benchmark Factory® is a database performance testing tool that allows you to conduct database workload replay, industry-standard benchmark testing, and scalability testing.

Review the following topics for a quick overview of Benchmark Factory.

• About Benchmark Factory
• Understanding Benchmark Factory Testing
• Understanding Benchmark Factory Terminology
• How Benchmark Factory Works
• The Benchmark Factory Environment
• Getting Started - the Benchmark Factory Workflow

What's New in Benchmark Factory

Benchmark Factory 8.3 is a minor release and includes the following new features and enhancements. For a list of resolved issues in this release, see the Benchmark Factory 8.3 Release Notes at the Quest Support Portal.

Microsoft SQL Server Support

The Benchmark Factory Agent for Linux can now be used when testing against the following database types (in addition to PostgreSQL, Oracle and MySQL):

• Microsoft SQL Server
• Microsoft SQL Server. If using the Benchmark Factory Agent for Linux to test against a Microsoft SQL Server database, ensure the unixODBC and msodbcsql Client are installed on the same Linux machine as the Agent. Installing the Microsoft SQL Server ODBC Driver, see Installing the Microsoft ODBC Driver for SQL Server on Linux and macOS.

Database Support

This release of Benchmark Factory includes support for the following database versions:

• Oracle 19c

Enhancements to the Benchmark Factory

• Benchmark Factory Agent for Linux. You can now gather CPU and Memory from the Benchmark Factory Agent for Linux. This display in Benchmark Factory Main window Bottom Agent panel.
• REST API. You can now use URL(api/jobs/tests/testindex/currentresult) to get current results of the test results are same as they display in UI.

New in Benchmark Factory 8.2.1

Benchmark Factory 8.2.1 is a minor release and includes resolved issues. For a complete list, see the Benchmark Factory 8.2.1 Release Notes at the Quest Support Portal.

New Features in 8.2

Connections

MySQL Native Connection. Beginning with this release, Benchmark Factory now provides the ability to create a MySQL connection using Native database connectivity.

1. To create a MySQL Native connection, click the New Connection button in the main toolbar. Then select MySQL from the Database Type drop-down list.
2. Select the Native tab in the New Connection dialog and enter the connection information.

Note: The following Benchmark tests are supported: TPC-C, TPC-E, and TPC-H.

Database Support

This release of Benchmark Factory includes support for the following database versions:

• Oracle 18c
• MySQL 8.0

Enhancements to the Benchmark Factory Agent for Linux

In release 8.1, Benchmark Factory introduced an agent which you can deploy in your Linux environment. Benchmark Factory 8.2 includes several significant enhancements to the Agent for Linux.

Automatically Install Remote Agent on Linux

You can now install an agent on a remote Linux machine through the Benchmark Factory Console. This provides an easy, UI-based method for installing a remote agent on a Linux platform.

1. To install an agent on Linux through the console, select Edit | Settings | Agent. Then click Setup New User Agent.
2. Select Linux and click OK. Enter the name or IP address and the connection information for the remote Linux machine on which you want to install the agent. If no agent is found, you are prompted to install one. Click Yes to automatically install an agent for Linux.
3. After the agent is successfully installed, you can select it from the list of available agents.

Installation Package

The Benchmark Factory Agent for Linux is now distributed as an RPM package. This provides a number of advantages over the previous archive file (.tar) format. This format allows BMF to provide automatic installation through the Benchmark Factory Console. You can also use YUM to install the Benchmark Factory Agent RPM package on Linux manually.

To download the RPM package for manual installation, go to https://bintray.com/quest/bmfrepo. For additional information, see "Install Remote Agent on Linux" in the Benchmark Factory Help. For best results, install the Agent for Linux automatically through the Benchmark Factory Console.

Oracle and MySQL Support

The Benchmark Factory Agent for Linux can now be used when testing against the following database types (in addition to PostgreSQL):

• Oracle
• MySQL

Oracle. If using the Benchmark Factory Agent for Linux to test against an Oracle database, ensure an Oracle Client is installed on the same Linux machine as the Agent. Review the following details:

• The Oracle Client and Oracle Instant Client, versions 10g R2 and later, are supported.
• If you use a version of the Oracle Instant Client later than 10g R2, you must create a symbolic link to the shared library for the Oracle Instant Client you intend to use.
• Additional Information—For additional prerequisites and more-detailed information, see "Install Remote Agent on Linux" in the Benchmark Factory Help.

New Features in 8.1

Benchmark Factory 8.1 is a minor release and includes the following new features and enhancements. For a list of resolved issues in this release, see the Benchmark Factory 8.1 Release Notes at the Quest Support Portal.

Benchmark Factory Agent for Linux

This release includes an exciting new feature. Benchmark Factory now provides an agent which you can deploy in your Linux environment. After installing the Linux version of Benchmark Factory Agent in your Linux environment, use the Benchmark Factory Console to set up the new agent. After setup, you can select the new agent from the list of agents available for load testing. In addition, the Benchmark Factory REST API supports configuring and modifying the Benchmark Factory Agent for Linux.

Supported Databases. The Benchmark Factory Agent for Linux can be used to test workloads against PostgreSQL databases.

Note: The Benchmark Factory Agent for Linux supports the following platforms only: Red Hat 7.x (64-bit), CentOs 7.x (64-bit), and Oracle Linux 7.x (64-bit).

The Benchmark Factory 8.1 Agent for Linux is available for download from the Benchmark Factory Support Page.

PostgreSQL Native Connection

This release of Benchmark Factory includes a PostgreSQL Native connection.

1. To create a PostgreSQL connection, click New Connection in the main toolbar.

2. Then select PostgreSQL from the Connection Type drop-down list.

   

3. Enter connection information in this dialog to create a PostgreSQL native connection.

   Alternatively, if you want to create a PostgreSQL ODBC connection, select the ODBC tab and enter connection information.

The following PostgreSQL database versions are supported: PostgreSQL 9.4, 9.5, 9.6 and 10, and EnterpriseDB Postgres 10.

Enhanced Licensing Dialog

This release includes an enhanced Licensing dialog which provides a simpler, easier-to-use interface.

• Select Help | Licensing to open the Licensing dialog. The Licensing dialog opens and displays the list of licenses for Benchmark Factory. Benchmark Factory supports multiple licenses. For example, a Toad for Oracle Edition license and a Toad for SQL Server Edition license may be active simultaneously.
• To enter a new license, click Add and enter the license key and the site message included with the license key.
• To modify a license, select the license to update and click Modify. Then update the key and site message.
• If Benchmark Factory was activated by a Toad Edition license, the Toad license is listed. Select the license and click Modify to update it.

• To delete a license, select the license to delete and click Delete License.

  Important: If you delete a license key, any features enabled by that license and not supported by another currently-active license are disabled.

BMFServer.exe Enhancements

• The default REST API port number for BMFServer.exe is now the same as the Benchmark Factory console, port 30100.
• The BMFServer log file (BMFServer.log) is now located in: \My Benchmark Factory\<version>\<bitness>\Error Logs

New Features in 8.0.1

Benchmark Factory 8.0.1 is a maintenance release and includes resolved issues and minor enhancements. See the Benchmark Factory 8.0.1 Release Notes for a complete list.

New Features in 8.0

Benchmark Factory® 8.0 includes the following new features and enhancements.

Enhanced REST API

With this release the Benchmark Factory REST API has been enhanced to allow users more control of jobs, tests, test runs, etc. This includes the ability to access the transactions in tests, agents used by jobs, test results of a test run, as well as the Benchmark Factory settings.

Support for JSON format. Responses can now be returned in JSON format. To return data using the JSON format, specify JSON in the request Accept header. Both XML and JSON are supported response or request formats.

Access to Additional REST Resources. Additional resources have been added to the Benchmark Factory REST API. For a complete list, see the Benchmark Factory Help or the BMF REST API on Toad World. The Help also includes a useful description of the REST resource structure and the URI hierarchy.

Integrate Benchmark Factory into Continuous Testing Process

This release includes additional features that allow you to easily integrate Benchmark Factory into your Continuous Integration or Continuous Testing process.

BMFServer.exe

This release includes a non-UI Benchmark Factory application, BMFServer.exe.

• BMFServer.exe is installed into the bin sub-directory of the installation directory when you install Benchmark Factory.
• To start BMFServer.exe, go to the installation directory. The default location is C:\Program Files\Quest Software\Benchmark Factory <version>\bin. Then double-click BMFServer.exe.
• BMFServer.exe performs the same functionality as BFactory.exe, except it has no graphic user interface.
• Use BMFServer.exe when automating your continuous improvement/testing process. You can run BMFServer.exe using a script, a custom application, or the Command Prompt window. BMFServer.exe can be used with the Benchmark Factory REST API.
• You cannot run the Benchmark Factory console and BMFServer.exe at the same time on the same machine.

BMFAgent.exe

Benchmark Factory now includes a non-UI Agent, BMFAgent.exe.

• BMFAgent.exe is installed into the bin sub-directory of the installation directory at the time you install Benchmark Factory.
• BMFAgent.exe has the same functionality as Agent.exe, except BMFAgent.exe has no graphic user interface.
• To modify BMFAgent.exe settings, use the BMFAgent.ini file or use the Command Prompt window. Settings you can modify include the console machine name/IP and the console port number.
• You can run multiple instances of BMFAgent.exe at the same time on the same machine.

Standard Benchmark Testing

Clear Cached Data between Iterations

You can now instruct Benchmark Factory to clear cached data between test iterations. Because cached data can improve performance, this new feature can prevent cached data from affecting subsequent iterations.

• Oracle—For Oracle connections, you can instruct Benchmark Factory to clear data buffer caches or shared pool (or both). To specify these options, select Edit | Connections. Then select a connection and click the Edit button. In the connection properties dialog, select the Miscellaneous tab. Then specify the Database Flush options for this connection.

  

  Tip: You can also specify default values to automatically apply to Oracle connections in Edit | Settings | Oracle.

  Note: Specific database privileges are required to perform this action. The privilege required is dependent on the database version and the option selected. See the online Help for more information.

• SQL Server®—For SQL Server connections, you can instruct Benchmark Factory to clear data buffer caches and procedure caches. To specify this option, select Edit | Connections. Then select a connection and click the Edit button. In the connection properties dialog, select the Miscellaneous tab. Then specify the Database Flush option for this connection.

  

  Tip: You can also specify a default value to apply to SQL Server connections in Edit | Settings | SQL Server.

  Note: This option is only applicable to SQL Server 2005 or later. The sysadmin fixed server role is required.

Connections

Microsoft® SQL Server 2017. This release includes support for SQL Server 2017. Benchmark Factory has been tested against SQL Server 2017 running on Windows or Linux.

IBM® DB2®. This release includes support for IBM DB2 11.1 for LUW and for z/OS.

General

Adding Bind Parameters

It is now easier to add a bind parameter/value pair in Test Options | Transactions.

• When adding a new statement in the Add SQL Transaction dialog or when editing an existing statement, you can now simply double-click within the Bind Parameters tab to add a bind parameter and parameter value.

Installation

Universal C Runtime Component

The Universal C Runtime component for Windows is required. See Universal C Runtime Update or Visual C++ Redistributable for Visual Studio 2015 to download this software.

Note: If you encounter an error when installing this software, install the missing prerequisite software. For Windows 8.1 or Windows Server 2012 R2, install the April 2014 update: https://support.microsoft.com/en-us/kb/2919355. For other operating systems, see the Universal C Runtime Update Prerequisites section.

Note: The features described in the New in This Release section apply to the commercial version of Benchmark Factory and may not be available in the freeware edition.

Additional Resources

Learn More and Connect with the Community

Use the following links to videos, blogs, and community resources to learn more about Benchmark Factory, to find answers to questions, and to advance your knowledge through Benchmark Factory tips and tricks.

• Benchmark Factory Community—Ask questions, join the discussion, find useful resources, and join the Beta program (when available).
• Product Documentation (Support Portal) —Find documents for current and past releases, as well as documents for other Toad products, on the Quest Support Portal.
• Benchmark Factory blogs—Find How To's, useful articles, and tips and tricks for using Benchmark Factory.
  • How to use the REST API (part 1)—Learn how to use the Benchmark Factory REST API.
  • How to use the REST API (part 2)—Learn how to modify an existing job or connection using the REST API.
  • REST API—This page lists the Benchmark Factory REST API resources and includes example request and response content.
  • Sample Powershell script usingREST API—This sample PowerShell script demonstrates some basic functionality of the Benchmark Factory REST API.
  • Benchmarking Best Practices—10 best practices to help you get started with database benchmarking.
  • Project Converter—Use this tool to convert an Oracle capture project file created in an earlier version of Benchmark Factory to .xml format.
• Toad World—Visit other Toad Communities, including Toad for Oracle. Find DBMS and SQL knowledge, find software downloads, and find answers to your database questions.
• Benchmark Factory Product Information—Find white papers, product demos, and purchasing information.