Benchmark Factory for Database 8.0 - User Guide

Getting Started

Getting Started With Benchmark Factory

Benchmark Factory® is a database performance testing tool that allows you to conduct database workload replay, industry-standard benchmark testing, and scalability testing.

Review the following topics for a quick overview of Benchmark Factory.

• About Benchmark Factory
• Understanding Benchmark Factory Testing
• Understanding Benchmark Factory Terminology
• How Benchmark Factory Works
• The Benchmark Factory Environment
• Getting Started - the Benchmark Factory Workflow

What's New in Benchmark Factory

New Features in 8.0

This release of Benchmark Factory® includes the following new features and enhancements.

Enhanced REST API

With this release the Benchmark Factory REST API has been enhanced to allow users more control of jobs, tests, test runs, etc. This includes the ability to access the transactions in tests, agents used by jobs, test results of a test run, as well as the Benchmark Factory settings.

Support for JSON format. Responses can now be returned in JSON format. To return data using the JSON format, specify JSON in the request Accept header. Both XML and JSON are supported response or request formats.

Access to Additional REST Resources. Additional resources have been added to the Benchmark Factory REST API. For a complete list, see the Benchmark Factory Help. The Help also includes a useful description of the REST resource structure and the URI hierarchy.

Integrate Benchmark Factory into Continuous Testing Process

This release includes additional features that allow you to easily integrate Benchmark Factory into your Continuous Integration or Continuous Testing process.

BMFServer.exe

This release includes a non-UI Benchmark Factory application, BMFServer.exe.

• BMFServer.exe is installed into the bin sub-directory of the installation directory when you install Benchmark Factory.
• To start BMFServer.exe, go to the installation directory. The default location is C:\Program Files\Quest Software\Benchmark Factory 8.0.0\bin. Then double-click BMFServer.exe.
• BMFServer.exe performs the same functionality as BFactory.exe, except it has no graphic user interface.
• Use BMFServer.exe when automating your continuous improvement/testing process. You can run BMFServer.exe using a script, a custom application, or the Command Prompt window. BMFServer.exe can be used with the Benchmark Factory REST API.
• You cannot run the Benchmark Factory console and BMFServer.exe at the same time on the same machine.

BMFAgent.exe

Benchmark Factory now includes a non-UI Agent, BMFAgent.exe.

• BMFAgent.exe is installed into the bin sub-directory of the installation directory at the time you install Benchmark Factory.
• BMFAgent.exe has the same functionality as Agent.exe, except BMFAgent.exe has no graphic user interface.
• To modify BMFAgent.exe settings, use the BMFAgent.ini file located in the ProgramData\Quest Software\BMF directory or use the Command Prompt window. Settings you can modify include the console machine name/IP and the console port number.
• You can run multiple instances of BMFAgent.exe at the same time on the same machine.

Standard Benchmark Testing

Clear Cached Data between Iterations

You can now instruct Benchmark Factory to clear cached data between test iterations. Because cached data can improve performance, this new feature can prevent cached data from affecting subsequent iterations.

• Oracle—For Oracle connections, you can instruct Benchmark Factory to clear data buffer caches or shared pool (or both). To specify these options, select Edit | Connections. Then select a connection and click the Edit button. In the connection properties dialog, select the Miscellaneous tab. Then specify the Database Flush options for this connection.

  

  Tip: You can also specify default values to automatically apply to Oracle connections in Edit | Settings | Oracle.

  Note: Specific database privileges are required to perform this action. The privilege required is dependent on the database version and the option selected. See the online Help for more information.

• SQL Server®—For SQL Server connections, you can instruct Benchmark Factory to clear data buffer caches and procedure caches. To specify this option, select Edit | Connections. Then select a connection and click the Edit button. In the connection properties dialog, select the Miscellaneous tab. Then specify the Database Flush option for this connection.

  

  Tip: You can also specify a default value to apply to SQL Server connections in Edit | Settings | SQL Server.

  Note: This option is only applicable to SQL Server 2005 or later. The sysadmin fixed server role is required.

Connections

Microsoft® SQL Server 2017. This release includes support for SQL Server 2017. Benchmark Factory has been tested against SQL Server 2017 running on Windows or Linux.

IBM® DB2®. This release includes support for IBM DB2 11.1 for LUW and for z/OS.

General

Adding Bind Parameters

It is now easier to add a bind parameter/value pair in Test Options | Transactions.

• When adding a new statement in the Add SQL Transaction dialog or when editing an existing statement, you can now simply double-click within the Bind Parameters tab to add a bind parameter and parameter value.

Installation

Universal C Runtime Component

The Universal C Runtime component for Windows is required. See Universal C Runtime Update or Visual C++ Redistributable for Visual Studio 2015 to download this software.

Note: If you encounter an error when installing this software, install the missing prerequisite software. For Windows 8.1 or Windows Server 2012 R2, install the April 2014 update: https://support.microsoft.com/en-us/kb/2919355. For other operating systems, see the Universal C Runtime Update Prerequisites section.

Note: The features described in the New in This Release section apply to the commercial version of Benchmark Factory and may not be available in the freeware edition.

Additional Resources

Learn More and Connect with the Community

Use the following links to videos, blogs, and community resources to learn more about Benchmark Factory, to find answers to questions, and to advance your knowledge through Benchmark Factory tips and tricks.

• Benchmark Factory Community—Ask questions, join the discussion, find videos and user guides, and join the Beta program (when available).
• Release Notes and Product Documentation (Benchmark Factory Community)
• Videos and Tutorials—Find video demos of Benchmark Factory features.
• Benchmark Factory Blog—Find How To's, useful articles, and tips and tricks for using Benchmark Factory.
  • How to use the REST API (part 1)— Learn how to use the Benchmark Factory REST API.
  • Benchmarking Best Practices—10 best practices to help you get started with database benchmarking.
• Product Documentation —Find documents for current and past releases, as well as documents for other Toad products, on the Quest Support Portal.
• Toad World—Visit other Toad Communities, including Toad for Oracle. Find DBMS and SQL knowledge, find software downloads, and find answers to your database questions.
• Benchmark Factory Product Information—Find white papers, product demos, and purchasing information.