Chat now with support
Chat with Support

Benchmark Factory for Database 7.6 - Release Notes

Release Notes

Benchmark Factory® 7.6

Release Notes

Thursday, March 16, 2017


About Benchmark Factory

Quest Software is pleased to make available Benchmark Factory® 7.6 to meet your database workload testing needs. Benchmark Factory is a database performance and code scalability testing tool that simulates users and transactions on the database and replays production workload in non-production environments. With Benchmark Factory developers, DBAs, and QA teams can validate that their databases will scale as user loads increase, application changes are made, and platform changes are implemented.

Benchmark Factory is available for Oracle®, SQL Server®, IBM® DB2®, SAP®, MySQL, and other databases via ODBC connectivity. Benchmark Factory:

  • Simulates thousands of concurrent users with a minimal amount of hardware.
  • Simulates database production workloads.
  • Reports tests results with built-in reporting features.

Upon completion of a test execution, all test results are collected and stored in the repository for data analysis and reporting. Benchmark Factory collects a variety of informative and detailed statistics that include:

  • Overall server throughput (measured in transactions per second/minute or bytes transferred).
  • Detailed transaction executed statistics by individual agent workstations producing a load.

 

New Features

This release of Benchmark Factory includes the following new features and enhancements.

Connections

Microsoft SQL Server Native Provider - New

You can now use a SQL Server native connection in Benchmark Factory to connect to a SQL Server database. The SQL Server native provider is included in the Benchmark Factory installation.

  • This new native provider delivers improved connection performance for SQL Server database connections.
  • To use the SQL Server native provider, install the version of Benchmark Factory (either 32-bit or 64-bit) that matches your operating system. This ensures that the version (bitness) of the SQL Server provider also matches your operating system.

Note: For a list of operating systems that support the SQL Server native provider distributed with Benchmark Factory (native client 11.0), see https://msdn.microsoft.com/en-us/library/cc280356(v=sql.120).aspx. If your client operating system is not supported, use ODBC connectivity.

REST API

This release of Benchmark Factory includes a REST API which will allow users to automate certain tasks and functionality in Benchmark Factory. Use a script, command-line tool, or custom application to interface with Benchmark Factory and to automate your load testing tasks. The API uses HTTP protocol and returns information in XML format.

In this release, the REST API supports the following functionality:

  • Create, run, and edit Benchmark Factory jobs
  • Create and edit connections
  • Import and replay Oracle and SQL Server captures

Using the Benchmark Factory REST API

HTTP Request MethodsBenchmark Factory REST API uses the four main HTTP methods: GET, POST, PUT, and DELETE.

Response Format—Information is returned in XML format.

Continuous Integration

  • Use the REST API to implement performance testing by Benchmark Factory and incorporate it into your Continuous Integration (CI) process.

General

Adding Bind Parameter Values for Prepared Statements

It is now much easier to add bind variable or parameter values for prepared statements.

  • In the Transaction tab, when adding a new statement or editing an existing statement, the Add bind parameter button is now enabled for prepared statements. Click this button to add bind parameter values for the selected SQL.

Specify Benchmark Factory Console Port Number

You can now specify a TCP/IP port for the Benchmark Factory console.

  1. Select Edit | Settings | General to open the Settings dialog.
  2. Then enter a port number in the TCP/IP Settings. The default value is port 4568.

Standard Benchmark Testing

Teradata Database Benchmark Testing

  • This release includes enhancements to the support for standard benchmark testing on Teradata databases.

PostgreSQL Database Benchmark Testing

  • This release includes enhancements to the support for standard benchmark testing on PostgreSQL databases.

Replication Benchmark Test

  • You can now edit the precision of the FLOAT data type when customizing table options in a Replication Benchmark test.

Capture and Replay

Easily Open Capture Scenario Wizard from Tools Menu

You can now open the Capture Scenario Wizard directly from the Tools menu.

  • Select Tools | Capture Workload. Then select either Oracle or SQL Server as the capture type.

Discontinued - Import from Foglight PA

The option to import a workload from a Foglight Performance Analysis Repository when creating a new Capture and Replay test is no longer available beginning with this release of Benchmark Factory.

Settings

Specify a Threshold for Memory Usage

You can now specify a memory usage threshold. If memory usage exceeds the percentage you specify, Benchmark Factory stops the job.

  • To specify a threshold, go to Edit | Settings | Error Handling and enter a percentage.
  • You can disable/enable this memory usage check.

Error Handling Page in Settings Dialog

The error handling options have been moved to Edit | Settings | Error Handling in the Settings dialog.

Run Reports

Improved Performance When Importing Large Test Results

  • This release includes markedly improved performance when importing a very large Test Result file.

 


Resolved Issues

Resolved Issues and Enhancements

The following is a list of issues addressed and enhancements implemented in this release of Benchmark Factory.

Feature Resolved Issue Defect ID
Benchmark Testing Teradata: Improved support for standard benchmark testing against a Teradata database. BMF-1254
Benchmark Testing Teradata: Corrected an issue that caused a "Duplicate row" error when loading tables in a Scalable Hardware benchmark test against a Teradata database. BMF-222
Benchmark Testing PostgreSQL: Improved support for standard benchmark testing against a PostgreSQL database. BMF-1265
Benchmark Testing In a benchmark test, if you choose "Refresh data using backup sets" as the data retention method but the scale factor of the backup sets does not match the scale factor selected in the test, Benchmark Factory now displays a friendly warning with instructions on how to proceed. BMF-1369
Benchmark Testing For a TPC-D test, the Custom Creation SQL page displays the region_idx1 index as expected. BMF-1509
Benchmark Testing User Interface: When attempting to run more than one job and the total virtual users requested exceeds the number of licensed virtual users currently available, a friendly warning instructs the user to reduce the number of virtual users / user load or reduce the number of jobs running. BMF-825
Benchmark Testing Enhancement: You can now edit the precision of the FLOAT data type when customizing table options in a Replication Benchmark test. BMF-1152
Benchmark Testing Replication Benchmark Test: Corrected an issue that caused an ORA-20000 error when re-running a test after changing a table option (data type precision) and switching retention from "Recreate objects…" to "Refresh data using inserts." BMF-1250
Capture and Replay Enhancement: Added the ability to import and replay a captured workload using automation / command line interface. BMF-1581
Capture and Replay Corrected an issue that, under certain conditions during the import of a SQL Server capture, resulted in an extra transaction with no SQL being added to the replay. BMF-929
Connections Corrected an issue that prevented data from loading correctly when using a SQL Server DB-Library connection. BMF-1338
Data Migration Wizard Corrected an issue which disabled the Next button on the "Choose a Data Source" page in the Migration Wizard even when the data source connection information was valid. BMF-963
Documentation Added clarification in the product documentation (online Help) as to when and why the AWR reported elapsed time and the replay time may not match the specified capture time. BMF-118
General - Settings Enhancement: You can now instruct Benchmark Factory to stop a job when memory usage exceeds a limit you specify (Edit | Settings | Error Handling). You can also disable the memory check. BMF-856
General - Settings You can no longer edit the repository credentials (user name and password) through Edit | Settings | Repository while a job is running. BMF-1229
Installation Corrected an issue that caused an error when installing Benchmark Factory using the .msi and a custom transform file. BMF-1157
Integration with other Toad Products Corrected an issue in which the Tools | Spotlight menu item was disabled in Benchmark Factory even if the Spotlight application was installed. BMF-969
Integration with Other Toad Products Corrected an issue that resulted in a "No scenarios to import" error when sending an Oracle trace file to Benchmark Factory from Toad for Oracle 12.8 or later. BMF-972
Run Reports Enhancement: Added an enhancement that markedly improves performance when importing a very large Test Result file. BMF-92
Scalability Testing After completing a Scalability test, the Results page now displays the Transactions vs User Load graph. BMF-1163
Test Stored Procedures for Scalability Corrected an issue which caused an error if the user selected a schema other than the default schema on the "Select Stored Procedures" page when selecting the stored procedures to test for scalability. BMF-1009
User Interface Corrected several text display issues encountered in the New/Edit Job Wizard when non-English locale settings were specified in Windows. BMF-1173, BMF-1185
User Interface Corrected an issue that prevented the scroll bar from displaying correctly in some cases in the Results page. BMF-918

 


Known Issues

The following is a list of issues known to exist at the time of this release.

Feature Known Issue Defect ID
Capture and Replay Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle 12c pluggable database (PDB).

N/A

Capture and Replay

Oracle workload: In Benchmark Factory 7.2 or later, if you attempt to use Oracle capture files created in an earlier version of Benchmark Factory (6.9.3 or earlier) and .NET Framework 2.0 is not installed, Windows may prompt you to install it (for example, in Windows 8 or later).

Workaround: Install .NET Framework 2.0 (3.5).

N/A

Capture and Replay Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle RAC system.

N/A

Capture and Replay

Issue: If you start a trace file capture using the current version of Benchmark Factory and then start another trace file capture using version 7.2 (or earlier) on the same database, the earlier version of Benchmark Factory cannot detect the current version performing a capture. This can result in an invalid capture.

Solution: For best results, do not perform trace file captures using both the current and an earlier version of Benchmark Factory concurrently on the same database.

N/A

Capture and Replay

SQL Server workload: After capturing activity from a user logged-in using Windows authentication, attempting to replay the capture using the ODBC profile results in the following error:

"Could not obtain information about Windows NT group/user '<group/user>', error code 0x5."

Workaround: Use a domain account to start SQL Server service and make sure the account can connect to the Active Directory of Domain.

N/A

Capture and Replay

Oracle workload: In an Oracle capture and replay, Benchmark Factory cannot generate the Statspack report if the local replay directory is mapped to drive "B." This is because Benchmark Factory must have drive B open to execute a background .bat file during oracle capture and replay.

Workaround: Do not map your local replay directory to drive "B" when running an Oracle capture and replay in which you chose to generate the Statspack report.

109221

Capture and Replay In an Oracle 12c container database (CDB), Benchmark Factory cannot replay an existing workload that was captured in an earlier version of an Oracle database. In an Oracle 12c container database, a user name (common user) is prefixed with "c##" by default.

N/A

Capture and Replay

Oracle Capture Using FGAC: When performing an Oracle Capture using the FGAC method, if the application has a trigger with a multi-schema select, additional transactions will be recorded. N/A

Connections -

Teradata ODBC

When creating or modifying a Teradata ODBC connection, if the Teradata DSN has incorrect connection information and you click Test Connection in the Connection dialog , Benchmark Factory might close unexpectedly. BMF-1497

Connections -

Transaction Wizard and IBM CLI Driver

 

When using the 64-bit version of Benchmark Factory, the Transaction Wizard stops working when using the IBM DB2 CLI Driver profile to connect the database which created objects using the ODBC Driver.

N/A

Connections -

Microsoft Oracle ODBC Driver

When using Microsoft’s Oracle ODBC driver for either testing or for Benchmark Factory repository access, there is an issue with the ODBC driver. It does not work with CLOB data. N/A

Connections -

Oracle ODBC driver

When using the Oracle 11g R2 Client and Oracle 32-bit ODBC driver in a Windows 64-bit environment, the Oracle 32-bit ODBC driver requires an 8-byte int data type.

Workaround: Use the Oracle native client or Oracle 32-bit ODBC driver.

N/A

Connections -

Microsoft SQL Server (Obsolete DB-Library) Connection

If you run a benchmark test using a Microsoft SQL Server (Obsolete DB-Library) connection to SQL Server 2014, and you have the "Recreate objects and reload data every execution" option selected, you might encounter the following error:

"DB-Library Message - Insert bulk is not supported over this access protocol"

Workaround: Use the Microsoft SQL Server ODBC driver.

N/A

Connections -

IBM CLI Driver

A profile created with a DB2 ODBC data source may crash Benchmark Factory if the IBM DB2 client catalog becomes broken/corrupted.

N/A

Connections

In Windows 10, if you open a New Connection dialog, create an SAP data source and then attempt to create a SQL Server data source, Benchmark Factory might close unexpectedly.

BMF-555

General -

Benchmark Factory Console

Benchmark Factory does not support data types that are created by the user. N/A
Industry Standard Benchmark Testing

Background: This issue applies to benchmark testing using a SQL Server or SAP native connection when the Refresh data retention option is selected in the Test Options tab.

Issue Description: In a test where the connection is lost while tables are loading, if you reconnect and attempt to rerun the job, Benchmark Factory might error with the following messages: “The Expected Total rows are not the same as the Actual Total rows,” or “a duplicate key was found” (when creating an index). The test is then canceled due to errors.

Resolution: Recreate and run a new job against your selected database using the native connection.

N/A

Industry Standard Benchmark Testing

If testing against a database other than Oracle, SQL Server, DB2, SAP, or MySQL, and using the Object/Data Retention option Refresh data using backup sets, backup sets might not be created resulting in a canceled test.

114266

Installation

(uninstall)

Beginning with release 7.1.1 in Windows Vista or later, if you have multiple editions of Benchmark Factory installed side-by-side (e.g., a Beta and a Commercial edition), and then uninstall one edition, you might encounter the following error when attempting to run a job:

" This object does not represent an OLE control"

Workaround: Do one of the following:

  • Open the Benchmark Factory application directory for the edition that is displaying the error. Then right-click the Benchmark Factory executable (BFactory.exe) and select Run As Administrator.
  • Or run the installer in Repair mode. To do this, right-click the .msi installer used to install the edition that is displaying the error and select Repair.

ST115773

Integration with other Toad products

Applies in Toad for Oracle suites (12.5 or later): When the database connection or the SQL statement contains Chinese characters, attempting to send SQL from SQL Optimizer for Oracle to Benchmark Factory results in an error. This issue only occurs when Spotlight on Oracle is launched prior to the other applications mentioned here.

Workaround: Always launch Benchmark Factory and SQL Optimizer for Oracle prior to launching Spotlight on Oracle if sending SQL from SQL Optimizer to Benchmark Factory.

N/A

Licensing

License keys compatible with Benchmark Factory 5.8.1 (or earlier) are no longer supported. See Product Licensing for instructions on how to upgrade an earlier-version license key.

N/A
Licensing If your current Benchmark Factory license key is a 21-digit format (x-xxxxx-xxxxx-xxxxx-xxxxx) and you want to use the 64-Bit installation, you will need to request a new license key. Please obtain the new key before installing the new release. To obtain a new key, please refer to the Licensing Assistance page on our support portal: https://support.quest.com/licensing-assistance. N/A
Oracle Database Privileges In Oracle 12c, the RESOURCE role no longer includes the UNLIMITED TABLESPACE system privilege. Therefore, Benchmark Factory users with the RESOURCE role must also be granted the UNLIMITED TABLESPACE privilege.

N/A

Repository

In a new installation (not an upgrade) of Benchmark Factory 7.2 or later, a Standard (Limited) user may encounter a Repository error when attempting to launch the application. The issue involves the SQLite database that is used as the default database in a new installation of Benchmark Factory 7.2 or later.

Workaround: Do one of the following:

  • Run Benchmark Factory using an Administrator account.
  • In the directory containing the SQLite database file, grant Write permission to all users. This directory is located in the following location in Windows 7: C:\ProgramData\Quest Software\BMF\Repository.

BMF-202

Repository If you created a Data Source Name for the BMF MySQL ODBC Driver in a previous release (6.9.3 or 7.0), attempting to edit this DSN results in an error.

N/A


Self Service Tools
Knowledge Base
Notifications & Alerts
Product Support
Software Downloads
Technical Documentation
User Forums
Video Tutorials
RSS Feed
Contact Us
Licensing Assistance
Technical Support
View All
Related Documents