Benchmark Factory for Database 8.0.1 - Release Notes

Release Notes

Benchmark Factory® 8.0.1

Release Notes

Wednesday, November 15, 2017


About Benchmark Factory

Quest Software is pleased to make available Benchmark Factory® 8.0.1 to meet your database workload testing needs. Benchmark Factory is a database performance and code scalability testing tool that simulates users and transactions on the database and replays production workload in non-production environments. With Benchmark Factory developers, DBAs, and QA teams can validate that their databases will scale as user loads increase, application changes are made, and platform changes are implemented.

Benchmark Factory is available for Oracle®, SQL Server®, IBM® DB2®, SAP®, MySQL, and other databases via ODBC connectivity. Benchmark Factory:

  • Simulates thousands of concurrent users with a minimal amount of hardware.
  • Simulates database production workloads.
  • Reports tests results with built-in reporting features.

Upon completion of a test execution, all test results are collected and stored in the repository for data analysis and reporting. Benchmark Factory collects a variety of informative and detailed statistics that include:

  • Overall server throughput (measured in transactions per second/minute or bytes transferred).
  • Detailed transaction executed statistics by individual agent workstations producing a load.

 

New Features

This release of Benchmark Factory includes the following new features and enhancements.

Benchmark Factory 8.0.1

Benchmark Factory 8.0.1 is a maintenance release and includes resolved issues and minor enhancements. See Resolved Issues and Enhancements for a complete list.

Benchmark Factory 8.0

Enhanced REST API

With this release the Benchmark Factory REST API has been enhanced to allow users more control of jobs, tests, test runs, etc. This includes the ability to access the transactions in tests, agents used by jobs, test results of a test run, as well as the Benchmark Factory settings.

Support for JSON format. Responses can now be returned in JSON format. To return data using the JSON format, specify JSON in the request Accept header. Both XML and JSON are supported response or request formats.

Access to Additional REST Resources. Additional resources have been added to the Benchmark Factory REST API. For a complete list of resources, see the Benchmark Factory Help or the BMF REST API on Toad World. The Help also includes a useful description of the REST resource structure and the URI hierarchy.

Integrate Benchmark Factory into Continuous Testing Process

This release includes additional features that allow you to easily integrate Benchmark Factory into your Continuous Integration or Continuous Testing process.

BMFServer.exe

This release includes a non-UI Benchmark Factory application, BMFServer.exe.

  • BMFServer.exe is installed into the bin sub-directory of the installation directory when you install Benchmark Factory.
  • To start BMFServer.exe, go to the installation directory. The default location is C:\Program Files\Quest Software\Benchmark Factory <version>\bin. Then double-click BMFServer.exe.
  • BMFServer.exe performs the same functionality as BFactory.exe, except it has no graphic user interface.
  • Use BMFServer.exe when automating your continuous improvement/testing process. You can run BMFServer.exe using a script, a custom application, or the Command Prompt window. BMFServer.exe can be used with the Benchmark Factory REST API.
  • You cannot run the Benchmark Factory console and BMFServer.exe at the same time on the same machine.

BMFAgent.exe

Benchmark Factory now includes a non-UI Agent, BMFAgent.exe.

  • BMFAgent.exe is installed into the bin sub-directory of the installation directory at the time you install Benchmark Factory.
  • BMFAgent.exe has the same functionality as Agent.exe, except BMFAgent.exe has no graphic user interface.
  • To modify BMFAgent.exe settings, use the BMFAgent.ini file located in the ProgramData\Quest Software\BMF directory or use the Command Prompt window. Settings you can modify include the console machine name/IP and the console port number.
  • You can run multiple instances of BMFAgent.exe at the same time on the same machine.

Standard Benchmark Testing

Clear Cached Data between Iterations

You can now instruct Benchmark Factory to clear cached data between test iterations. Because cached data can improve performance, this new feature can prevent cached data from affecting subsequent iterations.

  • Oracle—For Oracle connections, you can instruct Benchmark Factory to clear data buffer caches or shared pool (or both). To specify these options, select Edit | Connections. Then select a connection and click the Edit button. In the connection properties dialog, select the Miscellaneous tab. Then specify the Database Flush options for this connection.

    Tip: You can also specify default values to automatically apply to Oracle connections in Edit | Settings | Oracle.

    Note: Specific database privileges are required to perform this action. The privilege required is dependent on the database version and the option selected. See the online Help for more information.

  • SQL Server®—For SQL Server connections, you can instruct Benchmark Factory to clear data buffer caches and procedure caches. To specify this option, select Edit | Connections. Then select a connection and click the Edit button. In the connection properties dialog, select the Miscellaneous tab. Then specify the Database Flush option for this connection.

    Tip: You can also specify a default value to apply to SQL Server connections in Edit | Settings | SQL Server.

    Note: This option is only applicable to SQL Server 2005 or later. The sysadmin fixed server role is required.

Connections

Microsoft® SQL Server 2017. This release includes support for SQL Server 2017. Benchmark Factory has been tested against SQL Server 2017 running on Windows or Linux.

IBM® DB2®. This release includes support for IBM DB2 11.1 for LUW and for z/OS.

General

Adding Bind Parameters

It is now easier to add a bind parameter/value pair in Test Options | Transactions.

  • When adding a new statement in the Add SQL Transaction dialog or when editing an existing statement, you can now simply double-click within the Bind Parameters tab to add a bind parameter and parameter value.

Installation

Universal C Runtime Component

The Universal C Runtime component for Windows is required. See Universal C Runtime Update or Visual C++ Redistributable for Visual Studio 2015 to download this software.

Note: If you encounter an error when installing this software, install the missing prerequisite software. For Windows 8.1 or Windows Server 2012 R2, install the April 2014 update: https://support.microsoft.com/en-us/kb/2919355. For other operating systems, see the Universal C Runtime Update Prerequisites section.


Learn More

Remember, you can find blogs, videos, and forums at the Benchmark Factory Community.

  • Find answers to your questions in the forum.
  • Find links to Commercial, Trial, and Freeware downloads.
  • Find useful articles and product documentation.

Resolved Issues

Resolved Issues and Enhancements

The following is a list of issues addressed and enhancements implemented in this release of Benchmark Factory.

Resolved Issues in 8.0.1

Feature Resolved Issue Defect ID
Agent Corrected an issue that prevented the display of agent activity in the Agent view Output tab, for example when agents are being used to load table data. BMF-3220
BMFServer Corrected an issue that removed existing jobs and connections from the repository after the BMFServer (BMFServer.exe) failed to successfully connect to the repository. BMF-3377
Capture and Replay Corrected an issue that caused an error after attempting to modify the latency value for a transaction (or user scenario) in a captured SQL file, such as an Oracle capture or an Oracle, SQL Server, or ODBC trace file. BMF-3267
Capture and Replay If you attempt to delete a capture while it is running (Captures tab), you can now close the warning message, as expected. BMF-2998
REST API When adding a new job or test, a job/test name is now required or the default value is used. BMF-3224
REST API Improved content and detail of the error messages returned if data is missing or incorrect in the XML or JSON provided by the user. BMF-3223
REST API Removed some of the unnecessary or inapplicable properties from the results that are returned when using the REST API. BMF-2742, BMF-3271
REST API True/False values are now displayed as "True" or "False" instead of "1" or "0" in the results returned when using the REST API. BMF-3270
REST API Removed unnecessary information returned from a GET request for settings/repository. Also added a warning message if user attempts to specify or modify the repository driver name through the REST API. BMF-3266
REST API Corrected an issue that caused a TNS error when attempting to run a job with an Oracle connection that was created through the REST API. BMF-3263
REST API You can no longer delete the default agent through the REST API. BMF-2859
REST API The REST API now returns user scenario transaction information when requested, for example when using a request such as http://{server}:{port}/api/jobs/0/tests/0/Transactions/0/Transactions. BMF-2747

Resolved Issues in 8.0

Feature Resolved Issue Defect ID
Agent Corrected an issue which prevented the Agent (Agent.exe) from saving settings after the settings were modified through the Agent Settings dialog. BMF-2128
Agent Corrected an issue in which the number of agents was not correctly displayed in Benchmark Factory console after an agent was added (posted) using the REST API. BMF-2354
Benchmark Testing IBM DB2 z/OS: Corrected an issue that caused an error in the create objects step with a DB2 z/OS database connection. BMF-2645
Benchmark Testing Teradata: The "Distribute index creation" option is now disabled for Teradata connections to prevent errors during this step. BMF-1500
Benchmark Testing Test Options | Advanced tab | Database Checkpoints: You can now enter only a valid value in the "Number of Checkpoints" text box to avoid errors. BMF-1641
Capture and Replay If a re-capture is stopped by the user, Benchmark Factory prompts to delete the capture with no sessions, as expected. This issue was encountered during a SQL Server capture. BMF-1183
Capture and Replay If user started a second capture, the package file for the first and currently running capture was deleted (draft). Both captures were on the same database. BMF-1726
Capture and Replay Corrected an issue in which the Export Start Time was displayed as "1899-12-30 00:00:00" in the Export Status window of an aborted capture. BMF-2061
Capture and Replay Corrected an issue that caused the Grant Privileges dialog to open subsequent times when clicking Next in the Backup Scope page even if valid credentials had been entered. This issue was encountered when attempting to capture a SQL Server workload. BMF-2021
Capture and Replay SQL Server capture: If the user selects "Perform backup as part of the Capture Process" and the user does not have permissions to restore the database (“CREATE DATABASE” privilege), Benchmark Factory now displays a warning message. BMF-2019
Capture and Replay SQL Server capture: Corrected an issue with the Apply Privileges page in which Benchmark Factory attempted to authenticate the previously entered SQL Server credentials even though the user subsequently selected to use Windows Authentication. BMF-2130
Connections Added options to allow the user to instruct Benchmark Factory to clear cached data between iterations. This is useful since cached data can improve performance and therefore affect subsequent iterations. BMF-670
Connections This release includes support for IBM DB2 11 for LUW and z/OS. BMF-685
General Corrected an issue that caused a memory leak when running a custom SQL statement with BFScript in a bind parameter. BMF-2631
General Enhancement: It is easier to add a bind parameter/value pair in Test Options | Transactions. Double-click within the Bind Parameters tab to add a bind parameter and parameter value. BMF-1270
General Corrected an issue that caused Benchmark Factory to close unexpectedly when attempting to import SQL from a large number of stored procedures. In this case, the user imported the same procedures multiple times. BMF-2060
Installation Corrected an issue that prevented the installation of the SQL Server native provider during the installation of Benchmark Factory in a Windows Server 2008 SP2 environment. BMF-1947
Installation Corrected an issue that caused an error related to the SQL Server native provider for some users when uninstalling Benchmark Factory. BMF-2148
Integration with other Quest products Corrected an issue that caused Benchmark Factory to close unexpectedly after sending SQL from SQL Optimizer for SAP. This issue was encountered in a Windows Server 2012 R2 environment. BMF-2297
Job Wizard Job Wizard | Job Setup | Schedule tab: If you enter invalid scheduling details, the Edit Job Wizard no longer unexpectedly closes after you click OK in the warning message. BMF-1842
REST API In the REST API, connection is now a resource of Job. BMF-1723
REST API In the REST API, when you GET the job properties, the schedule properties are now under the Schedule object/node as expected. BMF-1725
REST API Enhancement: The REST API now returns more-descriptive error messages. BMF-1853
Run Reports Run Reports | Testbed Details: Corrected an issue in which Windows 8.1 was identified as Windows 8 in Testbed details. BMF-1228
Run Reports Corrected an issue which prevented the display of correct Time Phase (Sampling, Pre-Sampling) information on the Timing tab in Run Reports. BMF-2012
Run Reports Corrected an issue which prevented the Run Reports viewer (Tools | Run Reports) from opening if a "Run Reports" folder was open in Windows Explorer. BMF-2190
Settings Edit | Settings | Timing | User Startup: In the "Start a new user every" field, your specified setting is now saved, as expected. BMF-1843
System Requirements This release of Benchmark Factory has been tested with and supports Windows Server 2016. BMF-1575
System Requirements This release of Benchmark Factory was tested against SQL Server 2017 running on Windows or Linux. BMF-1878
Test Results Compare Results tab: Corrected an issue in which the best performing transaction was displayed twice after a custom scalability test was rerun. BMF-1423
User Interface Run Reports | Workload Details | Transactions tab: Corrected an issue which caused some overlapping of UI elements in this window when the transaction list was long. BMF-1032
User Interface SQL Server connection: Removed the Profile field from the Login dialog when repeating a capture from the Captures tab (right-click | Repeat Capture). BMF-1161
User Interface If you select multiple captures in the Captures tab and then right-click and select Delete, all selected captures are now deleted, as expected. Previously, this action deleted only one of the captures. BMF-1727
User Interface Capture and Replay: If you edit the Capture Thresholds fields and then move to the next page, your edits are now retained when you return to the Capture Thresholds page. BMF-1752
User Interface Edit | Settings | Latency: Changed "Set Transaction Rate" to "Transaction Rate." BMF-1845
User Interface Capture Scenario Wizard: Corrected a minor UI issue involving the placement of the warning message on the Filter Settings page. BMF-2132

Known Issues

The following is a list of issues known to exist at the time of this release.

Feature Known Issue Defect ID
Capture and Replay Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle 12c pluggable database (PDB).

N/A

Capture and Replay

Oracle workload: In Benchmark Factory 7.2 or later, if you attempt to use Oracle capture files created in an earlier version of Benchmark Factory (6.9.3 or earlier) and .NET Framework 2.0 is not installed, Windows may prompt you to install it (for example, in Windows 8 or later).

Workaround: Install .NET Framework 2.0 (3.5).

N/A

Capture and Replay Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle RAC system.

N/A

Capture and Replay

Issue: If you start a trace file capture using the current version of Benchmark Factory and then start another trace file capture using version 7.2 (or earlier) on the same database, the earlier version of Benchmark Factory cannot detect the current version performing a capture. This can result in an invalid capture.

Solution: For best results, do not perform trace file captures using both the current and an earlier version of Benchmark Factory concurrently on the same database.

N/A

Capture and Replay

SQL Server workload: After capturing activity from a user logged-in using Windows authentication, attempting to replay the capture using the ODBC profile results in the following error:

"Could not obtain information about Windows NT group/user '<group/user>', error code 0x5."

Workaround: Use a domain account to start SQL Server service and make sure the account can connect to the Active Directory of Domain.

N/A

Capture and Replay

Oracle workload: In an Oracle capture and replay, Benchmark Factory cannot generate the Statspack report if the local replay directory is mapped to drive "B." This is because Benchmark Factory must have drive B open to execute a background .bat file during oracle capture and replay.

Workaround: Do not map your local replay directory to drive "B" when running an Oracle capture and replay in which you chose to generate the Statspack report.

109221

Capture and Replay In an Oracle 12c container database (CDB), Benchmark Factory cannot replay an existing workload that was captured in an earlier version of an Oracle database. In an Oracle 12c container database, a user name (common user) is prefixed with "c##" by default.

N/A

Capture and Replay

Oracle Capture Using FGAC: When performing an Oracle Capture using the FGAC method, if the application has a trigger with a multi-schema select, additional transactions will be recorded. N/A

Connections -

Teradata ODBC

When creating or modifying a Teradata ODBC connection, if the Teradata DSN has incorrect connection information and you click Test Connection in the Connection dialog , Benchmark Factory might close unexpectedly. BMF-1497

Connections -

Transaction Wizard and IBM CLI Driver

 

When using the 64-bit version of Benchmark Factory, the Transaction Wizard stops working when using the IBM DB2 CLI Driver profile to connect the database which created objects using the ODBC Driver.

N/A

Connections -

Microsoft Oracle ODBC Driver

When using Microsoft’s Oracle ODBC driver for either testing or for Benchmark Factory repository access, there is an issue with the ODBC driver. It does not work with CLOB data. N/A

Connections -

Oracle ODBC driver

When using the Oracle 11g R2 Client and Oracle 32-bit ODBC driver in a Windows 64-bit environment, the Oracle 32-bit ODBC driver requires an 8-byte int data type.

Workaround: Use the Oracle native client or Oracle 32-bit ODBC driver.

N/A

Connections -

Microsoft SQL Server (Obsolete DB-Library) Connection

If you run a benchmark test using a Microsoft SQL Server (Obsolete DB-Library) connection to SQL Server 2014, and you have the "Recreate objects and reload data every execution" option selected, you might encounter the following error:

"DB-Library Message - Insert bulk is not supported over this access protocol"

Workaround: Use the Microsoft SQL Server ODBC driver.

N/A

Connections -

IBM CLI Driver

A profile created with a DB2 ODBC data source may crash Benchmark Factory if the IBM DB2 client catalog becomes broken/corrupted.

N/A

Connections

In Windows 10, if you open a New Connection dialog, create an SAP data source and then attempt to create a SQL Server data source, Benchmark Factory might close unexpectedly.

BMF-555

General -

Benchmark Factory Console

Benchmark Factory does not support data types that are created by the user. N/A
Industry Standard Benchmark Testing

Background: This issue applies to benchmark testing using a SQL Server or SAP native connection when the Refresh data retention option is selected in the Test Options tab.

Issue Description: In a test where the connection is lost while tables are loading, if you reconnect and attempt to rerun the job, Benchmark Factory might error with the following messages: “The Expected Total rows are not the same as the Actual Total rows,” or “a duplicate key was found” (when creating an index). The test is then canceled due to errors.

Resolution: Recreate and run a new job against your selected database using the native connection.

N/A

Industry Standard Benchmark Testing

If testing against a database other than Oracle, SQL Server, DB2, SAP, or MySQL, and using the Object/Data Retention option Refresh data using backup sets, backup sets might not be created resulting in a canceled test.

114266

Installation

When installing in a Windows Vista 32-bit environment as a limited user (not Administrator), installation fails.

Workaround: Install Benchmark Factory as Administrator.

N/A

Installation

(uninstall)

Beginning with release 7.1.1 in Windows Vista or later, if you have multiple editions of Benchmark Factory installed side-by-side (e.g., a Beta and a Commercial edition), and then uninstall one edition, you might encounter the following error when attempting to run a job:

" This object does not represent an OLE control"

Workaround: Do one of the following:

  • Open the Benchmark Factory application directory for the edition that is displaying the error. Then right-click the Benchmark Factory executable (BFactory.exe) and select Run As Administrator.
  • Or run the installer in Repair mode. To do this, right-click the .msi installer used to install the edition that is displaying the error and select Repair.

ST115773

Integration with other Toad products

Applies in Toad for Oracle suites (12.5 or later): When the database connection or the SQL statement contains Chinese characters, attempting to send SQL from SQL Optimizer for Oracle to Benchmark Factory results in an error. This issue only occurs when Spotlight on Oracle is launched prior to the other applications mentioned here.

Workaround: Always launch Benchmark Factory and SQL Optimizer for Oracle prior to launching Spotlight on Oracle if sending SQL from SQL Optimizer to Benchmark Factory.

N/A

Licensing

License keys compatible with Benchmark Factory 5.8.1 (or earlier) are no longer supported. See Product Licensing for instructions on how to upgrade an earlier-version license key.

N/A
Licensing If your current Benchmark Factory license key is a 21-digit format (x-xxxxx-xxxxx-xxxxx-xxxxx) and you want to use the 64-Bit installation, you will need to request a new license key. Please obtain the new key before installing the new release. To obtain a new key, please refer to the Licensing Assistance page on our support portal: https://support.quest.com/licensing-assistance. N/A
Oracle Database Privileges In Oracle 12c, the RESOURCE role no longer includes the UNLIMITED TABLESPACE system privilege. Therefore, Benchmark Factory users with the RESOURCE role must also be granted the UNLIMITED TABLESPACE privilege.

N/A

Repository

In a new installation (not an upgrade) of Benchmark Factory 7.2 or later, a Standard (Limited) user may encounter a Repository error when attempting to launch the application. The issue involves the SQLite database that is used as the default database in a new installation of Benchmark Factory 7.2 or later.

Workaround: Do one of the following:

  • Run Benchmark Factory using an Administrator account.
  • In the directory containing the SQLite database file, grant Write permission to all users. This directory is located in the following location in Windows 7: C:\ProgramData\Quest Software\BMF\Repository.

BMF-202

Repository If you created a Data Source Name for the BMF MySQL ODBC Driver in a previous release (6.9.3 or 7.0), attempting to edit this DSN results in an error.

N/A


Self Service Tools
Knowledge Base
Notifications & Alerts
Product Support
Software Downloads
Technical Documentation
User Forums
Video Tutorials
Contact Us
Licensing Assistance
Technical Support
View All
Related Documents