Release Notes
Benchmark Factory® 8.0
Release Notes
Friday, August 18, 2017
About Benchmark Factory
Quest Software is pleased to make available Benchmark Factory® 8.0 to meet your database workload testing needs. Benchmark Factory is a database performance and code scalability testing tool that simulates users and transactions on the database and replays production workload in non-production environments. With Benchmark Factory developers, DBAs, and QA teams can validate that their databases will scale as user loads increase, application changes are made, and platform changes are implemented.
Benchmark Factory is available for Oracle®, SQL Server®, IBM® DB2®, SAP®, MySQL, and other databases via ODBC connectivity. Benchmark Factory:
- Simulates thousands of concurrent users with a minimal amount of hardware.
- Simulates database production workloads.
- Reports tests results with built-in reporting features.
Upon completion of a test execution, all test results are collected and stored in the repository for data analysis and reporting. Benchmark Factory collects a variety of informative and detailed statistics that include:
- Overall server throughput (measured in transactions per second/minute or bytes transferred).
- Detailed transaction executed statistics by individual agent workstations producing a load.
New Features
This release of Benchmark Factory includes the following new features and enhancements.
Enhanced REST API
With this release the Benchmark Factory REST API has been enhanced to allow users more control of jobs, tests, test runs, etc. This includes the ability to access the transactions in tests, agents used by jobs, test results of a test run, as well as the Benchmark Factory settings.
Support for JSON format. Responses can now be returned in JSON format. To return data using the JSON format, specify JSON in the request Accept header. Both XML and JSON are supported response or request formats.
Access to Additional REST Resources. Additional resources have been added to the Benchmark Factory REST API. For a complete list, see the Benchmark Factory Help. The Help also includes a useful description of the REST resource structure and the URI hierarchy.
Integrate Benchmark Factory into Continuous Testing Process
This release includes additional features that allow you to easily integrate Benchmark Factory into your Continuous Integration or Continuous Testing process.
BMFServer.exe
This release includes a non-UI Benchmark Factory application, BMFServer.exe.
- BMFServer.exe is installed into the bin sub-directory of the installation directory when you install Benchmark Factory.
- To start BMFServer.exe, go to the installation directory. The default location is C:\Program Files\Quest Software\Benchmark Factory 8.0.0\bin. Then double-click BMFServer.exe.
- BMFServer.exe performs the same functionality as BFactory.exe, except it has no graphic user interface.
- Use BMFServer.exe when automating your continuous improvement/testing process. You can run BMFServer.exe using a script, a custom application, or the Command Prompt window. BMFServer.exe can be used with the Benchmark Factory REST API.
- You cannot run the Benchmark Factory console and BMFServer.exe at the same time on the same machine.
BMFAgent.exe
Benchmark Factory now includes a non-UI Agent, BMFAgent.exe.
- BMFAgent.exe is installed into the bin sub-directory of the installation directory at the time you install Benchmark Factory.
- BMFAgent.exe has the same functionality as Agent.exe, except BMFAgent.exe has no graphic user interface.
- To modify BMFAgent.exe settings, use the BMFAgent.ini file located in the ProgramData\Quest Software\BMF directory or use the Command Prompt window. Settings you can modify include the console machine name/IP and the console port number.
- You can run multiple instances of BMFAgent.exe at the same time on the same machine.
Standard Benchmark Testing
Clear Cached Data between Iterations
You can now instruct Benchmark Factory to clear cached data between test iterations. Because cached data can improve performance, this new feature can prevent cached data from affecting subsequent iterations.
-
Oracle—For Oracle connections, you can instruct Benchmark Factory to clear data buffer caches or shared pool (or both). To specify these options, select Edit | Connections. Then select a connection and click the Edit button. In the connection properties dialog, select the Miscellaneous tab. Then specify the Database Flush options for this connection.
Tip: You can also specify default values to automatically apply to Oracle connections in Edit | Settings | Oracle.
Note: Specific database privileges are required to perform this action. The privilege required is dependent on the database version and the option selected. See the online Help for more information.
-
SQL Server®—For SQL Server connections, you can instruct Benchmark Factory to clear data buffer caches and procedure caches. To specify this option, select Edit | Connections. Then select a connection and click the Edit button. In the connection properties dialog, select the Miscellaneous tab. Then specify the Database Flush option for this connection.
Tip: You can also specify a default value to apply to SQL Server connections in Edit | Settings | SQL Server.
Note: This option is only applicable to SQL Server 2005 or later. The sysadmin fixed server role is required.
Connections
Microsoft® SQL Server 2017. This release includes support for SQL Server 2017. Benchmark Factory has been tested against SQL Server 2017 running on Windows or Linux.
IBM® DB2®. This release includes support for IBM DB2 11.1 for LUW and for z/OS.
General
Adding Bind Parameters
It is now easier to add a bind parameter/value pair in Test Options | Transactions.
-
When adding a new statement in the Add SQL Transaction dialog or when editing an existing statement, you can now simply double-click within the Bind Parameters tab to add a bind parameter and parameter value.
Installation
Universal C Runtime Component
The Universal C Runtime component for Windows is required. See Universal C Runtime Update or Visual C++ Redistributable for Visual Studio 2015 to download this software.
Note: If you encounter an error when installing this software, install the missing prerequisite software. For Windows 8.1 or Windows Server 2012 R2, install the April 2014 update: https://support.microsoft.com/en-us/kb/2919355. For other operating systems, see the Universal C Runtime Update Prerequisites section.
Resolved Issues
Resolved Issues and Enhancements
The following is a list of issues addressed and enhancements implemented in this release of Benchmark Factory.
| Feature | Resolved Issue | Defect ID |
|---|---|---|
| Agent | Corrected an issue which prevented the Agent (Agent.exe) from saving settings after the settings were modified through the Agent Settings dialog. | BMF-2128 |
| Agent | Corrected an issue in which the number of agents was not correctly displayed in Benchmark Factory console after an agent was added (posted) using the REST API. | BMF-2354 |
| Benchmark Testing | IBM DB2 z/OS: Corrected an issue that caused an error in the create objects step with a DB2 z/OS database connection. | BMF-2645 |
| Benchmark Testing | Teradata: The "Distribute index creation" option is now disabled for Teradata connections to prevent errors during this step. | BMF-1500 |
| Benchmark Testing | Test Options | Advanced tab | Database Checkpoints: You can now enter only a valid value in the "Number of Checkpoints" text box to avoid errors. | BMF-1641 |
| Capture and Replay | If a re-capture is stopped by the user, Benchmark Factory prompts to delete the capture with no sessions, as expected. This issue was encountered during a SQL Server capture. | BMF-1183 |
| Capture and Replay | If user started a second capture, the package file for the first and currently running capture was deleted (draft). Both captures were on the same database. | BMF-1726 |
| Capture and Replay | Corrected an issue in which the Export Start Time was displayed as "1899-12-30 00:00:00" in the Export Status window of an aborted capture. | BMF-2061 |
| Capture and Replay | Corrected an issue that caused the Grant Privileges dialog to open subsequent times when clicking Next in the Backup Scope page even if valid credentials had been entered. This issue was encountered when attempting to capture a SQL Server workload. | BMF-2021 |
| Capture and Replay | SQL Server capture: If the user selects "Perform backup as part of the Capture Process" and the user does not have permissions to restore the database (“CREATE DATABASE” privilege), Benchmark Factory now displays a warning message. | BMF-2019 |
| Capture and Replay | SQL Server capture: Corrected an issue with the Apply Privileges page in which Benchmark Factory attempted to authenticate the previously entered SQL Server credentials even though the user subsequently selected to use Windows Authentication. | BMF-2130 |
| Connections | Added options to allow the user to instruct Benchmark Factory to clear cached data between iterations. This is useful since cached data can improve performance and therefore affect subsequent iterations. | BMF-670 |
| Connections | This release includes support for IBM DB2 11 for LUW and z/OS. | BMF-685 |
| General | Corrected an issue that caused a memory leak when running a custom SQL statement with BFScript in a bind parameter. | BMF-2631 |
| General | Enhancement: It is easier to add a bind parameter/value pair in Test Options | Transactions. Double-click within the Bind Parameters tab to add a bind parameter and parameter value. | BMF-1270 |
| General | Corrected an issue that caused Benchmark Factory to close unexpectedly when attempting to import SQL from a large number of stored procedures. In this case, the user imported the same procedures multiple times. | BMF-2060 |
| Installation | Corrected an issue that prevented the installation of the SQL Server native provider during the installation of Benchmark Factory in a Windows Server 2008 SP2 environment. | BMF-1947 |
| Installation | Corrected an issue that caused an error related to the SQL Server native provider for some users when uninstalling Benchmark Factory. | BMF-2148 |
| Integration with other Quest products | Corrected an issue that caused Benchmark Factory to close unexpectedly after sending SQL from SQL Optimizer for SAP. This issue was encountered in a Windows Server 2012 R2 environment. | BMF-2297 |
| Job Wizard | Job Wizard | Job Setup | Schedule tab: If you enter invalid scheduling details, the Edit Job Wizard no longer unexpectedly closes after you click OK in the warning message. | BMF-1842 |
| REST API | In the REST API, connection is now a resource of Job. | BMF-1723 |
| REST API | In the REST API, when you GET the job properties, the schedule properties are now under the Schedule object/node as expected. | BMF-1725 |
| REST API | Enhancement: The REST API now returns more-descriptive error messages. | BMF-1853 |
| Run Reports | Run Reports | Testbed Details: Corrected an issue in which Windows 8.1 was identified as Windows 8 in Testbed details. | BMF-1228 |
| Run Reports | Corrected an issue which prevented the display of correct Time Phase (Sampling, Pre-Sampling) information on the Timing tab in Run Reports. | BMF-2012 |
| Run Reports | Corrected an issue which prevented the Run Reports viewer (Tools | Run Reports) from opening if a "Run Reports" folder was open in Windows Explorer. | BMF-2190 |
| Settings | Edit | Settings | Timing | User Startup: In the "Start a new user every" field, your specified setting is now saved, as expected. | BMF-1843 |
| System Requirements | This release of Benchmark Factory has been tested with and supports Windows Server 2016. | BMF-1575 |
| System Requirements | This release of Benchmark Factory was tested against SQL Server 2017 running on Windows or Linux. | BMF-1878 |
| Test Results | Compare Results tab: Corrected an issue in which the best performing transaction was displayed twice after a custom scalability test was rerun. | BMF-1423 |
| User Interface | Run Reports | Workload Details | Transactions tab: Corrected an issue which caused some overlapping of UI elements in this window when the transaction list was long. | BMF-1032 |
| User Interface | SQL Server connection: Removed the Profile field from the Login dialog when repeating a capture from the Captures tab (right-click | Repeat Capture). | BMF-1161 |
| User Interface | If you select multiple captures in the Captures tab and then right-click and select Delete, all selected captures are now deleted, as expected. Previously, this action deleted only one of the captures. | BMF-1727 |
| User Interface | Capture and Replay: If you edit the Capture Thresholds fields and then move to the next page, your edits are now retained when you return to the Capture Thresholds page. | BMF-1752 |
| User Interface | Edit | Settings | Latency: Changed "Set Transaction Rate" to "Transaction Rate." | BMF-1845 |
| User Interface | Capture Scenario Wizard: Corrected a minor UI issue involving the placement of the warning message on the Filter Settings page. | BMF-2132 |
Known Issues
The following is a list of issues known to exist at the time of this release.
| Feature | Known Issue | Defect ID |
|---|---|---|
| Capture and Replay | Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle 12c pluggable database (PDB). |
N/A |
| Capture and Replay |
Oracle workload: In Benchmark Factory 7.2 or later, if you attempt to use Oracle capture files created in an earlier version of Benchmark Factory (6.9.3 or earlier) and .NET Framework 2.0 is not installed, Windows may prompt you to install it (for example, in Windows 8 or later). Workaround: Install .NET Framework 2.0 (3.5). |
N/A |
| Capture and Replay | Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle RAC system. |
N/A |
| Capture and Replay |
Issue: If you start a trace file capture using the current version of Benchmark Factory and then start another trace file capture using version 7.2 (or earlier) on the same database, the earlier version of Benchmark Factory cannot detect the current version performing a capture. This can result in an invalid capture. Solution: For best results, do not perform trace file captures using both the current and an earlier version of Benchmark Factory concurrently on the same database. |
N/A |
| Capture and Replay |
SQL Server workload: After capturing activity from a user logged-in using Windows authentication, attempting to replay the capture using the ODBC profile results in the following error: "Could not obtain information about Windows NT group/user '<group/user>', error code 0x5." Workaround: Use a domain account to start SQL Server service and make sure the account can connect to the Active Directory of Domain. |
N/A |
| Capture and Replay |
Oracle workload: In an Oracle capture and replay, Benchmark Factory cannot generate the Statspack report if the local replay directory is mapped to drive "B." This is because Benchmark Factory must have drive B open to execute a background .bat file during oracle capture and replay. Workaround: Do not map your local replay directory to drive "B" when running an Oracle capture and replay in which you chose to generate the Statspack report. |
109221 |
| Capture and Replay | In an Oracle 12c container database (CDB), Benchmark Factory cannot replay an existing workload that was captured in an earlier version of an Oracle database. In an Oracle 12c container database, a user name (common user) is prefixed with "c##" by default. |
N/A |
|
Capture and Replay |
Oracle Capture Using FGAC: When performing an Oracle Capture using the FGAC method, if the application has a trigger with a multi-schema select, additional transactions will be recorded. | N/A |
|
Connections - Teradata ODBC |
When creating or modifying a Teradata ODBC connection, if the Teradata DSN has incorrect connection information and you click Test Connection in the Connection dialog , Benchmark Factory might close unexpectedly. | BMF-1497 |
|
Connections - Transaction Wizard and IBM CLI Driver
|
When using the 64-bit version of Benchmark Factory, the Transaction Wizard stops working when using the IBM DB2 CLI Driver profile to connect the database which created objects using the ODBC Driver. |
N/A |
|
Connections - Microsoft Oracle ODBC Driver |
When using Microsoft’s Oracle ODBC driver for either testing or for Benchmark Factory repository access, there is an issue with the ODBC driver. It does not work with CLOB data. | N/A |
|
Connections - Oracle ODBC driver |
When using the Oracle 11g R2 Client and Oracle 32-bit ODBC driver in a Windows 64-bit environment, the Oracle 32-bit ODBC driver requires an 8-byte int data type. Workaround: Use the Oracle native client or Oracle 32-bit ODBC driver. |
N/A |
|
Connections - Microsoft SQL Server (Obsolete DB-Library) Connection |
If you run a benchmark test using a Microsoft SQL Server (Obsolete DB-Library) connection to SQL Server 2014, and you have the "Recreate objects and reload data every execution" option selected, you might encounter the following error: "DB-Library Message - Insert bulk is not supported over this access protocol" Workaround: Use the Microsoft SQL Server ODBC driver. |
N/A |
|
Connections - IBM CLI Driver |
A profile created with a DB2 ODBC data source may crash Benchmark Factory if the IBM DB2 client catalog becomes broken/corrupted. |
N/A |
|
Connections |
In Windows 10, if you open a New Connection dialog, create an SAP data source and then attempt to create a SQL Server data source, Benchmark Factory might close unexpectedly. |
BMF-555 |
|
General - Benchmark Factory Console |
Benchmark Factory does not support data types that are created by the user. | N/A |
| Industry Standard Benchmark Testing |
Background: This issue applies to benchmark testing using a SQL Server or SAP native connection when the Refresh data retention option is selected in the Test Options tab. Issue Description: In a test where the connection is lost while tables are loading, if you reconnect and attempt to rerun the job, Benchmark Factory might error with the following messages: “The Expected Total rows are not the same as the Actual Total rows,” or “a duplicate key was found” (when creating an index). The test is then canceled due to errors. Resolution: Recreate and run a new job against your selected database using the native connection. |
N/A |
| Industry Standard Benchmark Testing |
If testing against a database other than Oracle, SQL Server, DB2, SAP, or MySQL, and using the Object/Data Retention option Refresh data using backup sets, backup sets might not be created resulting in a canceled test. |
114266 |
| Installation |
When installing in a Windows Vista 32-bit environment as a limited user (not Administrator), installation fails. Workaround: Install Benchmark Factory as Administrator. |
N/A |
|
Installation (uninstall) |
Beginning with release 7.1.1 in Windows Vista or later, if you have multiple editions of Benchmark Factory installed side-by-side (e.g., a Beta and a Commercial edition), and then uninstall one edition, you might encounter the following error when attempting to run a job: " This object does not represent an OLE control" Workaround: Do one of the following:
|
ST115773 |
| Integration with other Toad products |
Applies in Toad for Oracle suites (12.5 or later): When the database connection or the SQL statement contains Chinese characters, attempting to send SQL from SQL Optimizer for Oracle to Benchmark Factory results in an error. This issue only occurs when Spotlight on Oracle is launched prior to the other applications mentioned here. Workaround: Always launch Benchmark Factory and SQL Optimizer for Oracle prior to launching Spotlight on Oracle if sending SQL from SQL Optimizer to Benchmark Factory. |
N/A |
| Licensing |
License keys compatible with Benchmark Factory 5.8.1 (or earlier) are no longer supported. See Product Licensing for instructions on how to upgrade an earlier-version license key. |
N/A |
| Licensing | If your current Benchmark Factory license key is a 21-digit format (x-xxxxx-xxxxx-xxxxx-xxxxx) and you want to use the 64-Bit installation, you will need to request a new license key. Please obtain the new key before installing the new release. To obtain a new key, please refer to the Licensing Assistance page on our support portal: https://support.quest.com/licensing-assistance. | N/A |
| Oracle Database Privileges | In Oracle 12c, the RESOURCE role no longer includes the UNLIMITED TABLESPACE system privilege. Therefore, Benchmark Factory users with the RESOURCE role must also be granted the UNLIMITED TABLESPACE privilege. |
N/A |
| Repository |
In a new installation (not an upgrade) of Benchmark Factory 7.2 or later, a Standard (Limited) user may encounter a Repository error when attempting to launch the application. The issue involves the SQLite database that is used as the default database in a new installation of Benchmark Factory 7.2 or later. Workaround: Do one of the following:
|
BMF-202 |
| Repository | If you created a Data Source Name for the BMF MySQL ODBC Driver in a previous release (6.9.3 or 7.0), attempting to edit this DSN results in an error. |
N/A |