Release Notes
Benchmark Factory® 7.6.1
Release Notes
Thursday, June 15, 2017
About Benchmark Factory
Quest Software is pleased to make available Benchmark Factory® 7.6.1 to meet your database workload testing needs. Benchmark Factory is a database performance and code scalability testing tool that simulates users and transactions on the database and replays production workload in non-production environments. With Benchmark Factory developers, DBAs, and QA teams can validate that their databases will scale as user loads increase, application changes are made, and platform changes are implemented.
Benchmark Factory is available for Oracle®, SQL Server®, IBM® DB2®, SAP®, MySQL, and other databases via ODBC connectivity. Benchmark Factory:
- Simulates thousands of concurrent users with a minimal amount of hardware.
- Simulates database production workloads.
- Reports tests results with built-in reporting features.
Upon completion of a test execution, all test results are collected and stored in the repository for data analysis and reporting. Benchmark Factory collects a variety of informative and detailed statistics that include:
- Overall server throughput (measured in transactions per second/minute or bytes transferred).
- Detailed transaction executed statistics by individual agent workstations producing a load.
New Features
This release of Benchmark Factory includes the following new features and enhancements.
Benchmark Factory 7.6.1
Benchmark Factory 7.6.1 is a maintenance release and includes several resolved issues and enhancements. See Resolved Issues and Enhancements for a complete list.
Benchmark Factory 7.6
Connections
Microsoft SQL Server Native Provider
You can now use a SQL Server native connection in Benchmark Factory to connect to a SQL Server database. The SQL Server native provider is included in the Benchmark Factory installation.
- This new native provider delivers improved connection performance for SQL Server database connections.
- To use the SQL Server native provider, install the version of Benchmark Factory (either 32-bit or 64-bit) that matches your operating system. This ensures that the version (bitness) of the SQL Server provider also matches your operating system.
Note: For a list of operating systems that support the SQL Server native provider distributed with Benchmark Factory (native client 11.0), see https://msdn.microsoft.com/en-us/library/cc280356(v=sql.120).aspx. If your client operating system is not supported, use ODBC connectivity.
REST API
This release of Benchmark Factory includes a REST API which will allow users to automate certain tasks and functionality in Benchmark Factory. Use a script, command-line tool, or custom application to interface with Benchmark Factory and to automate your load testing tasks. The API uses HTTP protocol and returns information in XML format.
In this release, the REST API supports the following functionality:
- Create, run, and edit Benchmark Factory jobs
- Create and edit connections
- Import and replay Oracle and SQL Server captures
Using the Benchmark Factory REST API
HTTP Request Methods—Benchmark Factory REST API uses the four main HTTP methods: GET, POST, PUT, and DELETE.
Response Format—Information is returned in XML format.
Continuous Integration
- Use the REST API to implement performance testing by Benchmark Factory and incorporate it into your Continuous Integration (CI) process.
General
Adding Bind Parameter Values for Prepared Statements
It is now much easier to add bind variable or parameter values for prepared statements.
-
In the Transaction tab, when adding a new statement or editing an existing statement, the Add bind parameter button is now enabled for prepared statements. Click this button to add bind parameter values for the selected SQL.
Specify Benchmark Factory Console Port Number
You can now specify a TCP/IP port for the Benchmark Factory console.
- Select Edit | Settings | General to open the Settings dialog.
- Then enter a port number in the TCP/IP Settings. The default value is port 4568.
Standard Benchmark Testing
Teradata Database Benchmark Testing
- This release includes enhancements to the support for standard benchmark testing on Teradata databases.
PostgreSQL Database Benchmark Testing
- This release includes enhancements to the support for standard benchmark testing on PostgreSQL databases.
Replication Benchmark Test
- You can now edit the precision of the FLOAT data type when customizing table options in a Replication Benchmark test.
Capture and Replay
Easily Open Capture Scenario Wizard from Tools Menu
You can now open the Capture Scenario Wizard directly from the Tools menu.
-
Select Tools | Capture Workload. Then select either Oracle or SQL Server as the capture type.
Discontinued - Import from Foglight PA
The option to import a workload from a Foglight Performance Analysis Repository when creating a new Capture and Replay test is no longer available beginning with this release of Benchmark Factory.
Settings
Specify a Threshold for Memory Usage
You can now specify a memory usage threshold. If memory usage exceeds the percentage you specify, Benchmark Factory stops the job.
- To specify a threshold, go to Edit | Settings | Error Handling and enter a percentage.
- You can disable/enable this memory usage check.
Error Handling Page in Settings Dialog
The error handling options have been moved to Edit | Settings | Error Handling in the Settings dialog.
Run Reports
Improved Performance When Importing Large Test Results
- This release includes markedly improved performance when importing a very large Test Result file.
Resolved Issues
Resolved Issues and Enhancements
The following is a list of issues addressed and enhancements implemented in this release of Benchmark Factory.
Resolved Issues in 7.6.1
| Feature | Resolved Issue | Defect ID |
|---|---|---|
| Benchmark Testing | New Job Wizard: When a Delete Objects step/test is included in a job, the test is now listed on the Summary page of the Job Wizard. | BMF-1768 |
| Benchmark Testing | Corrected an issue in which tables fail to load when the workload consists of only the "Create Objects" step and the "Distribute load using Benchmark Factory agents" option is selected. | BMF-2066 |
| Capture and Replay | SQL Server: Importing a SQL Server capture after moving the capture directory now works as expected. | BMF-1796 |
| Connections | Increased security related to user passwords. Contact Quest Support for more information. | BMF-1943 |
| Custom Replay Test | In a custom replay test, when attempting to add all Stored Procedures, cancelling the process or repeating the process now works as expected. | BMF-1856 |
| Licensing | Corrected an issue that caused Benchmark Factory to close unexpectedly after the user closed the Licensing dialog and the product had an expired license. | BMF-1709 |
| REST API | Corrected an issue in which an update to the user load through the REST API failed even though it returned a code of 204. | BMF-1722 |
| Run Reports | Corrected an issue in which the export/import of test runs caused lost data. | BMF-2005 |
| Settings | Corrected an issue that caused Benchmark Factory to close unexpectedly while running a job after the user changed the Result Logs directory path in Edit | Settings | General. | BMF-1977 |
| User Interface | Add SQL Transaction dialog: Corrected an issue that unexpectedly displayed the contents of the SQL tab even though the Bind Parameters tab was selected after resizing the dialog. | BMF-1724 |
Resolved Issues in 7.6
| Feature | Resolved Issue | Defect ID |
|---|---|---|
| Benchmark Testing | Teradata: Improved support for standard benchmark testing against a Teradata database. | BMF-1254 |
| Benchmark Testing | Teradata: Corrected an issue that caused a "Duplicate row" error when loading tables in a Scalable Hardware benchmark test against a Teradata database. | BMF-222 |
| Benchmark Testing | PostgreSQL: Improved support for standard benchmark testing against a PostgreSQL database. | BMF-1265 |
| Benchmark Testing | In a benchmark test, if you choose "Refresh data using backup sets" as the data retention method but the scale factor of the backup sets does not match the scale factor selected in the test, Benchmark Factory now displays a friendly warning with instructions on how to proceed. | BMF-1369 |
| Benchmark Testing | For a TPC-D test, the Custom Creation SQL page displays the region_idx1 index as expected. | BMF-1509 |
| Benchmark Testing | User Interface: When attempting to run more than one job and the total virtual users requested exceeds the number of licensed virtual users currently available, a friendly warning instructs the user to reduce the number of virtual users / user load or reduce the number of jobs running. | BMF-825 |
| Benchmark Testing | Enhancement: You can now edit the precision of the FLOAT data type when customizing table options in a Replication Benchmark test. | BMF-1152 |
| Benchmark Testing | Replication Benchmark Test: Corrected an issue that caused an ORA-20000 error when re-running a test after changing a table option (data type precision) and switching retention from "Recreate objects…" to "Refresh data using inserts." | BMF-1250 |
| Capture and Replay | Enhancement: Added the ability to import and replay a captured workload using automation / command line interface. | BMF-1581 |
| Capture and Replay | Corrected an issue that, under certain conditions during the import of a SQL Server capture, resulted in an extra transaction with no SQL being added to the replay. | BMF-929 |
| Connections | Corrected an issue that prevented data from loading correctly when using a SQL Server DB-Library connection. | BMF-1338 |
| Data Migration Wizard | Corrected an issue which disabled the Next button on the "Choose a Data Source" page in the Migration Wizard even when the data source connection information was valid. | BMF-963 |
| Documentation | Added clarification in the product documentation (online Help) as to when and why the AWR reported elapsed time and the replay time may not match the specified capture time. | BMF-118 |
| General - Settings | Enhancement: You can now instruct Benchmark Factory to stop a job when memory usage exceeds a limit you specify (Edit | Settings | Error Handling). You can also disable the memory check. | BMF-856 |
| General - Settings | You can no longer edit the repository credentials (user name and password) through Edit | Settings | Repository while a job is running. | BMF-1229 |
| Installation | Corrected an issue that caused an error when installing Benchmark Factory using the .msi and a custom transform file. | BMF-1157 |
| Integration with other Toad Products | Corrected an issue in which the Tools | Spotlight menu item was disabled in Benchmark Factory even if the Spotlight application was installed. | BMF-969 |
| Integration with Other Toad Products | Corrected an issue that resulted in a "No scenarios to import" error when sending an Oracle trace file to Benchmark Factory from Toad for Oracle 12.8 or later. | BMF-972 |
| Run Reports | Enhancement: Added an enhancement that markedly improves performance when importing a very large Test Result file. | BMF-92 |
| Scalability Testing | After completing a Scalability test, the Results page now displays the Transactions vs User Load graph. | BMF-1163 |
| Test Stored Procedures for Scalability | Corrected an issue which caused an error if the user selected a schema other than the default schema on the "Select Stored Procedures" page when selecting the stored procedures to test for scalability. | BMF-1009 |
| User Interface | Corrected several text display issues encountered in the New/Edit Job Wizard when non-English locale settings were specified in Windows. | BMF-1173, BMF-1185 |
| User Interface | Corrected an issue that prevented the scroll bar from displaying correctly in some cases in the Results page. | BMF-918 |
Known Issues
The following is a list of issues known to exist at the time of this release.
| Feature | Known Issue | Defect ID |
|---|---|---|
| Capture and Replay | Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle 12c pluggable database (PDB). |
N/A |
| Capture and Replay |
Oracle workload: In Benchmark Factory 7.2 or later, if you attempt to use Oracle capture files created in an earlier version of Benchmark Factory (6.9.3 or earlier) and .NET Framework 2.0 is not installed, Windows may prompt you to install it (for example, in Windows 8 or later). Workaround: Install .NET Framework 2.0 (3.5). |
N/A |
| Capture and Replay | Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle RAC system. |
N/A |
| Capture and Replay |
Issue: If you start a trace file capture using the current version of Benchmark Factory and then start another trace file capture using version 7.2 (or earlier) on the same database, the earlier version of Benchmark Factory cannot detect the current version performing a capture. This can result in an invalid capture. Solution: For best results, do not perform trace file captures using both the current and an earlier version of Benchmark Factory concurrently on the same database. |
N/A |
| Capture and Replay |
SQL Server workload: After capturing activity from a user logged-in using Windows authentication, attempting to replay the capture using the ODBC profile results in the following error: "Could not obtain information about Windows NT group/user '<group/user>', error code 0x5." Workaround: Use a domain account to start SQL Server service and make sure the account can connect to the Active Directory of Domain. |
N/A |
| Capture and Replay |
Oracle workload: In an Oracle capture and replay, Benchmark Factory cannot generate the Statspack report if the local replay directory is mapped to drive "B." This is because Benchmark Factory must have drive B open to execute a background .bat file during oracle capture and replay. Workaround: Do not map your local replay directory to drive "B" when running an Oracle capture and replay in which you chose to generate the Statspack report. |
109221 |
| Capture and Replay | In an Oracle 12c container database (CDB), Benchmark Factory cannot replay an existing workload that was captured in an earlier version of an Oracle database. In an Oracle 12c container database, a user name (common user) is prefixed with "c##" by default. |
N/A |
|
Capture and Replay |
Oracle Capture Using FGAC: When performing an Oracle Capture using the FGAC method, if the application has a trigger with a multi-schema select, additional transactions will be recorded. | N/A |
|
Connections - Teradata ODBC |
When creating or modifying a Teradata ODBC connection, if the Teradata DSN has incorrect connection information and you click Test Connection in the Connection dialog , Benchmark Factory might close unexpectedly. | BMF-1497 |
|
Connections - Transaction Wizard and IBM CLI Driver
|
When using the 64-bit version of Benchmark Factory, the Transaction Wizard stops working when using the IBM DB2 CLI Driver profile to connect the database which created objects using the ODBC Driver. |
N/A |
|
Connections - Microsoft Oracle ODBC Driver |
When using Microsoft’s Oracle ODBC driver for either testing or for Benchmark Factory repository access, there is an issue with the ODBC driver. It does not work with CLOB data. | N/A |
|
Connections - Oracle ODBC driver |
When using the Oracle 11g R2 Client and Oracle 32-bit ODBC driver in a Windows 64-bit environment, the Oracle 32-bit ODBC driver requires an 8-byte int data type. Workaround: Use the Oracle native client or Oracle 32-bit ODBC driver. |
N/A |
|
Connections - Microsoft SQL Server (Obsolete DB-Library) Connection |
If you run a benchmark test using a Microsoft SQL Server (Obsolete DB-Library) connection to SQL Server 2014, and you have the "Recreate objects and reload data every execution" option selected, you might encounter the following error: "DB-Library Message - Insert bulk is not supported over this access protocol" Workaround: Use the Microsoft SQL Server ODBC driver. |
N/A |
|
Connections - IBM CLI Driver |
A profile created with a DB2 ODBC data source may crash Benchmark Factory if the IBM DB2 client catalog becomes broken/corrupted. |
N/A |
|
Connections |
In Windows 10, if you open a New Connection dialog, create an SAP data source and then attempt to create a SQL Server data source, Benchmark Factory might close unexpectedly. |
BMF-555 |
|
General - Benchmark Factory Console |
Benchmark Factory does not support data types that are created by the user. | N/A |
| Industry Standard Benchmark Testing |
Background: This issue applies to benchmark testing using a SQL Server or SAP native connection when the Refresh data retention option is selected in the Test Options tab. Issue Description: In a test where the connection is lost while tables are loading, if you reconnect and attempt to rerun the job, Benchmark Factory might error with the following messages: “The Expected Total rows are not the same as the Actual Total rows,” or “a duplicate key was found” (when creating an index). The test is then canceled due to errors. Resolution: Recreate and run a new job against your selected database using the native connection. |
N/A |
| Industry Standard Benchmark Testing |
If testing against a database other than Oracle, SQL Server, DB2, SAP, or MySQL, and using the Object/Data Retention option Refresh data using backup sets, backup sets might not be created resulting in a canceled test. |
114266 |
|
Installation (uninstall) |
Beginning with release 7.1.1 in Windows Vista or later, if you have multiple editions of Benchmark Factory installed side-by-side (e.g., a Beta and a Commercial edition), and then uninstall one edition, you might encounter the following error when attempting to run a job: " This object does not represent an OLE control" Workaround: Do one of the following:
|
ST115773 |
| Integration with other Toad products |
Applies in Toad for Oracle suites (12.5 or later): When the database connection or the SQL statement contains Chinese characters, attempting to send SQL from SQL Optimizer for Oracle to Benchmark Factory results in an error. This issue only occurs when Spotlight on Oracle is launched prior to the other applications mentioned here. Workaround: Always launch Benchmark Factory and SQL Optimizer for Oracle prior to launching Spotlight on Oracle if sending SQL from SQL Optimizer to Benchmark Factory. |
N/A |
| Licensing |
License keys compatible with Benchmark Factory 5.8.1 (or earlier) are no longer supported. See Product Licensing for instructions on how to upgrade an earlier-version license key. |
N/A |
| Licensing | If your current Benchmark Factory license key is a 21-digit format (x-xxxxx-xxxxx-xxxxx-xxxxx) and you want to use the 64-Bit installation, you will need to request a new license key. Please obtain the new key before installing the new release. To obtain a new key, please refer to the Licensing Assistance page on our support portal: https://support.quest.com/licensing-assistance. | N/A |
| Oracle Database Privileges | In Oracle 12c, the RESOURCE role no longer includes the UNLIMITED TABLESPACE system privilege. Therefore, Benchmark Factory users with the RESOURCE role must also be granted the UNLIMITED TABLESPACE privilege. |
N/A |
| Repository |
In a new installation (not an upgrade) of Benchmark Factory 7.2 or later, a Standard (Limited) user may encounter a Repository error when attempting to launch the application. The issue involves the SQLite database that is used as the default database in a new installation of Benchmark Factory 7.2 or later. Workaround: Do one of the following:
|
BMF-202 |
| Repository | If you created a Data Source Name for the BMF MySQL ODBC Driver in a previous release (6.9.3 or 7.0), attempting to edit this DSN results in an error. |
N/A |