Spotlight on Oracle 10.10 - Release Notes

Brackets not permitted in installation path names: Prior to Oracle 11g, the Oracle client does not support an installation path for Spotlight that contains the brackets "(" and ")". For example: C:\Program Files (x86)\...

Tip: For 64-bit operating systems we recommend not installing Spotlight on Oracle (Windows 32) in "c:\Program Files" since Windows re-directs this back to "C:\Program Files (x86)".

From a Spotlight on Oracle connection where the Oracle database is installed on a Windows operating system, from drilldown Operating System | OS Processes right click on an Oracle process and select Top Sessions to open the Top Sessions drilldown - results in error: Unable to file Oracle session for this process.

• Connection lost during initial connection: Spotlight will not reconnect if the connection is lost while Spotlight is initializing its connection.
• Connection lost while viewing alert log: Spotlight will report a number of errors if the disconnection and reconnection occurs while the Alert Log drilldown is open.

• Microsoft Windows: "Waiting" graph within CPU Utilization chart is always zero. The "Waiting" graph in the CPU Utilization chart of the Operating System drilldown is included specifically to collect the relevant data from Unix systems. Windows systems do not provide the same information, and so the value of this metric is always zero for Windows systems.

• Unix AIX: Some Disk I/O and virtual memory statistics are not reported. These may include disk I/O dataflows on the home page, disk read and write statistics on the Activity Summary page of the Activity drilldown, and metrics such as % User, % Privileged, Work Set (MB), Page faults/s, Handles, Threads, Start Time, and % Processor.

• Unix HP-UX: Missing Total RAM value on Spotlight on Oracle RAC home page: The value for Total RAM can be displayed only if the cstm utility is installed and can be run on the HP-UX server.
• Unix HP-UX: Some Disk I/O and virtual memory statistics are not reported. These may include disk I/O dataflows on the home page, disk read and write statistics on the Activity Summary page of the Activity drilldown, and metrics such as % User, % Privileged, Work Set (MB), Page faults/s, Handles, Threads, Start Time, and % Processor.
• Unix HP-UX: Abnormal CPU figures. In some cases on HP-UX Unix, CPU user time has been diagnosed at -1% to -2%, which pushes the idle time to 101% to 102%. This appears to be a bug in the HP-UX operating system, which will result in abnormal Processor Utilization figures in Spotlight on Oracle.

• Linux: SUSE Linux — virtual memory statistics. Statistics that may not be reported include % User, % Privileged, Work Set (MB), Page faults/s, Handles, Threads, Start Time, and % Processor.
• Linux: Red Hat Linux — invalid network metrics reported by some versions of Red Hat Linux (2.4 kernel). Because of a known formatting bug in the output for the Unix netstat command, Spotlight may report inaccurate values for metrics (Network flows and Network charts, for example) that rely on netstat.

• Solaris Unix: Some virtual memory statistics are not reported. These include % User, % Privileged, Work Set (MB), Page faults/s, Handles, and Threads
• Solaris Unix: No data for Paging and Scan Rates in Solaris 9 systems. When Spotlight returns no data in the Paging and Scan Rates chart of the OS Performance page in the Operating System drilldown for a Solaris 9 system, you should apply the patch 116543-01 to that system.

Before doing this, you must configure the Solaris server under investigation. To do so:

1. Locate the SoO_DTrace_scripts.tar archive package in the Plug-ins\SoO sub-folder of the Spotlight installation folder.
2. Unpack the contents of the archive to a directory on the server. Ensure that the unpacked files have execute permission.
3. In the .profile of the account that will be used to run O/S commands from Spotlight, define a QUEST_SOO_HOME variable that points to that directory on the server. (In previous Spotlight versions, the variable was QUEST_HOME.)

4. Provide that user with DTrace authority by adding the following line to /etc/user_attr:

   <<username>>::::defaultpriv=basic,dtrace_proc,dtrace_kernel

For more information, see the DTrace User Guide.

Restarting Spotlight to change selected Oracle Home: Normally, you can change the Oracle Home setting for Spotlight on Oracle whenever the selected home is NOT being used for an active Spotlight connection.

In certain circumstances, however, disconnecting from a database causes the termination of one or more executing queries. When this happens, you can select a new Oracle Home in a connection profile (or in the Spotlight on Oracle Options window) ONLY AFTER Spotlight itself has been restarted.

Unable to create or open a Spotlight connection because of missing PATH entry for Oracle client:

• You cannot create a Spotlight on Oracle connection via a specified Oracle client if that client's home directory is not in the Windows PATH environment variable.
• Spotlight may not connect successfully to an Oracle database if the home directory for the specified Oracle client has been removed from the PATH variable (as can happen, for example, when the PATH entry has been removed by the Oracle Universal Installer).

Oracle error ORA-1031 when connecting to an Oracle database: To resolve the error ORA-1031: insufficient privileges in table Parameter when you try to connect to an Oracle database, add the following entry into the SQLNET.ORA file on the Spotlight client machine:

SQLNET.AUTHENTICATION_SERVICES= (NONE)

Note: This error is seen most often for the Parameter table, but can also occur for others.

Information shown for a single node only in Alert Log and Archive Log diagnostics: In a clustered environment where the global_names parameter is set to TRUE, alert log and archive log diagnostics within Spotlight on Oracle RAC will display data for one node only. To work around this, add the entry

alter session set global_names=FALSE

to the SPOTLIGHT.ORA file in the Plug-ins\SoO sub-folder of the Spotlight installation folder. This will change the global_names parameter for the relevant Spotlight session.

In an Oracle 12c Data Guard environment, in some circumstances the Transport Process Running alarm fails to fire when log shipping is stopped.

Spotlight on Oracle expects the Oracle process to stop when log shipping is stopped, but in some circumstances the Oracle process continues to run (idle).

Plugged database sessions sometimes don't show in the Top sessions drilldown.

To fix the issue, run alter system flush shared_pool

When the Spotlight connection is to the container database, the Container drop down menu may not populate with all the plugged in databases. This applies to the Activity | Summary and Pluggable Databases | Resource Usage drilldowns.

To fix the issue, run alter system flush shared_pool

On Oracle RAC, run alter system flush shared_pool on every node of the RAC.

When the Spotlight connection is to the container database:

• There may be No Data displayed on the IO Throughput Requests and IO Latency charts in the Activity | Summary drilldown when the selected Container is the plugged in database.

When the Spotlight connection is to the plugged in database:

• There will be No Data displayed in the I/O | Tablespaces and I/O | Datafiles drilldowns, and the I/O | Summary drilldown when File Type is selected.
• There will be No Data displayed on the IO Throughput Requests and IO Latency charts in the Activity | Summary drilldown.

The problem is v$iostat_file is empty on the pluggable database.

When the Spotlight connection is to the container database:

• There may be negative values displayed in the Activity | Summary drilldown when the selected Container is the plugged in database.
• There may be negative values displayed in the Pluggable Databases drilldown.

• There may be a delay in populating the Container drop down menu with all the plugged in databases on the Activity | Summary drilldown.

When the Spotlight connection is to the plugged in database:

• There may be negative/incorrect (self-correcting spiked) values displayed throughout the product including the Activity | Summary drilldown, the SQL & Application Workload | Parse Activity drilldown Parse Ratios and Parse Wait Times charts, and the SQL & Application Workload | Sort Activity | IO Wait Times chart.

The problem is with Oracle counters not always incrementing correctly for plugged in databases.

After heavy use of the cellcli utility, CPU utilization on the cell host may hit 100% and stay there. The runaway processes appear to be java and cellsrv. To fix, identify the cell host on the Exadata | Cell Balance drilldown then enter the following Unix command on the appropriate cell host:

cellcli -e "alter cell restart services MS"

• Program Name (Spotlight Console) - This may occur while monitoring Oracle Exadata. Exadata Storage Cells are generally only accessible to Spotlight via SSH port forwarding. If requested, allow Spotlight to communicate over the Network via SSH Port Forwarding.

REXEC and SSH protocols and Unix commands: Spotlight uses the REXEC or SSH protocols to submit various Unix commands (for example, netstat, vmstat, iostat, sar) to observe system activity. These commands must be accessible to an REXEC or SSH session for Spotlight to be able to observe Unix activity. If these commands are not located in the search path for REXEC or SSH sessions, Spotlight displays an error.

'iostat' command may not be installed.

Spotlight on Unix works with full functionality only when the iostat command is installed on the server under diagnosis. This may NOT be true for all Unix systems. If iostat is NOT installed, Spotlight may display an error message such as 'iostat: command not found'. If you encounter this error, you can:

Install the iostat command on the server. (On Red Hat Linux, this command is included in the sysstat package.)

Logging on as "root".

When connecting to a Unix OS, you need to do so from a logon other than "root". Remote login as "root" has been disallowed for security reasons.

Public-key cryptography available for SSH2 connections only:

Spotlight can now use public-key cryptography (via DSA or RSA) to communicate securely with Unix systems, but this functionality is made available only under the SSH2 protocol.

Password authentication for SSH connections that do NOT use public-key cryptography.

To allow Spotlight to make SSH connections to any Unix or Linux hosts that permit SSH connections, you may need to alter the PasswordAuthentication configuration item in the sshd_config file. Set the value of PasswordAuthentication to yes.

Specific implementations of AIX may limit Spotlight's ability to collect data. These include:

• The Procs Swapped button in the Memory panel on the home page is not supported.
• The Physical Memory metric in the Process Details sub-page of the Processes and Services drilldown is not supported.
• I/O data unavailable before AIX 5.3. For versions before AIX 5.3, disk I/O flows on the home page are disabled, as are disk read and disk write charts in drilldown pages.

Specific implementations of HP-UX may limit Spotlight's ability to collect data. These include:

• Missing Physical RAM and Virtual Memory values on Spotlight home page: Values for Physical RAM and Virtual Memory can be displayed only if the cstm utility is installed and can be run on the HP-UX server.
• Free RAM values may be incorrect. Spotlight uses the "free" column in vmstat(1) to show the amount of free RAM. This is not strictly correct for HP-UX servers, but it is the only figure available that is in any way useful.
• I/O data unavailable. Disk I/O flows on the home page are disabled, as are disk read and disk write charts in drilldown pages.
• The Physical Memory metric in the Process Details sub-page of the Processes and Services drilldown is not supported.

Specific implementations of Linux may limit Spotlight's ability to collect data. These include:

• The Procs Swapped button in the Memory panel on the home page is not supported.
• For versions of Linux based on the 2.4 kernel, Spotlight provides no information in the Waiting data series of the CPU Utilization chart on the Activity page of the Activity Summary drilldown.
• When running with the latest version of the Linux 2.4 kernel, Disk Activity metrics on the home page return invalid results due to erroneous statistics being returned from iostat in the sysstat package. To overcome this problem, the OS should be updated to a 2.6 kernel.
• In some Linux versions, Spotlight reports the virtual memory value of multi-threaded processes incorrectly. Each thread is reported as a separate process in the Processes drilldown, and the Virt Mem (MB) column is reported as the sum of the virtual memory values for all threads in the process.
• With some older versions of Linux based on the 2.4 kernel, the Disk Reads and Disk Writes charts may include negative values. These charts are based on output from iostat, some versions of which do not return the data required by Spotlight. To resolve this issue, upgrade to a more recent version of Linux.
• The Cache chart in the Memory page of the Activity Summary drilldown is not supported.

Unix OS commands. Unix OS commands need version 5.6 of perl or later. If this is not available you should revert to using a shell script to obtain operating system data for this server. To do this, edit the file ScriptConfig.ini (in the ..\Plug-ins\DatasourceUNIX sub-folder of the Spotlight installation folder) according to the instructions supplied in the file.

Problems with diagnosing Solaris 9 machines.

A number of Spotlight home page components will not be populated when Spotlight diagnoses lightly-loaded Solaris 9 machines. This is caused by the output from the vmstat command regularly recording a "State Change" event, which in turn resets values for the vmstat command. In this case, the CPU spinner, User % gauge, System % gauge, Queue Length gauge, Free RAM label, Blocked Processes button, Swapped Processes button, Paging In and Paging Out flows, and Swapping In and Swapping Out flows will appear grayed out.

Error Access denied on the Spotlight on Windows Home Page Open Sessions panel and the current user is in the Administrators group on both the local and the remote machine.

To fix this issue: Enable the AllowRemoteRPC property in the registry of the monitored system.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server] "

AllowRemoteRPC"=dword:00000001

or import the registry file AllowRemoteRPC.reg

Resolution: Ensure the user on the Spotlight Client has administrator privileges on the windows server you want to monitor.

Remote Registry service. The Spotlight machine and the machine being diagnosed must have the Remote Registry service running. If you attempt to connect to a machine that has the Remote Registry service stopped, Spotlight will return the following error message:

No Windows server called <machine name> could be found on the network. If the address is valid then the Remote Registry service may not be running on the remote server.

Make sure the Remote Registry service is running on the Spotlight machine and the machine you are connecting to.