Spotlight alarms indicate when the Oracle RAC exceeds acceptable performance thresholds.
Problems in a single instance may not always affect the operation of the cluster as a whole. In such cases, Spotlight on Oracle RAC can display an alarm in the relevant instance panel without reporting a specific Spotlight on Oracle RAC alarm.
However, if a problem in a single instance DOES affect the operation of the whole cluster – or is repeated in multiple instances so that the combined effect is significant – Spotlight on Oracle RAC reports the problem as an aggregated alarm.
These alarms are displayed on the Spotlight on Oracle RAC home page – usually in the Global Database heading of the Interconnect and Global Status panel, or on the individual components of the IO Subsystem panel.
Following are the alarms specific to Spotlight on Oracle RAC.
|Archive Critical Failure alarm||An Archive Critical Failure alarm indicates that the number of working archive destinations for the database cluster is about to fall below the minimum number needed to archive Oracle redo logs successfully. When you see this alarm, you should make space in the archive log destination by purging – or by backing up to an offline medium – any archived files that are not immediately needed.|
|Archive Destination Failed alarm||Spotlight raises an Archive Destination Failed alarm when at least one of the archive destinations specified for redo log files has become full. See Archive Destination Failure alarm below.|
|Archive Destination Failure alarm||An Archive Destination Failure alarm when at least one of the archive destinations specified for redo log files is about to become full. When you see this alarm, you should make space in the archive log destination by purging it, or by backing it up to an offline medium.|
|ASM Status alarm||When a Spotlight connection is lost, Spotlight automatically tries to re-establish the connection. The ASM Status alarm indicates that Spotlight is re-establishing its connection to the ASM diskgroup for the Oracle RAC cluster.|
|Availability alarm||This alarm indicates that the instances participating in the cluster are NOT all currently available for access. The database administrator should determine why the instances are unavailable, and restart them after taking the required actions.|
|(Global) Cache Miss Rate alarm||See Miss Rate alarm below.|
|Cluster Latency alarm||Spotlight raises the Cluster Latency alarm when cluster latency from all causes (interconnect, GES, LMS overhead) exceeds a specified threshold. Cluster latency is defined as the average elapsed time from when a requesting instance requests a block of data to the time when the requesting instance receives the block.|
|Cluster Overhead alarm||
Spotlight raises the Cluster Overhead alarm when the percentage of time dedicated to cluster co-ordination activities exceeds a specified threshold. Cluster overhead is the amount of time spent waiting for cluster related activities as a percentage of the total time spent waiting for all activities.
|Corrupt or Lost Blocks alarms||
Spotlight raises separate alarms for corrupt and lost blocks when a specified percentage of the blocks transferred across the interconnect are corrupted or lost during global cache consistent read requests.
High levels of lost or corrupt blocks should not happen in normal circumstances. If these alarms fire at all, it indicates either very serious overloading of the Oracle RAC cluster, or software or hardware faults in the cluster.
|Instance Failed alarm||Spotlight raises an Instance Failed alarm when a specified instance has failed because of an Archiver error. See Archive Critical Failure alarm above.|
|Load Imbalance alarm||
Spotlight raises the Load Imbalance alarm when:
Cluster balance is a fundamental indication of the health and productivity of an Oracle RAC environment. If load is not reasonably evenly distributed across the instances in a cluster, then the DBA is likely to regret having chosen a clustered environment.
Spotlight on Oracle RAC calculates balance by recording a user-definable metric of load over time on each instance (the default is Logical Reads, but you can switch this to CPU usage or Physical Writes via the Spotlight on Oracle RAC Options window) and calculating the relative balance of load across the cluster.
In the normal course of events, there are always minor variations in load between instances within the cluster, so Spotlight uses a statistical technique to determine if a variation is significant.
Note: The activation of specific Oracle Services can unbalance the workload across the Oracle RAC cluster. In these cases, Spotlight triggers the Load Imbalance alarm only when a more significant threshold is reached.
|Miss Rate alarm||
This alarm, more specifically known as the Cache Miss Rate alarm or Global Cache Miss Rate alarm, is raised when the global cache local miss rate exceeds a specified threshold. This rate is the percentage of time that a block required for a logical read is located in an inconsistent mode on ANOTHER instance in the cluster and must be transferred.
|Average Percentage CPU Load Across The Exadata Storage Cells alarm||
The Average Percentage CPU Load Across The Exadata Storage Cells alarm occurs when the total CPU utilization of the Exadata Storage Servers exceeds a threshold. One or more of the Server CPU may encounter a large number of requests, or you may have un-tuned SQL, which uses excessive amounts of CPU.
|Maximum Percentage CPU Load on The Exadata Storage Cells alarm||The Maximum Percentage CPU Load on The Exadata Storage Cells alarm occurs when the CPU utilization of one or more Exadata Storage Servers (Cells) exceeds a threshold.. One or more of the Server CPU may encounter a large number of requests, or you may have un-tuned SQL, which uses excessive amounts of CPU.|
|Single Block Read alarm||The Single Block Read alarm is raised if the average time taken to complete a request for a single block from the Exadata Storage Servers exceeds a given threshold. The source of the latency may be any database using the Exadata Storage Servers.|
|Cell Balance alarm||The Cell Balance alarm is raised when there is an unequal distribution of work-load between the Exadata Storage Servers.|
|Exadata Storage Server (Cell) Not Connected||Spotlight has lost its connection to at least one Exadata Storage Server, and is unable to re-connect. Check the status of your network connection. Check the status of the hosts of each Exadata Storage Server.|
|No Nodes Available alarm||This alarm indicates that Spotlight has lost its connection to the Oracle RAC cluster, and cannot re-connect to any of the nodes in the cluster. Check the status of your network connection, your host machines, and your database instances.|
Note: Spotlight on Oracle Data Guard alarms occur in the Alarm Log of the Primary Database in the Oracle Data Guard environment. See Spotlight on Oracle Data Guard Alarms for more information.