A false alarm or no alarm is triggered when an Oracle tablespace runs out of space or the tablespace metrics in the Oracle agent look incorrect.
The rule "DBO - Tablespace Used Percentage" may not fire.
An Oracle tablespace ran out of space, but no alarms have been raised.
CAUSE 1:
For tablespaces with autoextensible tablespaces, OS monitoring is required for the DBO Tablespace Usage to be accurately calculated.
If Foglight cannot collect the OS data for an autoextendable tablespace, the alert will not function because it does not have enough information to calculate to result.
CAUSE 2:
The parameter "Is ASM RAC" has been changed from "True" to "False" via "Administration | Agents | Agent Status". This causes duplicate topology objects and the needed table space metric will not be submitted to the correct "new" topology object. The rule is still reading from the old topology object which is empty now.
CAUSE 3:
This issue is caused by the maximum number of SSH sessions has been exceeded for this host. It resulted in the failure of the script in the DBO_OS_File_System_Disk collection. The failure caused some disk related metrics were not submitted to FMS, then the false alert was triggered.
CAUSE 4:
Missing ASM agent
CAUSE 5:
Alarm threshold configuration
CAUSE 6:
Rule firing configuration
CAUSE 7:
Tablespace offline data collection only runs every hour by default and did not capture this event.
TO TROUBLESHOOT ORACLE TABLESPACE ALARM ISSUES
Check the collection parameters from the Error in the Agent log to confirm that the collection ran
Check that the ASM agent and OS collection are both running (when the datafile is autoextensible on the disk or ASM)
Check the datafile number in the database using SELECT COUNT(*) FROM DBA_DATA_FILES;
Check the max row limit in the collection (this can cause some datafile information to not be retrieved from the database)
Set the FglAM or database agent to enable debug log writing to check the results for the "DBO - Tablespace free space ASM" with the result from SELECT * FROM DBA_DATA_FILES; to ensure that all datafiles are included in the calculation.
RESOLUTION 1:
Turn off the autoextend capability on the Oracle tablespace or configure and activate OS monitoring (IC agent and Oracle agent) for the Oracle instance.
RESOLUTION 2:
Drop the agent and recreate it with the correct settings.
Please Note
The DBO cartridge is querying ASM views that resides on monitored instance to retrieve basic information about ASM I/O activity, however the data is not always reported correctly due to multiple Oracle ASM related bugs.
The following are the Oracle ASM known bugs:
RESOLUTION 3
Increase the maximum number of SSH sessions (MaxSessions) on the Agent Manager host.
RESOLUTION 4
If it is extended to ASM, then the ASM agent must be set up to collect ASM related metrics. Also, run the following SQL statement: SELECT file_name, autoextensible, status FROM dba_data_files the file_name will identify if the data file is in os or ASM ( if the datafile is in ASM it will start with "+") and the autoextensible will identify if os or asm monitoring is needed or not.
RESOLUTION 5
The calculation is correct but due to the size of the DB the thresholds needs to be adjusted because it is based on percentage and not on numbers. When having 4 TB free space, 10% remaining free space is a high number, but as the rule is based on percentage an alarm will be triggered even there is still a lot of space available.
RESOLUTION 6
Expand Administration | Rules & Notifications | Rules | click on "Old Manage Rules" | Search for Rule Name "DBO-Tablespace Used Percentage" and click on it | check if "Fire Action if 3 consecutive are true | click on on the Save button.
RESOLUTION 7
By default in the Oracle cartridge the offline tablespace data collection is set to 3600 seconds / 1 hour. If the issue happen between 2 collections and the DBA solved the issue before the next collection happen, the issue will not be captured.
To avoid such situations, change the collection interval to a appropriate time interval, for example 15 minutes.
To do so, follow the steps below:
