DBO - Initialization Parameter Modified fires when Oracle database is restarted (4380320)

The Foglight alarm "DBO - Initialization Parameter Modified" is triggered when a monitored Oracle database initialization parameter (such as those in `init.ora` or the server parameter file) changes. These parameters control essential aspects of database behavior, including memory allocation, process management, and logging. Any change to these parameters can significantly affect database performance, availability, or security. The alarm helps ensure that modifications are tracked and reviewed for compliance and operational integrity.

Common reasons for this alarm include:

• Planned changes: Database administrators intentionally update parameters for tuning, patching, or maintenance.
• Unplanned changes: Unexpected modifications due to application deployments, patch updates, or automation scripts.
• Transient values after reboot: After an Oracle database reboot, some parameters may momentarily appear as `null` or unset during initial Foglight collection cycles. This can occur if the agent polls the database before all parameters are initialized.
• Collection frequency: The interval at which the Foglight agent collects parameter data plays a critical role. A longer offline interval (e.g., 3600 seconds) increases the risk that the agent's first poll after a restart will occur before all parameters are fully set, making transient `null` values and false alarms more likely. More frequent (shorter) intervals can help catch real changes quicker but may also increase the chance of capturing transient states right after a restart.
• Application-managed parameters: Some parameters (like `mfa_smtp_port`) are not Oracle core initialization parameters. They may be set by applications or jobs after database instance startup. Immediately after a restart, queries to `v$parameter` may return `NULL` for such parameters, which are later set by the application or background process. Foglight will see this as a change (`NULL` → actual value) and trigger the alarm, even though this is expected startup behavior. 
• New or deprecated parameters: Oracle patches may introduce new parameters or deprecate old ones; Foglight’s agent collects metadata (including the `is_deprecated` flag) and will detect any new parameters or changes in deprecation status. This is handled automatically by the agent and does not require additional configuration.
  
  Alarm Behavior for New Parameters
  • First Appearance: The alarm will trigger once when the new parameter is first detected after a patch and database restart.
  • Subsequent Restarts: The alarm will not trigger for the same parameter on future restarts, unless the parameter’s value or status changes, or if the agent baseline is reset.
  • Exceptions: If the parameter is removed, modified, or appears as `null` due to timing/frequency issues after a reboot, the alarm may trigger again.

To address this alarm

1). Verify the Parameter Change

• Use SQL to confirm the current value:

  SELECT name, value FROM v$system_parameter WHERE name = '<parameter_name>';

• Compare the output before and after restarts to verify if a real change has occurred.

2). Review Change Management

• Ensure any modifications were planned and documented.
• Check for recent patches, deployments, or maintenance activities.

3). Use Rule Filters for Known Volatile Parameters:

• If the parameter is non-critical or prone to transient changes, add a rule filter to suppress alarms for it:
  1. Navigate to Administration | Databases in the Foglight FMS console
  2. Select the agent or agents in the database dashboard by enabling the checkboxes next to the agent name(s)
  3. Select Settings | Administration
  4. Select Alarms
  5. Select the DBO - Installation Parameter Modified alarm
  6. Click the Rule Filters tab in the panel below the alarm description
  7. Click the Add button
  8. Set the radio button to Select from the list below the parameters to exclude, then click the Add button
  9. Add the parameter (such as mfa_smtp_port)
  10. Click the Save Changes button

4). Optimize Collection Frequency

• Consider increasing the collection interval right after reboots to avoid transient values, or manually align agent polling with maintenance windows to minimize false positives for `null` parameters.  
• Review and adjust offline interval settings (e.g., avoid 3600 seconds if frequent restarts are expected).
• Collection interval tuning can only partially reduce the risk of transient or false alarms but is not a root-cause solution for baseline-loss-related alarms.

Fundamental limitation with optimizing the collection frequency
Adjusting sampling frequency (how often the agent collects data) does not prevent alarms if the agent's baseline is lost after restart. The first post-restart collection will always treat all values as "new" if no previous data exists.

What tuning the frequency can do

• Longer intervals (less frequent sampling): May reduce the likelihood of collecting data during a database or application restart, so you might see fewer transient alarms if restarts are rare and predictable.
• Shorter intervals (more frequent sampling): May increase the chance of catching parameters in a transient (`NULL`) state after startup, possibly resulting in more false alarms.  

5). Delay Foglight Agent Start After DB Restart 
   After restarting the database, wait until all parameters are initialized (including those set by background jobs or applications) before starting or resuming collections with the Foglight agent.

6). Review Parameter Initialization Timing:  
Work with DBAs/application teams to document which parameters are set after startup and how long initialization typically takes.