DBSS - SQL Server Fatal Hardware Error (severity level 24) alarm fires for Error 823 in the SQL Server Error Log (4380435)

An alarm for the DBSS - SQL Server Fatal Hardware Error (severity level 24) rule fires for Error 823. What does this error indicate?

SQL Server Error 823 is a fatal system-level error that occurs when the database engine encounters an I/O failure during a read or write operation. This error may be triggered by a query or background process attempting to access a file that is missing, corrupted, or inaccessible. The error threatens database integrity and must be addressed immediately.

Additionally, the SQL Server Error Log may show a backup failure due to an I/O device error (Error 1117), which may be related to the same underlying issue.

Error 823 typically arises due to issues in the underlying storage system. Common causes include:

• Missing or deleted database or log files
• Disk corruption or hardware failure
• File system inconsistencies
• Driver or firmware issues
• Concurrent access conflicts or locked files

The error may also appear during backup operations if the system fails to flush I/O to the backup device.

Note: This error may also be caused by a missing .xel file used for Extended Events analysis in the SQL Server agent's Deadlock collection. Refer to Knowledgebase article 4379013 for details.

To resolve SQL Server Error 823, follow these steps:

1. Run DBCC CHECKDB on the affected database to check for consistency errors.
2. If errors are found, restore the database from a valid backup or use DBCC CHECKDB (DB_NAME, REPAIR_REBUILD) in single-user mode.
3. Verify the existence and accessibility of the file referenced in the error message.
4. Review SQL Server and system event logs for related I/O or disk errors.
5. Use the SQLIOSim utility to simulate and test disk I/O reliability.
6. Consult with your hardware or storage provider to ensure system health and configuration.

Note: If the error persists and no corruption is found, the issue may be intermittent or hardware-related. Further diagnostics may be required.

Futhrer Diagnostics

If SQL Server Error 823 persists or appears intermittently, consider the following advanced diagnostics to isolate the root cause:

• Run SQLIOSim to simulate SQL Server I/O and detect hardware or driver issues.
• Check Windows Event Logs for disk, controller, or driver errors.
• Inspect SQL Server Error Logs for repeated 823, 824, or 825 errors.
• Run DBCC CHECKDB with PHYSICAL_ONLY to detect physical corruption.
• Validate storage health using vendor diagnostics or hypervisor tools.
• Monitor disk I/O performance using PerfMon, Resource Monitor, or SQL DMVs.
• Ensure antivirus or backup software is not locking SQL Server files.
• Run chkdsk /f /r to check file system integrity.
• Enable trace flag 818 for deeper I/O diagnostics (use with Microsoft Support guidance).