Development of a scoring system to quantify errors from semantic characteristics in incident reports.

Area of Science:

• Healthcare Informatics
• Patient Safety Research
• Natural Language Processing Applications

Background:

• Incident reporting systems are crucial for identifying risks and fostering organizational learning.
• Free-text descriptions in incident reports contain valuable data on contributing factors.
• Existing methods may not fully leverage the rich information within free-text incident narratives.

Purpose of the Study:

• To develop and validate an automated error scoring system for incident reports.
• To extract information on error factors using a novel decision-making model incorporating natural language processing.
• To quantify incident severity and identify contributing factors for improved patient safety.

Main Methods:

• Retrospective analysis of over 114,000 free-text incident reports from Nagoya University Hospital (2012-2022).
• Application of morphological analysis and a decision-making model for term segmentation and error scoring.
• Validation of the 'report error score' against expert classifications using accuracy, recall, precision, and F-score metrics.

Main Results:

• A significant difference in error scores was observed between expert-identified error-related incidents and others (p<0.001).
• The proposed 'report error score' achieved high performance metrics: accuracy (0.8), recall (0.82), precision (0.85), and F-score (0.84).
• Group error scores demonstrated a strong positive association with expert ratings across all departments (correlation=0.66, p<0.001).

Conclusions:

• The developed error scoring system effectively utilizes aggregated incident report data to identify patient safety risks.
• This automated approach offers valuable insights for enhancing patient safety initiatives.
• The system supports organizational learning by providing a quantitative measure of incident-related errors.