Limited external reproducibility restricts the use of medical record review for benchmarking

Area of Science:

• Healthcare Quality and Safety
• Medical Informatics
• Patient Outcomes Research

Background:

• Medical record review (MRR) is a key method for assessing hospital quality and safety.
• Increasing use of MRR for institutional comparison necessitates high external reproducibility.
• This study investigates the external reproducibility of MRR for adverse event assessment.

Purpose of the Study:

• To evaluate the external reproducibility of medical record review for adverse event assessment.
• To determine the agreement in identifying adverse events, their preventability, and contribution to death.
• To assess the reliability of MRR when used for inter-institutional comparisons.

Main Methods:

• A sample of 40 medical records from deceased patients was randomly selected from two Dutch university medical centers.
• External review committees assessed these records using the same methodology as internal committees.
• Agreement was measured using overall and kappa statistics for adverse events, preventability, and contribution to death.

Main Results:

• Moderate agreement (kappa=0.47) was found for the presence of adverse events.
• Fair agreement (kappa=0.39) was observed for the preventability of adverse events.
• Poor agreement (kappa=-0.109) was noted for the contribution of adverse events to patient death.

Conclusions:

• While MRR is valuable for identifying general patient safety issues, its use for benchmarking requires caution.
• The findings suggest limitations in the external reproducibility of MRR for specific adverse event characteristics.
• Further research may be needed to refine MRR methods for reliable inter-institutional comparisons.