Leveraging a Large Language Model to Generate Quality Improvement Feedback for Clinical Notes

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Documentation Improvement

Background:

• Clinical documentation quality significantly impacts healthcare operations.
• Current feedback mechanisms for improving clinical notes are often time-consuming and insufficient.
• Large language models (LLMs) offer a potential solution for streamlining feedback processes.

Purpose of the Study:

• To evaluate if LLM-generated feedback on medical contingency and discharge planning (MCDP) is non-inferior to physician feedback.
• To assess the effectiveness of Generative Pre-trained Transformer 4 (GPT-4) in improving clinical documentation quality.

Main Methods:

• A cross-sectional study compared GPT-4 feedback with physician feedback on 64 inpatient progress notes.
• Notes were selected for low likelihood of containing MCDP using the AI Audit Tool.
• A/B testing evaluated understandability, usefulness, acceptability, and impartiality using Likert scales.

Main Results:

• GPT-4 feedback demonstrated non-inferiority to physician feedback across all evaluated measures.
• Understandability (mean 1.27), usefulness (mean 2.09), and acceptability (mean 2.07) were significantly better with GPT-4.
• Impartiality also showed non-inferior results (mean -0.20).

Conclusions:

• LLM-generated feedback is a viable alternative to expert clinician feedback for improving clinical note quality.
• This technology can enhance the efficiency and effectiveness of clinical documentation improvement processes.
• AI tools like LLMs show promise in addressing documentation quality challenges in healthcare.