Assessing Completeness of Clinical Histories Accompanying Imaging Orders Using Adapted Open-Source and Closed-Source...

Area of Science:

• Artificial Intelligence in Radiology
• Natural Language Processing for Clinical Data

Background:

• Incomplete clinical histories in radiology are a persistent issue, traditionally requiring manual analysis for quality improvement.
• Previous methods for assessing clinical history completeness are labor-intensive and not easily scalable.

Purpose of the Study:

• To adapt and evaluate open-source and closed-source large language models (LLMs) for automated extraction of clinical history elements from imaging orders.
• To utilize the best-performing open-source LLM to benchmark the completeness of a large dataset of clinical histories.

Main Methods:

• Retrospective analysis of 50,186 clinical histories from CT, MRI, US, and radiography orders.
• Adaptation of Llama 2-7B, Mistral-7B (open-source), and GPT-4 Turbo (closed-source) LLMs using prompt engineering, in-context learning, and fine-tuning.
• Evaluation of LLM performance against manual annotations by board-certified radiologists using accuracy, Cohen κ, and BERTScore.

Main Results:

• The fine-tuned open-source Mistral-7B model demonstrated substantial agreement with radiologists (κ = 0.73) and high semantic similarity (BERTScore = 0.96).
• Mistral-7B's performance closely rivaled that of GPT-4 Turbo in accuracy (91% vs. 92%) and semantic similarity.
• Using Mistral-7B, only 26.2% of clinical histories were found to contain all five essential elements (past medical history, what, when, where, clinical concern).

Conclusions:

• A fine-tuned open-source LLM (Mistral-7B) can effectively extract clinical history elements, matching the performance of larger closed-source models.
• This LLM provides a scalable solution for assessing clinical history completeness, establishing a crucial benchmark for clinical practice.
• The developed model and code will be fully open-sourced to promote wider adoption and further research.