Chain-of-verification prompting for NIH stroke scale extraction using small and frontier large language models

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Medicine
• Clinical Informatics

Background:

• The National Institutes of Health Stroke Scale (NIHSS) is crucial for acute stroke care but is often embedded in unstructured clinical notes.
• Automated extraction of NIHSS data using large language models (LLMs) is desirable, but smaller models typically underperform compared to advanced systems.
• Chain-of-Verification (CoVe) prompting is a novel technique designed to improve LLM performance through structured self-verification.

Purpose of the Study:

• To evaluate the impact of CoVe prompting on the performance of various LLMs for extracting NIHSS data from clinical notes.
• To compare the effectiveness of CoVe across different sizes of LLMs, specifically small models versus frontier models.

Main Methods:

• Eight LLMs, including four small models (LLaMA 3.2 3B, Mistral 3B, Gemma 3 4B, Qwen 3 4B) and four frontier models (GPT-5.2, Gemini 3 Pro, Claude Opus 4.5, Grok 4), were assessed.
• The models processed 312 patient discharge summaries, evaluated under both baseline and CoVe prompting conditions.
• Performance metrics included subscore and total score exact-match accuracy and mean absolute error (MAE).

Main Results:

• At baseline, frontier models significantly outperformed small models in subscore accuracy (88.5% vs 53.2%) and MAE (0.15 vs 0.84).
• CoVe prompting markedly improved small model performance, increasing subscore accuracy to 65.0% and reducing subscore MAE to 0.55.
• Frontier models demonstrated no significant performance improvement with CoVe prompting at the group level.

Conclusions:

• CoVe prompting substantially enhances the accuracy of NIHSS extraction by smaller LLMs.
• While CoVe does not significantly benefit frontier models, it presents a viable strategy for improving the utility of smaller LLMs in clinical data extraction.
• Current performance of even improved small models may not suffice for standalone clinical use, but CoVe offers a path for future development.