Related Experiment Videos
Sentence-Level Provenance for AI Medical Record Summarization: Formative Usability Evaluation of a Click-to-Inspect
Andrew Parambath1, Giordana Pulpo2, Vince Hartman3
1Stanford Medicine, 900 welch roadsuite 350, Palo Alto, US.
JMIR Human Factors
|July 30, 2026
Summary
A new sentence-level provenance interface significantly improves the verification of AI-generated medical summaries. This tool allows clinicians to quickly inspect AI claims, enhancing trust and reducing the burden of reviewing longitudinal patient records.
Area of Science:
- Medical Informatics
- Artificial Intelligence in Healthcare
- Clinical Decision Support
Background:
- Large language models (LLMs) generate medical summaries but face trust barriers due to verification challenges.
- Current provenance methods require manual searches in fragmented clinical notes, hindering time-constrained workflows.
Purpose of the Study:
- To design and evaluate a sentence-level provenance interface for rapid verification of AI-generated longitudinal medical record summaries.
- Enable clinicians to verify individual AI-generated statements efficiently.
Main Methods:
- A formative usability study evaluated a web-based sentence-level provenance interface.
- 46 clinicians (students to attending physicians across specialties) participated in remote moderated usability sessions.
- Usability assessed via System Usability Scale (SUS), Net Promoter Score (NPS), and qualitative feedback.
Main Results:
- High usability reported (mean SUS score 86.25) and positive overall experience (NPS 35).
- Clinicians found rapid access to evidence critical for trust calibration.
- Sentence-level inspectability was identified as a low-friction verification mechanism.
Conclusions:
- Sentence-level provenance transforms AI summaries into interactive verification tools.
- This approach may reduce verification burden and support calibrated reliance in clinical settings.
- Enables rapid, selective inspection of individual claims during longitudinal chart review.