Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

ER Retrieval Pathway01:45

ER Retrieval Pathway

4.9K
In the secretory pathway, vesicles transport proteins from one cellular compartment to another in forward transport to deliver the protein to its correct location. Occasionally, misfolded proteins and incorrect proteins escape their original compartments, and a retrieval pathway is used to return the escaped proteins to their original compartment.
The ER uses many checkpoints to prevent the entry of incorrectly folded or a resident protein as cargo onto a transport vesicle. These mechanisms...
4.9K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

SmartAlert - Implementing Machine Learning-Driven Clinical Decision Support for Inpatient Laboratory Utilization Reduction.

NEJM AI·2026
Same author

Why and How to Monitor Deployed AI Systems in Health Care.

NEJM catalyst innovations in care delivery·2026
Same author

BRIDGE: benchmarking large language models for understanding real-world clinical practice texts.

Nature biomedical engineering·2026
Same author

Patient perspectives on clinicians' use of ambient AI scribes.

JAMIA open·2026
Same author

Application of Machine Learning in Predicting the Properties of Two-Dimensional Semiconductor Materials.

Nanomaterials (Basel, Switzerland)·2026
Same author

Micro-randomization trial design under operational constraints.

Contemporary clinical trials·2026

Related Experiment Video

Updated: Feb 28, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

1.3K

Retrieval-Augmented Guardrails for AI-Drafted Patient-Portal Messages: Error Taxonomy Construction and Large-Scale

Wenyuan Chen1, Fateme Nateghi Haredasht2, Kameron C Black3

  • 1Stanford Center for Biomedical Informatics Research, Stanford, CA, USA, wenyuanc@stanford.edu.

Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing
|February 27, 2026
PubMed
Summary

Evaluating large language model (LLM) responses for patient messages is crucial. A new Retrieval-Augmented Error Checking (RAEC) pipeline improves accuracy and agreement in detecting clinical errors.

More Related Videos

Introduction of an Integrated Pathology Image Management, Artificial Intelligence, and Reporting System
05:33

Introduction of an Integrated Pathology Image Management, Artificial Intelligence, and Reporting System

Published on: July 11, 2025

1.3K
TBase - an Integrated Electronic Health Record and Research Database for Kidney Transplant Recipients
09:00

TBase - an Integrated Electronic Health Record and Research Database for Kidney Transplant Recipients

Published on: April 13, 2021

5.5K

Related Experiment Videos

Last Updated: Feb 28, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

1.3K
Introduction of an Integrated Pathology Image Management, Artificial Intelligence, and Reporting System
05:33

Introduction of an Integrated Pathology Image Management, Artificial Intelligence, and Reporting System

Published on: July 11, 2025

1.3K
TBase - an Integrated Electronic Health Record and Research Database for Kidney Transplant Recipients
09:00

TBase - an Integrated Electronic Health Record and Research Database for Kidney Transplant Recipients

Published on: April 13, 2021

5.5K

Area of Science:

  • Health Informatics
  • Artificial Intelligence in Medicine
  • Clinical Natural Language Processing

Background:

  • Asynchronous patient-clinician messaging via Electronic Health Record (EHR) portals increases clinician workload.
  • Large Language Models (LLMs) show promise for assisting with message responses but require robust evaluation due to potential inaccuracies.
  • Existing evaluation methods may not adequately capture the nuances of clinical communication.

Purpose of the Study:

  • To introduce a clinically grounded error ontology for evaluating LLM-generated patient message responses.
  • To develop a Retrieval-Augmented Error Checking (RAEC) pipeline to enhance the quality of LLM response evaluations.
  • To implement a scalable and interpretable error detection system using a two-stage prompting architecture.

Main Methods:

  • Developed a 5-domain, 59-code error ontology through inductive coding and expert adjudication.
  • Created a RAEC pipeline utilizing semantically similar historical message-response pairs for improved judgment.
  • Employed a two-stage prompting architecture with DSPy for hierarchical error detection on over 1,500 patient messages.

Main Results:

  • The RAEC pipeline demonstrated improved error identification, particularly in clinical completeness and workflow appropriateness.
  • Context-enhanced evaluations showed superior agreement (concordance = 50% vs. 33%) compared to baseline evaluations.
  • Performance metrics (F1 score) were significantly higher for context-enhanced labels (0.500 vs. 0.256) in human validation.

Conclusions:

  • The proposed clinically grounded error ontology and RAEC pipeline provide effective AI guardrails for patient messaging.
  • Retrieval augmentation significantly enhances the accuracy and reliability of LLM response evaluations.
  • This approach supports the safe and scalable deployment of LLMs in clinical communication workflows.