Related Experiment Video
Updated: Aug 13, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Methods of Evaluating Large Language Model-Based Health Care Applications Used by Nonprofessionals: Protocol for a
Maren Keuchel1,2, Pinar Bisgin2, Tom Strube2
1Health Care Informatics, Faculty of Health, School of Medicine, Witten/Herdecke University, Witten, North Rhine-Westphalia, Germany.
JMIR Research Protocols
|August 11, 2026
Summary
This scoping review maps how large language models (LLMs) in healthcare are evaluated for nonprofessional users. It identifies current methods to guide future research and ensure safe, effective AI applications.
Area of Science:
- Health Informatics
- Artificial Intelligence in Healthcare
- Human-Computer Interaction
Background:
- Large language models (LLMs) are increasingly used by nonprofessionals in healthcare.
- Evaluating these LLM applications is crucial to prevent misinformation and harmful decisions.
- Current guidance for evaluating LLM-based health applications for nonprofessionals is limited and fragmented.
Purpose of the Study:
- To conduct a scoping review mapping evaluation approaches for LLM-based health applications used by nonprofessionals.
- To identify and thematically map current evaluation methods, dimensions, metrics, and instruments.
- To provide a comprehensive overview of existing evaluation methodologies.
Main Methods:
- Adherence to Joanna Briggs Institute approach for scoping reviews and PRISMA-P/PRISMA-ScR guidelines.
- Inclusion of studies evaluating LLM-based health applications for nonprofessionals, with searches in PubMed, CINAHL, PsycInfo, and IEEE Xplore (results since 2021).
- Qualitative data summarization and interpretation, with rigorous screening and data extraction processes involving independent reviewers.
Main Results:
- Initial search yielded 8538 records; 17.8% were eligible for retrieval, with 88.3% successfully retrieved.
- Full-text screening excluded publications based on population, concept, and context mismatches, leaving 67.4% for data extraction.
- Final data extraction, coding, and synthesis are planned for Q4 2026.
Conclusions:
- The scoping review will systematically identify and map current evaluation methods for nonprofessional-use LLM health applications.
- Findings will offer insights into quality dimensions, metrics, and measurement instruments for LLM evaluation.
- The study aims to guide future research and development in quality assurance for healthcare LLMs used by the public.
Related Concept Videos
01:15Methods of Documentation VI: Case Management Model
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic illness...
For example, a patient with a chronic illness...
01:29Documentation in Long-Term and Home Healthcare Setting
Documentation in long-term care facilities and home healthcare settings is crucial for ensuring continuous, coordinated, and comprehensive care for patients. Each setting has its specific documentation processes and tools:
Long-Term Care Facilities
Long-Term Care Facilities