Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Health Information Technology and Healthcare Information System01:30

Health Information Technology and Healthcare Information System

993
Health Information Technology (HIT)
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
993

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Performance of AI in Predicting the Progression of Gestational Diabetes to Type 2 Diabetes: Systematic Review and Meta-Analysis.

Journal of medical Internet research·2026
Same author

Predicting Postpartum Depression Using Imbalance-Aware Machine Learning.

Studies in health technology and informatics·2026
Same author

Safeguarding AI-driven digital health - An adaptation of the Swiss cheese model for safety.

Digital health·2026
Same author

Artificial Intelligence-Based Prediction of Progression from Gestational Diabetes to Type 2 Diabetes.

Studies in health technology and informatics·2026
Same author

Ethical Implications of Small Language Models (SLMs) in Healthcare Applications.

Studies in health technology and informatics·2026
Same author

Automating Safety Surveillance for Software-Based Medical Devices: Insights from FDA MAUDE Data.

Studies in health technology and informatics·2026

Related Experiment Video

Updated: Oct 15, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

731

Evaluation Metrics for Health Chatbots: A Delphi Study.

Kerstin Denecke1, Alaa Abd-Alrazaq2, Mowafa Househ2

  • 1School of Engineering and Computer Science, Institute for Medical Informatics, Bern University of Applied Sciences, Biel, Switzerland.

Methods of Information in Medicine
|October 31, 2021
PubMed
Summary

Standardized evaluation metrics for health chatbots are needed. Experts reached consensus on key metrics for a new evaluation framework, emphasizing multifaceted assessments for chatbot reliability and user acceptance.

More Related Videos

Iterative Development of an Innovative Smartphone-Based Dietary Assessment Tool: Traqq
04:54

Iterative Development of an Innovative Smartphone-Based Dietary Assessment Tool: Traqq

Published on: March 19, 2021

4.8K
E-Patient Counseling Trial E-PACO: Computer Based Education versus Nurse Counseling for Patients to Prepare for Colonoscopy
06:28

E-Patient Counseling Trial E-PACO: Computer Based Education versus Nurse Counseling for Patients to Prepare for Colonoscopy

Published on: August 1, 2019

8.5K

Related Experiment Videos

Last Updated: Oct 15, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

731
Iterative Development of an Innovative Smartphone-Based Dietary Assessment Tool: Traqq
04:54

Iterative Development of an Innovative Smartphone-Based Dietary Assessment Tool: Traqq

Published on: March 19, 2021

4.8K
E-Patient Counseling Trial E-PACO: Computer Based Education versus Nurse Counseling for Patients to Prepare for Colonoscopy
06:28

E-Patient Counseling Trial E-PACO: Computer Based Education versus Nurse Counseling for Patients to Prepare for Colonoscopy

Published on: August 1, 2019

8.5K

Area of Science:

  • Health Informatics
  • Human-Computer Interaction
  • Digital Health

Background:

  • The proliferation of health chatbots necessitates rigorous evaluation due to sensitive data handling and clinical applications.
  • Current health chatbot evaluations lack standardization, hindering system comparison and user trust.
  • Inconsistent reporting of evaluation metrics impedes the assessment of health chatbot reliability.

Purpose of the Study:

  • To develop a health-specific chatbot evaluation framework by establishing expert consensus on relevant metrics.
  • To identify and refine a core set of metrics for assessing health chatbot performance and safety.

Main Methods:

  • An adapted Delphi study design was employed, involving three survey rounds.
  • Experts in research, healthcare, and health informatics rated potential metrics identified through a scoping review.
  • Metrics were categorized and consensus levels (high, moderate, low) were determined.

Main Results:

  • Twenty-four out of 26 initial metrics achieved high expert consensus.
  • The consensus set primarily includes global metrics (e.g., usability, security), response generation, and understanding metrics.
  • Aesthetics-related metrics (e.g., font, color) showed lower consensus among experts.

Conclusions:

  • A broad consensus exists among experts on essential health chatbot evaluation metrics.
  • The findings support the development of a comprehensive evaluation framework for health chatbots.
  • Multifaceted evaluation is crucial for ensuring the acceptability and reliability of health chatbots.