Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches01:23

Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches

Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Emotion Classification in Japanese Cancer Survivor Interview Narratives Using Sentiment Polarity and Plutchik Emotion Frameworks: Model Development and Evaluation Study.

JMIR formative research·2026
Same author

Comparison of Mask-Wearing Behavior on Social Media and Its Relationship With Demographic Characteristics During the COVID-19 Pandemic: Social Media Data Analysis Between the United States and Japan.

JMIR human factors·2026
Same author

Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis.

JMIR formative research·2026
Same author

Health-Related Quality of Life Before and After Sobriety in Combination With an Adjunctive Journaling App in Patients With Alcohol-Related Liver Disease: Prospective Single-Arm Study.

JMIR formative research·2026
Same author

Social harmony at work: A sharedness index linking team atmosphere to individual well-being in a Japanese company.

PloS one·2025
Same author

Elucidating Celecoxib's Preventive Effect in Capecitabine-Induced Hand-Foot Syndrome Using Medical Natural Language Processing.

JCO clinical cancer informatics·2025
Same journal

Pregnancy-Related Clinical Codes in Unlikely Populations in Primary Care.

JMIR medical informatics·2026
Same journal

Selecting, Scaling, and Measuring the Value of Ambient AI in a Nonacademic Health System: Multiphase Pilot Study.

JMIR medical informatics·2026
Same journal

Prediction of Early Hospital Admission (≤24 Hours) After Stroke Using Machine Learning and Deep Learning: Multicenter Study From China.

JMIR medical informatics·2026
Same journal

Assessing the Feasibility and Acceptability of Implementing a Preclinic Vital Signs Assessment in Primary Care: Cross-Sectional Pilot Study.

JMIR medical informatics·2026
Same journal

Candidate Passive Sensor Suite Technologies for Tactical Combat Casualty Care Environments: Comparative Assessment Study.

JMIR medical informatics·2026
Same journal

Relevance of the uMap Collaborative Platform as Support for Choropleth Mapping: A Traffic‒Light Statistical Signal Atlas of All-Cause Mortality-First French Lockdown.

JMIR medical informatics·2026
See all related articles

Related Experiment Video

Updated: May 17, 2026

Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models
07:14

Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models

Published on: December 23, 2025

A Multiassessment and Multiprofessional Agents Approach for Medical Chatbot Risk Estimation: Development and

Lenard Paulo Velasco Tamayo1, Tomohiro Nishiyama1, Shaowen Peng1

  • 1Nara Institute of Science and Technology, Ikoma-shi, Japan.

JMIR Medical Informatics
|May 15, 2026
PubMed
Summary
This summary is machine-generated.

This study introduces a multi-assessment (MA) and multiprofessional agents (MPA) framework to improve AI chatbot risk assessment in healthcare. The MA-MPA approach enhances accuracy, particularly in ethical and legal domains, by integrating specialized knowledge.

Keywords:
large language modelmedical question and answermultiassessmentmultiprofessional agentnatural language processing

More Related Videos

Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
07:31

Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack

Published on: May 15, 2020

Related Experiment Videos

Last Updated: May 17, 2026

Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models
07:14

Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models

Published on: December 23, 2025

Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
07:31

Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack

Published on: May 15, 2020

Area of Science:

  • Artificial Intelligence in Healthcare
  • Natural Language Processing
  • Risk Assessment Methodologies

Background:

  • Assessing AI chatbot responses in medical, ethical, and legal domains is crucial for safe healthcare applications.
  • Current large language models (LLMs) lack specialized domain knowledge for accurate risk assessment.
  • Existing ensemble methods struggle to resolve disagreements, leading to misclassification and risk assessment challenges.

Purpose of the Study:

  • To design, develop, and evaluate a synergistic multi-assessment (MA) and multiprofessional agents (MPA) approach for chatbot risk assessment.
  • To improve the accuracy and reliability of risk estimation in AI-driven healthcare interactions.
  • To address limitations of general LLMs and ensemble methods in specialized domains.

Main Methods:

  • Developed a multi-assessment (MA) framework with initial (MA1) and final (MA3) assessments, incorporating a verification assessment (MA2) using specialized agents (MPA) for medical, ethical, and legal domains.
  • Evaluated the approach on the MedNLP-CHAT corpus (N=226) using baseline, enhanced prompt, embedding-based search, and retrieval-augmented generation (RAG) methods.
  • Utilized macro F1-score and joint accuracy as primary metrics, supported by confidence intervals and paired macro F1-score differences.

Main Results:

  • The MA-MPA framework with RAG achieved a macro F1-score of 0.800 and 60.3% joint accuracy, outperforming existing systems in ethical and legal domains.
  • The MA approach significantly improved performance, with gains from MA1 to MA2 ranging from +0.176 to +0.214.
  • While MPA integration with MA and external knowledge showed gains, joint accuracy improvements were not consistently evident, and MA alone surpassed RAG in joint accuracy.

Conclusions:

  • The MA-MPA approach demonstrates potential for enhancing chatbot risk estimation, especially when combined with external knowledge.
  • The framework shows promise for improving balanced overall performance, though the medical domain remains a challenge.
  • Further improvements in contextually grounded risk estimation may be achieved through more specialized LLMs.