Related Experiment Video
Updated: Jun 9, 2026
07:14
Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models
Published on: December 23, 2025
Evaluating the performance of artificial intelligence in summarizing pre-coded text to support evidence synthesis: a
Kim Nordmann1, Stefanie Sauter1, Mirjam Stein1
1Kempten University of Applied Sciences, Bavarian Research Center for Digital Health and Social Care, Kempten, Germany.
BMC Medical Research Methodology
|May 30, 2025
Summary
Artificial intelligence chatbots demonstrated comparable correctness to human researchers in evidence synthesis. Chatbots provided more complete and contextually relevant answers, suggesting their potential to accelerate research processes.
Area of Science:
- Artificial Intelligence
- Medical Informatics
- Research Methodology
Background:
- The increasing capabilities of large language models (LLMs) are expanding AI applications in research.
- AI may accelerate specific research process stages, including evidence synthesis.
Purpose of the Study:
- To compare the accuracy, completeness, and relevance of chatbot-generated responses against human responses.
- To evaluate AI's role in evidence synthesis for scoping reviews.
Main Methods:
- A structured survey-based methodology was used.
- Responses from two human researchers and four chatbots (ZenoChat, ChatGPT 3.5, ChatGPT 4.0, ChatFlash) were analyzed.
- Questions were based on a sample of 407 articles for a scoping review on digitally supported healthcare worker interaction.
Main Results:
- No significant differences in correctness were found between human and chatbot answers.
- Chatbots demonstrated superior contextual recognition and provided more complete, though longer, responses.
- ZenoChat, ChatFlash, and ChatGPT (3.5 & 4.0) were ranked for performance, with ZenoChat rated highest.
Conclusions:
- AI-powered chatbots can potentially accelerate qualitative evidence synthesis.
- The continuous development of chatbots suggests expanding future applications in research facilitation.
