Related Experiment Video
Updated: Jul 15, 2025
07:13
Digital Home-Monitoring of Patients after Kidney Transplantation: The MACCS Platform
Published on: April 12, 2021
4.3K
Accuracy and Reliability of Chatbot Responses to Physician Questions
Rachel S Goodman1, J Randall Patrinely2, Cosby A Stone3
1Vanderbilt University School of Medicine, Nashville, Tennessee.
JAMA Network Open
|October 2, 2023
Summary
Artificial intelligence chatbots provide mostly accurate medical information for physicians, with accuracy improving over time and across different versions. However, limitations necessitate further research and development for clinical use.
Area of Science:
- Medical Informatics
- Artificial Intelligence in Healthcare
- Natural Language Processing
Background:
- Natural language processing tools like ChatGPT offer potential to improve medical information accessibility for professionals and patients.
- Assessing AI chatbot safety and efficacy in clinical settings is crucial for decision-making and healthcare efficiency.
Purpose of the Study:
- To evaluate the accuracy and completeness of AI chatbot responses to physician-generated medical questions.
- To identify the reliability and limitations of AI-generated medical information.
Main Methods:
- 33 physicians across 17 specialties generated 284 medical questions.
- Physicians rated chatbot answers for accuracy (6-point scale) and completeness (3-point scale).
- Responses were compared between GPT-3.5 and GPT-4, and re-evaluated over time.
Main Results:
- The median accuracy score was 5.5 (out of 6), and median completeness was 3.0 (out of 3).
- Accuracy slightly decreased with question difficulty but remained high for binary and descriptive questions.
- Re-querying and using GPT-4 significantly improved response accuracy.
Conclusions:
- AI chatbots generate largely accurate medical information for physician queries, with notable improvements over time and with newer versions.
- Despite high accuracy, limitations exist, indicating a need for further research, model development, and validation before widespread clinical adoption.
More Related Videos
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
261
03:14Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
610
Related Concept Videos
01:25Errors occurring during blood pressure monitoring
755
Blood pressure monitoring is a crucial clinical procedure in diagnosing and managing various cardiovascular conditions. Despite its significance, the accuracy of blood pressure measurements can be compromised by multiple factors, potentially leading to either falsely high or low readings. These inaccuracies are critical as they can significantly impact patient care. So, it is vital to understand these challenges deeply and adopt strategic approaches to minimize errors.
Several factors...
Several factors...
755
01:22Ethics and Bioethics
1.4K
Ethics is a philosophical study of moral actions. Ethics attempts to determine what is valuable for individuals and society. It examines the rational justification of moral judgments and analyzes what is morally just, fair, and right. Bioethics is a sub-discipline of applied ethics that analyzes the philosophical, social, and legal issues in life sciences and medicine. Ethical theories serve as a foundation for decision-making and represent the viewpoints from which people seek direction. They...
1.4K
01:23Nurses' Legal Responsibilities II
736
Establishing a secure, collaborative nurse-patient relationship is crucial for delivering high-quality care. This relationship, founded on trust, respect, and honesty, enhances the patient's comfort and willingness to share vital health information. For example, a nurse who listens actively and without judgment provides clear information about health conditions and treatment options and respects patient decisions, which builds a trusting relationship.
Communication between nurses and...
Communication between nurses and...
736
01:18Guidelines and Strategies for Safe Computer Charting
824
The guidelines and strategies provided by the American Nurses Association (ANA) and the Canadian Nurses Association (CNA) offer essential principles for ensuring safe and secure computer charting systems in healthcare settings. Let's break down each recommendation:
Maintain Confidentiality and Security:
Maintain Confidentiality and Security:
824
01:21Barriers to Effective Communication II
3.7K
The barriers to effective communication also include cultural barriers, semantic barriers, gender barriers, and time constraints.
Cultural barriers:
Differences in values, beliefs, religion, knowledge, and tradition can significantly impact communication. Awareness of nonverbal cues is critical, especially when conversing with a patient from a different culture. What appears appropriate in one culture may be inappropriate in another.
Semantic barriers:
As a result of their tendency to use...
Cultural barriers:
Differences in values, beliefs, religion, knowledge, and tradition can significantly impact communication. Awareness of nonverbal cues is critical, especially when conversing with a patient from a different culture. What appears appropriate in one culture may be inappropriate in another.
Semantic barriers:
As a result of their tendency to use...
3.7K
01:13Patient-centered Care
2.1K
Patient-centered care involves delivering care beyond inpatient hospitalization. Reflective practice can enhance a patient-centered approach. Reflective practice is a process of reasoning that considers all aspects of the present situation, including practicalities, learning from personal practice, and consideration of patient needs. Patients appreciate care decisions made while considering their input. Involving the patient in their care provides the patient with a sense of contribution rather...
2.1K