Related Experiment Video
Updated: Sep 11, 2025
05:47
Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
578
Evaluating chatbots in psychiatry: Rasch-based insights into clinical knowledge and reasoning
Yu Chang1,2, Si-Sheng Huang3,4, Wen-Yu Hsu2,3,4
1Department of Psychiatry, Chung Shan Medical University Hospital, Taichung, Taiwan.
Plos One
|August 14, 2025
Summary
Leading chatbots show potential for psychiatric clinical support, with ChatGPT-o1-preview excelling in knowledge and reasoning. However, limitations highlight the need for human oversight in AI integration for patient safety.
Area of Science:
- Psychiatry
- Artificial Intelligence
- Medical Informatics
Background:
- Chatbots are emerging as tools for clinical support in psychiatry.
- Evaluating the capabilities of these AI tools is crucial for safe integration.
Purpose of the Study:
- To systematically assess the clinical knowledge and reasoning abilities of 27 leading chatbots in psychiatry.
- To identify strengths and limitations of current AI in psychiatric decision support.
Main Methods:
- Utilized 160 multiple-choice questions from Taiwan Psychiatry Licensing Examinations.
- Employed Rasch analysis for quantitative performance evaluation.
- Conducted qualitative assessment of chatbot reasoning processes.
Main Results:
- OpenAI's ChatGPT-o1-preview achieved a Rasch ability score of 2.23, surpassing the passing threshold (p < 0.001).
- ChatGPT demonstrated strong diagnostic and therapeutic reasoning but had limitations in factual recall and niche topics.
- Identified reasoning biases in some chatbot responses.
Conclusions:
- Advanced chatbots show significant potential as clinical decision-support tools in psychiatry.
- Current AI limitations necessitate rigorous human oversight for patient safety.
- Continuous evaluation and domain-specific training are essential for safe AI implementation in clinical practice.
More Related Videos
Related Concept Videos
01:24Rational Emotive Behavior Therapy
111
Cognitive-behavioral therapies (CBTs) are grounded in the belief that our thoughts profoundly influence our emotions and actions. Advocates of CBT emphasize three core assumptions: first, that cognitions are identifiable and measurable; second, that they are central to psychological functioning; and third, that irrational or maladaptive beliefs can be replaced with rational and adaptive ones. This transformative approach to therapy has paved the way for specific models such as Albert...
111
01:37Reason and Intuition
6.9K
The human brain processes information for decision-making using one of two routes: an intuitive system and a rational system (Epstein, 1994; popularized by Kahneman, 2011 as System 1 and System 2, respectively). The intuitive system is quick, impulsive, and operates with minimal effort, relying on emotions or habits to provide cues for what to do next, while the rational system is logical, analytical, deliberate, and methodical. Research in neuropsychology suggests that the...
6.9K
01:30Psychosurgery
106
Psychosurgery, the surgical alteration or permanent removal of brain tissue to alleviate severe psychological conditions, stands as one of the most radical and controversial treatments in the history of mental health care. Its development and application have evolved significantly, marked by dramatic shifts in scientific understanding and ethical perspectives.
Historical Development of Psychosurgery
In the 1930s, Portuguese neurologist Antonio Egas Moniz introduced a surgical procedure designed...
Historical Development of Psychosurgery
In the 1930s, Portuguese neurologist Antonio Egas Moniz introduced a surgical procedure designed...
106
01:27Diagnostic and Statistical Manual of Mental Disorders (DSM)
149
The Diagnostic and Statistical Manual of Mental Disorders (DSM) serves as the primary classification system for mental health disorders, providing standardized diagnostic criteria for clinicians and researchers. First published by the American Psychiatric Association (APA) in 1952, the DSM has undergone several revisions to reflect evolving psychiatric understanding. The fifth edition, DSM-5, released in 2013, introduced key updates that expanded diagnostic categories and modified diagnostic...
149
01:13Patient-centered Care
2.3K
Patient-centered care involves delivering care beyond inpatient hospitalization. Reflective practice can enhance a patient-centered approach. Reflective practice is a process of reasoning that considers all aspects of the present situation, including practicalities, learning from personal practice, and consideration of patient needs. Patients appreciate care decisions made while considering their input. Involving the patient in their care provides the patient with a sense of contribution rather...
2.3K
01:20Introduction to Cognitive Psychology
818
Cognitive psychology is the field of psychology dedicated to examining how people think. It attempts to explain how and why we think the way we do by studying the interactions among human thinking, emotion, creativity, language, and problem-solving, as well as other cognitive processes. Cognitive psychology studies how information is processed and manipulated in remembering, thinking, and knowing.
This field emerged in the mid-20th century, following a period dominated by behaviorism, which...
This field emerged in the mid-20th century, following a period dominated by behaviorism, which...
818