Classifying Patient Complaints Using Artificial Intelligence-Powered Large Language Models: Cross-Sectional Study

Sky Wei Chee Koh ^1,2, Eunice Rui Ning Wong ^2,3, John Chong Min Tan ⁴

⁰Division of Family Medicine, Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, NUHS Tower Block Level 9, 1E Kent Ridge Road, Singapore, 119228, Singapore, 65 67163185.

Journal of Medical Internet Research +

|

August 6, 2025

View abstract on PubMed

Summary

Artificial intelligence (AI) shows promise in categorizing patient complaints using the Healthcare Complaint Analysis Tool (HCAT) General Practice (GP) taxonomy. Advanced large language models (LLMs) like GPT-4o mini and Claude 3.5 offer potential for improving patient safety and healthcare quality.

Area Of Science

Health Services Research
Artificial Intelligence in Healthcare
Patient Safety

Background

Patient complaints offer critical insights into healthcare system performance and patient safety risks.
Manual analysis of patient complaints is logistically challenging, limiting the extraction of valuable data.
Systemic changes driven by patient feedback can significantly enhance overall patient safety.

Purpose Of The Study

To evaluate the accuracy of AI-powered patient complaint categorization using the HCAT GP taxonomy.
To assess the utility of advanced LLMs in classifying patient complaints within primary care settings.
To identify key themes and areas for improvement from patient feedback data.

Main Methods

Analyzed 1816 anonymous patient complaints from Singaporean public primary care clinics.
Complaints were manually coded using the HCAT GP taxonomy by trained coders.
LLMs (GPT-3.5 turbo, GPT-4o mini, Claude 3.5 Sonnet) were employed for classification validation and thematic analysis.

Main Results

Most complaints concerned management and institutional processes, primarily of medium severity.
LLMs demonstrated moderate to good accuracy in HCAT GP field classifications (58.4%-95.5%).
GPT-4o mini and Claude 3.5 showed superior performance over GPT-3.5 turbo in several classification tasks.
Key complaint themes included long wait times, staff attitudes, and appointment booking issues.

Conclusions

LLMs show significant potential for classifying patient complaints in primary care using the HCAT GP taxonomy.
Further model fine-tuning is necessary to enhance AI accuracy in complaint analysis.
Integrating AI can support proactive identification of systemic issues, improving quality improvement and patient safety.

Keywords:

artificial intelligence family medicine health services large language models patient complaints primary care

Related Concept Videos

Classification of Illness

The meaning of illness is individualized to each person who experiences an alteration in health. In contrast, disease is a medical term indicating a pathological change in the structure and function of the body or mind. It is a condition that has specific symptoms and boundaries.
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...