Classifying Patient Complaints Using Artificial Intelligence-Powered Large Language Models: Cross-Sectional Study.

Area of Science:

• Health Services Research
• Artificial Intelligence in Healthcare
• Patient Safety

Background:

• Patient complaints offer critical insights into healthcare system performance and patient safety risks.
• Manual analysis of patient complaints is logistically challenging, limiting the extraction of valuable data.
• Systemic changes driven by patient feedback can significantly enhance overall patient safety.

Purpose of the Study:

• To evaluate the accuracy of AI-powered patient complaint categorization using the HCAT GP taxonomy.
• To assess the utility of advanced LLMs in classifying patient complaints within primary care settings.
• To identify key themes and areas for improvement from patient feedback data.

Main Methods:

• Analyzed 1816 anonymous patient complaints from Singaporean public primary care clinics.
• Complaints were manually coded using the HCAT GP taxonomy by trained coders.
• LLMs (GPT-3.5 turbo, GPT-4o mini, Claude 3.5 Sonnet) were employed for classification validation and thematic analysis.

Main Results:

• Most complaints concerned management and institutional processes, primarily of medium severity.
• LLMs demonstrated moderate to good accuracy in HCAT GP field classifications (58.4%-95.5%).
• GPT-4o mini and Claude 3.5 showed superior performance over GPT-3.5 turbo in several classification tasks.
• Key complaint themes included long wait times, staff attitudes, and appointment booking issues.

Conclusions:

• LLMs show significant potential for classifying patient complaints in primary care using the HCAT GP taxonomy.
• Further model fine-tuning is necessary to enhance AI accuracy in complaint analysis.
• Integrating AI can support proactive identification of systemic issues, improving quality improvement and patient safety.