Analyzing Patient Complaints in Web-Based Reviews of Private Hospitals in Selangor, Malaysia, Using Large Language...

Area of Science:

• Health Services Research
• Natural Language Processing
• Artificial Intelligence in Healthcare

Background:

• Traditional content analysis is time-consuming and resource-intensive.
• Large language models (LLMs) offer potential for automating and enhancing content analysis.
• LLM-assisted content analysis (LACA) is a novel approach leveraging LLMs for thematic coding.

Purpose of the Study:

• To develop and validate LACA for analyzing hospital web-based reviews.
• To identify key themes of issues from hospital web-based reviews using LACA.
• To assess the reliability and efficiency of LACA compared to traditional methods.

Main Methods:

• Acquired 14,938 web-based reviews for 53 private hospitals in Malaysia.
• Filtered fake reviews using NLP and machine learning; removed low-quality reviews with GPT-4o mini API.
• Developed a validated codebook via parallel human-LLM coding (κ=0.81) and applied factor analysis to themes.

Main Results:

• Identified 6 key themes: Service/Communication, Clinical Care, Facilities, Appointments, Financials, and Patient Rights.
• Achieved high interrater reliability (κ=0.81) between human coders and the LLM.
• Factor analysis revealed interpretable latent factors with high reliability (Cronbach's α 0.61-0.97).

Conclusions:

• A pipeline using Python Selenium, GPT-4o mini API, and factor analysis enables valid and reliable thematic analysis.
• LACA of web-based reviews is cost-effective, time-efficient, and supports real-time hypothesis generation for hospital managers.
• Despite potential biases, LACA provides a valuable tool for understanding patient experiences and improving hospital services.