Related Experiment Video
Updated: Jun 29, 2026
06:16
The Miniature Pig: A Large Animal Model for Cochlear Implant Research
Published on: July 28, 2022
3.7K
Patient Experiences in the Cochlear Implant Reddit Community: Comparing Human and Large Language Model Categorization
Daniel R S Habib1, Kiran Depala1, Jack Lin1
1School of Medicine, Vanderbilt University, Nashville, TN.
American Journal of Audiology
|February 26, 2026
Summary
Large language models (LLMs) show promise in analyzing cochlear implant (CI) patient experiences from online forums, offering faster insights than manual coding. While LLMs provide fair agreement, human expertise remains crucial for accurate interpretation of patient concerns.
Area of Science:
- Audiology and Speech-Language Pathology
- Artificial Intelligence in Healthcare
- Human-Computer Interaction
Background:
- Online patient communities offer valuable insights into real-world experiences with medical devices.
- Automated analysis using large language models (LLMs) has the potential to streamline the understanding of these experiences.
- A gap exists in comparing human versus automated analysis for nuanced patient-reported outcomes.
Purpose of the Study:
- To characterize patient experiences shared in the r/Cochlearimplants Reddit community.
- To compare the performance of human annotators with three different LLMs in categorizing these posts.
- To evaluate the efficiency and accuracy of LLM-based annotation for qualitative analysis.
Main Methods:
- Reflexive thematic analysis was used to manually code 996 posts from r/Cochlearimplants.
- Three LLMs (OpenAI o3, Gemini 2.5 Pro, Claude Sonnet 4) were employed to categorize posts using a human-generated codebook.
- Performance was assessed using metrics like Cohen's kappa, percent agreement, sensitivity, specificity, PPV, NPV, and time.
Main Results:
- Five key themes emerged: community support, medical/surgical journey, device issues, daily life adjustments, and media/outreach.
- OpenAI o3 and Gemini 2.5 Pro demonstrated the highest interrater reliability with human coders (κ = .35 and κ = .34).
- LLMs required less than 20 minutes for annotation, significantly faster than the 52 hours needed for human coders, with varying levels of accuracy.
Conclusions:
- LLMs can significantly accelerate qualitative analysis of online patient discourse, demonstrating fair agreement with human coders.
- Careful selection of LLM models and continued human oversight are essential for accurate interpretation of patient experiences.
- LLM annotation holds potential for real-time monitoring of patient concerns to inform clinical practice and device development.
