Publicly Accessible Large Language Model Responses to Frequently Asked Questions About Spondylodiscitis: Preliminary...

Patients increasingly use large language models (LLMs) to obtain medical information, but the quality of LLM-generated information on complex spinal infections such as spondylodiscitis remains uncertain. Existing evaluations in spine surgery have mainly addressed degenerative conditions or surgical procedures, and disease-specific data for spondylodiscitis are limited.

This preliminary study evaluated spine surgeons' ratings of single-turn LLM responses to 10 author-curated frequently asked questions (FAQs) about spondylodiscitis and compared answer sets generated from GPT-4, GPT-4o, and Google Gemini web interfaces under the authors' implemented prompting conditions.

A pool of patient-oriented questions was generated through a chronological workflow including publicly available FAQ sources, PubMed-informed terminology review, Google Trends topic checking, and LLM-generated candidate questions. Duplicate and semantically overlapping questions were removed, the remaining questions were grouped into thematic categories, and 10 final FAQ-style prompts were synthesized. Each prompt was submitted once to the publicly accessible web interfaces of GPT-4, GPT-4o, and Google Gemini. The study was interpreted as an expert evaluation of the resulting answer sets. Seven blinded board-certified spine surgeons rated the responses using a 4-level rating system ranging from excellent to unsatisfactory and additionally assessed comprehensiveness, clarity, empathy, and appropriateness of length. Descriptive statistics and nonparametric comparisons were performed. Interrater reliability was assessed using the intraclass correlation coefficient.

Across all responses, 38.6% (81/210) were rated as excellent, 39% (82/210) as satisfactory with minimal clarification needed, 16.7% (35/210) as satisfactory with moderate clarification needed, and 5.7% (12/210) as unsatisfactory. The most common reason for necessary clarification was insufficient information (58/141, 41.1%), followed by language-related issues (21/141, 14.9%) and overly detailed responses (18/141, 12.8%). The complication-related question received the highest mean rating (3.4/5), whereas treatment- and prognosis-related questions received lower ratings (2.7/5 and 2.9/5). Median overall ratings did not differ significantly among the 3 evaluated LLMs. Spine surgeons reported a generally positive attitude toward artificial intelligence-supported patient information but expressed remaining uncertainty regarding reliability and direct patient-physician communication.

In this preliminary expert evaluation, selected publicly accessible LLM web interfaces generated mostly satisfactory responses to author-curated spondylodiscitis FAQs. The findings reflect outputs produced at the time of access under the authors' implemented prompting conditions. LLMs may support patient education only with clinician oversight. Future research should explore advanced, domain-specific models to further improve the quality of communication between clinicians and patients.