Related Experiment Video
Updated: Jun 30, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Promising performance of locally deployed large language models for postoperative orthopaedic patient questions: An
Lea Lanter1, Sophie Masel1, Bettina Hochreiter1
1Department of Orthopaedic Surgery Balgrist University Hospital Zurich Switzerland.
Purpose:
Generative artificial intelligence (AI), particularly large language models (LLMs), are increasingly utilised in healthcare. While this may reduce the initial workload of healthcare professionals, unvalidated model outputs can pose a relevant risk to patient safety if they are inaccurate, incomplete or inconsistent with recommendations. This study evaluates and compares the performance of local and commercial LLMs on patient questions to inform institutional strategies for patient communication and management.
Methods:
Twenty postoperative patient questions were constructed and posed to GPT-5, Claude 4.5 Sonnet (commercial models) and GPT-OSS, Apertus (locally hosted). Responses were assessed using the QUEST framework evaluating quality, understanding, expression, safety and trust. Four blinded reviewers, two board-certified fellowship-trained orthopaedic surgeons and two orthopaedic surgery residents assessed responses. Overall model performance was compared using the Friedman test, with Wilcoxon signed-rank tests for pairwise post hoc comparisons. Metrics where lower values indicate better performance were inverted so that higher values uniformly represent better performance across all metrics.
Results:
The highest overall performance scores were observed for Claude 4.5 Sonnet (mean 0.949, SD 0.124) and GPT-5 (0.937, 0.142), followed by GPT-OSS (0.873, 0.189), while Apertus performed worst (0.693, 0.317). On QUEST dimensions, GPT-5 and Claude 4.5 Sonnet achieved consistently high ratings, whereas Apertus scored lower across information quality, reasoning and expression. Safety-relevant issues were concentrated in Apertus, with harmful content and fabrication each occurring in 22.5% of evaluated ratings (18/80). At the output level, this corresponded to harmful content in 9/20 unique model responses, and fabrication in 11/20 unique model responses. Inter-rater agreement was moderate (overall Fleiss' κ ≈ 0.50).
Conclusions:
In this structured evaluation setting, commercial LLMs showed higher overall performance and fewer safety-relevant ratings, but they remain externally controlled. Achieving privacy-compliant, real-time clinical integration will require advancing local LLMs through fine-tuning, rigorous validation and robust safety guardrails.
Level Of Evidence:
Level V.