Model-Dependent Prompt-Component Ablation for Medical LLM Decision Support: Application to ENT Biologics Screening

Large language models (LLMs) show promise for clinical decision support, but privacy constraints and unreliable free-text decisions limit deployment. We present Medotor, a locally deployed system for screening patients with chronic rhinosinusitis with nasal polyps (CRSwNP) for biologic therapy, and evaluate prompt design on 163 retrospective cases (132 eligible and 31 ineligible). Five models were tested on the same patients under five variants: a minimal prompt (P0), the full role/safety/structured nine-point prompt without retrieval (P1), removal of the role/safety block (P2), removal of the nine-point schema (P4), and addition to P1 of one fixed MemoRAG-derived knowledge block shared by all patients (M0). Every arm retained all 163 attempts; undecidable or malformed final decisions were errors. Balanced accuracy was the primary metric, with patient-level bootstrap intervals, exact paired McNemar tests, and Holm correction. The best observed balanced accuracies were 0.720 for DeepSeek V4 Flash/P0, 0.713 for Qwen 3.5 397B/P4, 0.704 for GPT-OSS 120B/P4, 0.657 for local Qwen 2.5/P2, and 0.549 for local Gemma 3n/P0. Removing the nine-point schema improved Qwen 3.5 relative to P1 ($p_{\text{Holm}}=0.015$), whereas the full prompt sharply degraded Gemma 3n and produced 84 undecidable outputs. M0 did not significantly improve P1 for any model after correction. Thus, prompt effects were strongly model-dependent, and a longer or retrieval-augmented prompt was not universally safer or more accurate. Because M0 was neither personalized nor updated between patients, no standard-RAG arm was run, and physician hallucination labels were qualitative rather than per output, the study supports conclusions about classification and output reliability-not measured hallucination reduction or superiority of MemoRAG over standard RAG.