Headache Diagnosis with Open Language Models on German Vignettes: Study Protocol

Headache disorders present diagnostic challenges due to their clinical heterogeneity and the extensive taxonomy of the ICHD-3 classification. While large language models (LLMs) have recently demonstrated impressive diagnostic reasoning capabilities, systematic evaluation for headache disorders remains elusive. This study protocol presents a retrospective, multicentre study designed to investigate whether anonymized clinical vignettes from specialized headache centres can be accurately classified according to the ICHD-3 by LLMs. Open-weight models selected for this study will operate in a GDPR-compliant environment and generate classifications that will be compared with expert diagnoses (the gold standard) using top-k accuracy, micro/macro precision, recall, F1 score and Cohen's κ. Secondary analyses will assess differences between centres and non-specialist raters. Preliminary pilot data (n = 50) already demonstrated exact accuracy of up to 62% and group-level accuracy of up to 92%. The study aims to confirm substantial agreement (κ > 0.6) and provide evidence of the usefulness of LLMs as diagnostic support tools in headache medicine.