Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Search research articles

Related Experiment Video

Updated: Jul 4, 2026

P300-Based Brain-Computer Interface Speller Performance Estimation with Classifier-Based Latency Estimation

P300-Based Brain-Computer Interface Speller Performance Estimation with Classifier-Based Latency Estimation

Published on: September 8, 2023

Benchmarking Prompting Strategies for Open-Source Language Models in ICHD-3 Headache Classification.

Dorian Zwanzig¹

¹Eberswalde University for Sustainable Development, Germany.

Studies in Health Technology and Informatics

|July 3, 2026

Summary

Open-source large language models (LLMs) struggle with headache classification, showing a significant gap due to poor knowledge retrieval. Improving retrieval accuracy is key to enhancing diagnostic performance.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

Headache Diagnosis with Open Language Models on German Vignettes: Study Protocol.

Studies in health technology and informatics·2026

Same author

Physical therapists' perspectives on a large language model-powered knowledge translation tool for guideline adherence: A qualitative focus group study.

Physiotherapy theory and practice·2026

Same author

Evaluating AI-Powered Q&A Systems: A Simple Approach to Determining the Need for Expert Ratings.

Studies in health technology and informatics·2025

Same author

Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models.

Studies in health technology and informatics·2025

Area of Science:

Medical Informatics
Artificial Intelligence
Natural Language Processing

Background:

Accurate headache classification is crucial for effective patient management.
Large language models (LLMs) show promise in clinical applications but require rigorous evaluation.
The International Classification of Headache Disorders, 3rd edition (ICHD-3) provides a standardized framework for diagnosis.

Purpose of the Study:

To benchmark the performance of open-source LLMs in ICHD-3 headache classification.
To identify the primary limitations hindering LLM diagnostic accuracy in this domain.
To quantify the impact of retrieval quality on classification performance.

Main Methods:

Five open-source LLMs (12-24B parameters) were evaluated on 305 synthetic headache vignettes from the HeadAI dataset.

Keywords:

ICHD-3 LLMs RAG headache classification

Related Experiment Videos

Last Updated: Jul 4, 2026

P300-Based Brain-Computer Interface Speller Performance Estimation with Classifier-Based Latency Estimation

P300-Based Brain-Computer Interface Speller Performance Estimation with Classifier-Based Latency Estimation

Published on: September 8, 2023

Standard prompting techniques were employed to assess baseline performance.

A three-tier oracle ablation was used to isolate the contribution of retrieval quality.

Main Results:

Standard prompting achieved 56-61% top-1 accuracy, with no significant differences between models.
The oracle ablation revealed a 36-percentage-point gap attributed to retrieval quality.
When the correct diagnosis was retrieved at rank 1 (Hit@1 = 23%), accuracy reached 94.9%.

Conclusions:

Knowledge retrieval quality, not rule application, is the primary bottleneck for LLM-based headache classification.
Enhancing the retrieval of relevant diagnostic information is critical for improving LLM performance.
Future research should focus on optimizing retrieval mechanisms for clinical decision support systems.