Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: Jul 4, 2026

P300-Based Brain-Computer Interface Speller Performance Estimation with Classifier-Based Latency Estimation
06:09

P300-Based Brain-Computer Interface Speller Performance Estimation with Classifier-Based Latency Estimation

Published on: September 8, 2023

Benchmarking Prompting Strategies for Open-Source Language Models in ICHD-3 Headache Classification.

Dorian Zwanzig1

  • 1Eberswalde University for Sustainable Development, Germany.

Studies in Health Technology and Informatics
|July 3, 2026
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Headache Diagnosis with Open Language Models on German Vignettes: Study Protocol.

Studies in health technology and informatics·2026
Same author

Physical therapists' perspectives on a large language model-powered knowledge translation tool for guideline adherence: A qualitative focus group study.

Physiotherapy theory and practice·2026
Same author

Evaluating AI-Powered Q&A Systems: A Simple Approach to Determining the Need for Expert Ratings.

Studies in health technology and informatics·2025
Same author

Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models.

Studies in health technology and informatics·2025
Same journal

A GenAI Pipeline for Violinist Kinematic Data Management.

Studies in health technology and informatics·2026
Same journal

AMAL-For-Qatar: A Comprehensive AI Ecosystem for Fetal Ultrasound Analysis - Project Overview and Achievements.

Studies in health technology and informatics·2026
Same journal

Longitudinal Treatment-Aware Multimodal AI for Dermatology: A Scoping Review.

Studies in health technology and informatics·2026
Same journal

Predicting Postpartum Depression Using Imbalance-Aware Machine Learning.

Studies in health technology and informatics·2026
Same journal

Validation of Deep-Learning Models for Autosegmentation of Brain Metastases.

Studies in health technology and informatics·2026
Same journal

Delay-Dependent Gating in Modular RNNs.

Studies in health technology and informatics·2026
See all related articles

Open-source large language models (LLMs) struggle with headache classification, showing a significant gap due to poor knowledge retrieval. Improving retrieval accuracy is key to enhancing diagnostic performance.

Area of Science:

  • Medical Informatics
  • Artificial Intelligence
  • Natural Language Processing

Background:

  • Accurate headache classification is crucial for effective patient management.
  • Large language models (LLMs) show promise in clinical applications but require rigorous evaluation.
  • The International Classification of Headache Disorders, 3rd edition (ICHD-3) provides a standardized framework for diagnosis.

Purpose of the Study:

  • To benchmark the performance of open-source LLMs in ICHD-3 headache classification.
  • To identify the primary limitations hindering LLM diagnostic accuracy in this domain.
  • To quantify the impact of retrieval quality on classification performance.

Main Methods:

  • Five open-source LLMs (12-24B parameters) were evaluated on 305 synthetic headache vignettes from the HeadAI dataset.
Keywords:
ICHD-3LLMsRAGheadache classification

Related Experiment Videos

Last Updated: Jul 4, 2026

P300-Based Brain-Computer Interface Speller Performance Estimation with Classifier-Based Latency Estimation
06:09

P300-Based Brain-Computer Interface Speller Performance Estimation with Classifier-Based Latency Estimation

Published on: September 8, 2023

  • Standard prompting techniques were employed to assess baseline performance.
  • A three-tier oracle ablation was used to isolate the contribution of retrieval quality.

    • Main Results:

    • Standard prompting achieved 56-61% top-1 accuracy, with no significant differences between models.
    • The oracle ablation revealed a 36-percentage-point gap attributed to retrieval quality.
    • When the correct diagnosis was retrieved at rank 1 (Hit@1 = 23%), accuracy reached 94.9%.

    Conclusions:

    • Knowledge retrieval quality, not rule application, is the primary bottleneck for LLM-based headache classification.
    • Enhancing the retrieval of relevant diagnostic information is critical for improving LLM performance.
    • Future research should focus on optimizing retrieval mechanisms for clinical decision support systems.