Related Experiment Video
Updated: Aug 5, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
From API to Action: A Multi-Model Comparison of OpenAI, Anthropic, Google, and Meta LLMs for Clinical Trial Data
Richard J Young1, Jorge Fonseca2, Brach Poston1,3
1Interdisciplinary Neuroscience, University of Nevada, Las Vegas, NV 89154, USA.
Bioengineering (Basel, Switzerland)
|July 28, 2026
Summary
Large Language Models (LLMs) show high accuracy in extracting brain stimulation data from clinical trials, with semantic evaluation revealing strong agreement among models. This supports their use in systematic reviews, highlighting the need for oversight on shared errors.
Area of Science:
- Neuroscience
- Computational Linguistics
Background:
- Clinical trial data extraction from registries is manual, time-consuming, and prone to errors.
- Unstructured protocol descriptions often contain critical, missed details.
- Large Language Models (LLMs) offer potential for automating data extraction, but systematic comparisons are limited.
Purpose of the Study:
- To compare the performance of four LLMs in extracting brain stimulation parameters from Parkinson's disease clinical trials.
- To assess inter-model agreement and accuracy against an expert gold standard.
Main Methods:
- Four LLMs (OpenAI o4-mini-high, Anthropic Claude-Sonnet-4, Google Gemini 2.5-Pro, Meta Llama-4-Maverick) extracted transcranial direct current stimulation (tDCS) parameters from 67 trials.
- Data were extracted into a structured JSON schema.
- Pairwise inter-model agreement was quantified using Cohen's Kappa and percentage agreement.
- Accuracy was validated against an expert gold standard.
Main Results:
- Exact-string matching showed high agreement for simple data (100% for binary, 96.4% for categorical) but lower agreement for complex targets (48.6%).
- Semantic matching significantly improved inter-model agreement (97.0%) and aligned with expert accuracy (95.5%).
- Model-specific strengths were observed for numeric parameters like duration.
Conclusions:
- Inter-model agreement decreases with task complexity using exact-string matching, but this reflects varied wording, not reduced accuracy.
- Semantic evaluation demonstrates high and aligned agreement and accuracy across LLMs.
- Hybrid human-AI approaches are recommended, focusing expert oversight on shared errors and minority classes.
