CNS-Obsidian: A Neurosurgical Vision-Language Model Built From Scientific Publications.

Area of Science:

• Artificial Intelligence in Medicine
• Neurosurgical Applications
• Vision-Language Models

Background:

• General-purpose vision-language models (VLMs) trained on uncurated data have limitations for high-stakes medical applications.
• Neurosurgery requires specialized AI tools for reliable decision-making.
• CNS-Obsidian is a novel VLM developed for neurosurgical applications.

Purpose of the Study:

• To develop and evaluate CNS-Obsidian, a neurosurgical VLM trained on peer-reviewed literature.
• To compare the clinical utility of CNS-Obsidian against GPT-4o in a real-world neurosurgical setting.
• To establish a framework for creating specialized AI models in scientific domains.

Main Methods:

• Compiled 23,984 articles from Neurosurgery Publications, generating 78,853 figures and captions.
• Utilized GPT-4o and Claude Sonnet-3.5 to create 263,064 training samples in three formats.
• Trained CNS-Obsidian, a fine-tuned 34-billion parameter LLAVA-Next model, and conducted a blinded, randomized trial comparing it with a HIPAA-compliant GPT-4o endpoint.

Main Results:

• CNS-Obsidian achieved similar performance to GPT-4o on synthetic questions but lower accuracy (46.81% vs. 65.70%) on human-generated questions.
• In a randomized trial, CNS-Obsidian received positive ratings in 40.62% of cases compared to 57.89% for GPT-4o.
• Both models included the correct diagnosis in approximately 60% of evaluated consultations.

Conclusions:

• Domain-specific VLMs trained on curated literature can achieve performance close to frontier models, despite smaller size and lower cost.
• This work provides a transparent framework for developing specialized AI models within scientific communities.
• Low clinical utilization highlights the need for alternative AI integration strategies beyond chatbot interfaces to align with specialist workflows.