Local LLMs as cooperative agents for low-cost surgical navigation support

Area of Science:

• Medical Technology
• Artificial Intelligence
• Surgical Navigation

Background:

• Computer-integrated surgical navigation systems require skilled technicians, limiting their use in resource-constrained settings.
• Low-cost surgical navigation solutions face barriers due to limited OR staff and technical training.
• Advancements in local large language models (LLMs) offer potential for user assistance and task performance.

Purpose of the Study:

• To explore the feasibility of using local LLM-based agents as a natural language interface for low-cost surgical navigation.
• To facilitate hands-free UI manipulation and provide documentation-grounded technical guidance.
• To reduce technician dependence in surgical navigation.

Main Methods:

• Proposed the Navigation Offline Virtual Agent (NOVA) architecture integrating distinct local LLM agents.
• Utilized agents for knowledge tasks, action tasks, and inter-agent delegation.
• Developed a prototype integrating NOVA agents into the open-source NousNav neuronavigation system.

Main Results:

• Local LLM agents performed comparably to commercial, cloud-hosted LLMs.
• A user study demonstrated NOVA's capability for hands-free patient registration.
• Achieved a mean end-to-end latency of 10.4 ± 2.4 s and an 81% command success rate.

Conclusions:

• Local LLM-based agents can be deployed to support users of low-cost navigation systems.
• NOVA provides a context-aware natural language interface, reducing technician dependence.
• This represents a significant step toward sustainable, accessible surgical navigation.