Local LLMs as cooperative agents for low-cost surgical navigation support

Computer-integrated surgical navigation systems are often run in the OR with the assistance of a technician for controlling the user interface and advising on technical details of the system. In lower-resource healthcare settings, limited access to additional OR staff and technical training for operating navigation systems can represent a barrier to sustainably deploying a low-cost surgical navigation solution. Recent advancements in locally deployable large language models (LLMs) have improved their ability to answer technical questions based on source materials and safely perform limited tasks on behalf of a user.

The objective of this paper is to explore the feasibility of using a network of local LLM-based agents to act as a natural language interface for a low-cost surgical navigation system, facilitating hands-free manipulation of the user interface and providing documentation-grounded technical guidance. We propose the navigation offline virtual agent (NOVA), an end-to-end architecture that integrates distinct local LLM-based agents for knowledge tasks, action tasks, and delegation between the agents. Two semisynthetic benchmark datasets were generated for ablation studies of individual agents, and a prototype was built which integrates these agents into NousNav, an open-source neuronavigation system.

The agents based on local LLMs were found to perform comparably to closed-source, commercially hosted LLMs. In a user study with nine participants, NOVA facilitated hands-free patient registration with a mean end-to-end latency of $10.4\pm 2.4$  s and an 81% command success rate.

This work demonstrates that local LLM-based agents can be deployed to support users of low-cost navigation systems by providing a context-aware natural language interface, representing an important step toward reducing technician dependence in low-cost surgical navigation.