Autonomous skeletal landmark localization toward agentic C-arm control.

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Robotics

Background:

• Automated C-arm positioning is crucial for timely emergent interventions.
• Conventional deep learning (DL) C-arm control failures necessitate manual operation, causing delays.
• An agentic C-arm control framework using multimodal large language models (MLLMs) is needed for improved accuracy and clinician feedback integration.

Purpose of the Study:

• Investigate the adaptation of MLLMs for autonomous skeletal landmark localization.
• Evaluate MLLMs' capability in C-arm control for emergent interventions.
• Explore MLLMs' potential to incorporate clinician feedback and reasoning for precise positioning.

Main Methods:

• Fine-tuned two MLLMs on annotated synthetic and real X-ray datasets with skeletal landmarks.
• Tasked MLLMs with retrieving closest skeletal landmarks from X-rays.
• Performed quantitative comparisons against a leading DL approach and qualitative experiments on reasoning and navigation.

Main Results:

• Fine-tuned MLLMs achieved competitive performance in skeletal landmark localization compared to DL methods.
• MLLMs demonstrated evidence of reasoning and spatial awareness in qualitative assessments.
• The MLLMs showed capability in correcting initial prediction errors and sequential C-arm navigation.

Conclusions:

• Fine-tuned MLLMs accurately localize skeletal landmarks, showing potential for agentic autonomous C-arm control.
• MLLMs offer a promising avenue for enhancing C-arm positioning systems.
• The study provides code for MLLM-based C-arm localization.