LLaMIA: Few-Shot Medical Image Lesion Localization Analysis with Large Language Models

Accurate lesion localization in medical imaging is essential for reliable diagnosis and treatment planning, yet existing methods often require large-scale annotations that are costly and time-consuming. Few-shot lesion localization alleviates this challenge by requiring only a small number of annotated samples; however, performance remains highly dependent on limited support information and annotation availability. To address this, we propose LLaMIA, the first large language model LLM-driven framework designed specifically for few-shot lesion localization. Unlike prior approaches that only treat LLMs as auxiliary tools, LLaMIA adopts an LLM-guided coarse-to-fine localization framework, where CLIP and Q-Former provide lesion-aware multimodal representations, the LLM predicts coarse lesion coordinates through structured prompts, and the Localization Refinement Network further improves spatial precision. We design three modules: a lesion-aware regional encoding network for fine-grained multimodal guidance, an instruction construction pipeline for disambiguating lesion attributes, and a localization prediction mechanism with bipartite matching, coordinate regression, and confidence supervision. Furthermore, a contrastive inference strategy is introduced to mitigate hallucination. Extensive experiments are conducted under multiple few-shot settings (1-shot, 3-shot, 5-shot, and 10-shot) on three public datasets to comprehensively evaluate localization capability under annotation scarcity. LLaMIA achieves state-of-the-art lesion localization performance, outperforming existing baselines in terms of accuracy, robustness, and generalization. The results demonstrate that LLaMIA effectively leverages lesion-aware multimodal reasoning to address annotation scarcity, while mitigating hallucination through contrastive inference. This highlights the advantages of integrating structured prompts with spatial reasoning in medical imaging tasks. LLaMIA introduces a novel paradigm for LLM-driven lesion localization, showing the potential of multimodal instruction-based reasoning to advance medical image analysis with minimal annotation cost.