Towards Interactive and Interpretable Image Retrieval-Based Diagnosis: Enhancing Brain Tumor Classification with LLM...

When evaluating patient scans, clinicians make use of their previous experience to make a diagnosis. However, for complex conditions such as brain tumors, the availability of relevant information beyond a clinician's personal context becomes more valuable. While the application of content-based image retrieval (CBIR) for medical images is not new, their introduction into practice has been relatively minimal. Where older CBIR systems relying on manually extracted image features suffered from poor performance, newer systems incorporating deep learning may have better performance but decreased interpretability. In this study, we present an interactive image retrieval system that enables accurate and interpretable brain tumor classification. We show that image encoders trained with supervised contrastive learning preserve latent structure within the retrieval space and exhibit classification performance on par with, or exceeding, that of conventional black box classifiers. We integrate off-the-shelf LLMs to enhance the system's accessibility through retrieval report summarization and user Q&A interactions. We recognize the importance of developing clinician-ML systems by providing a framework that clinicians can not only trust the performance of but can interact with. Our findings provide for the seed of a system that can augment the performance of human clinicians in a process that mirrors their natural thought patterns, while increasing the speed of their interactions with medical CBIR through LLMs.