Instruction tuning for colorectal histopathology: a multimodal vision-language assistant on human-evaluated data

Despite the previous research on general-purpose Multimodal Large Language Models (MLLM) and histopathology multimodal chatbots, there has been limited exploration of their applications, particularly concerning colorectal cancer (CRC) histopathology slides. The success demonstrated by Language and Vision Assistant (LLaVA) as an MLLM in the natural image domain suggests its suitability for downstream tuning in histopathology slide analysis. In this study, we present CChat, a novel adaptation of the LLaVA model, to investigate its utility in CRC computational pathology. We accomplish this by fine-tuning LLaVA on a custom human-validated dataset curated from CRC histopathology slides. We also generate a benchmark dataset, to empirically evaluate CChat with other state-of-the-art chatbots, on which our model achieved final BERTScore, BLEU and Rouge-L scores of 93.23, 44.91, and 44.85, respectively, illustrating our methodology's potential for generation of high-quality instruction datasets.