Leveraging Image-Text Pairs for Generalized Category Discovery in Medical Image Classification.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Computer Vision

Background:

• Generalized Category Discovery (GCD) is vital for identifying known and novel medical categories from unlabeled data.
• Existing GCD methods primarily use image data, neglecting valuable textual information for disease understanding.

Purpose of the Study:

• To propose M³GCD (Medical Multi-Modal Generalized Category Discovery), a framework leveraging image-text pairs for joint recognition of known and discovery of novel medical categories.
• To enhance multi-modal robustness by adaptively balancing modality contributions using Dynamic Expert Fusion and Local Experts Balancing mechanisms.

Main Methods:

• Developed a Dynamic Expert Fusion module for sample-specific modality weight learning.
• Implemented a Local Experts Balancing mechanism to maintain individual modality discriminative power.
• Introduced a Category Diffusion module based on the Metropolis-Hastings framework for adaptive category merging and splitting during training.

Main Results:

• M³GCD demonstrated consistent improvements in clustering performance for both known and unknown categories across multiple datasets (MIMIC-CXR, PatchGastric, MM-Retina).
• The proposed framework effectively integrates global and local perspectives to enhance multi-modal robustness.

Conclusions:

• M³GCD offers a significant advancement in medical generalized category discovery by effectively utilizing multi-modal data.
• The method enables simultaneous recognition of known classes and discovery of previously unseen categories without prior knowledge, advancing precision medicine.