Combining Real and Synthetic Data to Overcome Limited Training Datasets in Multimodal Learning

Area of Science:

• Biomedical informatics
• Artificial intelligence in medicine
• Dermatology

Background:

• Biomedical data is often multimodal, offering complementary patient insights.
• Multimodal deep learning (DL) can enhance clinical decision-making but requires paired data.
• Publicly available biomedical datasets are often unimodal, hindering multimodal DL development.

Purpose of the Study:

• To develop a strategy for creating robust multimodal representations of skin lesion data.
• To address the challenge of limited paired data in public skin lesion datasets.
• To improve the performance of automated skin lesion classification using multimodal DL.

Main Methods:

• A multimodal architecture was designed to integrate image embeddings with fine-grained text representations.
• Large language models (LLMs) were utilized to synthesize textual descriptions from image metadata.
• The synthesized text data was paired with original skin lesion images for model training.

Main Results:

• The proposed multimodal representation significantly outperformed unimodal approaches in skin lesion classification.
• Superior performance was achieved across nine diverse internal and external datasets.
• The integration of synthesized text data enhanced the robustness of the model.

Conclusions:

• The developed strategy effectively leverages synthesized data to create powerful multimodal representations for skin lesion analysis.
• This approach overcomes limitations of unimodal datasets and advances the application of multimodal DL in dermatology.
• The findings suggest a promising direction for improving automated diagnostic tools in clinical practice.