Related Experiment Videos
Synthesized clinical notes enable training robust multimodal AI models from unimodal dermatology datasets
Niccolo Marini1, Zhaohui Liang2, Sivaramakrishnan Rajaraman2
1National Library of Medicine, National Institutes of Health, Bethesda, MD, USA. niccolo.marini@nih.gov.
NPJ Digital Medicine
|July 17, 2026
Summary
Synthesizing clinical notes with large language models (LLMs) improves multimodal algorithms for skin lesion analysis. This approach enhances model robustness and generalization across diverse dermatology datasets.
Area of Science:
- Dermatology
- Artificial Intelligence
- Medical Informatics
Background:
- Multimodal (MM) algorithms show promise for automated skin lesion analysis.
- Clinical translation is limited by small, heterogeneous dermatology datasets.
- Large Language Models (LLMs) can synthesize clinical notes but may hallucinate.
Purpose of the Study:
- Investigate strategies for generating reliable clinical notes using LLMs and metadata.
- Mitigate LLM hallucinations to improve MM training for dermatology.
- Enhance performance on downstream tasks like cross-modal retrieval and zero-shot learning.
Main Methods:
- Trained a MM architecture using real dermatology images paired with synthesized clinical notes.
- Evaluated the MM model on 15 datasets (6 internal, 9 external).
- Assessed performance on cross-modal retrieval and zero-shot learning tasks.
Main Results:
- Synthesized notes under specific conditions improved MM model robustness and generalization.
- Outperformed state-of-the-art medical foundation models.
- Demonstrated the potential of reliable synthesized data for MM dermatology applications.
Conclusions:
- Strategies for generating reliable clinical notes can overcome data limitations in dermatology AI.
- LLM-synthesized data, when carefully managed, can enhance MM dermatology models.
- This work advances the clinical applicability of AI in skin lesion analysis.