High-Quality Medical Image Generation from Free-hand Sketch

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Deep Learning

Background:

• Generating medical images from free-hand sketches is valuable but challenging due to data scarcity.
• Existing deep learning models struggle to generalize from synthesized sketches to real free-hand drawings.
• Current methods often fail to produce satisfactory results on real medical sketches.

Purpose of the Study:

• To develop a practical deep learning model for generating medical images from free-hand sketches.
• To enable cost-effective training using synthesized sketches while achieving robust generalization.
• To improve the quality and realism of generated medical images from user-drawn inputs.

Main Methods:

• Proposed Sketch2MedI, a novel sketch-to-image generation model.
• Leveraged StyleGAN's latent space to represent sketches effectively.
• Trained the model using synthesized sketches, demonstrating cost-effectiveness.
• Evaluated generalization capabilities on real free-hand sketches.

Main Results:

• Sketch2MedI demonstrated robust generalization to free-hand sketches.
• Achieved high-quality and realistic medical image generation, specifically for pharyngeal images.
• Outperformed existing models like pix2pix, CycleGAN, UNIT, and U-GAT-IT in comparative evaluations.
• Showcased superior quantitative and qualitative performance across various metrics.

Conclusions:

• Sketch2MedI offers a practical and effective solution for medical image generation from free-hand sketches.
• The model's ability to encode sketches in StyleGAN's latent space is key to its generalization.
• This approach overcomes the limitations of training solely on synthesized data, paving the way for wider applications.