Related Experiment Video
Updated: May 8, 2026
Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography
Published on: November 30, 2022
Deep Learning Models Connecting Images and Text: A Primer for Radiologists
An Ni Wu1, Merve Kulbay1, Phillip M Cheng1
1From the Departments of Radiology, Radiation Oncology, and Nuclear Medicine, Centre hospitalier de l'Université de Montréal, Université de Montréal, 1000 rue Saint-Denis, D03.5431, Montreal, QC, Canada H2X 0C1 (A.N.W., A.C.C., L.L.G., A.T.); Centre de recherche du Centre hospitalier de l'Université de Montréal, Montreal, Quebec, Canada (A.N.W., M.K., L.L.G., E.M., I.B.A., A.T.); Department of Ophthalmology and Visual Sciences, McGill University, Montreal, Quebec, Canada (M.K.); Department of Radiology, Keck School of Medicine of the University of Southern California, Los Angeles, Calif (P.M.C.); Department of Medical Imaging, CISSS Lanaudiére, Université Laval, Joliette, Quebec, Canada (A.C.C.); AFX Medical, Montreal, Quebec, Canada (G.C.); Department of Medical Imaging, Western University, London, Ontario, Canada (J.C.); École de Technologie Supérieure, Montreal, Quebec, Canada (I.B.A.); and Institute of Biomedical Engineering, Université de Montréal, Montreal, Quebec, Canada (A.T.).
None:
In radiology practice, medical images are described and interpreted by radiologists in text reports. Recent technical developments enabling deep learning models to connect images and text may facilitate the radiologic workflow. These developments include advances in data embedding, self-supervised learning, zero-shot learning, and transformer-based model architectures. Models connecting images and text can be divided into four categories: (a) Text-image alignment models associate text descriptions with corresponding images. (b) Image-to-text models create text descriptions from images. (c) Text-to-image models generate images from text descriptions. (d) Multimodal models integrate and interpret multiple types of data such as images, videos, text, and numbers simultaneously. Potential clinical applications of these models include automated captioning of medical images, generation of the preliminary radiology report, and creation of educational images. These advances may enable case prioritization, streamlining of clinical workflows, and improvements in diagnostic accuracy. Published under a CC BY 4.0 license.
