Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies
  1. Home
  3. Xrayswingen: Automatic Medical Reporting For X-ray Exams With Multimodal Model.
  1. Home
  3. Xrayswingen: Automatic Medical Reporting For X-ray Exams With Multimodal Model.

Related Concept Videos

X-ray Imaging01:24

X-ray Imaging

5.5K
German physicist Wilhelm Röntgen (1845–1923) was experimenting with electrical current when he discovered that a mysterious and invisible "ray" would pass through his flesh but leave an outline of his bones on a screen coated with a metal compound. In 1895, Röntgen made the first durable record of the internal parts of a living human: an "X-ray" image (as it came to be called) of his wife’s hand. Scientists worldwide quickly began their own experiments with...
5.5K
Computed Tomography01:10

Computed Tomography

4.5K
Tomography refers to imaging by sections. Computed tomography (CT) is a non-invasive imaging technique that uses computers to analyze several cross-sectional X-rays to reveal minute details about structures in the body.
The technique was invented in the 1970s and is based on the principle that as X-rays pass through the body, they are absorbed or reflected at different levels. In the technique, a patient lies on a motorized platform while a computerized axial tomography (CAT) scanner rotates...
4.5K
Imaging Studies for Cardiovascular System III: X-Ray01:20

Imaging Studies for Cardiovascular System III: X-Ray

180
The most common cardiovascular diagnostic test is an X-ray. It produces images of the heart, blood vessels, and adjacent structures.
Definition and Purpose
An X-ray, or radiograph, is a non-invasive method that uses ionizing radiation to take images of internal structures. It is mainly used in cardiac imaging to examine the heart, lungs, and major blood vessels, aiming to identify abnormalities in the heart's size, shape, and position, such as heart failure, congenital defects, and vascular...
180
Positron Emission Tomography01:29

Positron Emission Tomography

4.2K
Positron emission tomography (PET) is a medical imaging technique involving radiopharmaceuticals — substances that emit short-lived radiation. Although the first PET scanner was introduced in 1961, it took 15 more years before radiopharmaceuticals were combined with the technique and revolutionized its potential.
One of the main requirements of a PET scan is a positron-emitting radioisotope, which is produced in a cyclotron and then attached to a substance used by the part of the body...
4.2K
Magnetic Resonance Imaging01:24

Magnetic Resonance Imaging

5.1K
Magnetic resonance imaging (MRI) is a noninvasive medical imaging technique based on a phenomenon of nuclear physics discovered in the 1930s, in which matter exposed to magnetic fields and radio waves was found to emit radio signals. In 1970, a physician and researcher named Raymond Damadian noticed that malignant (cancerous) tissue gave off different signals than normal body tissue. He applied for a patent for the first MRI scanning device in clinical use by the early 1980s. The early MRI...
5.1K

Related Experiment Video

Author Spotlight: An Efficient and Robust Software for Automated Fusion of Multiple Preclinical Imaging Modalities
07:13

Author Spotlight: An Efficient and Robust Software for Automated Fusion of Multiple Preclinical Imaging Modalities

Published on: October 27, 2023

1.1K

XRaySwinGen: Automatic medical reporting for X-ray exams with multimodal model.

Gilvan Veras Magalhães1, Roney L de S Santos1, Luis H S Vogado1

  • 1Departamento de Computação, Universidade Federal do Piauí, Teresina, Brazil.

Heliyon
|April 1, 2024

View abstract on PubMed

Summary
This summary is machine-generated.

This study introduces an AI model for generating detailed radiology reports from X-rays, improving diagnostic accuracy. The multimodal approach enhances medical image captioning for better patient condition insights.

Keywords:
Computer visionMedical reportMultimodalNatural language processingTransformers

More Related Videos

X-ray Dose Reduction through Adaptive Exposure in Fluoroscopic Imaging
08:30

X-ray Dose Reduction through Adaptive Exposure in Fluoroscopic Imaging

Published on: September 11, 2011

14.4K
3D Printing of Preclinical X-ray Computed Tomographic Data Sets
11:06

3D Printing of Preclinical X-ray Computed Tomographic Data Sets

Published on: March 22, 2013

40.5K

Related Experiment Videos

Author Spotlight: An Efficient and Robust Software for Automated Fusion of Multiple Preclinical Imaging Modalities
07:13

Author Spotlight: An Efficient and Robust Software for Automated Fusion of Multiple Preclinical Imaging Modalities

Published on: October 27, 2023

1.1K
X-ray Dose Reduction through Adaptive Exposure in Fluoroscopic Imaging
08:30

X-ray Dose Reduction through Adaptive Exposure in Fluoroscopic Imaging

Published on: September 11, 2011

14.4K
3D Printing of Preclinical X-ray Computed Tomographic Data Sets
11:06

3D Printing of Preclinical X-ray Computed Tomographic Data Sets

Published on: March 22, 2013

40.5K

Area of Science:

  • Medical Imaging and Artificial Intelligence
  • Computer Vision and Natural Language Processing

Background:

  • Manual radiology report generation is time-consuming and prone to errors.
  • Current AI in medical imaging often focuses on classification, missing crucial patient condition details.
  • Generating descriptive reports from X-rays remains a challenge for AI.

Purpose of the Study:

  • To develop a multimodal AI model for generating detailed, descriptive radiology reports from X-ray images.
  • To overcome limitations in current AI medical image captioning by focusing on comprehensive report generation.
  • To improve the accuracy and efficiency of medical reporting through advanced AI techniques.

Main Methods:

  • Utilized a multimodal model combining Computer Vision (Swin Transformer) for image encoding and Natural Language Processing (GPT-2) for text decoding.
  • Implemented Swin Transformer for hierarchical image representation, enhancing perception without increasing computational cost.
  • Employed cross-attention layers and bilingual training (Portuguese PT-BR and English) for robust report generation.

Main Results:

  • Achieved promising performance on a proposed database with ROUGE-L of 0.748 and METEOR of 0.741.
  • Demonstrated effectiveness on the NIH CHEST X-ray dataset, yielding ROUGE-L of 0.404 and METEOR of 0.393.
  • The model successfully generated descriptive textual reports from X-ray images.

Conclusions:

  • The proposed multimodal AI model effectively generates detailed radiology reports, addressing limitations in current medical image captioning.
  • The innovative use of Swin Transformer and GPT-2 offers an efficient and effective approach to AI-driven medical report generation.
  • Bilingual training and promising results indicate the model's potential for improving radiology workflows and diagnostic capabilities.