A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities

Area of Science:

• Biomedical image analysis
• Artificial intelligence in medicine
• Foundation models

Background:

• Traditional biomedical image analysis separates tasks like segmentation, detection, and recognition.
• This fragmented approach limits holistic understanding and efficiency in biomedical discovery.

Purpose of the Study:

• To introduce BiomedParse, a unified foundation model for joint biomedical image segmentation, detection, and recognition.
• To enhance accuracy across multiple imaging modalities and enable novel applications.

Main Methods:

• Developed BiomedParse, a foundation model for joint analysis of biomedical images.
• Trained the model on a large dataset of over 6 million image, segmentation mask, and textual description triples.
• Leveraged natural language labels from existing datasets for training.

Main Results:

• BiomedParse demonstrated superior performance in image segmentation across nine imaging modalities compared to existing methods.
• Achieved greater improvements on segmenting objects with irregular shapes.
• Successfully enabled simultaneous segmentation and labeling of all objects via textual descriptions.

Conclusions:

• BiomedParse offers an all-in-one solution for biomedical image analysis across major modalities.
• The model facilitates efficient and accurate image-based biomedical discovery.
• Joint learning improves individual task performance and unlocks new analytical capabilities.