Related Experiment Video
Updated: Jul 4, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models
Katherine Elfer1, Emma Gardecki2, Victor Garcia2
1United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland; National Institutes of Health, National Cancer Institute, Division of Cancer Prevention, Cancer Prevention Fellowship Program, Bethesda, Maryland.
A new framework, CLEARR-AI, ensures reproducible reporting for AI in medical imaging. It evaluates image annotations and metadata crucial for training and testing AI detection and diagnostic models.
Area of Science:
- Medical Imaging AI
- Artificial Intelligence in Healthcare
- Biomedical Informatics
Background:
- Artificial intelligence (AI) models for medical image analysis require robust reporting standards for training and testing data.
- Existing reporting guidelines (CONSORT, SPIRIT, STARD, TRIPOD) are being extended to address AI-specific needs.
- Previous work proposed an annotation workflow and quality checklist for computational pathology.
Purpose of the Study:
- To operationalize an annotation workflow into an evaluable quality checklist for reader-interpreted medical images.
- To demonstrate the utility of this framework for AI model development in digital pathology.
- To introduce the Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence (CLEARR-AI) framework.
Main Methods:
- Developed an evaluable quality checklist based on a prior annotation workflow.
- Applied the checklist to an annotation effort involving digital pathology images.
- Focused on collecting and evaluating annotations and metadata for AI training and testing.
Main Results:
- The CLEARR-AI framework provides a practical tool for assessing the quality of medical image annotations.
- Demonstrated the framework's applicability to reader-interpreted medical images, specifically in digital pathology.
- Highlighted the importance of transparent and reproducible reporting for AI in medical diagnostics.
Conclusions:
- The CLEARR-AI framework enhances the quality and reproducibility of annotations used in medical AI.
- This reporting framework is essential for reliable AI model development and deployment in healthcare.
- CLEARR-AI supports transparent reporting, crucial for the validation and clinical integration of diagnostic AI.
