Related Experiment Videos
Advancing biomedical image retrieval: development and analysis of a test collection
William R Hersh1, Henning Müller, Jeffery R Jensen
1Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University, BICC, Portland, OR 97239, USA. hersh@ohsu.edu
Summary
This study developed an image retrieval test collection, finding that combined visual and textual methods yielded the best results. Textual methods proved more robust across diverse topics than purely visual approaches.
Area of Science:
- Information Retrieval
- Computer Vision
- Natural Language Processing
Background:
- Developing effective image retrieval systems requires robust evaluation methodologies.
- Existing test collections may not adequately assess diverse retrieval strategies.
Purpose of the Study:
- To develop and analyze results from a novel image retrieval test collection.
- To evaluate the performance of various image retrieval approaches, including visual, textual, and mixed methods.
Main Methods:
- Collected and analyzed results from 13 research groups participating in the Cross-Language Evaluation Forum image retrieval task.
- Categorized topics and system runs based on visual, textual, or mixed approaches and query methods (automated vs. manual).
- Assessed the impact of duplicate relevance judgments on retrieval performance.
Main Results:
- Systems combining visual and textual methods achieved the best performance.
- Significant performance variations were observed across different topics.
- Textual methods demonstrated greater resilience to visually oriented topics compared to visual methods on textually oriented topics.
- Mean Average Precision (MAP) did not always correlate with user-centric measures like precision at 10 or 30 images.
Conclusions:
- A new test collection was established for evaluating visual and textual image retrieval.
- Further research is needed to understand the influence of topic and run types on retrieval performance.
- User studies are essential for identifying optimal evaluation metrics for image retrieval systems.