Learning semantic and visual similarity for endomicroscopy video retrieval

Area of Science:

• Computer Vision
• Medical Informatics
• Biomedical Engineering

Background:

• Content-based image retrieval (CBIR) aids medical diagnosis but lacks physician interpretability.
• Traditional CBIR systems provide only visual outputs, limiting clinical utility.
• Endomicroscopy video analysis requires retrieval systems that offer both visual and semantic information.

Purpose of the Study:

• To develop an endomicroscopy video retrieval system providing consistent visual and semantic outputs.
• To enhance CBIR by integrating semantic knowledge extraction with visual similarity learning.
• To improve the interpretability and clinical relevance of retrieval results for physicians.

Main Methods:

• Developed "Dense-Sift," an adapted bag-of-visual-words method for endomicroscopy video signature computation.
• Leveraged a semantic ground truth of eight binary concepts to transform visual signatures into semantic signatures.
• Employed an intuitive Fisher-based method for semantic detection, outperforming Support Vector Machine (SVM) methods.
• Learned an adjusted similarity distance from perceived similarity ground truth to improve retrieval relevance.

Main Results:

• The Fisher-based method significantly outperformed SVMs in semantic detection accuracy.
• The distance learning method statistically improved the correlation with perceived similarity.
• Semantic signatures achieved recall performance close to visual signatures and superior to state-of-the-art CBIR methods.
• Semantic signatures effectively communicate high-level medical knowledge, consistent with visual data.

Conclusions:

• The proposed system successfully integrates visual and semantic retrieval for endomicroscopy videos.
• Semantic signatures offer a concise, interpretable, and clinically relevant output for physicians.
• This approach enhances CBIR systems, providing valuable diagnostic support in medical imaging.