Towards a unified search: Improving PubMed retrieval with full text

Area of Science:

• Biomedical Informatics
• Information Retrieval

Background:

• PubMed offers growing full-text access via PubMed Central, but integrated searching remains a challenge.
• Current search methods cannot simultaneously query abstracts and full-text articles for unified results.

Purpose of the Study:

• To develop and evaluate methods for scoring full-text articles and integrating these scores with abstract-based retrieval.
• To improve overall retrieval performance by combining information from multiple sources within PubMed.

Main Methods:

• Proposed converting BM25 scores to log odds ratio scores for uniform combination across different article sections.
• Developed a probabilistic transformation to balance contributions from heterogeneous retrieval sources (full-text vs. abstract-only).
• Utilized PubMed click data for training and evaluating retrieval effectiveness.

Main Results:

• BM25 ranking on abstracts improved Mean Average Precision (MAP) by 10.6%.
• Integrating full-text articles yielded a 7% relative improvement over abstract-only retrieval.
• The proposed log odds ratio method effectively normalized and combined query token occurrences across sections.

Conclusions:

• Combining scores from different sections using log odds ratios enhances retrieval accuracy.
• A method balancing full-text and abstract scores significantly improves search performance in PubMed.
• Summing the top three section scores demonstrated the most effective retrieval strategy.