Threshold Average Precision (TAP-k): a measure of retrieval designed for bioinformatics

Area of Science:

• Bioinformatics
• Computational Biology
• Information Retrieval

Background:

• Accurate measurement of retrieval efficacy is crucial for bioinformatics progress.
• Existing methods, such as the pooled receiver operating characteristic for n irrelevant records (ROC(n)) score, have limitations and do not accurately reflect real-world usage.
• The pooled ROC(n) score can be overly sensitive to individual query results.

Purpose of the Study:

• To address the shortcomings of current retrieval efficacy measures in bioinformatics.
• To propose a novel metric that better reflects the practical application of retrieval algorithms.
• To establish criteria for an ideal measure of retrieval efficacy.

Main Methods:

• Introduction of the Threshold Average Precision (TAP-k) metric, an adaptation of average precision from information retrieval.
• Development of three new criteria for evaluating retrieval efficacy measures.
• Utilizing PSI-BLAST, GLOBAL, HMMER, and RPS-BLAST for comparative analysis.

Main Results:

• The pooled ROC(n) score can yield evaluations that deviate significantly from expected outcomes.
• The proposed TAP-k metric demonstrates superior performance and satisfies most desired criteria for retrieval efficacy.
• Real-world data examples highlight the deficiencies of the pooled ROC(n) score.

Conclusions:

• The Threshold Average Precision (TAP-k) offers a more reliable and accurate method for assessing bioinformatics retrieval algorithm performance.
• TAP-k provides a robust alternative to the pooled ROC(n) score, addressing its sensitivity and usage reflection issues.
• A freely available TAP-k web server and script facilitate its adoption in the bioinformatics community.