Besides precision & recall: exploring alternative approaches to evaluating an automatic indexing tool for MEDLINE

Area of Science:

• Biomedical Informatics
• Information Retrieval
• Medical Librarianship

Background:

• Traditional evaluation of automatic indexing tools, such as the Medical Text Indexer (MTI) for MEDLINE, relies heavily on precision and recall.
• There is a need for complementary evaluation methods to assess the performance of automatic indexing in producing MeSH (Medical Subject Headings) recommendations.
• Evaluating semantic similarity and retrieval performance offers alternative perspectives on indexing tool effectiveness.

Purpose of the Study:

• To explore and evaluate alternative approaches for assessing the performance of an automatic indexing tool for MEDLINE.
• To complement the traditional precision and recall metrics with new evaluation strategies.
• To examine the utility of semantic similarity and retrieval comparison for indexing tool evaluation.

Main Methods:

• The Medical Text Indexer (MTI) was evaluated on a random set of MEDLINE citations.
• Semantic similarity of indexing terms was assessed at the term level.
• Documents retrieved using MTI index terms were compared against PubMed related citations for relevance.

Main Results:

• Semantic similarity scores between index term sets were found to be higher than Dice similarity scores.
• Queries based on automatic indexing retrieved 75% of the original documents.
• 58% of the top ten related citations were retrieved by queries based on automatic indexing.

Conclusions:

• Alternative evaluation measures, including semantic similarity and retrieval comparison, support previous findings on indexing tool performance.
• These methods can effectively identify specific documents within a test set requiring more in-depth analysis.
• The study validates the utility of these complementary approaches for evaluating automatic indexing systems in biomedical contexts.