Manual versus machine: How accurately does the Medical Text Indexer (MTI) classify different document types into...

Area of Science:

• Biomedical Informatics
• Medical Text Analysis
• Controlled Vocabularies

Background:

• The Medical Subject Headings (MeSH) thesaurus is a vital controlled vocabulary for classifying biomedical literature.
• Automated indexing using algorithms like the Medical Text Indexer (MTI) assists researchers in categorizing text.
• The reliability of MTI for document types beyond traditional publications requires evaluation.

Purpose of the Study:

• To assess the reliability of the Medical Text Indexer (MTI) algorithm for classifying text from grants, patents, and drug indications.
• To compare MTI's automated classification performance against expert manual classification.
• To identify methods for improving MTI's recall and precision across different document types.

Main Methods:

• Collected text samples from research grants, patents, and drug indications.
• Utilized the Medical Text Indexer (MTI) to assign MeSH descriptors to the collected texts.
• Compared MTI's classifications with expert manual classifications, calculating recall and precision metrics.
• Analyzed the impact of MTI ranking scores, document length, and MeSH category aggregation on performance.

Main Results:

• MTI demonstrated high recall (78-86%) across grants, patents, and drug indications.
• Precision varied, with 53% for grants, 73% for patents, and 64% for drug indications.
• MTI achieved >94% recall and >87% precision for detecting the presence of any disease.
• Recall and precision could be enhanced by using MTI ranking scores, excluding long documents, and aggregating MeSH categories.

Conclusions:

• The Medical Text Indexer (MTI) is a potentially valuable tool for researchers classifying diverse texts into MeSH categories.
• MTI's performance is promising, especially for identifying disease-related terms, but requires careful application.
• Optimizing MTI usage through ranking scores and document filtering can improve its accuracy in medical text classification.