Study Design Indexing in Transition: A Focused Comparison of manual NLM Indexing vs. Transformer-based Automated...

Area of Science:

• Biomedical informatics
• Clinical study design
• Evidence synthesis

Background:

• Accurate indexing of biomedical publications is vital for evidence retrieval and synthesis.
• Existing automated systems rely on National Library of Medicine (NLM) indexing, which has known errors and limitations.
• Transformer-based models (TM) offer a potential alternative for study design indexing.

Purpose of the Study:

• To evaluate the accuracy and suitability of a transformer-based model (TM) for indexing clinical study designs.
• To compare TM indexing performance against NLM indexing.
• To address challenges in automated indexing, including NLM assignment errors and differing indexing goals.

Main Methods:

• A transformer-based model (TM) was evaluated on its ability to index four study designs: cohort, case-control, cross-sectional, and case report.
• The evaluation focused on confident TM predictions (high and low scores) that disagreed with NLM assignments.
• Dual annotators established ground truth by independently indexing articles, with comparisons made for 2016 (manual NLM) and 2025 (automated NLM) publications.

Main Results:

• For case-control, case report, and cross-sectional designs, TM demonstrated high accuracy (86-100%) in identifying articles exhibiting the design when NLM failed.
• Conversely, TM's low-confidence predictions correctly identified articles *not* exhibiting the design in 0-21% of cases when NLM assigned it.
• TM's confident predictions were highly accurate and distinct from automated NLM indexing, except for cohort studies where both methods showed significant errors.

Conclusions:

• Transformer-based models show promise for identifying articles that exhibit specific study designs, crucial for clinical decision-making and evidence synthesis.
• NLM indexing for cohort studies is unreliable as a gold standard for training or evaluating automated systems.
• Further research is needed to create a new, manually annotated corpus for improving automated indexing of cohort studies.