Hybrid Ensemble-Rule Algorithm for Improved MEDLINE® Sentence Boundary Detection

Area of Science:

• Computational Linguistics
• Bioinformatics
• Natural Language Processing

Background:

• Sentence boundary detection (SBD) is crucial for Natural Language Processing (NLP).
• Standard SBD methods struggle with complex corpora like MEDLINE citations due to ambiguities.
• Errors in SBD negatively affect downstream NLP tasks.

Purpose of the Study:

• To develop a robust SBD algorithm for challenging scientific text.
• To improve the accuracy of sentence segmentation in MEDLINE citations.
• To overcome limitations of existing SBD engines.

Main Methods:

• Implemented a majority voting system combining three SBD engines: Python NLTK, pySBD, and Syntok.
• Developed custom post-processing rules using spaCy for part-of-speech tagging, abbreviation, and capitalization detection.
• Utilized general sentence statistics for refinement.

Main Results:

• The proposed hybrid approach demonstrated superior performance compared to individual SBD engines.
• Experiments on thousands of MEDLINE citations validated the algorithm's effectiveness.
• Significant improvements in SBD accuracy were observed.

Conclusions:

• Combining multiple SBD engines with tailored post-processing rules enhances accuracy for complex corpora.
• The developed algorithm offers a more reliable solution for SBD in biomedical text.
• This advancement supports more effective NLP applications in scientific domains.