iSentenizer-μ: multilingual sentence boundary detection model.

Area of Science:

• Natural Language Processing
• Computational Linguistics

Background:

• Sentence boundary detection (SBD) systems are typically sensitive to training data genres and languages.
• Retraining SBD models for new data requires discarding previous work and starting from scratch.

Purpose of the Study:

• To introduce iSentenizer-μ, a novel multilingual SBD system.
• To develop an adaptable SBD system capable of handling diverse text genres and languages.

Main Methods:

• Utilized an incremental tree learning architecture, specifically the i(+)Learning algorithm.
• Developed a system adaptable to various text topics and Roman-alphabet languages through incremental knowledge merging.
• Designed iSentenizer-μ to revise existing models rather than requiring complete retraining.

Main Results:

• iSentenizer-μ demonstrated high accuracy in detecting sentence boundaries across a mixture of text genres and languages.
• The system was extensively evaluated on Danish, German, English, Spanish, Dutch, French, Italian, Portuguese, Greek, Finnish, and Swedish.
• Outperformed two state-of-the-art SBD systems, Punkt and MaxEnt, on all tested datasets.

Conclusions:

• The proposed iSentenizer-μ system offers a robust and adaptable solution for multilingual SBD.
• Incremental learning enables efficient adaptation to new data, overcoming limitations of traditional retraining approaches.
• iSentenizer-μ provides superior performance compared to existing SBD systems in diverse linguistic and topical contexts.