Sequence-to-sequence pretraining for a less-resourced Slovenian language

Area of Science:

• Natural Language Processing
• Machine Learning
• Computational Linguistics

Background:

• Large pretrained language models like BERT and T5 have advanced NLP.
• T5's sequence-to-sequence objective is well-suited for text generation.
• Existing T5 models are often limited to well-resourced languages.

Purpose of the Study:

• To develop and evaluate T5-type sequence-to-sequence models for the Slovene language.
• To assess the performance of these models on various classification and generation tasks.
• To compare SloT5 models against multilingual and monolingual baselines.

Main Methods:

• Trained two sizes of T5-type models for Slovene (SloT5).
• Evaluated models on 11 tasks: 8 classification (NER, sentiment, QA, NLI, coreference, lemmatization) and 3 generation (simplification, summarization).
• Compared SloT5 against mT5, mBART-50, multilingual BERT, XLM-RoBERTa, a trilingual BERT, and monolingual SloBERTa.

Main Results:

• SloT5 models generally underperformed the monolingual SloBERTa on classification tasks.
• SloT5 models demonstrated effectiveness in text generation tasks, yielding useful results.
• Model performance is influenced by size, and insufficient Slovene training data limits large model pretraining.

Conclusions:

• SloT5 models are valuable for Slovene text generation, despite limitations in classification.
• The findings suggest potential generalizability to other low-resource languages.
• Training code and models are publicly released to facilitate further research.