Low resource word sense disambiguation in Oromo with fine tuned small transformers

A key task in natural language processing is word sense disambiguation (WSD), which attempts to determine the accurate meaning of ambiguous words based on their context. While transformer-based designs have achieved significant results in high-resource languages, WSD for low-resource languages such as Oromo remains hard due to inadequate annotated corpora and lexical resources. Contextual representation learning has been greatly enhanced by recent advancements in transformer-based language models, allowing for more reliable disambiguation in situations with limited input. This study uses a manually created dataset from the Oromo-English Dictionary to examine the efficacy of transformer-based models for lexical-sample WSD in Oromo. The dataset contains sentences annotated by two native speakers, attaining an inter-annotator agreement of 0.82, indicating good annotation reliability. The dataset was filtered for experimental usage following preprocessing, normalization, and elimination of noisy cases. 472 training sentences, 71 validation sentences (15%), and 140 test sentences made up the final dataset. The dataset has a highly unbalanced long-tail distribution and encompasses 43 sense classes. BERT-base-cased gets the best performance with an accuracy of 0.862 and a macro-F1 score of 0.2897, according to an experimental evaluation of transformer-based models, including BERT, RoBERTa, DistilBERT, Davlan/afro-xlmr-base, and multilingual variations. Significant differences between models with χ² = 34.03 and p = 4.0 × 10^-1 are confirmed by statistical analysis using the Friedman test. BERT-base-cased performs much better than most transformer variations and classical baselines, according to post-hoc Wilcoxon signed-rank tests. These results show that contextual transformer representations are quite successful for low-resource WSD, although there is still a significant class imbalance that limits performance.