Tucano: Advancing neural text generation for Portuguese

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Artificial Intelligence

Background:

• Deep learning models require extensive data and computational power, creating a resource gap for low-resource languages.
• Existing natural language processing (NLP) advancements primarily benefit high-resource languages, leaving others behind.

Purpose of the Study:

• To bridge the resource gap in NLP by developing resources for Portuguese neural text generation.
• To introduce GigaVerbo, a large-scale Portuguese text corpus, and Tucano, a family of transformer models.

Main Methods:

• Development of GigaVerbo, a Portuguese text corpus comprising 200 billion tokens.
• Training of Tucano, a series of decoder-only transformer models utilizing the GigaVerbo corpus.
• Evaluation of Tucano models against Portuguese and multilingual baselines on various NLP benchmarks.

Main Results:

• Tucano models demonstrated superior performance compared to existing Portuguese and multilingual models across multiple benchmarks.
• The developed resources, including models and datasets, facilitate reproducible research in NLP for Portuguese.

Conclusions:

• The GigaVerbo corpus and Tucano models represent a significant advancement for neural text generation in Portuguese.
• Openly releasing these resources promotes further research and development in low-resource language NLP.