Transformer-based NLP approaches for credit risk prediction: a systematic review

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Financial Technology

Background:

• Traditional credit scoring models rely on structured data, limiting their ability to capture nuanced risk factors.
• Unstructured textual data in finance contains valuable information for more accurate credit risk assessment.

Purpose of the Study:

• To systematically review transformer-based Natural Language Processing (NLP) and Large Language Model (LLM) approaches for credit risk prediction.
• To address the limitations of conventional credit scoring methods by exploring advanced AI techniques.

Main Methods:

• A PRISMA-guided systematic literature review was conducted across major academic databases (Scopus, ScienceDirect, Web of Science).
• Studies published between 2015 and 2025 were screened using semantic similarity, resulting in 63 eligible papers for qualitative synthesis.

Main Results:

• Transformer architectures (BERT, RoBERTa, LLaMA) consistently outperform traditional models in financial prediction tasks.
• Attention-based LSTM and hybrid CNN-Transformer models show significant improvements in AUC, KS statistics, accuracy, and F1 scores.
• Multimodal and transformer systems achieve over 95% accuracy in financial risk monitoring, though explainability and fairness evaluations are limited.

Conclusions:

• Transformer-based NLP and LLMs offer superior credit risk prediction by utilizing unstructured data.
• Key challenges include interpretability, transparency, regulatory compliance, and ethical considerations for deployment in financial environments.
• Future research must focus on bias mitigation and governance-aware model design for responsible AI in finance.