Transformer-based NLP approaches for credit risk prediction: a systematic review

This study systematically reviews transformer based Natural Language Processing (NLP) and Large Language Model (LLM) approaches for credit risk prediction, addressing limitations of traditional structured data credit scoring models.

A PRISMA guided systematic literature review was conducted across Scopus, ScienceDirect, and Web of Science, covering English language studies published between 2015 and 2025. A total of 284 records were identified and screened using semantic similarity filtering, resulting in 63 eligible studies for qualitative synthesis.

The findings show that transformer based architectures, including Bidirectional Encoder Representations from Transformers (BERT), Robustly Optimized BERT Pretraining Approach (RoBERTa), and Large Language Model Meta AI (LLaMA), consistently outperform traditional statistical and machine learning baselines across financial prediction tasks. Attention based Long Short Term Memory (LSTM) models reported Area Under the Curve (AUC) improvements of 3.08% and KS increases of 10.3% over classical methods, while hybrid Convolutional Neural Network (CNN) Transformer architectures achieved accuracy levels up to 96.9% and F1 scores of 0.995 in credit risk datasets. Multimodal and transformer based systems also reported accuracy levels exceeding 95% in financial risk monitoring tasks. However, only a limited subset of studies incorporates formal explainability frameworks or fairness evaluations.

Transformer based NLP approaches improve credit risk prediction by effectively leveraging unstructured textual data. Nevertheless, challenges remain in interpretability, transparency, regulatory alignment, and ethical deployment. Future research should prioritize bias mitigation and governance aware model design to support responsible use in regulated financial environments.