Long-range context modeling for software vulnerability detection using an XLNet-based approach

Software vulnerability detection is a critical area of research in cybersecurity. Recently, various Language Model (LM)-based approaches have shown strong potential in this domain. However, most existing methods rely on Transformer architectures that, while powerful, struggle to capture very long-range code dependencies essential for identifying subtle vulnerabilities. To address this limitation, we introduce XLNetVD, an XLNet-based function-level Vulnerability Detection framework leveraging a bidirectional Transformer-XL model for extended context modeling. XLNet effectively captures long code sequences encompassing data flow, control flow, and variable dependencies that are key factors in vulnerability identification. We benchmark XLNet against six mainstream contextual embedding models and three non-contextual embedding models to evaluate its representation capability for vulnerability detection. Experimental results show that XLNet surpasses CodeBERT and GPT-2, achieving the best F1-score of 68%. Furthermore, by applying the Low-Rank Adaptation (LoRA) fine-tuning technique, we demonstrate that XLNet-LoRA achieves the best trade-off between performance and efficiency among LoRA-enhanced LMs. We further integrate XLNet into an end-to-end framework, XLNetVD, and conduct extensive evaluations on two datasets: a real-world dataset with a highly imbalanced vulnerable-to-non-vulnerable ratio of 1:65, and the SARD dataset, which contains balanced, synthetic samples. Results confirm that XLNetVD consistently delivers competitive performance across both real-world and synthetic datasets, establishing it as one of the state-of-the-art vulnerability detection solutions.