Promises and perils of using Transformer-based models for SE research

Area of Science:

• Computer Science
• Software Engineering

Background:

• Transformer-based pre-trained models are increasingly used for code-related tasks.
• A comprehensive analysis of these models' performance, resource consumption, and generalization is needed.

Purpose of the Study:

• To empirically analyze 519 papers on transformer models for code (2017-2023).
• To evaluate model architectures (CodeBERT, CodeGPT, CodeT5) on key software engineering tasks (Bug Fixing, Bug Detection, Code Summarization, Code Search).
• To assess resource consumption and generalization capabilities.

Main Methods:

• Systematic literature review of 519 papers.
• Empirical experiments with CodeBERT, CodeGPT, and CodeT5.
• Evaluation across four software engineering tasks.

Main Results:

• Encoder-only models (CodeBERT) can outperform encoder-decoder models on general tasks.
• Decoder-only models (CodeGPT) show promise for specific generation tasks.
• Less popular models may offer superior performance.
• CodeBERT is efficient for understanding tasks; CodeT5's efficiency is inconsistent for generation.
• Poor model generalization observed on popular Bug Fixing and Code Summarization benchmarks.

Conclusions:

• Transformer models offer significant potential but face practical challenges.
• Model architecture choice is critical and task-dependent.
• Generalization remains a key area for improvement in code intelligence research.