Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review

Area of Science:

• Computer Science
• Artificial Intelligence
• Software Engineering

Background:

• Natural Language Processing (NLP) techniques, particularly transformer-based large language models (LLMs), are increasingly vital in AI-assisted programming.
• LLMs trained on Big Code, incorporating software naturalness, power applications like GitHub Copilot and DeepMind AlphaCode.
• These models are integral to diverse programming tasks including code generation, completion, translation, refinement, summarization, and defect/clone detection.

Purpose of the Study:

• To provide a comprehensive literature review on the application of NLP and LLMs in AI-assisted programming.
• To overview major LLMs and their downstream applications in software development.
• To explore challenges and opportunities in integrating NLP with software naturalness for enhanced coding assistance.

Main Methods:

• Literature review of NLP techniques and transformer-based LLMs in AI-assisted programming.
• Analysis of LLM applications in code generation, completion, translation, refinement, summarization, defect detection, and clone detection.
• Exploration of challenges and opportunities, including extending capabilities to mobile development environments like Xcode.

Main Results:

• LLMs trained on Big Code significantly enhance AI-assisted programming tasks.
• Applications like GitHub Copilot demonstrate the practical impact of these models.
• The integration of NLP with software naturalness offers substantial opportunities for developer empowerment.

Conclusions:

• LLMs trained on Big Code are transformative for AI-assisted programming.
• Further research into NLP integration can unlock advanced coding assistance and streamline development.
• Extending these capabilities to mobile development platforms like Xcode presents a significant future opportunity.