Generating reliable software project task flows using large language models through prompt engineering and robust...

Area of Science:

• Artificial Intelligence
• Software Engineering
• Natural Language Processing

Background:

• Large Language Models (LLMs) show potential for transforming unstructured software documentation into structured task flows.
• However, LLM-generated outputs often lack the procedural reliability essential for software engineering tasks.

Purpose of the Study:

• To benchmark leading LLMs (Gemini 2.5 Pro, Grok 3, GPT-Omni, DeepSeek-R1, LLaMA-3) using diverse prompting strategies.
• To introduce and validate a novel evaluation metric, the Hybrid Semantic Similarity Metric (HSSM), for assessing procedural reliability.

Main Methods:

• Utilized real-world software tutorials from the "Build Your Own X" repository for benchmarking.
• Implemented five prompting strategies: Zero-Shot, Chain-of-Thought, and ISO 21502-Guided.
• Developed HSSM, combining SentenceTransformer embeddings and context-aware key-term overlap for semantic and procedural evaluation.

Main Results:

• HSSM demonstrated superior performance over traditional metrics (BERTScore, SBERT, USE) with lower variance (1.5-2.9% CV) and higher correlation with human judgments.
• Even Zero-Shot prompting achieved high alignment (96.33% HSSM) for task flow generation when evaluated with HSSM.
• LLMs showed varying performance based on prompting strategies and model architecture.

Conclusions:

• The study provides a scalable framework for evaluating LLM-generated task flows in software engineering.
• HSSM offers a robust method for assessing procedural coherence, crucial for reliable AI-assisted software planning.
• Findings suggest potential for LLMs in AI-driven project management, prompt engineering, and procedural generation tools.