ReqTestAgent: execution-grounded language-model agents for acceptance testing under requirement ambiguity
Related Experiment Video
Updated: Nov 4, 2026

Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models
Published on: December 23, 2025
1Faculty of Engineering, The University of Hong Kong, Hong Kong, Hong Kong SAR, China.
Introduction:
Large language models (LLMs) can translate ambiguous requirements into executable tests whose assertions conflict with the accepted behavior. Evaluating such tests requires distinguishing behavioral compatibility from fault-detection strength.
Methods:
We present ReqTestAgent, an execution-grounded workflow that connects requirement interpretation, testing-knowledge retrieval, test generation, and feedback from canonical execution and mutation testing. We evaluate 100 requirement instances formed from 25 Orchid/BigCodeBench tasks, each with lexical, semantic, syntactic, and vagueness variants, using a reference implementation available to the evaluator. Comparisons include three generation baselines, their verifier-enhanced counterparts, and component ablations.
Results:
The generation baselines fail the canonical implementation on 51%-68% of instances, whereas all verifier-enhanced methods achieve 100% canonical pass. ReqTestAgent achieves 92% qualified suites and a macro-averaged effective mutation score of 0.678, compared with 0.655-0.670 for the verifier-enhanced baselines. Removing verification reduces its effective score to 0.216. Across instance-mutant evaluations, ReqTestAgent kills 492 of 756 mutants, a pooled rate of 0.651; this rate weights instances by their mutant counts. A comparative execution trace shows how explicit domain assertions detect faults missed by the verifier-enhanced baselines.
Discussion:
Execution feedback provides the principal validity improvement in this controlled setting. The workflow links interpretation records to executable checks, while the evaluation establishes canonical compatibility as a prerequisite for interpreting mutation-based test strength.
Related Concept Videos
Goodness-of-Fit Test
Hypothesis: Accept or Fail to Reject?
There are two ways to indicate that the null hypothesis is not rejected. 'Accept' the null hypothesis and 'fail to...
Detection of Gross Error: The Q Test
Stereotype Content Model
Quality Assurance
Complementation Tests
Organisms heterozygous for different mutations are crossed pairwise in all combinations. If present on different genes, the mutations can complement each other by providing the missing...