Related Experiment Videos
How students use generative AI for software testing: An observational study
Baris Ardic1, Quentin Le Dilavrec1, Andy Zaidman1
1Computer Science, Delft University of Technology, Delft, 2628XE Netherlands.
Empirical Software Engineering
|June 23, 2026
Summary
Novice developers using generative AI for unit testing reported time savings and reduced cognitive load. However, they also expressed concerns about trust, quality, and ownership, with interaction strategies not significantly impacting test effectiveness.
Area of Science:
- Software Engineering
- Artificial Intelligence
- Human-Computer Interaction
Background:
- Generative AI tools like ChatGPT are increasingly integrated into software engineering.
- AI-assisted workflows may alter the developer's role, impacting control, quality, and learning, especially for novices.
Purpose of the Study:
- Investigate how novice software developers interact with generative AI for unit test engineering.
- Examine novice developers' strategies, reliance on AI, and perceived benefits/challenges in AI-assisted test engineering.
Main Methods:
- Observational study with 12 undergraduate students using ChatGPT (GPT-3.5) for unit testing tasks.
- Identified four interaction strategies based on the origin of test ideas and implementation (AI vs. participant).
- Analyzed prompting styles (one-shot vs. iterative) and their alignment with interaction strategies.
Main Results:
- Four distinct interaction strategies and associated prompting styles were identified.
- Students reported benefits: time-saving, reduced cognitive load, and aid in test ideation.
- Students reported drawbacks: diminished trust, quality concerns, and lack of ownership.
Conclusions:
- Interaction and prompting strategies influenced workflow dynamics but did not significantly impact test effectiveness or code quality (mutation score, test smells).
- Generative AI offers productivity gains but necessitates careful consideration of novice developer experience, trust, and ownership.
Related Concept Videos
01:16Non-equilibrium in the Cell
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
01:11Observational Studies
Observational studies are a type of analytical study where researchers observe events without any interventions. In other words, the researcher does not influence the response variable or the experiment's outcome.
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One example of...
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One example of...
01:28Automatic Processing and Automatic Social Behavior
Automatic processing refers to the cognitive operations that occur without conscious intent or awareness, playing a fundamental role in shaping social cognition and behavior. These processes enable individuals to navigate complex social environments efficiently by relying on mental shortcuts and pre-existing knowledge structures known as schemas. One of the most influential mechanisms underlying automatic processing is priming, which subtly activates mental representations through exposure to...