Pedagogically Aligned Objectives Create Reliable Automatic Cloze Tests
Brian Ondov1, Dina Demner-Fushman1, Kush Attal2
1National Library of Medicine, Bethesda, MD, USA.
Proceedings of the Conference. Association for Computational Linguistics. North American Chapter. Meeting
|September 10, 2025
Summary
This study introduces a new method for creating cloze test distractors, improving upon masked language models. The developed approach generates more reliable and valid educational assessments for human comprehension.
Area of Science:
- Natural Language Processing
- Educational Technology
- Computational Linguistics
Background:
- Masked Language Models (MLMs) are suitable for generating plausible distractors in cloze tests.
- Existing neural methods do not adequately ensure distractors are distinct and incorrect.
- Automated metrics for evaluating cloze tests lack reliability and validity for human comprehension.
Purpose of the Study:
- To formulate pedagogically motivated objectives (plausibility, incorrectness, distinctiveness) for distractor generation.
- To develop an unsupervised, interpretable method for jointly optimizing distractor sets.
- To evaluate the reliability and validity of generated cloze tests with human participants.
Main Methods:
- Formulated distractor objectives using conditional distributions from language models.
- Developed an unsupervised method for jointly optimizing distractor sets based on formulated objectives.
- Conducted human evaluations to compare cloze test reliability and validity against other methods.
Main Results:
- The proposed method demonstrates stronger correlation with teacher-created comprehension tests than state-of-the-art neural methods.
- The generated cloze tests exhibit greater internal consistency.
- The implementation is freely available for creating multiple-choice cloze tests.
Conclusions:
- The developed method effectively generates plausible, distinct, and incorrect distractors for cloze tests.
- This approach enhances the reliability and validity of automated cloze test creation.
- The tool offers a practical solution for educators seeking to create effective comprehension assessments.
Related Concept Videos
00:49Complementation Tests
6.1K
A complementation test is a simple cross to identify whether the two mutations are located on the same gene or different genes. It was first performed by Edward Lewis in the 1940s while working on fruit flies. He developed the test to identify the location and arrangement of different mutations on chromosomes.
Organisms heterozygous for different mutations are crossed pairwise in all combinations. If present on different genes, the mutations can complement each other by providing the missing...
Organisms heterozygous for different mutations are crossed pairwise in all combinations. If present on different genes, the mutations can complement each other by providing the missing...
6.1K
01:22Purposive Learning
451
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
451
01:21Cognitive Learning
1.0K
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
1.0K
01:29Reliability and Validity
13.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.7K
01:28Automatic Processing and Automatic Social Behavior
219
Automatic processing refers to the cognitive operations that occur without conscious intent or awareness, playing a fundamental role in shaping social cognition and behavior. These processes enable individuals to navigate complex social environments efficiently by relying on mental shortcuts and pre-existing knowledge structures known as schemas. One of the most influential mechanisms underlying automatic processing is priming, which subtly activates mental representations through exposure to...
219
01:37Social Loafing
38.9K
Another way in which a group presence can affect performance is social loafing—the exertion of less effort by a person working together with a group. Social loafing occurs when our individual performance cannot be evaluated separately from the group. Thus, group performance declines on easy tasks (Karau & Williams, 1993). Essentially individual group members loaf and let other group members pick up the slack. Because each individual’s efforts cannot be evaluated,...
38.9K