Development of Immediate Self-Feedback Very Short Answer Questions: Implementing Testlet Response Theory in Formative
Sethapong Lertsakulbunlue1, Anupong Kantiwong1
1Department of Pharmacology, Phramongkutklao College of Medicine, Bangkok, 10400 Thailand.
Background:
The novel Immediate Self-feedback Very Short Answer Question (ISF-VSAQ) format exhibited testlet effect due to its instant feedback feature. This study utilized the testlet-as-a-polytomous-item-modeling (TPIM), an extension of item response theory, to tackle the testlet-related challenges.
Methods:
Ninety-four pre-clinical students participated in two ten-item ISF-VSAQ exams on cardiovascular drugs based on a parallel blueprint. Each question involves clinical vignettes followed by expected answers, self-marking, and understanding levels. Messick's validity framework guided the collection of validity evidence.
Results:
Validity evidence included five sources: (1) Content: Experts reviews and alignment with the standard examination blueprint. (2) Response process: Students received an example and guide of ISF-VSAQ. Self-assessment steps were also explained, and all students were provided adequate time. (3) Internal structure: Higher marginal reliability was demonstrated in the TPIM model. While unidimensionality was violated in the second ISF-VSAQ, the TPIM maintained unidimensionality. Item discrimination was generally acceptable, except for one item on the second occasion with minor vignette changes. Items with new knowledge showed better discrimination and item information functions on the second occasion. (4) Relations to other variables: Path analysis shows that the first occasion's understanding and scoring abilities strongly influenced the second occasion, highlighting the testlet effect. (5) Consequences: Path analysis revealed that the understanding ability in the second attempt directly affected the summative cardiovascular (β = 0.26, p = 0.066) and comprehensive examination ability (β = 0.34, p = 0.010).
Conclusion:
The TPIM model effectively tackles the testlet effect of the present ISF-VSAQ. Thus, when employing multiple instances of self-feedback format, analyzing it as a testlet is recommended.
Supplementary Information:
The online version contains supplementary material available at 10.1007/s40670-024-02167-w.
More Related Videos
00:08A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
10:26Problem-Solving Before Instruction PS-I: A Protocol for Assessment and Intervention in Students with Different Abilities
Published on: September 11, 2021
Related Concept Videos
Self-Evaluation: Self-Enhancement and Self-Verification
Metacognition
Surveys
Self-Report Tests of Personality
Transient and Steady-state Response
These test signals are integral in designing control systems to exhibit two key performance aspects: transient response and steady-state...
Reliability and Validity
