A Hierarchical Model for Accuracy and Choice on Standardized Tests.
Steven Andrew Culpepper1, James Joseph Balamuta2
1Department of Statistics, University of Illinois at Urbana-Champaign, 725 South Wright Street, Champaign, IL, 61820 , USA. sculpepp@illinois.edu.
Psychometrika
|November 27, 2015
Summary
Allowing test-takers choice in standardized testing can improve score precision. An
Area of Science:
- Psychometrics
- Educational Measurement
- Statistics
Background:
- Standardized testing often lacks flexibility, potentially impacting score accuracy.
- Existing methods for improving score precision face limitations.
Purpose of the Study:
- To assess the psychometric value of offering test-takers choice in standardized tests.
- To introduce a novel framework and methodology for analyzing cognitive responses and item choices.
- To evaluate a new test administration design for enhanced score precision.
Main Methods:
- Developed a hierarchical framework for jointly modeling cognitive accuracy and item choices.
- Introduced the 'answer two, choose one' (A2C1) test administration design.
- Utilized the 'cIRT' R package for statistical analysis.
Main Results:
- The A2C1 design and payout structure successfully encouraged choices aligned with cognitive abilities.
- Item choices provided information and discrimination comparable to cognitive items.
- Theoretical results identified conditions where choice enhances score precision.
Conclusions:
- Test-taker choice, particularly with the A2C1 design, can be a viable mechanism to improve score precision in standardized testing.
- The 'cIRT' R package offers a practical tool for implementing and analyzing such designs.
- This approach provides a potential solution for enhancing measurement accuracy without solely relying on item writing expertise.
Related Concept Videos
01:52Accuracy and Precision
3.1K
3.1K
01:52Accuracy and Precision
17.6K
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value. Highly accurate...
17.6K
01:13Accuracy and Errors in Hypothesis Testing
680
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
680
01:29Measures of Intelligence
8.9K
Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
8.9K
01:29Reliability and Validity
14.4K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
14.4K
01:19Testing a Claim about Standard Deviation
3.2K
A complete procedure to test a claim about population standard deviation or population variance is explained here.
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
3.2K