Consequences Validity Evidence: Evaluating the Impact of Educational Assessments
David A Cook1, Matthew Lineberry
1D.A. Cook is professor of medicine and medical education, associate director, Mayo Clinic Online Learning, and consultant, Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota. M. Lineberry is assistant professor of medical education, Department of Medical Education, and assistant director for research, Graham Clinical Performance Center, University of Illinois at Chicago, Chicago, Illinois.
Summary
Consequences validity evidence, which examines assessment impacts, is crucial but poorly understood. This study explains this evidence type and offers a framework for its collection and interpretation.
Area of Science:
- Educational Measurement
- Health Professions Education
- Psychometrics
Background:
- Assessment validity is critical for educational and clinical settings.
- Consequences validity evidence, examining assessment impacts, is underutilized and poorly understood.
- Traditional validity frameworks lack a direct counterpart to consequences validity.
Purpose of the Study:
- To clarify the concept of consequences validity evidence.
- To propose a framework for collecting and interpreting consequences validity evidence.
- To highlight the importance of assessment impact in decision-making.
Main Methods:
- Literature review and conceptual analysis.
- Examination of assessment as an intervention.
- Proposal of a framework for organizing consequences evidence.
Main Results:
- Consequences validity evidence assesses impacts on examinees, educators, and practice outcomes.
- Impacts can be intended or unintended, beneficial or harmful, stemming from score use or the assessment activity.
- Both quantitative and qualitative methods are suitable for gathering this evidence.
Conclusions:
- Consequences validity evidence is essential for robust assessment validation.
- A structured approach is needed to systematically collect and interpret this evidence.
- The rigor and type of evidence depend on the specific assessment and its intended use.
Related Concept Videos
01:29Reliability and Validity
14.3K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
14.3K
01:20Nursing Process for Patient and Caregiver Teaching III: Evaluation and Documentation
2.7K
Evaluation of the teaching process enables the nurse to determine if the patient's learning needs were met and if training was effective. If the expected outcomes are not met, the care plan is revised, and additional education or reinforcement is provided. Nurses can ask questions after the session or obtain feedback to assess the patient's understanding of the topic.
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning,...
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning,...
2.7K
01:15Nursing Evaluation
4.6K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
4.6K
01:27Halo Effect
643
The halo effect is a cognitive bias in which an individual's overall impression influences judgments about their specific traits. This psychological phenomenon leads people to associate positive characteristics with those they perceive as generally good and negative characteristics with those they view as bad. This effect is particularly influential in social perception, professional evaluations, and decision-making processes.The Psychological Basis of the Halo EffectThe halo effect is rooted...
643
02:16Surveys
17.2K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
17.2K
01:24Sources of Self-Esteem II: Performance Feedback
211
Self-esteem is intricately tied to our perception of competence and our ability to exert control over our lives. One of the primary sources of this perception is performance feedback — the ongoing evaluation of our actions in terms of success and failure. According to Franks and Marolla (1976), people derive self-worth from experiencing themselves as causal agents, capable of achieving goals and overcoming obstacles. This process nurtures a critical component of self-esteem:...
211