Related Experiment Video
Updated: Jan 19, 2026
Reliability- Inter-rater Reliability in Psychology Experiments
Published on: April 30, 2023
Variations in reliability and validity do not influence judge, attorney, and mock juror decisions about psychological
Jacqueline Austin Chorn1, Margaret Bull Kovera1
1Department of Psychology.
Objective:
We tested whether the reliability and validity of psychological testing underlying an expert's opinion influenced judgments made by judges, attorneys, and mock jurors.
Hypotheses:
We predicted that the participants would judge the expert's evidence more positively when it had high validity and high reliability.
Method:
In Experiment 1, judges (N = 111) and attorneys (N = 95) read a summary of case facts and proffer of expert testimony on the intelligence of a litigant. The psychological testing varied in scientific quality; either there was (a) blind administration (i.e., the psychologist did not have an expectation for the test result) of a highly reliable test, (b) nonblind administration (i.e., the psychologist did have an expectation for the test result) of a highly reliable test, or (c) blind administration of a test with low reliability. In a trial simulation (Experiment 2), we varied the scientific quality of the intelligence test and whether the cross-examination addressed the scientific quality of the test.
Results:
The variations in scientific quality did not influence judges' admissibility decisions nor their ratings of scientific quality nor did it influence attorneys' decisions about whether to move to exclude the evidence. Attorneys' ratings of scientific quality were sensitive to variations in reliability but not the testing conditions. Scientifically informed cross-examinations did not help mock jurors (N = 192) evaluate the validity or the reliability of a psychological test.
Conclusion:
Cross-examination was an ineffective method for educating jurors about problems associated with nonblind testing and reliability, which highlights the importance of training judges to evaluate the quality of expert evidence. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
Related Concept Videos
01:29Reliability and Validity
05:13Reliability in Psychology Experiments
08:08Comparing the Frequency Effect Between the Lexical Decision and Naming Tasks in Chinese
08:34Thinking Too Much Impairs Decision-Making
When we are considering a tough choice between two or more attractive options, we often end up actively weighing the pros and cons of each alternative. By reflecting on their advantages and disadvantages, we attempt to fit a complex, subjective decision into an orderly set of criteria. However, research in psychology suggests that this sort of introspective approach might not always yield the most optimal outcomes.1
In...
01:23Factors Affecting Consumer Decision Process - Psychological
Here are some key psychological factors affecting consumer behavior:
01:29Psychological and Descriptive Understanding of Ethical Decision Making
The normative approach defines explicit standards for ethical behavior, emphasizing principles such as honesty, fairness, and accountability. This perspective provides a clear framework for distinguishing right from wrong and ensures that actions...