Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Deductive Reasoning01:16

Deductive Reasoning

55.3K
Deductive reasoning, or deduction, is the type of logic used in hypothesis-based science. In deductive reasoning, the pattern of thinking moves in the opposite direction as compared to inductive reasoning, which means that it uses a general principle or law to predict specific results. From those general principles, a scientist can deduce and predict the specific results that would be valid as long as the general principles are valid.
For example, a researcher can deduce specific predictions...
55.3K
Reasoning01:30

Reasoning

83
Reasoning is the action of thinking about something in a logical, sensible way. It is integral to problem-solving, decision-making, and critical thinking. Reasoning can be inductive or deductive. Reasoning involves transforming information into conclusions, which is essential for problem-solving, decision-making, and critical thinking.
Inductive reasoning involves deriving generalizations from specific observations. This type of reasoning helps form beliefs about the world. For example,...
83
Inductive Reasoning00:59

Inductive Reasoning

60.5K
Inductive reasoning is a form of logical thinking that uses related observations to arrive at a general conclusion. It is uncertain and operates in degrees to which the conclusions are credible. As such, inductive arguments can be weak or strong, rather than valid or invalid, and conclusions can be used to formulate testable, falsifiable hypotheses.
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
60.5K
Decision Making: Traditional Method01:14

Decision Making: Traditional Method

4.0K
The process of hypothesis testing based on the traditional method includes calculating the critical value, testing the value of the test statistic using the sample data, and interpreting these values.
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
4.0K
Decision Making: P-value Method01:09

Decision Making: P-value Method

5.4K
The process of hypothesis testing based on the P-value method includes calculating the P- value using the sample data and interpreting it.
First, a specific claim about the population parameter is proposed. The claim is based on the research question and is stated in a simple form. Further, an opposing statement to the claim  is also stated. These statements can act as null and alternative hypotheses:  a null hypothesis would be a neutral statement while the alternative hypothesis can...
5.4K
Reason and Intuition01:37

Reason and Intuition

6.5K
The human brain processes information for decision-making using one of two routes: an intuitive system and a rational system (Epstein, 1994; popularized by Kahneman, 2011 as System 1 and System 2, respectively). The intuitive system is quick, impulsive, and operates with minimal effort, relying on emotions or habits to provide cues for what to do next, while the rational system is logical, analytical, deliberate, and methodical. Research in neuropsychology suggests that the...
6.5K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science.

Proceedings of the National Academy of Sciences of the United States of America·2026
Same author

A virtual consultative forum on future reforms to Better Access.

The Australian and New Zealand journal of psychiatry·2026
Same author

On the relationship between indirect measures of Black versus White racial attitudes and discriminatory outcomes: An adversarial collaboration using a sample of White Americans.

Journal of personality and social psychology·2026
Same author

Private speech: similarities between a large language model and children.

Frontiers in artificial intelligence·2026
Same author

Reflections on adversarial collaboration from the adversaries: was it worth it<b>?</b>

Theory and society·2026
Same author

The Cytotoxicity Profile of Silver Carboxylate in a TiO<sub>2</sub>/Polydimethylsiloxane Matrix in Osteoblasts, Keratinocytes, Endothelial Cells, and Skeletal Muscle Cells.

Surgical infections·2025
Same journal

Exploring psychological tradeoffs: Developing and demonstrating an R Shiny app for Pareto optimization.

Behavior research methods·2026
Same journal

The performance of Bayesian fit measures in detecting misspecified multilevel structural equation modeling.

Behavior research methods·2026
Same journal

Psychometric functions from multiple responses : Dedicated to the memory of Colin L. Mallows.

Behavior research methods·2026
Same journal

Low-cost, open-source, full-stack software and Arduino-based hardware for control of commercially available animal behavior systems.

Behavior research methods·2026
Same journal

PyNeon: A Python package for the analysis of Neon multimodal mobile eye-tracking data.

Behavior research methods·2026
Same journal

Talking surveys: How photorealistic embodied conversational agents shape response quality, engagement, and satisfaction.

Behavior research methods·2026
See all related articles

Related Experiment Video

Updated: Jul 13, 2025

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
08:12

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

2.5K

Validating a forced-choice method for eliciting quality-of-reasoning judgments.

Alexandru Marcoci1, Margaret E Webb2, Luke Rowe3

  • 1Centre for the Study of Existential Risk, University of Cambridge, 16 Mill Lane, Cambridge, CB2 1SB, UK. am3159@cam.ac.uk.

Behavior Research Methods
|October 13, 2023
PubMed
Summary
This summary is machine-generated.

Forced-choice comparisons effectively assess written argument quality, with novices and experts accurately identifying better reasoning. This method offers a valid, reliable, and efficient approach for large-scale quality assessments.

Keywords:
Automatic reasoning assessmentComparative judgmentForced choiceQuality of reasoningReasoning

More Related Videos

A Two-interval Forced-choice Task for Multisensory Comparisons
07:13

A Two-interval Forced-choice Task for Multisensory Comparisons

Published on: November 9, 2018

11.0K
Working Memory Training for Older Participants: A Control Group Training Regimen and Initial Intellectual Functioning Assessment
07:01

Working Memory Training for Older Participants: A Control Group Training Regimen and Initial Intellectual Functioning Assessment

Published on: September 20, 2020

4.8K

Related Experiment Videos

Last Updated: Jul 13, 2025

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
08:12

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

2.5K
A Two-interval Forced-choice Task for Multisensory Comparisons
07:13

A Two-interval Forced-choice Task for Multisensory Comparisons

Published on: November 9, 2018

11.0K
Working Memory Training for Older Participants: A Control Group Training Regimen and Initial Intellectual Functioning Assessment
07:01

Working Memory Training for Older Participants: A Control Group Training Regimen and Initial Intellectual Functioning Assessment

Published on: September 20, 2020

4.8K

Area of Science:

  • Cognitive Psychology
  • Educational Measurement
  • Artificial Intelligence

Background:

  • Evaluating the quality of written arguments is crucial but often subjective.
  • Traditional methods can be time-consuming and require expert raters.
  • Developing scalable and reliable assessment tools is an ongoing challenge.

Purpose of the Study:

  • To investigate the criterion validity of forced-choice comparisons for assessing argument quality.
  • To determine the reliability and efficiency of this assessment method for novices and experts.
  • To explore methods for optimizing the assessment process and utilizing linguistic features.

Main Methods:

  • Two studies were conducted using a forced-choice design where participants compared arguments.
  • Inter-rater reliability was calculated to assess agreement between raters.
  • Efficiency was improved using transitivity and an AVL tree method; a regression model analyzed linguistic features.

Main Results:

  • Novices and experts accurately identified arguments supporting more accurate solutions (62.2% and 74.4% respectively).
  • Participants correctly identified arguments from larger teams (82% for novices, 85% for experts) with high inter-rater reliability.
  • Efficiency methods reduced judgment numbers with minimal accuracy loss; a regression model showed high correlation with objective scores.

Conclusions:

  • Forced-choice comparisons provide a valid and reliable method for assessing reasoning quality, even for novices.
  • The method is efficient and scalable for large-volume assessments.
  • Leveraging transitivity and linguistic analysis offers further optimization for argument quality evaluation.