Related Experiment Video
Updated: Aug 16, 2026
12:49
Transcranial Direct Current Stimulation (tDCS) of Wernicke's and Broca's Areas in Studies of Language Learning and Word Acquisition
Published on: July 13, 2019
Designing vocabulary multiple-choice tests for exploring word frequency effects in distributional semantics
Joseph P Levy1, John A Bullinaria2
1Independent Researcher, London, UK. joe.levy@cantab.net.
Behavior Research Methods
|August 14, 2026
Summary
Designing specific vocabulary multiple-choice question (VMCQ) tests for distributional semantic vectors (DSVs) yields better results than human-centric tests. This study introduces improved VMCQ methods for evaluating DSVs, particularly word frequency effects.
Area of Science:
- Computational Linguistics
- Natural Language Processing
- Cognitive Science
Background:
- Distributional semantic vectors (DSVs) represent word meanings in vector spaces, crucial for AI and psychology.
- Vocabulary multiple-choice question (VMCQ) tests are common for evaluating DSVs, often adapted from human linguistic tests.
Purpose of the Study:
- To demonstrate that VMCQs designed specifically for DSVs outperform human-centric tests.
- To develop improved VMCQ scoring and evaluation methods for DSVs.
- To explore the impact of word frequency on DSV performance using novel VMCQ approaches.
Main Methods:
- Developed an improved VMCQ scoring method for statistically reliable DSV evaluation.
- Introduced new evaluation methods to analyze word frequency effects on DSVs.
- Tested the proposed VMCQ approach on nine diverse DSVs.
Main Results:
- DSVs benefit from VMCQ designs tailored to their specific properties.
- The new scoring method enhances the reliability of DSV performance assessment.
- The study identified an effective VMCQ strategy for exploring word frequency effects in DSVs.
Conclusions:
- Tailored VMCQs offer superior evaluation of distributional semantic vectors compared to adapted human tests.
- The developed methods provide more reliable insights into DSV performance and word frequency impacts.
- This research advances the methodology for assessing and understanding semantic vector representations.
Related Concept Videos
01:19Expected Frequencies in Goodness-of-Fit Tests
A goodness-of-fit test is conducted to determine whether the observed frequency values are statistically similar to the frequencies expected for the dataset. Suppose the expected frequencies for a dataset are equal such as when predicting the frequency of any number appearing when casting a die. In that case, the expected frequency is the ratio of the total number of observations (n) to the number of categories (k).
01:27Language and Cognition
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
02:01Group Design
The most basic experimental design involves two groups: the experimental group and the control group. The two groups are designed to be the same except for one difference— experimental manipulation. The experimental group gets the experimental manipulation—that is, the treatment or variable being tested—and the control group does not. Since experimental manipulation is the only difference between the experimental and control groups, we can be sure that any differences between the two are due to...
01:24Components of Language
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs. “eh”). Phonemes combine to...
01:21Frequency-dependent Selection
When the fitness of a trait is influenced by how common it is (i.e., its frequency) relative to different traits within a population, this is referred to as frequency-dependent selection. Frequency-dependent selection may occur between species or within a single species. This type of selection can either be positive—with more common phenotypes having higher fitness—or negative, with rarer phenotypes conferring increased fitness.Positive Frequency-Dependent SelectionIn positive...
01:25Choosing Between z and t Distribution
The z and the Student t distribution estimate the population mean using the sample mean and standard deviation. However, to decide which distribution to use for a calculation, one needs to determine the sample size, the nature of the distribution, and whether the population standard deviation is known. If the population standard deviation is known and the population is normally distributed, or if the sample size is greater than 30, the z distribution is preferred. The Student t distribution is...