Related Experiment Video
Updated: Jun 5, 2025
00:08
A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
7.0K
Response Rates, Reliability, and Convergent Validity in Experience Sampling Data.
Jeffrey S Simons1,2, Stephen A Maisto3, Raluca M Simons1
1University of South Dakota, Vermillion, USA.
Assessment
|December 10, 2024
Summary
Study findings indicate that the number of completed assessments and reliability, not just compliance, impact data quality in experience sampling method (ESM) studies. Compliance alone does not guarantee valid or reliable intoxication and negative affect measures.
Area of Science:
- Psychological Science
- Methodology
- Data Science
Background:
- Experience sampling method (ESM) is widely used to study psychological phenomena in daily life.
- Assessing the reliability and validity of ESM data is crucial for accurate research findings.
- Compliance with ESM protocols can influence data quality, but its precise impact is debated.
Purpose of the Study:
- To examine the associations between compliance rates and the reliability and convergent validity of intoxication and negative affect assessments in ESM data.
- To investigate these associations across diverse populations: Veterans, Sexual Minority Men, and College Students.
- To determine if a specific compliance threshold reliably predicts data validity.
Main Methods:
- Utilized data from three distinct samples (Veterans, Sexual Minority Men, College Students).
- Operationalized convergent validity as within-person associations between in situ and retrospective assessments, and transdermal alcohol measures.
- Assessed reliability using intraclass correlation (ICC) and the number of completed assessments.
Main Results:
- Reliability of ESM aggregates (e.g., daily means) depends more on the number of completed assessments and ICC than mere compliance.
- Convergent validity showed a correlation with compliance rate, but associations were weak.
- No clear compliance threshold (e.g., 80%) emerged to distinguish between good and poor data validity.
Conclusions:
- Compliance rate is not the sole determinant of data reliability or validity in ESM studies.
- Researchers should consider the number of assessments and ICC for reliability, and acknowledge that high compliance does not always equate to high validity.
- Findings suggest a nuanced view of compliance in ESM research, applicable across different demographic groups.
Related Concept Videos
02:16Surveys
14.7K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
14.7K
01:29Reliability and Validity
12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K
00:55Convenience Sampling Method
8.6K
Sampling is a technique to select a portion (or subset) of the larger population and study that portion (the sample) to gain information about the population. Data are the result of sampling from a population. The sampling method ensures that samples are drawn without bias and accurately represent the population.
Convenience sampling is a non-random method of sample selection; this method selects individuals that are easily accessible and may result in biased data. For example, a marketing...
Convenience sampling is a non-random method of sample selection; this method selects individuals that are easily accessible and may result in biased data. For example, a marketing...
8.6K
01:07Data Collection by Survey
6.4K
The systematic method of obtaining and analyzing accurate information of a population is called data collection. A survey is a standard method of data collection that involves collecting information from a target human population about their experience, opinion, or knowledge of a product, service, or process. The responses are recorded and interpreted. The most common survey examples are written questionnaires, face-to-face or telephonic conversations, focus groups, and electronic (e-mail or...
6.4K
02:20Longitudinal Research
11.8K
Sometimes we want to see how people change over time, as in studies of human development and lifespan. When we test the same group of individuals repeatedly over an extended period of time, we are conducting longitudinal research. Longitudinal research is a research design in which data-gathering is administered repeatedly over an extended period of time. For example, we may survey a group of individuals about their dietary habits at age 20, retest them a decade later at age 30, and then again...
11.8K
02:30Naturalistic Observations
15.4K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
15.4K