Related Experiment Video
Updated: May 17, 2026
11:21
Methodology for Establishing a Community-Wide Life Laboratory for Capturing Unobtrusive and Continuous Remote Activity and Health Data
Published on: July 27, 2018
Evaluating Crowdsourced Data Collection for Carceral Death Surveillance: Pilot Study Using Amazon Mechanical Turk
Emily Wang1, Julia Healey-Parera1, Amy Duan1
1Department of Population Health Sciences, School of Medicine, Duke University, Durham, NC, United States.
JMIR Formative Research
|May 15, 2026
Summary
Crowdsourcing using Amazon Mechanical Turk (MTurk) rapidly extracted data on deaths in custody but yielded low-quality results due to poor worker agreement. Improved task design or AI support is needed for reliable data collection on deaths in correctional facilities.
Area of Science:
- Public Health
- Criminology
- Data Science
Background:
- Incarcerated populations face elevated health risks, yet deaths in custody are underreported and poorly monitored.
- Existing reporting mechanisms like the Death in Custody Reporting Act have inconsistent, delayed, and inaccessible data.
- Researchers often rely on correctional agency press releases, which lack standardized formats for data extraction.
Purpose of the Study:
- To evaluate the utility of Amazon Mechanical Turk (MTurk) for extracting structured information from press releases concerning deaths in custody.
- To assess the feasibility of crowdsourcing for timely data collection on deaths within correctional facilities.
Main Methods:
- 144 press releases on deaths in custody (2000-2023) from state prisons and ICE were assigned to 3 MTurk workers each.
- Workers completed a 16-question form based on Death in Custody Reporting Act variables.
- Data quality was assessed via strict and 2-way concordance, with qualitative review of errors.
Main Results:
- The task was completed rapidly (within 48 hours) but showed low agreement among crowd workers.
- Strict concordance rates were low: 14.2% for age, 12.3% for race/ethnicity, and 11.4% for date of birth.
- Qualitative review revealed frequent errors, missing data, and inattentive responses, indicating insufficient data quality for complex abstraction.
Conclusions:
- While MTurk facilitated quick data collection, the extracted information from carceral press releases was of low quality.
- General crowdsourcing platforms require enhanced training or oversight for complex data abstraction tasks.
- Future improvements may involve refined task design, AI integration, and ultimately, standardized reporting by correctional institutions to improve surveillance of deaths in custody.
