Related Experiment Video
Updated: Nov 8, 2025
05:02
Comparing Bibliometric Analysis Using PubMed, Scopus, and Web of Science Databases
Published on: October 24, 2019
32.7K
Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study
Anna H Noel-Storr1, Patrick Redmond2, Guillaume Lamé3,4
1Cochrane Dementia and Cognitive Improvement Group, Radcliffe Department of Medicine, University of Oxford, Oxford, OX3 9DU, UK. anna.noel-storr@rdm.ox.ac.uk.
BMC Medical Research Methodology
|April 28, 2021
Summary
Crowdsourcing citation screening for systematic reviews is feasible, with non-specialist contributors achieving high accuracy. This approach can significantly reduce review production times.
Area of Science:
- Health Sciences
- Information Science
- Citizen Science
Background:
- Crowdsourcing leverages public participation, often via the internet, for various tasks.
- Non-specialist crowds have demonstrated reliability in identifying quantitative studies through title/abstract screening.
- This study explored crowdsourcing for topic-based citation screening in a mixed-studies systematic review.
Purpose of the Study:
- To investigate the feasibility and performance of crowdsourcing for citation screening in a mixed-studies systematic review.
- To assess the accuracy (sensitivity and specificity) of crowd-based screening compared to a review team.
- To evaluate the time efficiency and task engagement of crowd contributors.
Main Methods:
- The study utilized Cochrane Crowd, an online citizen science platform, for citation screening.
- Volunteers underwent task-specific online training before independently screening 9546 titles and abstracts.
- Crowd decisions were compared against a 'gold standard' set by the review team, with a repeat task using an adjusted algorithm.
Main Results:
- The crowd achieved 84% sensitivity and 99% specificity in the initial screening task.
- Task completion time for the crowd was significantly less (33 hours) than for the review team (410 hours).
- A subsequent task with an updated algorithm yielded 94% sensitivity and 98% specificity, with positive contributor experiences.
Conclusions:
- Crowdsourcing citation screening for mixed-studies systematic reviews appears feasible, requiring consideration of customized training resources.
- Crowd screening offers a potential method for more time-efficient systematic review conduct with minimal impact on accuracy.
- Further research is recommended to fully establish the utility and optimize the application of crowdsourcing in systematic reviews.
Related Concept Videos
01:15Study Design in Statistics
9.7K
A study design is a set of techniques that allow a researcher to collect and analyze data from different variables defined for a specific research problem. Statistics is commonly for effective study design and more robust experiments,
Does aspirin reduce the risk of heart attacks? Is one brand of fertilizer more effective at growing roses than another? Is fatigue as dangerous to a driver as the influence of alcohol? Questions like these are answered using randomized experiments with proper...
Does aspirin reduce the risk of heart attacks? Is one brand of fertilizer more effective at growing roses than another? Is fatigue as dangerous to a driver as the influence of alcohol? Questions like these are answered using randomized experiments with proper...
9.7K
01:17Systematic Sampling Method
11.8K
Sampling is a technique to select a portion (or subset) of the larger population and study that portion (the sample) to gain information about the population. Data are the result of sampling from a population. The sampling method ensures that samples are drawn without bias and accurately represent the population. Because measuring the entire population in a study is not practical, researchers use samples to represent the population of interest.
Systematic sampling is one of the simplest methods...
Systematic sampling is one of the simplest methods...
11.8K
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
220
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
220
01:15Systematic Error: Methodological and Sampling Errors
6.0K
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
6.0K
01:09Random Sampling Method
13.4K
Sampling is a technique to select a portion (or subset) of the larger population and study that portion (the sample) to gain information about the population. Data are the result of sampling from a population. The sampling method ensures that samples are drawn without bias and accurately represent the population. Because measuring the entire population in a study is not practical, researchers use samples to represent the population of interest. Among the various sampling methods used by...
13.4K
00:55Convenience Sampling Method
10.3K
Sampling is a technique to select a portion (or subset) of the larger population and study that portion (the sample) to gain information about the population. Data are the result of sampling from a population. The sampling method ensures that samples are drawn without bias and accurately represent the population.
Convenience sampling is a non-random method of sample selection; this method selects individuals that are easily accessible and may result in biased data. For example, a marketing...
Convenience sampling is a non-random method of sample selection; this method selects individuals that are easily accessible and may result in biased data. For example, a marketing...
10.3K