Incorporating public values into evaluative criteria: Using crowdsourcing to identify criteria and standards

Area of Science:

• Evaluation methodology
• Social sciences
• Human-computer interaction

Background:

• Evaluation relies on value judgments comparing performance to criteria and standards.
• Traditional methods for setting criteria and standards can be time-consuming and costly.

Purpose of the Study:

• To investigate the feasibility of using crowdsourcing to establish evaluation criteria and standards.
• To assess the quality and characteristics of crowd-generated criteria and standards.

Main Methods:

• A four-phase study involving participant tasks to identify, weigh criteria, and set standards.
• Crowdsourcing approach to collect public opinion data for setting evaluation benchmarks.
• Analysis of crowd-generated criteria for quality attributes like clarity, completeness, and realism.

Main Results:

• Crowd-generated criteria were found to be of high quality.
• Specific quality attributes identified: clear, concise, complete, non-overlapping, and realistic.
• The study demonstrated the potential for crowdsourcing in evaluation.

Conclusions:

• Crowdsourcing is a viable and effective method for setting evaluation criteria and standards.
• This approach offers a potentially cost- and time-effective alternative to traditional methods.
• The generated criteria and standards are stable and of high quality, suitable for evaluation.