The Curse of Performative User Studies.

Area of Science:

• Computer Graphics
• Human-Computer Interaction
• Research Methodology

Background:

• User studies are common in computer graphics research for evaluating tools and understanding user behavior.
• Online crowdworking platforms are frequently used for conducting these evaluations.

Purpose of the Study:

• To critically examine the common practice of user studies in computer graphics research.
• To argue that many user studies are performative rather than genuinely insightful.
• To highlight the negative impacts of casually performed studies on authors, reviewers, and research innovation.

Main Methods:

• Critical analysis of current practices in computer graphics user studies.
• Argumentation based on the perceived motivations and outcomes of these studies.

Main Results:

• Many user studies are conducted due to reviewer expectations rather than a genuine need for evaluation.
• Casually performed studies can be uninformative or misleading, creating unnecessary work.
• The pressure for positive results may stifle creativity and novel research directions.

Conclusions:

• A call for greater thoughtfulness from reviewers and authors regarding the necessity and execution of user studies.
• Emphasizes the need for new research and guidelines to ensure user studies are genuinely useful.
• Advocates for a shift towards more meaningful and impactful user evaluation in computer graphics.