Generative AI can and should accelerate research evaluation reform to better recognize 'distinctly human...

Area of Science:

• Scholarly communication
• Research evaluation
• Artificial intelligence

Background:

• Generative artificial intelligence (GenAI) is transforming research practices.
• Traditional scholarly evaluation metrics (e.g., publication and citation counts) are insufficient and becoming less reliable.
• GenAI disproportionately impacts text-based and computational work, potentially worsening existing biases in research evaluation.

Purpose of the Study:

• To propose reforms in scholarly evaluation that address the challenges posed by GenAI.
• To identify and emphasize 'distinctly human contributions' crucial for research but not captured by current metrics.
• To argue that GenAI necessitates a return to fundamental principles of good evaluation rather than entirely new systems.

Main Methods:

• Conceptual analysis of GenAI's impact on research labor and evaluation.
• Identification of two key categories of human contributions: epistemic-ethical and socio-relational.
• Proposal of practical mechanisms for evaluation reform, including modified Contributor Role Taxonomy (CRediT) statements, broader output recognition, and enhanced narrative CVs/testimonies.

Main Results:

• Current metrics like publication and citation counts are poor proxies for research impact, especially with GenAI.
• Two essential categories of human contributions are identified: epistemic-ethical (situated judgment under accountability) and socio-relational (mentoring, teaching, community engagement).
• Proposed reforms, while valuable, face challenges like GenAI manipulation, underscoring the need for human judgment in evaluation.

Conclusions:

• Scholarly evaluation must adapt to recognize 'distinctly human contributions' that resist automation.
• GenAI does not invalidate existing evaluation principles but highlights the enduring importance of human judgment, accountability, and relationships.
• Effective research evaluation requires acknowledging and valuing human efforts that cannot be easily quantified or automated.