Man versus machine? Self-reports versus algorithmic measurement of publications

Area of Science:

• Bibliometrics
• Scientometrics
• Research Evaluation

Background:

• Accurate measurement of scientific publications is crucial for research evaluation.
• Existing methods rely on either researcher self-reports (surveys) or algorithmic data extraction.
• The quality and potential biases of these two data sources are not fully understood.

Purpose of the Study:

• To compare the quality of scientific publication data collected via surveys versus algorithmic approaches.
• To identify and illustrate the types of measurement errors inherent in self-reported and machine-generated publication data.
• To evaluate how these different data collection methods relate to career outcomes.

Main Methods:

• Utilized newly available linked data from Web of Science and the Survey of Doctorate Recipients.
• Estimated the relationship between publication measures from survey and algorithmic data with career outcomes (salaries, faculty rankings).
• Applied statistical methods to evaluate data quality without requiring gold standard data.

Main Results:

• Self-reported publication data exhibited smaller potential biases compared to algorithmically generated data.
• Measurement errors in algorithmic data were linked to name frequency and data availability for matching.
• Measurement errors in self-reported data increased over a researcher's career due to complexity and recall issues.

Conclusions:

• Self-reported publication data may offer a more reliable measure of publication output, despite algorithmic data's potential for broader coverage.
• Understanding the distinct error patterns in survey and algorithmic data is essential for accurate research assessment.
• Provides guidance on utilizing linked survey and publication databases for improved research evaluation.