Research impact assessment. Principles and applications to proposed, ongoing, and completed projects

Area of Science:

• Research evaluation
• Science and technology policy
• Bibliometrics

Background:

• Federal agencies employ various Research Impact Assessment (RIA) approaches, including retrospective, peer review, and quantitative methods.
• Peer review is the most frequently utilized method for assessing research impact.
• Each RIA method possesses inherent limitations and challenges.

Purpose of the Study:

• To describe generic RIA approaches used by the federal government.
• To highlight the shortcomings of existing RIA methods.
• To identify opportunities for improving research impact assessment.

Main Methods:

• Review of existing RIA approaches used by the federal government.
• Analysis of the strengths and weaknesses of retrospective, peer review, and quantitative methods.
• Examination of the application of bibliometrics in research impact assessment.

Main Results:

• Peer review is the predominant RIA method, despite its limitations.
• A wide array of research impact targets exist, including effects on research fields, technology, and education.
• The weighting of research impact is subjective, influenced by reviewer perspectives.
• Limited evidence exists for the federal government's use of multiple RIA techniques, particularly bibliometrics supporting peer review.

Conclusions:

• Existing RIA methods have significant shortcomings.
• The complexity of research impact targets necessitates a multifaceted assessment approach.
• There is a substantial gap in the federal government's utilization of combined RIA techniques, especially bibliometrics, indicating an area for development.