Contextualizing Evaluation in Research Consortia: A Reflective Case Study from the Research Centers in Minority...

In 2020, evaluators within the Research Centers in Minority Institutions (RCMIs) program proposed a conceptual framework identifying four primary evaluation targets: scientific productivity, scientific collaboration, professional growth, and research resources. This study extends prior work by capturing the contextual and process-oriented dimensions of program impact. This reflective practice-based project examines how non-quantitative approaches complement traditional metrics to better characterize RCMI outcomes. Evaluators representing ten RCMI sites participated in a multi-site case study guided by three questions addressing: (1) qualitative evidence of impact beyond metrics; (2) challenges and successes in implementation of non-quantitative methods; and (3) potential expansion of evaluation targets. Evaluators provided descriptive responses, generating a 22-page dataset that was analyzed thematically. Thirteen non-quantitative evaluation domains emerged: investigator consultations, investigator productivity, investigator success, community partnerships, intra-RCMI collaborations, implementation of team science, career progression, programmatic support, mentoring support, impact on RCMI affiliates, intellectual resources, physical resources, and faculty hires. Key challenges included inconsistent data capture and limited evaluation resources, while successes highlighted improved cross-site learning and visibility of program impact. Findings support retaining the original evaluation targets while expanding the framework to include institutional transformation, equitable research environments, and longitudinal societal impact. A conceptual map was developed to depict how mixed methods that include non-quantitative approaches can yield RCMI evaluations that expand upon the current approach, which relies primarily on quantitative data. The authors recommend quantitative targets and non-quantitative strategies to provide context, communicate evidence of success, and inform programmatic changes to deepen the findings and strengthen the rigor of RCMI evaluation practices.