Unlocking the Hidden Value of Evaluation Reports: A Semi-Automated Workflow to Inform Strategic Planning and...

Evaluation reports contain critical, practice-based evidence relevant to policy and strategic decision-making, particularly for monitoring progress toward the Sustainable Development Goals (SDGs). However, limited discoverability, richness of information content, and structure can act as barriers to their use in literature analyses, especially compared to peer-reviewed literature. This paper addresses the challenge of discovering evaluation reports for synthesis through a scalable, semi-automated methodology.

We present a pipeline to process and map information from 631 publicly available UNICEF evaluation reports. It includes PDF-to-text conversion, document parsing, neural network-based sentence mining and outcome-specific filtering using adjacency-query search in SWIFT-Review. The target framework was UNICEF's 2022-2025 Strategic Plan, encompassing 64 defined outcomes. Tools we used were free and methods we developed are open-source.

The approach achieved a 92% reduction in overall text volume and good recall (0.93) in identifying relevant sentences. A median search precision of 0.6 was achieved across outcomes, requiring an average review of only 21 evaluation reports per outcome to extract insights. These results indicate a potential to increase feasibility of discovering high-quality information from evaluation reports, although efficiency might vary when applied to other datasets.

Our findings demonstrate that evaluation reports - despite their structural heterogeneity - can be systematically searched, extracted, and mapped. This work highlights the untapped value of evaluation reports for evidence-based planning and offers a replicable approach for researchers working at the intersection of development research, policy, and artificial intelligence.