A reproducible optimisation protocol for calibrating prompt-based large language model workflows in evidence...

This methods article presents a reproducible workflow for optimising and testing improving large language models (LLMs) prompts in evidence-synthesis tasks with defined inputs and outputs. The method separates the fixed rules that define the scientific task from the editable prompt instructions that frame and apply them. It optimises those instructions against labelled or reference examples and an explicit task metric, then saves the final workflow with its specification, metric, settings, and evaluation traces so others can inspect and reuse it. The example code uses DSPy and GEPA tools, but the same logic can transfer to other prompt-optimisation frameworks that support structured task definitions, metric-guided search, and reusable outputs. Title and abstract screening is the illustrative validation case here as it provides labelled benchmark data and clear evaluation metrics. The demonstrated workflow uses a smaller student LLM for the scientific task and a larger reflection LLM to steer prompt optimisation during calibration. This work shows compilation, saving and reloading, and how optimisation budget affects a smaller student model.•Separate what the model must decide (the fixed rules), from how it is prompted to decide (the mutable prompt instructions).•Optimise the prompt automatically against labelled examples and a declared scoring function, rather than iterating manually.•Package the calibrated prompt with its rules, metric, settings, traces, and evaluation logs for inspection and reuse.