Benchmarking and behavioral characterization of LLM agents for protein design

Large language models (LLMs) are increasingly deployed as agents for scientific discovery, but standardized frameworks for evaluating their performance and behavior in scientific workflows are lacking. Protein design provides a demanding test case because modern workflows combine stochastic generative models, structure prediction systems, and physics-based evaluation tools that require extensive candidate exploration and filtering. Here we introduce BioDesignBench, a benchmark of 76 expert-curated protein design tasks spanning antibodies, enzymes, fluorescent proteins, binders, and scaffolds, together with human and non-LLM baselines and behavioral metrics derived from tool-use traces. We evaluate four frontier LLM agents across diverse protein design workflows and find that the strongest agents surpass deterministic hardcoded pipelines but consistently underperform expert practice. Decomposing agent behavior along orthogonal axes of tool coverage and evaluation depth, we localize the gap to evaluation rather than tool selection: agents generally pick appropriate tools but score each candidate against only a narrow set of metrics, rarely compare alternatives, and terminate exploration prematurely. Guided workflows improve tool coverage but not evaluation depth. A compute-matched forced-depth intervention that requires agents to evaluate every candidate across multiple complementary metric categories (structure, interface, physics, and learned affinity) substantially improves performance and rules out generic scaffolding effects, demonstrating that the gap is behavioral rather than a fundamental capability constraint. All evaluations are performed in silico. We release BioDesignBench, open-source reference agents, and a public leaderboard as a community resource for evaluating and improving AI agents for protein engineering.