Benchmarking LLM-based agents for single-cell omics analysis.

Area of Science:

• Computational Biology
• Bioinformatics
• Artificial Intelligence

Background:

• Single-cell omics data analysis faces challenges with traditional manual workflows.
• Artificial intelligence (AI) agents offer adaptive planning, code generation, and knowledge fusion for complex analyses.
• A lack of comprehensive benchmarks impedes the progress of AI agents in this field.

Purpose of the Study:

• To introduce a novel benchmarking evaluation system for assessing AI agent capabilities in single-cell omics analysis.
• To provide a standardized framework for comparing different AI agent approaches.

Main Methods:

• Developed a unified platform compatible with various agent frameworks and large language models (LLMs).
• Defined multidimensional metrics for evaluating cognitive program synthesis, collaboration, execution efficiency, bioinformatics knowledge integration, and task completion quality.
• Created a dataset of 50 diverse real-world single-cell omics analysis tasks across different omics types, species, and technologies.

Main Results:

• Grok3-beta demonstrated state-of-the-art performance among the evaluated agent frameworks.
• Multi-agent frameworks showed enhanced collaboration and execution efficiency compared to single-agent systems via specialized role division.
• Attribution analysis indicated high-quality code generation is critical for task success, with self-reflection having the most significant impact, followed by retrieval-augmented generation (RAG) and planning.

Conclusions:

• Persistent challenges remain in AI agent code generation, long-context handling, and context-aware knowledge retrieval.
• This work establishes a critical empirical foundation for developing robust AI agents in computational biology.
• Best practices for AI agent development in bioinformatics are provided based on empirical findings.