Benchmarking cell type and gene set annotation by large language models with AnnDictionary.

Area of Science:

• Computational Biology
• Bioinformatics
• Artificial Intelligence

Background:

• Single-cell RNA sequencing (scRNA-seq) generates large anndata datasets.
• Automated analysis of anndata, particularly cell-type annotation, is crucial.
• Large language models (LLMs) show potential for biological data analysis.

Purpose of the Study:

• Introduce AnnDictionary, an open-source package for parallel anndata analysis using LLMs.
• Benchmark major LLMs for de novo cell-type annotation accuracy.
• Evaluate LLM performance in functional gene set annotation.

Main Methods:

• Developed AnnDictionary, integrating LangChain and AnnData, supporting multiple LLM providers.
• Implemented multithreading optimizations for efficient analysis of large anndata.
• Conducted benchmarking studies comparing LLM annotations against manual annotations and gene set databases.

Main Results:

• LLM performance in cell-type annotation varies with model size, with high agreement (>80-90%) for major cell types.
• Inter-LLM agreement also correlates with model size.
• Claude 3.5 Sonnet achieved high accuracy (>80%) in functional gene set annotation.

Conclusions:

• AnnDictionary facilitates efficient, parallel analysis of anndata using LLMs.
• LLMs demonstrate significant potential for accurate cell-type and functional annotation in scRNA-seq data.
• Ongoing benchmarking and leaderboard maintenance will track LLM advancements in this field.