PertReason: A Knowledge-Grounded Benchmark and Framework for Cell-State-Conditioned Mechanistic Reasoning of...

Area of Science:

• Computational Biology
• Machine Learning in Science

Background:

• Evaluating machine learning (ML) models in scientific domains requires assessing both prediction accuracy and the mechanistic faithfulness of their reasoning.
• Realistic distribution shifts, such as novel cell types or perturbations, pose significant challenges to current ML models.

Purpose of the Study:

• To introduce PertReason, a knowledge-grounded benchmark and framework for evaluating cell-state-conditioned reasoning about perturbation effects.
• To assess the ability of ML models to generate mechanistically faithful explanations and their robustness to complex distribution shifts.

Main Methods:

• Developed PertReasonQA, a benchmark combining single-cell genetic and chemical perturbation data with knowledge graphs.
• Dynamically conditioned pathways on cell-specific basal states to prevent generic memorization.
• Evaluated state-of-the-art ML models on their predictive accuracy and mechanistic reasoning capabilities.

Main Results:

• Identified systematic gaps between predictive accuracy and mechanistic reasoning in current ML models.
• Observed failure modes such as correct predictions via flawed logic, ignored cellular context, and directionally inconsistent mechanisms.
• Introduced PertReasonLM, a model trained to align predictions with context-specific mechanistic reasoning.

Conclusions:

• Current ML models exhibit significant limitations in generating faithful mechanistic explanations, even when predictions are accurate.
• PertReason provides a diagnostic framework to expose and mitigate failures in scientific reasoning within ML systems.
• Developing models that ground rationales in context-specific pathways is crucial for reliable scientific AI.