GAME: Genomic API for Model Evaluation.

Area of Science:

• Genomics
• Machine Learning
• Computational Biology

Background:

• Genomics datasets and machine learning models are rapidly expanding.
• Existing benchmarking methods for sequence-to-activity models are ad hoc and lack standardization, hindering progress.
• Heterogeneity in model architectures and evaluation tasks complicates comparative analysis.

Purpose of the Study:

• To introduce GAME, a system for large-scale, community-led standardized benchmarking of genomics models.
• To enable seamless integration of pre-trained models and user-defined evaluation tasks.
• To accelerate the development and application of accurate sequence-to-activity models.

Main Methods:

• Developed GAME, a system using an Application Programming Interface (API) paradigm for model-task communication.
• Implemented a Matcher module utilizing a large language model (LLM) for automated task alignment.
• Employed containerization for enhanced reproducibility and cross-platform deployment.
• Focused benchmarks on predicting technology-independent biochemical phenomena.

Main Results:

• GAME provides a flexible framework for integrating diverse models and benchmarks.
• The system ensures consistent evaluation protocols through standardized communication.
• Automated task alignment and containerization simplify benchmarking processes.
• Initial examples demonstrate the framework's utility and extensibility.

Conclusions:

• GAME addresses the critical need for standardized benchmarking in genomics machine learning.
• The system facilitates community-driven contributions, fostering an evolving resource.
• GAME will accelerate genomics research by clarifying model performance and guiding future development.