eHive: an artificial intelligence workflow system for genomic analysis.

Area of Science:

• Comparative genomics
• Bioinformatics
• Computational biology

Background:

• The Ensembl project requires significant computational resources for comparative genomics, with analysis complexity growing quadratically with the number of species.
• Current Ensembl releases involve generating genomic alignments and protein homology predictions, demanding substantial processing power.
• The Ensembl project supports 50 species, with plans for future expansion, necessitating scalable computational solutions.

Purpose of the Study:

• To develop a novel, fault-tolerant distributed processing system to manage computationally intensive comparative genomics analyses.
• To enhance the efficiency and reliability of Ensembl's comparative genomics resource production pipelines.
• To create a flexible system capable of supporting diverse dataflow and branching rules for various bioinformatics tasks.

Main Methods:

• Introduction of eHive, a distributed processing system utilizing blackboard systems, autonomous agents, and dataflow graphs.
• Implementation of a MySQL database as a central blackboard and autonomous agent controller.
• Development of Perl scripts as autonomous agents to query the system and execute jobs based on defined rules.

Main Results:

• Successful implementation of three key pipelines: pairwise whole genome alignments, multiple whole genome alignments, and gene trees with protein homology inference.
• Demonstration of eHive's efficiency in real-world comparative genomics scenarios.
• Validation of eHive as a fault-tolerant system capable of handling complex computational demands.

Conclusions:

• eHive provides a reliable and efficient solution for producing computationally demanding results in comparative genomics.
• The system enables high-throughput analysis with minimal user supervision.
• eHive is crucial for the continued scalability and success of the Ensembl project's comparative genomics resources.