A comprehensive evaluation of self-attention for detecting regulatory feature interactions

Area of Science:

• Computational Biology
• Bioinformatics
• Machine Learning

Background:

• Deep learning models are increasingly used in computational biology.
• Interpreting these models is crucial for extracting meaningful biological insights.
• Attention maps from self-attention layers show promise in predicting transcription factor cooperativity.

Purpose of the Study:

• To enhance the interpretability and precision of attention maps for predicting transcription factor cooperativity.
• To evaluate different attention-based methods for discovering transcription factor cooperativity.

Main Methods:

• Incorporating an entropy term into self-attention layers to generate sparse attention maps.
• Developing and evaluating various attention-based models for transcription factor cooperativity discovery.
• Comprehensive performance comparison of different attention model flavors.

Main Results:

• The addition of an entropy term resulted in high-precision, interpretable sparse attention maps.
• Entropy-enhanced attention models demonstrated significant benefits in transcription factor cooperativity discovery.
• A comprehensive evaluation provided insights into the relative performance of different attention-based approaches.

Conclusions:

• Entropy-enhanced attention models offer a valuable tool for biological discovery in computational biology.
• The developed methods improve the interpretability and precision of deep learning models for gene regulatory network analysis.
• Practitioners can effectively utilize these insights for advanced biological discoveries.