A pitfall for machine learning methods aiming to predict across cell types.

Area of Science:

• Genomics
• Bioinformatics
• Machine Learning

Background:

• Machine learning models are crucial for predicting genomic activity.
• Accurate predictions across diverse cell types are essential for model utility.
• Overfitting can lead to misleading performance metrics in genomic prediction models.

Purpose of the Study:

• To identify and explain a common pitfall in machine learning models for genomic activity prediction.
• To demonstrate how models can falsely appear accurate due to memorizing genomic loci.
• To propose methods for diagnosing and avoiding this overfitting issue.

Main Methods:

• Training and testing machine learning models on genomic data.
• Analyzing model performance when training and test sets share identical genomic loci.
• Evaluating predictions of gene expression and chromatin domain boundaries.

Main Results:

• Models trained and tested on the same genomic loci can exhibit inflated performance metrics.
• This phenomenon arises from the model memorizing locus-specific average activity.
• The issue was observed in both gene expression and chromatin boundary prediction tasks.

Conclusions:

• Over-reliance on identical genomic loci in training and testing sets can create a false impression of model generalizability.
• Awareness and implementation of diagnostic methods are crucial for developing robust genomic prediction models.
• Future large-scale genomic projects must proactively address this memorization pitfall to ensure reliable predictions across cell types.