A cautionary tale about properly vetting datasets used in supervised learning predicting metabolic pathway...

Area of Science:

• Biochemistry
• Bioinformatics
• Machine Learning

Background:

• Metabolite-specific data mapping to cellular metabolism pathways is crucial for biochemical interpretation.
• Machine learning, especially deep learning, is used for metabolite-to-pathway mapping prediction using known datasets.
• The Kyoto Encyclopedia of Genes and Genomes (KEGG) is a source for such training datasets.

Purpose of the Study:

• To describe and evaluate the erroneous KEGG-SMILES dataset.
• To identify publications that used this flawed dataset.
• To demonstrate the impact of duplicate entries on machine learning model performance.

Main Methods:

• Analysis of the KEGG-SMILES dataset for duplicate entries.
• Identification of prior research utilizing the KEGG-SMILES dataset.
• Evaluation of machine learning model performance before and after de-duplicating the dataset.

Main Results:

• The KEGG-SMILES dataset contains a significant proportion (~26%) of duplicate entries.
• The presence of duplicates grossly inflates the k-fold cross-validation performance of machine learning models.
• De-duplicating the dataset leads to a reduction in reported model performance.

Conclusions:

• The KEGG-SMILES dataset is erroneous and should not be used for training machine learning models.
• Prior machine learning results based on this dataset require critical re-evaluation.
• Proper vetting of benchmark datasets is essential to avoid inflated performance metrics in machine learning research.