An evaluation methodology for machine learning-based tandem mass spectra similarity prediction

Area of Science:

• Analytical Chemistry
• Computational Chemistry
• Bioinformatics

Background:

• Untargeted tandem mass spectrometry (MS/MS) is crucial for small molecule analysis and organization.
• Molecular networking visualizes structurally related compounds but relies on MS/MS spectral comparison, a key bottleneck.
• Machine learning (ML) shows promise for predicting MS/MS structural similarity, yet lacks standardized evaluation methods and addresses data leakage concerns.

Purpose of the Study:

• To develop a standardized methodology for evaluating ML models in MS/MS spectral similarity.
• To create a robust training and evaluation framework that mirrors real-world applications.
• To assess the impact of MS-specific insights on ML model performance and generalizability.

Main Methods:

• Introduced a novel evaluation methodology with a train/test split allowing variable structural similarity.
• Developed a framework measuring prediction accuracy using domain-inspired annotation and retrieval metrics.
• Investigated the effect of MS-specific training insights (instrumentation, collision energy, adduct) on performance.

Main Results:

• Demonstrated the ability to evaluate ML models at varying degrees of structural similarity.
• Showcased the effectiveness of domain-inspired metrics for practical performance reflection.
• Highlighted the significant role of collision energy in prediction errors and confirmed metric orthogonality.

Conclusions:

• The developed benchmark provides a foundation for future ML development in MS/MS similarity.
• Facilitates standardized comparison between different ML models for MS/MS data.
• The proposed evaluation metrics offer a more realistic reflection of practical performance in small molecule analysis.