Related Experiment Video
Updated: Aug 31, 2025
Semi-automated Biopanning of Bacterial Display Libraries for Peptide Affinity Reagent Discovery and Analysis of Resulting Isolates
Published on: December 6, 2017
Benchmarks in antimicrobial peptide prediction are biased due to the selection of negative data
Katarzyna Sidorczuk1, Przemysław Gagat1, Filip Pietluch1
1University of Wrocław, Faculty of Biotechnology, Poland.
Abstract:
Antimicrobial peptides (AMPs) are a heterogeneous group of short polypeptides that target not only microorganisms but also viruses and cancer cells. Due to their lower selection for resistance compared with traditional antibiotics, AMPs have been attracting the ever-growing attention from researchers, including bioinformaticians. Machine learning represents the most cost-effective method for novel AMP discovery and consequently many computational tools for AMP prediction have been recently developed. In this article, we investigate the impact of negative data sampling on model performance and benchmarking. We generated 660 predictive models using 12 machine learning architectures, a single positive data set and 11 negative data sampling methods; the architectures and methods were defined on the basis of published AMP prediction software. Our results clearly indicate that similar training and benchmark data set, i.e. produced by the same or a similar negative data sampling method, positively affect model performance. Consequently, all the benchmark analyses that have been performed for AMP prediction models are significantly biased and, moreover, we do not know which model is the most accurate. To provide researchers with reliable information about the performance of AMP predictors, we also created a web server AMPBenchmark for fair model benchmarking. AMPBenchmark is available at http://BioGenies.info/AMPBenchmark.
Insights
Negative data sampling significantly impacts antimicrobial peptide (AMP) prediction models. Ensuring consistent data generation for training and benchmarking is crucial for accurate AMP discovery and reliable performance evaluation.
Area of Science:
- Bioinformatics
- Computational Biology
- Drug Discovery
Background:
- Antimicrobial peptides (AMPs) show promise against microbes, viruses, and cancer cells, offering an alternative to traditional antibiotics due to lower resistance.
- Machine learning (ML) is a cost-effective approach for discovering novel AMPs, leading to the development of numerous computational prediction tools.
- The performance of these AMP prediction tools is heavily influenced by the data used for training and benchmarking.
Purpose of the Study:
- To investigate the impact of negative data sampling methods on the performance of machine learning models for antimicrobial peptide (AMP) prediction.
- To evaluate the reliability of current benchmarking practices for AMP prediction software.
- To provide a standardized platform for fair and accurate benchmarking of AMP predictors.
Main Methods:
- Generated 660 predictive models using 12 distinct ML architectures and 11 different negative data sampling strategies.
- Defined model architectures and sampling methods based on established AMP prediction software.
- Developed a web server, AMPBenchmark, for standardized and unbiased performance evaluation.
Main Results:
- Model performance is positively affected when training and benchmark datasets are generated using the same or similar negative data sampling methods.
- Current benchmarking analyses for AMP prediction models are significantly biased due to inconsistent data sampling.
- The true accuracy of existing AMP prediction models remains uncertain.
Conclusions:
- Inconsistent negative data sampling in training and benchmarking leads to biased evaluations of antimicrobial peptide (AMP) prediction models.
- A standardized approach to data generation is essential for reliable performance assessment.
- The AMPBenchmark web server offers a solution for fair and accurate benchmarking, aiding researchers in selecting optimal AMP prediction tools.
More Related Videos
11:56Antimicrobial Peptides Produced by Selective Pressure Incorporation of Non-canonical Amino Acids
Published on: May 4, 2018
08:31Biosensor-based High Throughput Biopanning and Bioinformatics Analysis Strategy for the Global Validation of Drug-protein Interactions
Published on: December 1, 2020
Related Concept Videos
00:57Antibiotic Selection
01:23Antimicrobial Proteins
Interferons
Interferons (IFNs) are proteins produced by lymphocytes, macrophages, and fibroblasts infected with viruses. While IFNs cannot prevent viruses from entering and...