Explaining the optimistic performance evaluation of newly proposed methods: A cross-design validation experiment

Area of Science:

• Bioinformatics
• Computational Biology
• Biostatistics

Background:

• New data analysis methods are frequently introduced with superior performance claims.
• Subsequent studies often report diminished performance for these same methods.
• This discrepancy raises concerns about the reproducibility and reliability of method evaluations.

Purpose of the Study:

• To investigate the reasons behind performance discrepancies between original and subsequent studies of data analysis methods.
• To systematically assess method performance across different study designs.

Main Methods:

• Cross-design validation of methods was employed.
• Two data analysis tasks were used: cancer subtyping with multiomic data and differential gene expression analysis.
• Methods were re-evaluated using the study design (datasets, competing methods, evaluation criteria) of other methods.

Main Results:

• Three out of four methods showed reduced performance on the new study design.
• Differences in datasets were identified as a primary cause for performance decline.
• The study illustrates the impact of varying assessment degrees of freedom on method performance.

Conclusions:

• Performance discrepancies may stem from factors beyond author non-neutrality, including expertise and application field differences.
• Transparent evaluation and comprehensive method documentation are crucial for ensuring reliable use of new methods in subsequent studies.