Assessing Random Forest self-reproducibility for optimal short biomarker signature discovery

Area of Science:

• Biomedical informatics
• Machine learning in diagnostics
• Biomarker discovery

Background:

• Biomarker signature discovery is crucial for clinical diagnostics, especially with limited pathological knowledge.
• Shorter biomarker signatures are preferred for cost-effectiveness in diagnostics.
• Algorithm selection for biomarker discovery often relies on average Area Under the Curve (AUC), but AUC stability is not guaranteed.

Purpose of the Study:

• To introduce two novel AUC-derived hyper-stability scores: Hyper-stability Resampling Sensitive (HRS) and Hyper-stability Signature Sensitive (HSS).
• To propose these scores as complementary metrics to average AUC for enhancing confidence in classification algorithm selection.
• To evaluate the performance and stability of various Random Forest implementations.

Main Methods:

• Comparison of 15 different Random Forest implementations.
• Assessment of classification algorithms using average AUC performance.
• Introduction and application of the proposed HRS and HSS hyper-stability scores.

Main Results:

• No single Random Forest implementation is universally optimal for all datasets and classification tasks.
• Algorithm performance is highly dependent on the specific data and classification question.
• The proposed HRS and HSS scores provide valuable insights into algorithm stability beyond average AUC.

Conclusions:

• The choice of Random Forest implementation for biomarker signature discovery should be data- and task-dependent.
• Average AUC alone is insufficient for selecting robust classification algorithms.
• Prior testing of both average AUC performance and AUC-derived stability (HRS, HSS) is essential for reliable diagnostic tool development.