On the risk of extracting relevant information from random data

Area of Science:

• Neuroscience
• Biomedical Engineering
• Signal Processing

Background:

• Re-assessment of a study using near-infrared spectroscopy (NIRS) to decode decision-making.
• Critique of the original study's feature selection methodology without independent validation.
• Introduction of a simulation using random time series to demonstrate potential risks.

Discussion:

• The original study performed feature selection on all data, risking overfitting.
• Lack of independent assessment in the original study hinders result validation.
• Simulation results mirrored the original study's findings, suggesting potential methodological flaws.

Key Insights:

• Classification accuracy from the original study may not represent genuine decision-making information.
• The methodology employed risks inflating classification accuracy through data dredging.
• Independent validation is crucial for reliable neuroimaging study findings.

Outlook:

• Further research should prioritize robust validation techniques in NIRS-based decision-making studies.
• Development of standardized protocols for feature selection and validation is recommended.
• Re-evaluation of existing NIRS studies with similar methodologies may be warranted.