On the distribution of performance from multiple neural-network trials.

Area of Science:

• Computational neuroscience
• Machine learning performance analysis

Background:

• Neural network simulation performance is typically reported using mean and standard deviation.
• This statistical approach assumes a Gaussian distribution, which is often not accurate for simulation results.

Purpose of the Study:

• To investigate the distribution of neural network simulation results for practical problems.
• To demonstrate how assuming Gaussian distributions can distort the interpretation of results, particularly in comparative studies.
• To propose improved guidelines for reporting simulation performance.

Main Methods:

• Analysis of result distributions from neural network simulations on practical tasks.
• Comparison of results assuming Gaussian distributions versus actual observed distributions.
• Controlled task evaluation to assess performance skew based on target function complexity.

Main Results:

• Observed distributions of simulation results are frequently non-Gaussian, asymmetric, or multimodal.
• Assuming Gaussian distributions can significantly impact the interpretation of comparative study outcomes.
• Performance distributions exhibit skewness: towards better performance for smoother functions and worse for complex functions.

Conclusions:

• Standard reporting of neural network simulation performance using mean and standard deviation can be misleading.
• The characteristics of the target function influence the skewness of performance distributions.
• Adopting new reporting guidelines that detail the actual distribution is crucial for accurate interpretation and comparison of neural network performance.