A Framework for Comparison and Interpretation of Machine Learning Classifiers to Predict Autism on the ABIDE Dataset

Area of Science:

• Neuroscience
• Computer Science
• Biomedical Engineering

Background:

• Autism Spectrum Disorder (ASD) is a neurodevelopmental condition affecting approximately 1% of the population.
• Machine learning (ML) models are increasingly used to classify autism using neuroimaging data, but performance varies across studies.
• Discrepancies in experimental setups hinder direct comparisons of ML model efficacy for autism diagnosis.

Purpose of the Study:

• To standardize and compare the performance of five prominent ML models for autism classification.
• To identify factors contributing to performance variations in autism neuroimaging research.
• To evaluate feature stability across different ML models for autism classification.

Main Methods:

• Utilized the Autism Brain Imaging Data Exchange (ABIDE) dataset, including functional connectivity, structural volumes, and phenotypic information.
• Trained and evaluated five ML models: Graph Convolutional Networks (GCN), Edge-Variational Graph Convolutional Networks (EV-GCN), Fully Connected Networks (FCN), Autoencoder followed by FCN (AE-FCN), and Support Vector Machine (SVM).
• Employed a unified evaluation standard to compare model performance, including classification accuracy and Area Under the Curve (AUC). Feature stability was assessed using the SmoothGrad method.

Main Results:

• All tested ML models achieved comparable classification accuracy around 70%.
• Ensemble models, particularly GCN, showed the highest accuracy (72.2%) and AUC (0.77), though not significantly better than SVM (70.1% accuracy, 0.77 AUC) under identical testing conditions.
• Fully Connected Networks (FCN) demonstrated the most stable feature selection for autism classification, as indicated by SmoothGrad analysis.

Conclusions:

• Variations in published autism classification accuracies may stem from differences in inclusion criteria, data modalities, and evaluation pipelines, rather than inherent differences between ML algorithms.
• While ensemble GCN models showed slightly superior performance, SVM remains a competitive and robust choice for autism classification.
• Feature stability analysis is crucial for understanding model interpretability and reliability in neuroimaging-based autism research.