Rigorous evaluation of five models for e-diary-only seizure forecasting: Retrospective and prospective datasets do...

Area of Science:

• Neurology
• Data Science
• Medical Informatics

Background:

• Electronic diaries (e-diaries) offer a potential tool for seizure forecasting to aid patient daily life management.
• Previous seizure forecasting models lacked rigorous testing against simple benchmarks.
• The Napkin method, a basic moving window average, serves as a benchmark for forecasting simplicity.

Purpose of the Study:

• To evaluate if machine learning models for seizure forecasting using e-diaries outperform the Napkin method.
• To assess the clinical effectiveness of seizure forecasting models based on e-diary data.

Main Methods:

• Analysis of retrospective and prospective cohorts from Seizure Tracker, including e-diary and seizure type data.
• Implementation and comparison of five machine learning models (Perceptron, 1D-convolution, Multilayer Perceptron, Cycle, point-process generalized linear model) against the Napkin forecast.
• Prediction of 24-hour seizure probability using a 90-day historical window and evaluation via standard metrics (AUC-PR, AUC-ROC, Brier score).

Main Results:

• A total of 5501 retrospective and 36 prospective patients were analyzed.
• No implemented machine learning model demonstrated significantly superior performance compared to the Napkin method across various metrics and seizure frequencies.
• Models were deemed clinically ineffective if they did not surpass the Napkin method's performance.

Conclusions:

• Current seizure forecasting techniques utilizing only e-diary data may not achieve clinically effective 24-hour risk prediction beyond the simplicity of the Napkin method.
• The feasibility of effective seizure forecasting with e-diaries alone remains questionable with existing methodologies.