Mixture of TSMixer Experts for Time Series Forecasting.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Time Series Analysis

Background:

• Multi-Layer Perceptron (MLP) mixer models show state-of-the-art performance in time series forecasting.
• Extending MLP mixers to Mixture-of-Experts (MoE) architectures increases model capacity but also trainable parameters, posing training challenges.

Purpose of the Study:

• To propose an efficient method for creating Mixture-of-Experts (MoE) models for time series forecasting.
• To mitigate the increased parameter count and training complexity associated with MoE architectures.

Main Methods:

• A novel MoE approach is proposed, comprising one fully trainable global expert and multiple non-trainable local experts.
• Local experts are generated by cloning the global expert's weights and applying moment learning for weight distribution modification.
• The method utilizes a lightweight Time Series Mixer (TSMixer) architecture.

Main Results:

• The proposed method achieves forecasting accuracy competitive with fully trainable MoE models.
• The approach significantly reduces the increase in trainable parameters compared to traditional MoE models.
• Efficiency is further validated by memory footprint measurements and effect-size analysis.

Conclusions:

• The proposed method offers an efficient way to enhance time series forecasting models using MoE architectures.
• Expert specialization is achieved without the need for extensive independent training of local experts.
• This approach provides a favorable trade-off between performance, parameter efficiency, and memory usage.