Spatial-Temporal Transformer Networks for Traffic Flow Forecasting Using a Pre-Trained Language Model.

Area of Science:

• Artificial Intelligence
• Transportation Engineering
• Data Science

Background:

• Current methods for traffic analysis rely on spatial-temporal graph neural networks (STGNNs), which combine graph neural networks (GNNs) and sequence models.
• The hybrid nature of STGNNs leads to increased model complexity, posing challenges for efficient traffic flow analysis.

Purpose of the Study:

• To propose a novel framework for traffic flow analysis and forecasting using a simplified Transformer architecture.
• To efficiently extract spatial-temporal dependencies in traffic data by designing specialized embeddings.
• To leverage pre-trained language models for enhanced traffic forecasting performance.

Main Methods:

• Developed a framework based solely on the original Transformer architecture, avoiding complex hybrid structures.
• Designed specific embeddings to capture intricate spatial-temporal dependencies within traffic flow data.
• Utilized pre-trained language models to improve the accuracy of traffic flow predictions.
• Evaluated the proposed framework against state-of-the-art STGNNs and Transformer-based models on four real-world traffic datasets (PEMS04, PEMS08, METR-LA, PEMS-BAY).

Main Results:

• The proposed Transformer-based framework demonstrated superior performance across most evaluation metrics compared to existing STGNNs.
• The framework effectively extracted spatial-temporal dependencies, outperforming other models in traffic flow forecasting.
• Experiments on diverse real-world datasets confirmed the robustness and efficiency of the new approach.

Conclusions:

• The proposed Transformer-only framework offers a more efficient and effective solution for spatial-temporal traffic flow analysis and forecasting.
• This approach simplifies model complexity while enhancing predictive accuracy, providing a promising alternative to current STGNN methods.
• The findings suggest that carefully designed embeddings and pre-trained language models can significantly boost traffic forecasting capabilities.