https://arxiv.org/abs/2409.03282
Interpretable mixture of experts for time series prediction under recurrent and non-recurrent conditions
Time Features:
- Time of day (cyclic encoding)
- Day of week (cyclic encoding)
- Month of year (cyclic encoding)
- Holiday indicator (binary)
The Temporal Fusion Transformer (TFT) is a deep learning model designed for multi-horizon time series forecasting, which means it predicts future values over multiple time steps. It is particularly useful in scenarios involving complex temporal dependencies and multiple input features. TFT is both powerful for prediction and interpretable, making it stand out among time series forecasting models.
Key Components of TFT:
Variable Selection:
- TFT dynamically selects relevant features (both static and time-varying) that are important for making predictions at each time step. This is done using a gated residual network which assigns importance weights to different features. It allows the model to focus on the most relevant inputs for prediction, enhancing interpretability.
LSTM-based Encoder-Decoder:
- TFT employs long short-term memory (LSTM) networks, a type of recurrent neural network, for encoding past data (in a context window) and decoding future data (in the prediction window). The LSTM captures temporal patterns from the input data, which are crucial for accurate forecasting.
Multi-Head Attention:
- One of the standout features of TFT is the use of multi-head attention, inspired by the Transformer model. This mechanism helps the model focus on different parts of the time series and various time steps. Attention helps identify important temporal dependencies, such as sudden changes or long-term trends, at multiple time points.
Gating Mechanisms:
- TFT uses gating mechanisms throughout the model to regulate how information flows through its layers. These gates help prevent irrelevant information from propagating forward, improving efficiency and reducing noise in predictions.
Quantile Regression:
- Instead of just predicting a single point estimate, TFT can output quantile predictions (e.g., predictions at the 10th, 50th, and 90th percentiles), making it possible to estimate uncertainties in the forecast. This is particularly helpful when making forecasts under uncertain or volatile conditions.
Interpretability in TFT:
TFT is designed with interpretability in mind. Two main methods of interpretation are:
- Feature Importance: TFT quantifies the importance of each input feature in predicting the target value. This allows users to understand which features, such as weather conditions, traffic incidents, or the time of day, play the most crucial role in predictions.
- Temporal Attention: By utilizing multi-head attention, TFT can show which time steps in the past (within the context window) are the most influential for making predictions at future time steps.
Why TFT is Suitable for Traffic Prediction:
- Capturing Complex Temporal Dependencies: Traffic patterns often involve recurring trends (like rush hours) as well as non-recurring events (like accidents or severe weather). TFT’s attention mechanism helps capture both short-term and long-term dependencies between these events and traffic speed.
- Interpretability: Understanding the factors that influence traffic speeds, such as weather or incidents, is crucial for decision-making. TFT’s interpretability allows for insights into how these features affect predictions in different conditions (recurrent vs. non-recurrent).
- Multi-Source Inputs: TFT can efficiently handle multiple sources of data (like traffic incidents, weather conditions, etc.), making it well-suited for multi-variable prediction problems like traffic speed forecasting.
In this paper, TFT is used as the backbone for expert models in both recurrent and non-recurrent traffic prediction, benefiting from its ability to handle temporal dependencies and provide interpretability.
