Intriguing Properties of Positional Encoding in Time Series Forecasting
arxiv(2024)
摘要
Transformer-based methods have made significant progress in time series
forecasting (TSF). They primarily handle two types of tokens, i.e., temporal
tokens that contain all variables of the same timestamp, and variable tokens
that contain all input time points for a specific variable. Transformer-based
methods rely on positional encoding (PE) to mark tokens' positions,
facilitating the model to perceive the correlation between tokens. However, in
TSF, research on PE remains insufficient. To address this gap, we conduct
experiments and uncover intriguing properties of existing PEs in TSF: (i) The
positional information injected by PEs diminishes as the network depth
increases; (ii) Enhancing positional information in deep networks is
advantageous for improving the model's performance; (iii) PE based on the
similarity between tokens can improve the model's performance. Motivated by
these findings, we introduce two new PEs: Temporal Position Encoding (T-PE) for
temporal tokens and Variable Positional Encoding (V-PE) for variable tokens.
Both T-PE and V-PE incorporate geometric PE based on tokens' positions and
semantic PE based on the similarity between tokens but using different
calculations. To leverage both the PEs, we design a Transformer-based
dual-branch framework named T2B-PE. It first calculates temporal tokens'
correlation and variable tokens' correlation respectively and then fuses the
dual-branch features through the gated unit. Extensive experiments demonstrate
the superior robustness and effectiveness of T2B-PE. The code is available at:
\href{https://github.com/jlu-phyComputer/T2B-PE}{https://github.com/jlu-phyComputer/T2B-PE}.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要