QKFormer: Hierarchical Spiking Transformer using Q-K Attention
arxiv(2024)
摘要
Spiking Transformers, which integrate Spiking Neural Networks (SNNs) with
Transformer architectures, have attracted significant attention due to their
potential for energy efficiency and high performance. However, existing models
in this domain still suffer from suboptimal performance. We introduce several
innovations to improve the performance: i) We propose a novel spike-form Q-K
attention mechanism, tailored for SNNs, which efficiently models the importance
of token or channel dimensions through binary vectors with linear complexity.
ii) We incorporate the hierarchical structure, which significantly benefits the
performance of both the brain and artificial neural networks, into spiking
transformers to obtain multi-scale spiking representation. iii) We design a
versatile and powerful patch embedding module with a deformed shortcut
specifically for spiking transformers. Together, we develop QKFormer, a
hierarchical spiking transformer based on Q-K attention with direct training.
QKFormer shows significantly superior performance over existing
state-of-the-art SNN models on various mainstream datasets. Notably, with
comparable size to Spikformer (66.34 M, 74.81
groundbreaking top-1 accuracy of 85.65
outperforming Spikformer by 10.84
time that directly training SNNs have exceeded 85
code and models are publicly available at
https://github.com/zhouchenlin2096/QKFormer
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要