arXiv · cs.AI · Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan · 9 小时前模型评分41
距离自适应表示DAR：按token距离分配KV维度以压缩缓存
Dual Dimensionality for Local and Global Attention
导读
研究提出距离自适应表示（DAR，Distance-Adaptive Representation），在局部上下文窗口内保留全维度Key/Value，窗口外token降至1/2或1/4维度。在70M至410M参数预训练及1B模型监督微调中，DAR性能接近全维度基线，而全局统一降维效果更差。其高效实现将8K上下文平均KV缓存缩小68.8%，128K上下文解码提速最高1.9倍。
分类置信度
中 · 待人工复核
- 只有单一媒体来源，需要人工复核。
来源级别：媒体与个人
事件类型：研究发布
证据摘录
- “We formalize this idea as Distance-Adaptive Representation (DAR), which preserves full-dimensional representations within a local context window while assigning reduced-dimensional representations (e.g. 1/2 or 1/4 of the original dimensionality) to tokens beyond that window.”
- “Across multiple pretraining scales (70M to 410M parameters), as well as continued supervised fine-tuning on a 1B-scale model, this approach closely matches the performance of full-dimensional baselines.”
- “We further develop an efficient realization of DAR that achieves model quality comparable to the MLA baseline while reducing the average KV-cache size by 68.8\% at 8K context and achieving up to a $1.9\times$ decoding speedup at 128K context on the evaluated models.”
来源：arXiv · cs.AI · arxiv.org