arXiv · cs.AI · Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Chaitanya Lakhchaura, Rishitej Reddy Vyalla, Erik Cambria, Zhengkui Wang, Rajiv Ratn Shah · 10 小时前模型评分38
Sonar-MoE：面向内存高效 MoE 训练的反向传播内专家更新方法
Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training
导读
Sonar-MoE 在反向传播过程中直接更新每个专家张量并立即释放其梯度，使 图形处理器（GPU，Graphics Processing Unit） 最多只保留一层的专家梯度。
分类置信度
中 · 待人工复核
- 只有单一媒体来源，需要人工复核。
来源级别：媒体与个人
事件类型：研究发布
证据摘录
- “We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away.”
- “When we fine-tune OLMoE-1B-7B and Granite-3.1-3B-A800M in bf16 with two expert-kernel backends, Sonar-MoE lowers peak memory by 19% to 22%.”
- “Its updates are bitwise identical to standard training without expert gradient clipping, and held-out loss stays within 0.003 of all baselines.”
来源：arXiv · cs.AI · arxiv.org