arXiv · cs.AI · Zain Naboulsi · 9 小时前模型评分57
虚假引用是否属于奖励劫持？一项关于后训练与引用提示的预注册研究
Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues
导读
研究在900道固定问题上对三个模型家族四个后训练阶段的十二个开放权重检查点做了四项预注册实验，检验后训练语言模型编造引用是否属于奖励劫持。
分类置信度
中 · 待人工复核
- 只有单一媒体来源，需要人工复核。
来源级别：媒体与个人
应用领域：法律与合规、医疗与生命科学、科学研究
事件类型：研究发布
证据摘录
- “We ran four pre-registered experiments on 900 frozen questions (US federal case law, computationally adjacent scientific literature, and medical literature) across twelve open-weights checkpoints covering four post-training stages of three model families, then refit on fifteen.”
- “Asking for citations raises fabrication sharply in post-trained checkpoints and not in base ones: a 47.8-point interaction on twelve checkpoints (Holm-adjusted p = 2e-34) and 41.5 points on fifteen, in all three domains.”
- “Post-training also raises fabrication overall, from 4.8% to 56.7%, but our registered prediction that it would rise most where fabrication is most tempting did not hold.”
- “A preference model trained only on surface citation features prefers post-trained output 86% of the time, and holding citation count fixed shows it rewards quantity.”
来源：arXiv · cs.AI · arxiv.org