arXiv · cs.AI · Konstantin Krestnikov · 5 小时前模型评分46
语言模型训练中的真相是压缩的副产品：Transformer 对错误规则的偏好研究
Truth as a Compression Artifact in Language Model Training
导读
一项 arXiv 研究用概率质量而非成对比较重新评估 Transformer 对正确与错误数学解法的偏好：86M 参数模型对正确补全仅占 59.0% 概率质量，而成对准确率 87.4% 高估了超出随机的部分。模型并未过滤一致的或可学习的错误结构，压缩仍只是动机而非实测机制。
分类置信度
中 · 待人工复核
- 只有单一媒体来源，需要人工复核。
来源级别：媒体与个人
事件类型：研究发布
证据摘录
- “We instead score the correct completion against every string the error generator can produce, a share on which frequency matching scores exactly 0.500.”
- “At 86M parameters the correct completion holds 59.0% +/- 0.8% of the mass against random errors (59.4% on an independent corpus), where the single realized error gives 77.6% and pair accuracy 87.4%, a three- to fourfold overstatement of the excess above chance; below 86M the primary corpus shows no preference.”
- “The coherent false rules of the main corpora are not filtered at any size (43.5--48.8%; the shortfall below 0.5 is a completion-length effect).”
- “With the scored false strings held fixed, a false rule whose selector the model can learn moves the correct share from 5 points above the null to 7 below (418M--3.08B), though the correct completion still outweighs the selected string.”
来源：arXiv · cs.AI · arxiv.org