arXiv · cs.AI · Volodymyr Ovcharov2026-11-03 13:00 · 10 小时前模型评分49

法律选择题基准测试暴露“只看选项”漏洞：GPT-5.6 与 Gemini 3.1 Pro 仍显著高于随机水平

Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks

导读

研究在乌克兰法官资格考试数据集 UA-JudgeExam 的 11,990 道四选项题目上发现，隐藏题干只给选项时 Claude Haiku 4.5 得分 0.383，高于随机水平，其中 11.8% 的题目在全部八种选项顺序下都能盲答。

分类置信度

中 · 待人工复核

来源级别：媒体与个人

应用领域：法律与合规

事件类型：研究发布

证据摘录

“We release the corpus, predictions and harness.”
“Gating those out retains 8,128 items, on which the gating model itself now scores 0.204, while GPT-5.6, which took no part in the selection, still answers 0.515 of them with the question hidden.”
“Rewriting distractors instead overshoots to 0.168, below chance and as exploitable.”

来源：arXiv · cs.AI · arxiv.org