arXiv · cs.AI · Ahmad Jarjis Hasan, Jonathan Lwowski2026-11-03 13:00 · 7 小时前模型评分65

同输出不同分数：多模态大模型基准评分器审计研究

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks

导读

研究复现了八个同行评审多模态基准的官方评分器，在五个可装入单张 16 GB 图形处理器（GPU，Graphics Processing Unit）的开源视觉语言模型（VLM，Vision-Language Model）上发现，仅评分规则一项就能让相同输出的准确率相差最多 22 分。

分类置信度

中 · 待人工复核

来源级别：媒体与个人

事件类型：研究发布

证据摘录

“We reproduce the official scorers of eight peer-reviewed benchmarks on five open vision-language models (VLMs) that fit a single 16 GB GPU, and introduce a blinded human-plus-LLM-judge adjudication (Cohen's kappa=0.956) that recovers what the model actually answered.”
“We release all code, labels, and the protocol, and argue benchmarks should report a non-compliance rate and at least two rules, not one unexamined grade.”

来源：arXiv · cs.AI · arxiv.org