arXiv · cs.AI · Raghu Venkat, Tricha Anjali · 11 小时前模型评分46
研究：置信度分数能否判断 文本转结构化查询语言（text-to-SQL，Text to Structured Query Language） 答案正确性？基于 BIRD 与 Spider 的探索性研究
Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider
导读
一项研究审计了生产级对话分析系统中置信度分数的设计，在 BIRD dev 的 1,534 道题上由 GPT-5.1 每题采样五次作答，该分数区分对错的能力较弱（AUROC 0.624），并会扣留 35.7% 的正确答案。
分类置信度
中 · 待人工复核
- 只有单一媒体来源，需要人工复核。
来源级别：媒体与个人
事件类型：研究发布
证据摘录
- “This paper audits that design.”
- “On the 1,534 questions of BIRD dev, answered by GPT-5.1 with five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624) and withholds 35.7% of correct answers.”
- “Before any Spider answer existed we froze and hashed a rule in which agreement with the shown answer is the score, with deterministic vetoes and deterministic and judge-based caps.”
- “The frozen rule was not the best agreement-based rule on Spider, and the harness and prompts were rebuilt after registration.”
- “Labels are model-judged; a ten-answer human pilot found no disagreement, and a full audit has not been done.”
来源：arXiv · cs.AI · arxiv.org