arXiv · cs.AI · Dayam Nadeem, Safdar Mustafa, Ameen Ur Rehman, Shahab Saquib Sohail, Mohammad Nadeem, Usman Naseem2026-11-03 13:00 · 9 小时前模型评分42

双语基准测试：五款开源大模型复述圣训全部未达精确匹配

They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs

导读

一项双语基准测试让 Mistral-7B、Gemma3-4B、Jais-7B、ALLaM-7B 和 Phi4-mini-3.8B 仅凭书目引用复述六大圣训集的原文，3 万条回答中 0% 同时达到高词汇保真度（BLEU）与高语义保真度。

分类置信度

中 · 待人工复核

来源级别：媒体与个人

应用领域：其他

事件类型：研究发布

证据摘录

“This study introduces a bilingual (English/Arabic) benchmark that probes five open-weight, locally-deployable language models (Mistral-7B, Gemma3-4B, Jais-7B, ALLaM-7B, and Phi4-mini-3.8B) across all six canonical Sunni Hadith collections”
“Each model is given only the bibliographic reference and asked to reproduce the exact Hadith text, with no retrieval augmentation, and is scored on lexical fidelity (BLEU) and semantic fidelity (multilingual cosine similarity), with refusals flagged separately.”
“The complete code, prompts, evaluation pipeline, and scored outputs are publicly available at this https URL .”

来源：arXiv · cs.AI · arxiv.org