arXiv · cs.AI · Javad Mohammad Alizadeh, Huanmei Wu · 11 小时前模型评分47
THReD 基准测试：评估 大语言模型（LLM，Large Language Model） 智能体及其预处理工具将电子健康记录转为 人工智能（AI，Artificial Intelligence） 可用数据集的能力
Benchmark of THReD: Evaluating an LLM Agent and Its Preprocessing Tools for Turning Electronic Health Records into AI-Ready Datasets
导读
研究团队对基于聊天智能体的 THReD 系统开展基准测试，该系统将电子健康记录（EHR，Electronic Health Record）提取数据转为 人工智能（AI，Artificial Intelligence） 可用数据集，智能体提出的步骤需用户确认后才执行。
分类置信度
中 · 待人工复核
- 存在未解决的相邻类别：ai-tooling，需要人工复核。
- 只有单一媒体来源，需要人工复核。
来源级别：媒体与个人
应用领域：医疗与生命科学
事件类型：研究发布
证据摘录
- “We report a benchmark of THReD, a web-based system that turns electronic health record (EHR) extracts into AI-ready datasets through a chat agent whose proposed steps run only after user confirmation.”
- “The agent benchmark has 141 tasks (100 templates) in seven categories, including goal-level requests, failure recovery, requests calling for no action, schema variation and prompt injection through the data, on synthetic tables calibrated to a regional health information exchange; success is judged by checks on the produced data.”
- “Over 2,550 episodes with Claude Haiku 5.5 in four context configurations, the deployed agent completed 92.7% (95% CI 88.1-96.4) of tasks on its first answer, abstained on every advice request, recovered from every injected fault, and never proposed an action requested by planted text.”
- “Without the data profile, completion fell to 58.0%; goal-level requests were hardest (80.6%).”
- “Code, task definitions, and per-episode results are available on Zenodo via the DOI provided in the DOI field”
来源：arXiv · cs.AI · arxiv.org