arXiv · cs.AI · Yichen Zhao, Minghao Shao, Zhuoyan Yu, Yijun Shen, Boyuan Chen, Yik-Cheung Tam, Muhammad Shafique · 11 小时前模型评分49
ROSTrans：面向 ROS1 到 ROS2 迁移的大语言模型基准测试
ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration
导读
ROSTrans 基准用 108 个任务评测大语言模型把 ROS1 代码迁移到 ROS2 的能力，任务来自 70 多个开源仓库，按接口、行为和系统三级正确性划分。六个前沿模型 pass@1 均低于 10.9%，56.5% 的样本编译成功却在运行时失败。执行引导修复三轮后，Claude Opus 4.6 和 GPT-5.5 的任务成功率分别升至 49.1% 和 47.2%。
分类置信度
中 · 待人工复核
- 只有单一媒体来源，需要人工复核。
来源级别：媒体与个人
应用领域：制造与工业、科学研究
事件类型：研究发布
证据摘录
- “We present ROSTrans, a benchmark for evaluating LLMs on ROS1-to-ROS2 code migration, a representative and practically important setting for ROS code generation.”
- “Candidates are scored by injecting only their target files into an otherwise fixed copy of the reference package, so a candidate can neither supply nor weaken the tests it is judged by.”
- “Evaluating six frontier LLMs over five stochastic generations per task, we find pass@1 below 10.9%, and that 56.5% of all samples build successfully and then fail at runtime,and 32.0% fail runtime assertions on the migrated interfaces, a class of error no compile-only oracle can detect.”
- “Execution-guided repair further improves task success, reaching 49.1% and 47.2% for Claude Opus 4.6 and GPT-5.5 after three rounds, respectively.”
来源：arXiv · cs.AI · arxiv.org