arXiv Daily Digest

150

Papers

Uncertainty-Aware Motion Planning for Autonomous Driving in Mixed Traffic Environment

混合交通环境下的自主驾驶不确定性感知运动规划

Cheng, Ming, Chen, Hao, Yang, Ziyi, Luo, Ziluowen, Wang, Senzhang

Abstract

In mixed-traffic environments where autonomous and human-driven vehicles may co-exist, motion planning for autonomous vehicles requires anticipating the future behaviors of surrounding human drivers. Existing reinforcement learning-based methods generally directly incorporate the predicted human intents into the observation to enable a proactive planning. However, human intent is inherently uncertain due to the behavioral diversity, perception noise, and partial observability. Treating predicted intends as deterministic states can result in unsafe decisions for autonomous vehicles. To address this problem, we propose Uncertainty-Aware Motion Planning (UAMP), which incorporates uncertainty in human intent prediction for AV decision-making. Specifically, UAMP first introduces a proximity-aware uncertainty estimator to quantify the interaction-conditioned intent uncertainty and constructs an uncertainty-guided joint intent distribution over surrounding human-driven vehicles. Within this uncertainty set, UAMP further introduces Uncertainty-Calibrated Value Learning (UCVL) to correct value function learning biases arising from directly incorporating uncertain human intent predictions into the observation. Extensive experiments in various mixed-traffic scenarios show that UAMP significantly improves safety and driving comfort, while maintaining traffic efficiency compared with existing approaches. The code is released at https://anonymous.4open.science/r/UAMP-5638.

Chinese Translation

在自主驾驶车辆与人类驾驶车辆共存的混合交通环境中，自主车辆的运动规划需要预测周围人类驾驶者的未来行为。现有的基于强化学习的方法通常直接将预测的人类意图纳入观察中，以实现主动规划。然而，由于行为多样性、感知噪声和部分可观测性，人类意图本质上是不确定的。将预测的意图视为确定性状态可能导致自主车辆做出不安全的决策。为了解决这一问题，我们提出了不确定性感知运动规划（Uncertainty-Aware Motion Planning, UAMP），该方法在自主车辆决策中纳入了人类意图预测的不确定性。具体而言，UAMP首先引入了一种基于接近度的不确定性估计器，以量化交互条件下的意图不确定性，并构建了一个围绕周围人类驾驶车辆的不确定性引导的联合意图分布。在这个不确定性集合中，UAMP进一步引入了不确定性校准价值学习（Uncertainty-Calibrated Value Learning, UCVL），以纠正因直接将不确定的人类意图预测纳入观察而产生的价值函数学习偏差。在各种混合交通场景中的广泛实验表明，与现有方法相比，UAMP显著提高了安全性和驾驶舒适性，同时保持了交通效率。代码已发布在 https://anonymous.4open.science/r/UAMP-5638。

View on arXiv Download PDF AI Translation

cs.RO / 2 / 2606.10025

GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation

GHOST：用于机器人操作泛化的分层子目标策略

Krishna, Sriram, Eisner, Ben, Zhan, Haotian, Yuan, Ying, Zhen, Haoyu, Gan, Chuang, Tulsiani, Shubham, Held, David

Abstract

We present GHOST, a framework for learning visuomotor manipulation policies that generalize beyond the training distribution. GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses from multi-view RGB-D observations, and (ii) a low-level goal-conditioned controller that executes embodiment-specific actions. To condition image-based policies on 3D goals, we introduce a simple spatial interface that projects predicted goals into the image plane and represents them as end-effector heatmaps. Across a suite of manipulation tasks, this hierarchical factorization consistently improves performance and robustness compared to a flat Diffusion Policy. Further, we show that this hierarchical interface also makes it easy to incorporate human demonstrations without relying on (noisy) action retargeting. As sub-goals are largely embodiment-agnostic, we train the high-level policy on human video to specify how learned skills should be applied and composed, while keeping the low-level policy trained purely on robot data. This hierarchy enables adaptation to novel objects and task variations using a small number of human demonstrations.

Chinese Translation

我们提出了GHOST，一个学习视觉运动操作策略的框架，该策略能够超越训练分布进行泛化。GHOST将控制因素分解为（i）一个高层策略，该策略根据多视角RGB-D观察预测下一个子目标，作为3D末端执行器姿态的分布，以及（ii）一个低层目标条件控制器，该控制器执行特定于体现的动作。为了将基于图像的策略条件化于3D目标，我们引入了一个简单的空间接口，该接口将预测的目标投影到图像平面，并将其表示为末端执行器热图。在一系列操作任务中，这种分层分解始终比平坦的扩散策略（Diffusion Policy）提高了性能和鲁棒性。此外，我们还展示了这种分层接口使得在不依赖（嘈杂的）动作重定向的情况下，轻松地融入人类演示。由于子目标在很大程度上与体现无关，我们在人的视频上训练高层策略，以指定学习到的技能应如何应用和组合，同时保持低层策略仅在机器人数据上进行训练。这种层次结构使得使用少量人类演示适应新物体和任务变体成为可能。

View on arXiv Download PDF AI Translation

cs.RO / 3 / 2606.10039

Robotic Nonprehensile Object Transportation with a Hanging Tray

带悬挂托盘的机器人非抓握物体运输

Heins, Adam, Schoellig, Angela P.

Abstract

We consider the nonprehensile object transportation task known as the waiter's problem, in which a robot must move an object balanced on a tray from one location to another. In contrast to prior works on the robotic waiter's problem, which make the robot tilt a tray rigidly held by its end effector (EE), we use a tray suspended from the EE by ropes, such that it behaves like a three-dimensional pendulum. Some prior works have actuated the robot so that the EE simulates the behavior of a pendulum, because pendular motion reduces the shear forces acting on the transported objects, minimizing the sliding of rigid objects and sloshing in containers of liquid. In contrast, our use of a real hanging tray allows us to obtain the benefits of pendular motion while only actuating a 3 degree-of-freedom (DOF) mobile base, rather than requiring a full 6-DOF manipulator arm. Our experiments in simulation and on real hardware show that the hanging tray substantially reduces both sliding and sloshing compared to a static, rigidly-grasped tray. Furthermore, we integrate the hanging tray into an interactive robot waiter demonstration, which uses computer vision to identify people with a raised hand and visual servoing to steer toward them and allow them to access the tray.

Chinese Translation

我们考虑一种非抓握物体运输任务，称为服务员问题，其中机器人必须将一个平衡在托盘上的物体从一个位置移动到另一个位置。与之前关于机器人服务员问题的研究不同，后者使机器人以刚性方式倾斜由末端执行器（EE）握持的托盘，我们采用通过绳索悬挂于EE的托盘，使其表现得像一个三维摆。一些先前的研究使机器人运动，以便EE模拟摆的行为，因为摆动运动减少了作用于运输物体的剪切力，从而最小化了刚性物体的滑动和液体容器中的晃动。相较之下，我们使用真实的悬挂托盘，使我们能够获得摆动运动的好处，同时只需驱动一个具有3个自由度（DOF）的移动底座，而不需要完整的6-DOF机械臂。我们在仿真和真实硬件上的实验表明，与静态、刚性抓握的托盘相比，悬挂托盘显著减少了滑动和晃动。此外，我们将悬挂托盘集成到一个互动机器人服务员演示中，该演示利用计算机视觉识别举手的人，并通过视觉伺服引导机器人朝他们移动，使他们能够接触托盘。

View on arXiv Download PDF AI Translation

cs.RO / 4 / 2606.10040

Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imagination

高效世界-动作模型（Efficient-WAM）：一种具有低成本未来想象的10亿参数模型

Li, Jiajun, Guo, Tiecheng, Ye, Yifan, Zhang, Rongyu, Chi, Xiaowei, Sun, Qianpu, Li, Ying, Lou, Yunfan, Huang, Yan, Lu, Zhihe, Guo, Meng, Zhang, Shanghang

Abstract

World-Action Models (WAMs) have emerged as a promising paradigm for embodied control by coupling future visual prediction with action generation. However, most existing WAMs rely on photorealistic future prediction, which incurs high inference latency and makes real-time robot deployment difficult. This motivates a more efficient WAM design that preserves the control benefits of future visual prediction while reducing its inference cost. We introduce Efficient-WAM, a World-Action Model that reduces the cost of future imagination while preserving its control benefit. Efficient-WAM improves inference efficiency via a compact video expert transferred from WAN-2.2-5B, token-sparse video latents, and asymmetric video-action denoising that allocates fewer sampling steps to video than to actions. Instead of optimizing the future branch for visual fidelity, Efficient-WAM treats future video prediction as a compact guidance signal for action generation. Comprehensive experiments on RoboTwin 2.0 and real-world manipulation tasks show that Efficient-WAM maintains strong action performance despite visibly coarse future predictions. While maintaining competitive control capabilities, our 1B-parameter model can reduce per-chunk latency to around 100 ms during physical deployment, achieving a 30x speedup over existing WAMs.

Chinese Translation

世界-动作模型（World-Action Models, WAMs）通过将未来视觉预测与动作生成相结合，成为了具身控制的一个有前景的范式。然而，现有的大多数WAMs依赖于逼真的未来预测，这导致高推理延迟，使得实时机器人部署变得困难。这促使我们设计一个更高效的WAM，以在降低推理成本的同时保留未来视觉预测的控制优势。我们提出了高效WAM（Efficient-WAM），一种在降低未来想象成本的同时保留其控制效益的世界-动作模型。高效WAM通过从WAN-2.2-5B转移的紧凑视频专家、稀疏的令牌视频潜变量以及不对称视频-动作去噪来提高推理效率，这种方法为视频分配的采样步骤少于动作。高效WAM将未来视频预测视为动作生成的紧凑指导信号，而不是优化未来分支以提高视觉逼真度。在RoboTwin 2.0和真实世界操作任务上的全面实验表明，尽管未来预测明显粗糙，高效WAM仍能保持强大的动作性能。在保持竞争控制能力的同时，我们的10亿参数模型在物理部署期间将每个块的延迟降低至约100毫秒，实现了相较于现有WAMs的30倍加速。

View on arXiv Download PDF AI Translation

cs.RO / 5 / 2606.10180

Flow Control: Steering Vision-Language-Action Models with Simple Real-Time Inputs

流控制：通过简单的实时输入引导视觉-语言-动作模型

Kao, Jonathan C., Chan, Jason, Wang, Andy

Abstract

We introduce flow control of vision-language-action (VLA) models, a simple and effective way to steer VLA actions in real-time through generic inputs, such as a keyboard. This method can be used out-of-the-box and does not require retraining or fine-tuning VLAs. It enables relatively crude user inputs to steer a VLA to align with user intent. The VLA transforms these inputs into action samples drawn from the VLA expert action distribution learned during training, so that the generated actions are high quality (conformity to the action expert distribution) and high fidelity (reflecting the user's intent). We demonstrate that flow control has many desirable properties: (1) flow control accurately and responsively steers robot actions with user inputs, (2) it is robust to suboptimal user inputs, (3) it enables users to steer VLAs to achieve significantly higher success rates and faster task completion, and (4) fine-tuning a VLA on flow control trajectories improves the autonomous policy. Together, these results provide a simple and intuitive way for users to help steer VLA actions, increasing task performance.

Chinese Translation

我们介绍了视觉-语言-动作（VLA）模型的流控制，这是一种通过通用输入（如键盘）在实时中引导VLA动作的简单而有效的方法。该方法可以开箱即用，无需对VLA进行重新训练或微调。它使得相对粗糙的用户输入能够引导VLA与用户意图对齐。VLA将这些输入转化为从训练过程中学习到的VLA专家动作分布中抽取的动作样本，从而生成的动作具有高质量（符合动作专家分布）和高保真度（反映用户意图）。我们展示了流控制具有许多理想特性：（1）流控制能够准确且快速地根据用户输入引导机器人动作，（2）对次优用户输入具有鲁棒性，（3）使用户能够引导VLA以显著提高成功率和加快任务完成速度，以及（4）在流控制轨迹上微调VLA可以改善自主策略。综合这些结果，为用户提供了一种简单直观的方式来帮助引导VLA动作，从而提高任务性能。

View on arXiv Download PDF AI Translation

cs.RO / 6 / 2606.10208

Exploration of Foundation Model-Based Robots in Patient and Elderly Care

基于基础模型的机器人在患者和老年人护理中的探索

Qiu, Zhiwen, Liu, Wei, Hao, Yuexing

Abstract

Demand for older-adult and patient care is growing rapidly as populations age worldwide. Foundation models are increasingly being integrated into robots and interactive agents, with the promise of more flexible communication and personalized assistance. However, care settings require reliable and workflow-compatible systems with accountable human oversight, and it remains unclear whether current embodied systems can translate technical advances into clinical impact. This Perspective synthesizes foundation model-based care robots across three areas: design features, user experience, and evidence for care-related outcomes. Current systems most commonly use foundation models as conversational and reasoning layers within voice-centered socially assistive embodiments, while multimodal grounding and physical autonomy remain limited. Empirical evaluations report positive usability and engagement benefits, but reliability failures persist across the interaction pipeline such as hallucinations and conversational breakdowns. Evidence for care impact remains concentrated in proximal outcomes such as cognitive engagement and participation, with limited evidence for validated clinical or care-related changes. We argue that future research should transition toward care-specific evaluation standards, accountable autonomy, and integration into care workflows to support more responsive and responsible care technologies.

Chinese Translation

随着全球人口老龄化，老年人和患者护理的需求正在迅速增长。基础模型越来越多地被集成到机器人和互动代理中，承诺提供更灵活的沟通和个性化的帮助。然而，护理环境需要可靠且与工作流程兼容的系统，并且需要有可追溯的人类监督，目前尚不清楚现有的具身系统是否能够将技术进步转化为临床影响。本文综合了基于基础模型的护理机器人在三个领域的应用：设计特征、用户体验和护理相关结果的证据。目前的系统最常将基础模型用作以语音为中心的社会辅助具身体的对话和推理层，而多模态基础和物理自主性仍然有限。实证评估报告了积极的可用性和参与度收益，但在互动流程中仍然存在可靠性故障，例如幻觉和对话中断。护理影响的证据仍然集中在认知参与和参与等近端结果上，针对经过验证的临床或护理相关变化的证据有限。我们认为，未来的研究应转向特定于护理的评估标准、可追溯的自主性以及与护理工作流程的整合，以支持更具响应性和责任感的护理技术。

View on arXiv Download PDF AI Translation

cs.RO / 7 / 2606.10229

What Demonstration Curation Metrics Do to Your Policy

演示策划指标对您的策略的影响

Bedi, Aarav

Abstract

We study whether demonstration-curation metrics that detect defective training episodes also improve the downstream behavior-cloning policy that trains on the curated data. On a contact-rich LIBERO pick-and-place benchmark with a controlled structural defect (early gripper release during the carry phase), we find that the two quantities are sharply decoupled. The metric with the highest defect-detection AUROC (0.804) produces the worst curated policy (13.3% task success), while a metric with a substantially lower AUROC (0.638) produces a policy that nearly matches the oracle trained on ground-truth clean data (90.0% vs. 93.3%). We further show that five of the seven metrics we evaluate exploit episode length as a trivial proxy for the defect label, a confound that inflates reported AUROCs to near-perfect values and disappears once episode length is controlled. Across all conditions, the contaminated baseline succeeds on only 3.3% of rollouts, and the two best curation methods close this to within 3 percentage points of the 93.3% oracle ceiling. Our results argue that curation methods should be evaluated by the policy they produce, not the defects they flag, and that any curation benchmark must control for episode length before reporting detection accuracy. We release the testbed, all metric implementations, and the evaluation pipeline.

Chinese Translation

我们研究了检测缺陷训练情节的演示策划指标是否也能改善在策划数据上训练的下游行为克隆策略。在一个具有受控结构缺陷（在搬运阶段早期释放夹具）的接触丰富的LIBERO拾取与放置基准测试中，我们发现这两个量是明显解耦的。具有最高缺陷检测AUROC（0.804）的指标产生了最差的策划策略（任务成功率为13.3%），而一个AUROC显著较低（0.638）的指标产生的策略几乎与在真实干净数据上训练的oracle相匹配（90.0%对93.3%）。我们进一步显示，我们评估的七个指标中有五个将情节长度作为缺陷标签的简单代理，这种混淆因素使得报告的AUROC膨胀至接近完美的值，并在控制情节长度后消失。在所有条件下，受污染的基线仅在3.3%的回合中成功，而两种最佳策划方法将这一成功率缩小到离93.3%的oracle上限仅相差3个百分点。我们的结果表明，策划方法应根据其产生的策略进行评估，而不是根据其标记的缺陷，并且任何策划基准在报告检测准确性之前必须控制情节长度。我们发布了测试平台、所有指标实现和评估管道。

View on arXiv Download PDF AI Translation

cs.RO / 8 / 2606.10244

YUBI: Yielding Universal Bidigital Interface for Bimanual Dexterous Manipulation at Scale

YUBI：用于大规模双手灵巧操作的通用双指接口

Ohkawa, Takehiko, Arima, Jumpei, Noguchi, Yuki, Tateno, Masatoshi, Sugiura, Makoto, Okubo, Takuya, Ikeuchi, Kengo, Shin, Yuma, Nishizawa, Hiroki, Kanazawa, Naoaki, Wakayama, Yuki, Fukunaga, Daiki, Makihara, Koshi, Motoda, Tomohiro, Erich, Floris, Domae, Yukiyasu, Matsushima, Tatsuya, Okumatsu, Yohishiro, Ota, Kei

Abstract

We introduce Yielding Universal Bidigital Interface (YUBI), a finger-aligned gripper designed to enable intuitive, ergonomic, and scalable data collection for bimanual dexterous manipulation. While handheld data collection systems such as Universal Manipulation Interface (UMI) enable affordable data collection, their bulky pistol-grip designs can pose ergonomic and usability challenges for fine-grained, dexterous manipulation tasks. To address this, YUBI presents a distinct design principle: yielding, finger-driven actuation that directly maps human finger movements to gripper jaw motion. Using the YUBI devices, we set up a data collection system with integrated VR-based 6 DoF tracking of the gripper, ensuring high-fidelity trajectory data acquisition. We curate a UMI-based dataset of unprecedented scale: 8,434 hours across 1.20M episodes and 119 tasks. Experiments show that YUBI offers advantages over the UMI gripper in versatility for complex bimanual tasks, dexterity, and operational efficiency. A single policy trained on the YUBI dataset transfers across multiple bimanual robots (UR, Franka, and ELEY) simply by mounting the gripper on each platform, confirming that the collected data are directly executable as policy supervision. We release the gripper hardware, data-collection software, and dataset as one integrated stack, offering the open community a reproducible path to large-scale data acquisition for advancing robotic foundation models.

Chinese Translation

我们介绍了Yielding Universal Bidigital Interface (YUBI)，这是一种与手指对齐的夹持器，旨在实现直观、符合人体工程学且可扩展的数据收集，以支持双手灵巧操作。尽管诸如Universal Manipulation Interface (UMI)的手持数据收集系统能够实现经济的数据收集，但其笨重的手枪握把设计可能在细致的灵巧操作任务中带来人体工程学和可用性挑战。为了解决这一问题，YUBI提出了一种独特的设计原则：采用屈服的、由手指驱动的驱动方式，直接将人类手指的运动映射到夹持器的夹爪运动上。通过使用YUBI设备，我们建立了一个数据收集系统，集成了基于虚拟现实的6自由度（6 DoF）夹持器跟踪，确保高保真度的轨迹数据采集。我们整理了一个前所未有规模的基于UMI的数据集：包含8,434小时的数据，涵盖1.20M个实验和119个任务。实验表明，YUBI在复杂双手任务的多样性、灵巧性和操作效率方面优于UMI夹持器。一个在YUBI数据集上训练的单一策略可以通过简单地将夹持器安装在每个平台上，跨多个双手机器人（UR、Franka和ELEY）进行迁移，确认所收集的数据可以直接作为策略监督执行。我们将夹持器硬件、数据收集软件和数据集作为一个集成的整体发布，为开放社区提供了一条可重复的大规模数据采集路径，以推动机器人基础模型的发展。

View on arXiv Download PDF AI Translation

cs.RO / 9 / 2606.10267

What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agents

在协调机器人策略中什么是重要的：层次化 VLA 代理的系统研究

Hu, Jiaheng, Shridhar, Mohit, Lu, Caden, Shah, Dhruv, Chiang, Hao-Tien Lewis, Tan, Jie, Xie, Annie

Abstract

Hierarchical vision-language-action (Hi-VLA) systems have emerged as a promising paradigm for complex robot manipulation, by using high-level VLM planners to decompose tasks into language subgoals executed by low-level VLA controllers. Despite recent empirical progress, there is a lack of unified design principles for these systems: existing Hi-VLA systems differ in how they choose and connect planners, controllers, mechanisms to switch between the two, and how observations and memory are represented in the planner. In this paper, we present a systematic study of Hi-VLA design for robot manipulation. We unify representative Hi-VLA agents under an options-style control framework and benchmark core design choices across short-horizon, long-horizon, and reasoning-intensive tasks. Our analysis distills practical principles for building Hi-VLA systems, showing how model choices and interface mechanisms jointly shape performance. Applying these principles yields a substantially stronger system than either flat VLA control or a naively designed hierarchy, across experiments both in simulation and on a real ALOHA robot. Overall, our results provide a foundation for building more capable, robust, and principled hierarchical VLA agents. More information and video at jiahenghu.github.io/hi-vla.

Chinese Translation

层次化视觉-语言-行动（Hi-VLA）系统作为复杂机器人操作的一种有前景的范式，通过使用高层次的 VLM 规划器将任务分解为由低层次 VLA 控制器执行的语言子目标而出现。尽管最近在实证研究上取得了一定进展，但对于这些系统缺乏统一的设计原则：现有的 Hi-VLA 系统在选择和连接规划器、控制器、切换机制以及如何在规划器中表示观察和记忆方面存在差异。本文对机器人操作的 Hi-VLA 设计进行了系统研究。我们在选项风格控制框架下统一了代表性的 Hi-VLA 代理，并在短期、长期和推理密集型任务中对核心设计选择进行了基准测试。我们的分析提炼出构建 Hi-VLA 系统的实用原则，展示了模型选择和接口机制如何共同影响性能。应用这些原则在模拟和真实 ALOHA 机器人实验中，产生了比平面 VLA 控制或简单设计的层次结构更强大的系统。总体而言，我们的结果为构建更具能力、稳健性和原则性的层次化 VLA 代理提供了基础。更多信息和视频请访问 jiahenghu.github.io/hi-vla。

View on arXiv Download PDF AI Translation

cs.RO / 10 / 2606.10273

Locomotion analysis of a quadruped interacting with the lunar granular surface

与月球颗粒表面交互的四足动物运动分析

Vyas, Yash J

Abstract

Deploying legged robots in extra-terrestrial environments includes many challenges due to complex terrain interactions, energy, and thermal constraints. For effective mechanical design of a lunar exploration quadrupedal robot, careful consideration of motor torques, energy expenditure, and cost of transport is required. The lunar surface is composed of granular regolith, which impacts the locomotion of legged robots and their performance. Locomotion algorithms trained with rigid contact assumptions are also ineffective when applied to environments with soft contacts, such as granular surfaces, which can result in instability and poor tracking. In this report, the physical modelling of the granular lunar surface-robot foot contacts is applied to a simulation environment with locomotion trained using Reinforcement Learning. A comparison is conducted between the policy trained on rigid contact and soft contact environments, analysing the gait and locomotion performance metrics. The analysis demonstrates that soft contacts simulating regolith surfaces pose additional challenges for Reinforcement Learning based training, result in a qualitatively different gait, and increase the overall energy expenditure.

Chinese Translation

在外星环境中部署腿式机器人面临许多挑战，主要由于复杂的地形交互、能量和热量限制。为了有效设计月球探索四足机器人，需要仔细考虑电机扭矩、能量消耗和运输成本。月球表面由颗粒状风化层组成，这对腿式机器人的运动和性能产生影响。基于刚性接触假设训练的运动算法在应用于软接触环境（如颗粒表面）时也表现不佳，这可能导致不稳定和跟踪效果差。在本报告中，将颗粒状月球表面与机器人足部接触的物理建模应用于一个使用强化学习训练的仿真环境。对比了在刚性接触和软接触环境中训练的策略，分析了步态和运动性能指标。分析结果表明，模拟风化层表面的软接触为基于强化学习的训练带来了额外挑战，导致步态在质量上有所不同，并增加了整体能量消耗。

View on arXiv Download PDF AI Translation

cs.RO / 11 / 2606.10276

Hierarchical Policies from Verbal and Egocentric Human Signals for Natural Human-Robot Interaction

基于语言和自我中心人类信号的分层策略用于自然人机交互

Lee, Dongjun, Choi, Juheon, Shin, Dong Kyu, Kang, Sinjae, Lee, Kimin

Abstract

For natural human-robot interaction, a robot must understand human intent expressed not only through language but also through nonverbal signals such as gestures and gaze. However, current robot policies rely on language instructions as the sole interface for conveying intent, leaving nonverbal signals unused and placing the full burden of communication. In this work, we present EDITH, a robot framework that captures the human's nonverbal signals through continuous streams of first-person view and gaze from smart glasses, and uses them alongside language instructions as inputs to the robot policy. Our hardware system streams the human's first-person view, gaze, and speech to the robot in real time, transcribing the speech into language instructions. To handle these rich but noisy signals, we design a hierarchical policy in which a high-level policy infers the human's intent and produces a sequence of subtasks, where each subtask is represented as a fine-grained instruction paired with a keyframe that grounds the intent in the scene (e.g., the frame where the human points at the target object). A low-level policy then executes these subtasks. In our experiments on human-robot interactive tasks, EDITH enables the robot to act on the human's nonverbal signals even when intent is expressed only briefly, and significantly reduces user effort to convey intent compared to using language instructions alone. Visit our project page for source code and real-robot demo videos.

Chinese Translation

为了实现自然的人机交互，机器人必须理解人类通过语言以及非语言信号（如手势和目光）表达的意图。然而，目前的机器人策略仅依赖语言指令作为传达意图的唯一接口，未能利用非语言信号，从而使沟通的全部负担落在用户身上。在本研究中，我们提出了EDITH，一个机器人框架，通过智能眼镜捕捉人类的非语言信号，包括第一人称视角和目光，并将其与语言指令一起作为机器人策略的输入。我们的硬件系统实时将人类的第一人称视角、目光和语音流传输给机器人，并将语音转录为语言指令。为了处理这些丰富但嘈杂的信号，我们设计了一种分层策略，其中高层策略推断人类的意图并生成一系列子任务，每个子任务都表示为一个细粒度指令，并配有一个关键帧，将意图与场景中的具体内容相结合（例如，人类指向目标物体的画面）。然后，低层策略执行这些子任务。在我们关于人机交互任务的实验中，EDITH使机器人能够根据人类的非语言信号进行操作，即使意图仅被简短表达，并且与仅使用语言指令相比，显著减少了用户传达意图的努力。请访问我们的项目页面获取源代码和真实机器人演示视频。

View on arXiv Download PDF AI Translation

cs.RO / 12 / 2606.10288

MARCH: Model-Assisted Reinforcement Learning for the Perceptive Control of Humanoids over Sparse Footholds

MARCH：用于在稀疏支撑点上进行人形机器人感知控制的模型辅助强化学习

Crismariu, Codrin, Cosner, Ryan K.

Abstract

Perceptive bipedal locomotion over sparse terrain remains a difficult challenge: model-based methods are precise but brittle to uncertainty, while model-free methods are robust but struggle to discover the precise, constrained motions required for safety-critical locomotion where small errors can cause catastrophic failures. We propose a model-assisted reinforcement learning (RL) framework that combines both perspectives in three steps: (1) generate a safe reference trajectory using simplified models; (2) train a privileged teacher policy guided by a control Lyapunov function (CLF) reward built around the safe reference trajectory; and (3) distill the teacher into a vision-based student policy. We show that this model-assistance procedure produces physically grounded locomotion, improving sample efficiency, reducing the need for a complex learning curriculum, and achieving smoother locomotion behavior alongside stepping stone performance comparable to model-free baselines. We validate our approach in simulation and demonstrate successful deployment on a Unitree G1 humanoid robot navigating sparse footholds with lateral constraints.

Chinese Translation

在稀疏地形上进行感知双足行走仍然是一个困难的挑战：基于模型的方法虽然精确，但对不确定性较为脆弱；而无模型的方法虽然鲁棒，但在发现安全关键行走所需的精确且受限的运动方面存在困难，因为小错误可能导致灾难性失败。我们提出了一种模型辅助强化学习（RL）框架，通过三个步骤结合了这两种视角：（1）使用简化模型生成安全参考轨迹；（2）训练一个由控制李雅普诺夫函数（CLF）奖励引导的特权教师策略，该奖励围绕安全参考轨迹构建；（3）将教师策略提炼为基于视觉的学生策略。我们展示了这一模型辅助过程能够产生物理上合理的行走，改善样本效率，减少对复杂学习课程的需求，并实现更平滑的行走行为，同时在跨步表现上与无模型基线相当。我们在仿真中验证了我们的方法，并成功地在Unitree G1人形机器人上部署，导航于具有侧向约束的稀疏支撑点。

View on arXiv Download PDF AI Translation

cs.RO / 13 / 2606.10289

Improved Representation of Matrix Lie Group Operations through Tensor Notation

通过张量符号改进矩阵李群运算的表示

Taylor, Clark

Abstract

Several recent papers have demonstrated the utility of using Lie groups within estimation problems, yielding improved accuracy and consistency. This paper introduces a new tool for describing operations with matrix Lie groups: tensors and the Einstein summation notation. While tensors and Einstein notation are well-known in other research fields, applying this mathematical notation to represent and compute matrix Lie derivatives is novel. More importantly, this new notation greatly clarifies the derivatives and operations necessary to work with matrix Lie Groups in (gradient-based) estimation frameworks. Therefore, the main contribution of this paper is not a new capability, but a more perspicuous mathematical notation for working with matrix Lie groups.

Chinese Translation

最近几篇论文展示了在估计问题中使用李群的实用性，从而提高了准确性和一致性。本文引入了一种新的工具，用于描述矩阵李群的运算：张量及爱因斯坦求和符号。虽然张量和爱因斯坦符号在其他研究领域中已广为人知，但将这种数学符号应用于表示和计算矩阵李导数是新颖的。更重要的是，这种新符号极大地澄清了在（基于梯度的）估计框架中处理矩阵李群所需的导数和运算。因此，本文的主要贡献不是一种新的能力，而是一种更清晰的数学符号，用于处理矩阵李群。

View on arXiv Download PDF AI Translation

cs.RO / 14 / 2606.10305

SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation

SARM2：自我改进机器人操作的多任务阶段感知奖励建模

Chen, Qianzhong, Zheng, Hau, Yu, Justin, Huang, Suning, Sun, Jiankai, Goldberg, Ken, Wen, Chuan, Abbeel, Pieter, Shentu, Yide, Wu, Philipp, Schwager, Mac

Abstract

Fine-tuning vision-language-action (VLA) policies for long-horizon manipulation still relies heavily on behavior cloning, which requires costly high-quality demonstrations and keeps policies near the demonstration distribution. Reward models can reduce this dependence by reweighting demonstrations and providing dense supervision for on-robot reinforcement learning (RL), but they must be dense, accurate, and general. Existing methods fall short: task-specific stage-aware models are accurate but require per-task annotations, while general vision-language-model (VLM) reward models are broadly applicable but too coarse for fine-grained long-horizon progress. We introduce RM, a multi-task stage-aware reward model that combines an action-primitive-based stage estimator with a multi-gate Mixture-of-Experts (MMoE) value head to produce dense per-step rewards across manipulation tasks. Building on RM, we further propose SPIRAL (Self-Policy Improvement via Reward-Aligned Learning), an on-policy reward-guided framework that improves VLA policies from cheap autonomous rollouts. On a 10-task benchmark, RM reduces value-estimation MSE by 80% over the strongest baselines; when used in SPIRAL, it improves task success from around 50% to near-perfect performance on Folding Shorts (58% to 100%) and Cleaning Whiteboard (50% to 90%), showing that high-quality dense rewards are key to a stable robot data flywheel. Project website: https://qianzhong-chen.github.io/sarm2.github.io/.

Chinese Translation

针对长时间操作的视觉-语言-动作（VLA）策略的微调仍然高度依赖于行为克隆，这需要昂贵的高质量示范，并使策略保持在示范分布附近。奖励模型可以通过重新加权示范并为机器人强化学习（RL）提供密集监督，从而减少这种依赖，但它们必须是密集的、准确的和通用的。现有方法存在不足：特定任务的阶段感知模型虽然准确，但需要每个任务的注释，而通用的视觉-语言模型（VLM）奖励模型适用范围广泛，但对于细粒度的长时间进展来说过于粗糙。我们提出了RM，一种多任务阶段感知奖励模型，它结合了基于动作原语的阶段估计器和多门混合专家（MMoE）价值头，以在操作任务中生成密集的逐步奖励。在RM的基础上，我们进一步提出了SPIRAL（通过奖励对齐学习进行自我策略改进），这是一个基于策略的奖励引导框架，可以通过廉价的自主回放来改进VLA策略。在一个10任务基准测试中，RM将价值估计均方误差（MSE）降低了80%，相较于最强基线；在SPIRAL中使用时，它将任务成功率从约50%提高到几乎完美的表现，在折叠短裤（从58%提高到100%）和清洁白板（从50%提高到90%）上显示出高质量密集奖励是稳定机器人数据飞轮的关键。项目网站：https://qianzhong-chen.github.io/sarm2.github.io/

View on arXiv Download PDF AI Translation

cs.RO / 15 / 2606.10340

OMG: Omni-Modal Motion Generation for Generalist Humanoid Control

OMG：通用人形机器人控制的全模态运动生成

Huang, Siqiao, Lee, Kun-Ying, Qiao, Dongming, He, Guanqi, Wang, Zhenyu, Li, Yitang, Zhu, Shaoting, Zhao, Hang

Abstract

Humanoid whole-body control has made significant progress in recent years, yet existing approaches remain limited to few-skill policies with heavy reward engineering, or motion trackers that are difficult to extend to new input modalities. We argue that the key to general-purpose humanoid control is to build a scalable brain, a module capable of reasoning with diverse conditioning modalities, atop a reactive motion tracking cerebellum, mirroring the hierarchical structure of biological motor systems. Two challenges arise in realizing this vision: acquiring a vast amount of high-quality data to achieve general purpose control, and equipping the generator with the capability to condition on compositional, extensible multi-modal inputs. We present OMG, which addresses these challenges with a meticulous data curation, filtering and labeling pipeline, as well as a diffusion-based motion generation backbone that conditions on language, audio, and human reference motions. Extensive experiments validate OMG as an omni-modal whole-body controller exhibiting state-of-the-art performance, model scaling behavior and efficient adaptation to new distributions and modalities, marking a concrete step toward foundation models for humanoid robots.

Chinese Translation

近年来，人形全身控制取得了显著进展，但现有方法仍然局限于少量技能策略，需大量的奖励工程，或是难以扩展到新输入模态的运动跟踪器。我们认为，通用人形控制的关键在于构建一个可扩展的大脑，一个能够以多样的条件模态进行推理的模块，建立在反应式运动跟踪小脑之上，反映生物运动系统的层次结构。在实现这一愿景时面临两个挑战：获取大量高质量数据以实现通用控制，以及为生成器提供基于组合的、可扩展的多模态输入的条件能力。我们提出了OMG，针对这些挑战，采用了精细的数据策划、过滤和标注流程，以及一个基于扩散的运动生成骨干，能够对语言、音频和人类参考运动进行条件处理。大量实验验证了OMG作为一个全模态全身控制器的有效性，展现了最先进的性能、模型扩展能力以及对新分布和模态的高效适应，标志着向人形机器人基础模型迈出的重要一步。

View on arXiv Download PDF AI Translation

cs.RO / 16 / 2606.10348

Rethinking Embodied Navigation via Relational Inductive Bias

通过关系归纳偏置重新思考具身导航

An, Weitao, Xu, Chenghao, Yang, Xu, Deng, Cheng

Abstract

Object navigation requires an agent to locate a target in an unknown environment through visual observations. Existing methods typically rely on open-vocabulary detectors or vision-language models (VLMs) to answer where to search, but often overlook what not to trust - which semantic cues are unreliable. Open-vocabulary perception is prone to systematic misleading evidence: false positives, outdated static priors, and repeated failed exploration due to lack of embodied verification, which contaminates mapping and decision-making. Such errors are rooted in structured object relations in real-world scenes. To address this, we propose DB-Nav, a framework that reshapes the search space via dual relational biases. It factorizes target-centric relations into an Activation Bias (propagates contextual evidence) and an Inhibition Bias (suppresses unreliable regions via perceptual confusion and action-level falsification). These biases are unified into a Relational Activation-Inhibition Exploration Graph that modulates frontier exploration values using online observations and failed accesses. Experiments on ObjectNav benchmarks show that DB-Nav significantly outperforms existing methods in success rate (SR) and Success weighted by Path Length (SPL), offering a lightweight, interpretable, and robust navigation framework without costly online VLM reasoning.

Chinese Translation

物体导航要求代理在未知环境中通过视觉观察定位目标。现有方法通常依赖开放词汇检测器或视觉-语言模型（VLM）来回答搜索位置，但往往忽视了不可信的内容——哪些语义线索是不可靠的。开放词汇感知容易受到系统性误导证据的影响：假阳性、过时的静态先验以及由于缺乏具身验证而导致的重复失败探索，这些都污染了映射和决策。这些错误根源于现实场景中的结构化物体关系。为了解决这个问题，我们提出了DB-Nav，一个通过双重关系偏置重塑搜索空间的框架。它将以目标为中心的关系分解为激活偏置（传播上下文证据）和抑制偏置（通过感知混淆和行动层面的虚假验证抑制不可靠区域）。这些偏置统一为一个关系激活-抑制探索图，利用在线观察和失败访问来调节前沿探索值。在ObjectNav基准测试中的实验表明，DB-Nav在成功率（SR）和按路径长度加权的成功率（SPL）方面显著优于现有方法，提供了一种轻量级、可解释且稳健的导航框架，无需昂贵的在线VLM推理。

View on arXiv Download PDF AI Translation

cs.RO / 17 / 2606.10363

HiMem-WAM: Hierarchical Memory-Gated World Action Models for Robotic Manipulation

HiMem-WAM：用于机器人操作的分层记忆门控世界动作模型

Sun, Xiaoquan, Zhang, Ruijian, Cao, Chen, Sun, Yihan, Chen, Jiahui, Xu, Zetian, Chen, Bo, Chen, Haijier, Yang, Zhen, Zhu, Jiarun, Hong, Yijun, Xu, JingZhe, Pang, Jingrui, Yuan, Mingqi, Chen, Jiayu

Abstract

World Action Models (WAMs) have emerged as a new powerful paradigm for embodied intelligence, learning action-relevant visual dynamics that significantly enhance generalization and robustness. However, existing WAMs still struggle with task-relevant memory in long-horizon robotic manipulation. To address this, we present HiMem-WAM, a Hierarchical Memory-Gated WAM that integrates motion-centric latent actions, high-level skill latents, and boundary-triggered memory updates. Specifically, we develop a hierarchical latent action framework that jointly learns low-level motion and high-level skill latents, providing structured temporal abstraction. Meanwhile, a boundary-aware memory gate writes compact task states at predicted skill transitions, enabling causal inference without test-time generation of future video or optical flow estimation. Evaluated on LIBERO, LIBERO-PLUS, RMBench and real-world tasks, HiMem-WAM shows that hierarchical latents improve robustness under deployment perturbations, and the memory module substantially benefits memory-dependent long-horizon manipulation.

Chinese Translation

世界动作模型（WAMs）作为一种新兴的强大范式，已在具身智能领域崭露头角，学习与动作相关的视觉动态，显著增强了泛化能力和鲁棒性。然而，现有的WAM在长时间跨度的机器人操作中仍然面临与任务相关的记忆挑战。为了解决这一问题，我们提出了HiMem-WAM，一种分层记忆门控WAM，它集成了以运动为中心的潜在动作、高级技能潜在变量和边界触发的记忆更新。具体而言，我们开发了一个分层潜在动作框架，联合学习低级运动和高级技能潜在变量，提供结构化的时间抽象。同时，边界感知的记忆门在预测的技能过渡时写入紧凑的任务状态，使因果推断得以实现，而无需在测试时生成未来视频或光流估计。在LIBERO、LIBERO-PLUS、RMBench和实际任务上的评估表明，HiMem-WAM的分层潜在变量在部署扰动下提高了鲁棒性，而记忆模块则显著有利于依赖记忆的长时间跨度操作。

View on arXiv Download PDF AI Translation

cs.RO / 18 / 2606.10366

A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation

提高视觉-语言-行动（VLA）评估的仿真与现实相关性的实用方法

Wang, Shuo, Xu, Hanyuan, Hu, Yingdong, Lin, Fanqi, Gao, Yang

Abstract

Simulation has become an essential tool for evaluating and improving vision-language-action (VLA) policies, offering scalable, reproducible, and controllable alternatives to costly real-world robot evaluation. Recent simulation benchmarks have made substantial progress on realism and diversity, yet these platforms have not been widely adopted as reliable proxies for real-world policy evaluation. In this work, we investigate this issue through the lens of sim-and-real correlation. We conduct a systematic study across multiple simulation platforms, VLA policies, tasks, and perturbation factors, measuring whether simulated evaluation preserves real-world conclusions in terms of policy ranking consistency, performance correlation, and perturbation-wise failure patterns. This analysis allows us to characterize the limitations of existing simulators and identify what kinds of simulation signals are more aligned with real-world deployment. We further examine how users should exploit simulation for policy improvement, including when simulator-based finetuning is beneficial and how the amount of post-training data affects sim-and-real alignment. Overall, our work provides a unified framework for measuring, interpreting, and improving the usefulness of simulation for VLA policies, offering guidance both for simulator designers and for practitioners who use simulation as part of the policy development pipeline.

Chinese Translation

仿真已成为评估和改进视觉-语言-行动（VLA）策略的重要工具，提供了可扩展、可重复和可控的替代方案，以取代成本高昂的现实世界机器人评估。最近的仿真基准在现实性和多样性方面取得了显著进展，但这些平台尚未被广泛采用作为可靠的现实世界策略评估代理。在本研究中，我们通过仿真与现实相关性的视角来探讨这一问题。我们在多个仿真平台、VLA 策略、任务和扰动因素上进行系统研究，测量仿真评估在策略排名一致性、性能相关性和扰动失败模式方面是否保留了现实世界的结论。该分析使我们能够表征现有仿真器的局限性，并识别哪些类型的仿真信号与现实世界部署更为一致。我们进一步探讨用户应如何利用仿真来改进策略，包括何时基于仿真的微调是有益的，以及后训练数据的数量如何影响仿真与现实的对齐。总体而言，我们的工作提供了一个统一的框架，用于测量、解释和改善仿真对 VLA 策略的实用性，为仿真器设计者和将仿真作为策略开发流程一部分的实践者提供指导。

View on arXiv Download PDF AI Translation

cs.RO / 19 / 2606.10371

Test-time Adversarial Takeover: A Real-time Hijacking Interface against Robotic Diffusion Policies

测试时对抗接管：针对机器人扩散策略的实时劫持接口

Yin, Zi, Chai, Peilin, Huang, Siyuan, Hu, Zhanhao

Abstract

Diffusion-based action generation has become a foundational component of embodied AI, but its reliance on visual conditioning leaves deployed visuomotor policies vulnerable to adversarial manipulation. Most prior attacks focus on disruption: they perturb the observation stream to reduce task success or induce erratic behavior. We study a stronger threat, Test-time Adversarial Takeover (TAKO), in which an attacker obtains a real-time steering interface over a frozen robot policy and turns it into a remotely piloted instrument. TAKO learns a small vocabulary of reusable universal patches through differentiable diffusion inference; at test time, the attacker switches among these patches in the camera stream to compose attacker-chosen trajectories. This works because the perturbation acts on the visual conditioning pathway, where the induced bias can persist through iterative generative inference. We further show that the natural targeted baseline, target-policy matching, fails because the victim policy cannot reliably supervise itself on out-of-distribution target shifts. Across four tasks (2D manipulation, simulated aerial delivery, simulated ground navigation, and physical-world ground navigation), two visual encoders (ResNet-18 and EfficientNet-B0 + Transformer), and three generative inference families (DDPM, DDIM, and flow matching), human operators achieve 100\% takeover success on attacker-defined objectives in every evaluated setting. The project page is available at https://tako-attack.github.io.

Chinese Translation

基于扩散的动作生成已成为具身人工智能的基础组成部分，但其对视觉条件的依赖使得已部署的视觉运动策略易受对抗性操控。大多数先前的攻击集中于干扰：它们扰动观察流以降低任务成功率或引发不稳定行为。我们研究了一种更强的威胁，测试时对抗接管（Test-time Adversarial Takeover, TAKO），在这种情况下，攻击者获得了对冻结机器人策略的实时控制接口，并将其转变为远程操控的工具。TAKO通过可微分的扩散推理学习一小部分可重用的通用补丁；在测试时，攻击者在摄像头流中切换这些补丁，以组合攻击者选择的轨迹。这之所以有效，是因为扰动作用于视觉条件路径，所引发的偏差可以在迭代生成推理中持续存在。我们进一步表明，自然的目标基线，即目标策略匹配，失败了，因为受害者策略无法可靠地对分布外目标变化进行自我监督。在四个任务（二维操作、模拟空中投递、模拟地面导航和物理世界地面导航）、两个视觉编码器（ResNet-18 和 EfficientNet-B0 + Transformer）以及三种生成推理家族（DDPM、DDIM 和流匹配）中，人类操作者在每个评估设置中都能在攻击者定义的目标上实现100%的接管成功率。项目页面可访问 https://tako-attack.github.io。

View on arXiv Download PDF AI Translation

cs.RO / 20 / 2606.10382

UMI-Bench 1.0: An Open and Reproducible Real-World Benchmark for Tabletop Robotic Manipulation with UMI Data

UMI-Bench 1.0：一个开放且可重复的真实世界桌面机器人操作基准，基于UMI数据

Jin, Shi, Wang, Yuntian, Duan, Yuhui, Wu, Di, Dong, Gaoqi, Liu, Xiaohang, Li, Xiaotong, Jia, Hongfei, Zhang, Zehao, Wang, Tianyu, Jia, Zhongjie, Yao, Yuanqi, Bai, Chenjia, Zhaxizhuoma, Liu, Siao, Cao, Nieqing, Wang, Jin, Yu, Chao, Ding, Yan

Abstract

Real-robot evaluation is essential for understanding whether learned manipulation policies can operate reliably outside curated demonstrations. This need is particularly pressing for Universal Manipulation Interface (UMI)-style policies, whose performance depends on the coupling between wrist-view observations, action representation, data collection, and physical deployment. Existing real-world benchmarks have made important progress, but they are not designed around this UMI data-to-deployment setting. We present UMI-Bench 1.0, a local-first real-robot benchmark for standardized evaluation of UMI-style manipulation policies. To the best of our knowledge, this is the first benchmark dedicated to real-world evaluation of UMI-based manipulation models. UMI-Bench aligns data collection, scene reset, policy execution, result logging, and task-factor analysis within a unified protocol. By making the full evaluation process reproducible and auditable, UMI-Bench provides a practical testbed for measuring how UMI-trained policies generalize to real physical manipulation.

Chinese Translation

真实机器人评估对于理解学习到的操作策略是否能够在策划的演示之外可靠运行至关重要。这一需求对于通用操作接口（Universal Manipulation Interface, UMI）风格的策略尤为迫切，因为其性能依赖于腕部视角观察、动作表示、数据收集和物理部署之间的耦合。现有的真实世界基准虽然取得了重要进展，但并未围绕这一UMI数据到部署的设置进行设计。我们提出了UMI-Bench 1.0，这是一个以本地优先的真实机器人基准，用于对UMI风格操作策略进行标准化评估。根据我们所知，这是第一个专门用于UMI基础操作模型的真实世界评估基准。UMI-Bench在统一协议下对数据收集、场景重置、策略执行、结果记录和任务因素分析进行了对齐。通过使整个评估过程可重复和可审计，UMI-Bench为测量UMI训练策略如何推广到真实物理操作提供了一个实用的测试平台。

View on arXiv Download PDF AI Translation

cs.RO / 21 / 2606.10442

Information-Preserving Continuous Occupancy Mapping with Variance-Weighted Submap Joining

信息保留的连续占用映射与方差加权子图合并

Bai, Zhuhua, Wang, Yingyu, Zhao, Liang, Huang, Shoudong

Abstract

Large-scale SLAM remains challenging due to accumulated trajectory drift and the increasing computational cost of maintaining global consistency. Submap joining alleviates these issues by constructing locally consistent submaps and subsequently fusing them into a global map. However, existing occupancy-based submap joining methods operate on discrete grids, resulting in non-smooth gradients during optimization and neglecting the uncertainty associated with occupancy estimates. We propose the first continuous probabilistic submap joining framework that jointly optimizes submap poses and a global occupancy field in the latent log-odds space. The framework employs an information-preserving sparse Bayesian formulation that compresses raw occupancy observations into sufficient-statistic log-odds tuples while retaining the posterior information of the original observations. This yields closed-form predictive mean and variance estimates for occupancy mapping, which directly enable a submap joining formulation with analytical Jacobians, leading to more accurate submap joining and yielding a closed-form optimal global map upon pose convergence. Experiments on both simulated and large-scale real-world datasets demonstrate that the proposed method achieves higher pose accuracy and improved global consistency than state-of-the-art grid-based submap joining approaches, while producing more compact map representations and better-calibrated uncertainty estimates than existing continuous occupancy mapping methods.

Chinese Translation

大规模SLAM（同步定位与地图构建）因轨迹漂移的累积和维护全局一致性的计算成本增加而面临挑战。子图合并通过构建局部一致的子图并随后将其融合为全局地图来缓解这些问题。然而，现有的基于占用的子图合并方法在离散网格上操作，导致优化过程中出现不平滑的梯度，并忽视了与占用估计相关的不确定性。我们提出了第一个连续概率子图合并框架，该框架在潜在对数几率空间中联合优化子图姿态和全局占用场。该框架采用信息保留的稀疏贝叶斯形式，将原始占用观测压缩为充分统计量对数几率元组，同时保留原始观测的后验信息。这产生了占用映射的封闭形式预测均值和方差估计，直接实现了具有解析雅可比矩阵的子图合并公式，从而实现更准确的子图合并，并在姿态收敛时产生封闭形式的最优全局地图。在模拟和大规模真实数据集上的实验表明，所提出的方法在姿态精度和全局一致性方面优于最先进的基于网格的子图合并方法，同时生成更紧凑的地图表示和更好校准的不确定性估计，超越了现有的连续占用映射方法。

View on arXiv Download PDF AI Translation

cs.RO / 22 / 2606.10449

GuideWalk: Learning Unified Autonomous Navigation and Locomotion for Humanoid Robots across Versatile Terrains

GuideWalk：学习统一的自主导航与人形机器人在多样地形上的运动

Han, Haoxuan, Chen, Chen, Gong, Linao, Yang, Xin, Hu, Hao, Guo, Junhong, He, Zhicheng, Su, Yao, He, Fenghua

Abstract

Humanoid robots have achieved strong locomotion capabilities, but reliable navigation on versatile terrains remains challenging because obstacle avoidance must be coordinated with dynamically feasible motion. In this work, we present GuideWalk, a unified end-to-end framework that integrates traversability-aware navigation guidance with terrain-adaptive locomotion teacher for humanoid navigation. Specifically, we introduce a navigation module that provides explicit velocity guidance, decoupling obstacle avoidance from terrain conditions to enable robust planning across diverse environments. We propose a composite teacher distillation scheme, where goal-directed commands and dynamically consistent actions are aggregated and distilled into a single policy. To further improve robustness, the distilled policy is refined with reinforcement learning and an auxiliary behavior cloning objective, which promotes exploration while preserving desirable teacher behaviors. Experiments demonstrate that GuideWalk achieves stable and effective navigation while maintaining stable humanoid locomotion.

Chinese Translation

人形机器人已经具备了强大的运动能力，但在多样地形上实现可靠的导航仍然具有挑战性，因为障碍物规避必须与动态可行的运动协调。在本研究中，我们提出了GuideWalk，一个统一的端到端框架，集成了考虑可通行性的导航指导与适应地形的运动教师，以实现人形机器人的导航。具体而言，我们引入了一个导航模块，提供明确的速度指导，将障碍物规避与地形条件解耦，从而能够在多样环境中实现稳健的规划。我们提出了一种复合教师蒸馏方案，其中目标导向的指令和动态一致的动作被聚合并蒸馏为单一策略。为了进一步提高稳健性，蒸馏出的策略通过强化学习和辅助行为克隆目标进行优化，这促进了探索，同时保留了期望的教师行为。实验表明，GuideWalk在保持稳定的人形运动的同时，实现了稳定和有效的导航。

View on arXiv Download PDF AI Translation

cs.RO / 23 / 2606.10495

Act on What You See: Unlocking Safe Social Navigation in Vision-Language-Action Models

行动基于所见：解锁视觉-语言-行动模型中的安全社交导航

Wang, Qingzi, Wu, Xiyang, Shi, Guangyao, Chen, Dianwei, Yang, Xianfeng, Manocha, Dinesh

Abstract

Safe social navigation requires robots to distinguish people from ordinary obstacles and to react before danger becomes imminent. We show that pretrained Vision-Language-Action (VLA) models already encode pedestrian-object distinctions and future collision signals in their internal representations, but behavior cloning fails to translate these signals into socially appropriate actions. To address this mismatch, we propose SALSA, a two-stage annotation-free post-training framework: (1) social behavioral alignment bridges intermediate-layer social features to the action head and trains on counterfactual human-object scene pairs to break visual saliency shortcuts; (2) temporal safety alignment provides automatically generated future-risk supervision to enable anticipatory collision avoidance. On SCAND and real-world deployment, SALSA reduces near-collisions by 86.4% and improves social counterfactual accuracy from 53% to 93%, demonstrating that safer social navigation can be achieved by teaching VLA policies to act on representations they already possess. These results show that pretrained VLA policies can be adapted for safer social navigation by better aligning their latent representations with action generation.

Chinese Translation

安全的社交导航要求机器人能够区分人类与普通障碍物，并在危险即将来临之前做出反应。我们展示了预训练的视觉-语言-行动（VLA）模型已经在其内部表示中编码了行人与物体的区分以及未来碰撞信号，但行为克隆无法将这些信号转化为社会适当的行为。为了解决这一不匹配，我们提出了SALSA，一个两阶段的无注释后训练框架：（1）社交行为对齐将中间层的社交特征与动作头连接，并在反事实的人-物场景对上进行训练，以打破视觉显著性捷径；（2）时间安全对齐提供自动生成的未来风险监督，以实现预期的碰撞避免。在SCAND和实际部署中，SALSA将近碰撞减少了86.4%，并将社交反事实准确率从53%提高到93%，证明通过教导VLA策略基于其已有的表示进行行动，可以实现更安全的社交导航。这些结果表明，预训练的VLA策略可以通过更好地对齐其潜在表示与动作生成来适应更安全的社交导航。

View on arXiv Download PDF AI Translation

cs.RO / 24 / 2606.10501

Uncovering Vulnerability of Vision-Language-Action Models under Joint-Level Physical Faults

揭示视觉-语言-动作模型在关节级物理故障下的脆弱性

Jo, Minsoo, Kwon, Taeju, Chun, Junha, Jeong, Youngjoon, Kim, Taesup

Abstract

Deploying Vision-Language-Action (VLA) models in real robotic systems requires robustness not only to semantic and perceptual variations, but also to embodiment-side faults that change how actions are physically realized. Real robots can experience joint-level changes caused by actuator degradation, hardware faults, safety limits, collision damage, or wear-induced friction. These faults are critical because they alter the action-to-motion interface of a policy, disrupting the learned closed-loop relationship between commanded actions, realized motion, and subsequent observations. In this work, we study realistic joint-level physical faults and show that VLA models are vulnerable when predicted actions are executed through a perturbed robot body. Our analysis reveals joint-dependent effects, with heterogeneous degradation in task success across affected joints. We also show that performance drops cannot be attributed solely to physical infeasibility, since feasible faults such as increased joint friction can still substantially reduce success rates and induce closed-loop execution mismatch. Motivated by these findings, we propose Joint-level Physical-fault Aware Residual Calibrator (J-PARC), a lightweight residual calibration framework built on top of a frozen VLA policy. J-PARC infers a latent joint-fault regime from recent joint dynamics and conditions a shared residual calibrator on this regime, enabling adaptive action correction across faulty joints. Experiments show that J-PARC improves robustness under joint-level faults while preserving fault-free environment performance.

Chinese Translation

在真实机器人系统中部署视觉-语言-动作（VLA）模型不仅需要对语义和感知变化具有鲁棒性，还需要对改变动作物理实现方式的体现侧故障具有鲁棒性。真实机器人可能会经历由执行器退化、硬件故障、安全限制、碰撞损伤或磨损引起的关节级变化。这些故障至关重要，因为它们改变了策略的动作与运动接口，破坏了命令动作、实现运动和后续观察之间的学习闭环关系。在本研究中，我们研究了现实中的关节级物理故障，并表明当预测的动作通过受扰动的机器人身体执行时，VLA模型是脆弱的。我们的分析揭示了关节依赖效应，受影响关节的任务成功率存在异质性退化。我们还表明，性能下降不能仅归因于物理不可行性，因为可行的故障（如关节摩擦增加）仍然可以显著降低成功率并导致闭环执行不匹配。基于这些发现，我们提出了关节级物理故障感知残差校准器（J-PARC），这是一个轻量级的残差校准框架，建立在冻结的VLA策略之上。J-PARC从最近的关节动态中推断潜在的关节故障状态，并在该状态上条件化共享的残差校准器，从而实现对故障关节的自适应动作修正。实验表明，J-PARC在关节级故障下提高了鲁棒性，同时保持了无故障环境下的性能。

View on arXiv Download PDF AI Translation

cs.RO / 25 / 2606.10568

VeriSpace: Spatially Grounded Action Verification for Vision-Language-Action Models

VeriSpace：面向视觉-语言-动作模型的空间基础动作验证

Zhao, Guiyu, Guo, Longteng, Zhu, Junyou, Fu, Jun, Mei, Yanghong, Cao, Bin, Jiang, Jie, He, Xingjian, Liu, Jing

Abstract

Vision-language-action (VLA) models have shown strong promise for robotic manipulation, but their reliability at test time remains limited by one-shot action prediction, where even small action errors can cause grasp failure, collision, or incorrect task progression. A natural alternative is to equip VLA systems with test-time verification, allowing multiple candidate actions to be proposed and evaluated before execution. However, reliable action verification is challenging because it requires not only distinguishing subtle geometric differences between candidate actions, but also assessing whether an action makes meaningful progress toward the task goal. We present VeriSpace, a 3D-aware action verifier for test-time action selection in VLA systems. VeriSpace evaluates candidate actions through two key components: Dual-Path 3D-Injected Scene Encoding, which constructs a scene representation that jointly preserves visual semantics and explicit 3D geometry, and Spatially-Grounded Action Reasoning, which evaluates each action by reasoning over task-relevant spatial relations, geometric validity, and expected goal progress. Together, these components enable more reliable discrimination between subtle yet outcome-critical action candidates while remaining fully compatible with existing VLA policies. Experiments on public benchmarks and real-world robotic manipulation tasks show that VeriSpace consistently improves decision reliability over both underlying VLA policies and prior verification-based methods, yielding substantial gains in both in-distribution and out-of-distribution settings.

Chinese Translation

视觉-语言-动作（VLA）模型在机器人操作中展现出了良好的前景，但其在测试时的可靠性仍受限于一次性动作预测，即使是微小的动作错误也可能导致抓取失败、碰撞或任务进展不正确。一个自然的替代方案是为VLA系统配备测试时验证功能，允许在执行前提出并评估多个候选动作。然而，可靠的动作验证具有挑战性，因为它不仅需要区分候选动作之间的微妙几何差异，还需要评估某个动作是否朝着任务目标取得了有意义的进展。我们提出了VeriSpace，一种用于VLA系统测试时动作选择的3D感知动作验证器。VeriSpace通过两个关键组件评估候选动作：双路径3D注入场景编码（Dual-Path 3D-Injected Scene Encoding），该组件构建了一个同时保留视觉语义和明确3D几何的场景表示；空间基础动作推理（Spatially-Grounded Action Reasoning），该组件通过推理与任务相关的空间关系、几何有效性和预期目标进展来评估每个动作。这些组件共同使得在微妙但结果关键的动作候选之间进行更可靠的区分成为可能，同时与现有的VLA策略完全兼容。在公共基准和真实世界的机器人操作任务上的实验表明，VeriSpace在决策可靠性方面始终优于基础VLA策略和先前的基于验证的方法，在分布内和分布外的设置中均取得了显著的提升。

View on arXiv Download PDF AI Translation

cs.RO / 26 / 2606.10577

AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness

AgenticNav：作为工具调用接口的零-shot视觉与语言导航

Li, Yijian, Li, Changze, Shi, Hantian, Luo, Jiaying, Cai, Jiyuan, Yang, Ming, Qin, Tong

Abstract

Zero-shot vision-and-language navigation in continuous environments (VLN-CE) has recently become feasible with large vision-language models (VLMs). However, existing methods typically rely on learned waypoint predictors to propose navigable actions. This severely limits the model's action space and fails to leverage depth inputs effectively. Moreover, memory is commonly handled by accumulating long textual or visual histories with substantial irrelevant context, or by retrieving cross-episode experiences, which weakens the zero-shot setting. In this paper, we rethink zero-shot VLN-CE as an agentic interface between the VLM and the environment, and present AgenticNav, a lightweight navigation harness that exposes action, depth, and memory as callable tools. Instead of choosing from predicted waypoints, the action tool allows the VLM to directly select a target pixel in RGB observations, converting it into executable motion. Depth is exposed through an on-demand pixel-depth tool, enabling the VLM to request precise metric distances only where they matter. For memory, AgenticNav provides a compact map image summarizing the historical trajectory, paired with a recall tool that allows the VLM to selectively revisit past visual observations without overwhelming the prompt context. On the R2R-CE benchmark, AgenticNav establishes new state-of-the-art (SOTA) performance among zero-shot methods given the same VLM backbone. Real-world validation further highlights its zero-shot generalization compared to prior methods. Ablations show that our action tool design outperforms traditional waypoint predictors, and that depth tool and agentic memory further contribute to navigation performance.

Chinese Translation

在连续环境中的零-shot视觉与语言导航（VLN-CE）最近随着大型视觉-语言模型（VLMs）的出现变得可行。然而，现有的方法通常依赖于学习的航点预测器来提出可导航的动作。这严重限制了模型的动作空间，并未有效利用深度输入。此外，记忆通常通过积累长文本或视觉历史以及大量无关上下文来处理，或者通过检索跨情节的经验，这削弱了零-shot设置。在本文中，我们重新思考零-shot VLN-CE，将其视为VLM与环境之间的代理接口，并提出AgenticNav，这是一种轻量级导航工具，能够将动作、深度和记忆作为可调用工具暴露出来。行动工具允许VLM直接在RGB观测中选择目标像素，而不是从预测的航点中选择，从而将其转换为可执行的运动。深度通过按需像素深度工具暴露，使VLM能够仅在重要的地方请求精确的度量距离。对于记忆，AgenticNav提供了一幅紧凑的地图图像，概括了历史轨迹，并配备了一个回忆工具，允许VLM选择性地重新访问过去的视觉观测，而不会淹没提示上下文。在R2R-CE基准测试中，AgenticNav在相同VLM骨干下建立了零-shot方法中的新最先进（SOTA）性能。现实世界的验证进一步突显了其与先前方法相比的零-shot泛化能力。消融实验表明，我们的行动工具设计优于传统的航点预测器，而深度工具和代理记忆进一步促进了导航性能。

View on arXiv Download PDF AI Translation

cs.RO / 27 / 2606.10579

LieIPM: Lie Group Interior Point Method for Direct Trajectory Optimization of Rigid Bodies

LieIPM：用于刚体直接轨迹优化的李群内点法

Teng, Sangli, Zhang, Ruiqi, Lin, Tzu-Yuan, Clark, William A, Mueller, Mark, Vasudevan, Ram, Ghaffari, Maani, Sreenath, Koushil

Abstract

Designing dynamically feasible trajectories for rigid bodies is a fundamental problem in robotics. While direct methods are widely used, the existing constrained optimizers typically operate in Euclidean space and ignore the manifold structure of rigid body motions. This mismatch may introduce singularities or lead to poorly conditioned optimization problems. To bridge this gap, we develop a structure-aware framework for constrained trajectory optimization directly on matrix Lie groups. Our approach is based on the second-order rigid body models utilizing Lie group structures, which enables efficient Newton-type updates while preserving the underlying geometry. Building on this model, we propose a line-search Lie Group Interior Point Method (LieIPM) to handle constraints on the manifolds. We instantiate the framework for rigid body motion planning using Lie group variational integrators and derive closed-form intrinsic derivatives that exploit group symmetries. The LieIPM preserves the topology of rotation motions by construction and avoids singularities. Numerical results demonstrate superior robustness and faster convergence compared to general-purpose solvers and structure-exploiting optimal control methods.

Chinese Translation

为刚体设计动态可行的轨迹是机器人学中的一个基本问题。尽管直接方法被广泛使用，但现有的约束优化器通常在欧几里得空间中操作，忽略了刚体运动的流形结构。这种不匹配可能引入奇异性或导致优化问题的条件不良。为了解决这一问题，我们开发了一种结构感知的框架，直接在矩阵李群上进行约束轨迹优化。我们的方法基于利用李群结构的二阶刚体模型，使得在保持底层几何结构的同时能够高效地进行牛顿型更新。在此模型基础上，我们提出了一种线搜索李群内点法（LieIPM），以处理流形上的约束。我们利用李群变分积分器实例化该框架用于刚体运动规划，并推导出利用群对称性的封闭形式内在导数。LieIPM通过构造保持了旋转运动的拓扑结构，避免了奇异性。数值结果表明，与通用求解器和利用结构的最优控制方法相比，LieIPM展现出更强的鲁棒性和更快的收敛速度。

View on arXiv Download PDF AI Translation

cs.RO / 28 / 2606.10614

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations

灵巧点策略：从人类示范中学习基于点的灵巧手策略

Kim, Beomjun, Park, Seong Hyeon, Sim, Seunghoon, Moon, Seungjun, Lee, Sanghyeok, Shin, Jinwoo

Abstract

Robotic foundation models pre-trained on human demonstration videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real robots. A common remedy is to fine-tune these models on robot-specific demonstrations. However, robot data collection can be prohibitively expensive and time-consuming, which is particularly acute in dexterous manipulation, e.g., teleoperating a multi-fingered hand for even a single atomic task can take days. To address this, we introduce Dexterous Point Policy, a framework that learns dexterous manipulation policies directly from human videos and requires no robot demonstrations. Our core insight is that a unified 3D keypoint representation can bridge human and robot embodiments when used for both observations and actions. Specifically, we extract 3D keypoints of task-relevant objects and human hands from raw videos, and train an autoregressive transformer over these keypoints. We observe that at the keypoint level, specifically the wrist and fingertips, human and robot behaviors closely align, enabling direct policy transfer. On a suite of real-robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%. Furthermore, our method generalizes strongly to unseen scenarios, including multi-object environments and novel object categories.

Chinese Translation

基于人类示范视频预训练的机器人基础模型显示出良好的前景，但在将这些策略部署到真实机器人上时仍存在显著的体现差距。常见的解决方案是对这些模型进行机器人特定示范的微调。然而，收集机器人数据可能会非常昂贵且耗时，尤其是在灵巧操作中，例如，仅仅进行一次原子任务的多指手的远程操作可能需要数天时间。为了解决这个问题，我们提出了灵巧点策略（Dexterous Point Policy），这是一个直接从人类视频中学习灵巧操作策略的框架，并且不需要机器人示范。我们的核心见解是，统一的3D关键点表示可以在观察和动作中架起人类与机器人体现之间的桥梁。具体而言，我们从原始视频中提取与任务相关的物体和人手的3D关键点，并在这些关键点上训练自回归变换器。我们观察到，在关键点层面，尤其是手腕和指尖，人类与机器人行为高度一致，从而实现直接的策略转移。在涵盖抓取与放置及工具使用的一系列真实机器人任务中，灵巧点策略的成功率达到了75.0%，而最先进的VLA基线仅为1.0%。此外，我们的方法在未见场景中也表现出强大的泛化能力，包括多物体环境和新物体类别。

View on arXiv Download PDF AI Translation

cs.RO / 29 / 2606.10639

Planar-Sector LOS Guidance for Interception of Agile Targets with Lifting-Wing Quadcopters

用于灵活目标拦截的平面扇形视线引导方法：提升翼四旋翼无人机的应用

Liu, Linkai, Yang, Kun, Zou, Han, Min, Chen, Lv, Shuli, Wang, Shuai, Quan, Quan

Abstract

Autonomous visual interception of agile aerial targets is challenging due to unpredictable target motion, limited sensing, and the strong coupling between target visibility and interceptor maneuverability. Most existing strapdown-camera interception methods preserve visibility using conic line-of-sight (LOS) constraints that keep the target near the image center. While safe, such symmetric constraints unnecessarily restrict maneuverability and can significantly reduce the usable thrust for pursuit. Motivated by the observation that aggressive FPV pilots do not maintain equal visibility margins in all image directions, this paper proposes a Planar-Sector Line-of-Sight (PS-LOS) guidance framework for autonomous interception using a lifting-wing quadcopter equipped with only a strapdown monocular camera. PS-LOS tightly constrains lateral image error while relaxing longitudinal image error within a safe field-of-view margin, preserving visibility while releasing maneuverability for acceleration-intensive pursuit. Under the lifting-wing quadcopter model, PS-LOS provides nearly 50% more available thrust near the LOS direction than conventional conic LOS constraints. To realize LOS-only interception without direct depth measurements, a delay-compensated state-estimation framework and a nonlinear guidance-and-control architecture are developed for lifting-wing quadcopters. Extensive outdoor flight experiments demonstrate autonomous interception of agile targets exhibiting large-amplitude, high-frequency, and unpredictable motion under real wind disturbances. The proposed system achieves successful interceptions at ranges up to 138 m while maintaining continuous visual tracking throughout the engagement. The results validate PS-LOS as a visibility-preserving, maneuverability-aware guidance framework for long-range visual interception of agile aerial targets.

Chinese Translation

自主视觉拦截灵活的空中目标面临诸多挑战，主要由于目标运动不可预测、传感能力有限以及目标可见性与拦截器机动性之间的强耦合关系。现有的大多数固定摄像头拦截方法通过锥形视线（LOS）约束来保持可见性，确保目标接近图像中心。虽然这种方法安全，但对称约束不必要地限制了机动性，并可能显著减少追击时可用的推力。受观察到的激进第一人称视角（FPV）飞行员在所有图像方向上并不保持相等可见性边际的启发，本文提出了一种平面扇形视线（PS-LOS）引导框架，用于仅配备固定单目摄像头的提升翼四旋翼无人机的自主拦截。PS-LOS在安全视野边际内严格约束横向图像误差，同时放宽纵向图像误差，从而在保持可见性的同时释放机动性，以适应加速密集的追击。在提升翼四旋翼模型下，PS-LOS在视线方向上提供的可用推力比传统锥形视线约束多出近50%。为了实现仅基于视线的拦截而无需直接深度测量，本文为提升翼四旋翼无人机开发了延迟补偿状态估计框架和非线性引导与控制架构。广泛的户外飞行实验表明，该系统能够在真实风扰动下自主拦截表现出大幅度、高频率和不可预测运动的灵活目标。该系统在距离高达138米的范围内成功实现了拦截，并在整个交战过程中保持了连续的视觉跟踪。结果验证了PS-LOS作为一种保持可见性、关注机动性的引导框架，适用于灵活空中目标的远程视觉拦截。

View on arXiv Download PDF AI Translation

cs.RO / 30 / 2606.10683

UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data

UniDexTok：来自真实数据的统一灵巧手标记器

Fang, Dong, Wu, Youjun, Zhong, Yuanxin, Zhang, Rui, Wang, Yunlong, Jia, Xiaosong, Jiang, Yu-Gang

Abstract

Dexterous hands are essential for fine-grained manipulation, but their hardware designs vary substantially across embodiments. Differences in kinematics, joint definitions, and degrees of freedom make it difficult to define a shared state representation compared with parallel grippers. As a result, dexterous-hand data remains fragmented and difficult to use for joint training. In this work, we propose the Unified Dexterous Hand Model (UDHM), which maps human and robot hand states into a shared 22-DoF semantic interface. Based on UDHM, we introduce UniDexTok, a retargeting-free state tokenizer that learns embodiment-conditioned discrete tokens from standardized real joint states. UniDexTok provides a unified representation for heterogeneous dexterous hands without relying on retargeting or simulation data. Compared with the recent baseline UniHM, UniDexTok reduces MPJAE from 15.63 degrees to 0.16 degrees and MPJPE from 18.51 mm to 0.18 mm, corresponding to error reductions of 98.98% and 99.03%, respectively. These results improve reconstruction from centimeter-scale to sub-millimeter accuracy. Experiments further show that data from other embodiments improves target-embodiment reconstruction accuracy, demonstrating the benefit of cross-embodiment tokenization. UniDexTok also shows strong zero-shot and few-shot reconstruction ability when new dexterous hands are introduced.

Chinese Translation

灵巧手对于精细操作至关重要，但其硬件设计在不同实现之间存在显著差异。运动学、关节定义和自由度的差异使得与并行夹具相比，定义共享状态表示变得困难。因此，灵巧手数据仍然是碎片化的，难以用于联合训练。在本研究中，我们提出了统一灵巧手模型（Unified Dexterous Hand Model, UDHM），该模型将人类和机器人手的状态映射到一个共享的22自由度语义接口。基于UDHM，我们引入了UniDexTok，这是一种无重定向的状态标记器，能够从标准化的真实关节状态中学习具体现实条件的离散标记。UniDexTok为异构灵巧手提供了一种统一的表示，无需依赖重定向或模拟数据。与最近的基线模型UniHM相比，UniDexTok将MPJAE从15.63度降低到0.16度，MPJPE从18.51毫米降低到0.18毫米，分别对应98.98%和99.03%的误差减少。这些结果将重建精度从厘米级提高到亚毫米级。实验进一步表明，来自其他实现的数据提高了目标实现的重建精度，展示了跨实现标记化的优势。当引入新的灵巧手时，UniDexTok还表现出强大的零样本和少样本重建能力。

View on arXiv Download PDF AI Translation

cs.RO / 31 / 2606.10688

Self-Supervised Relevance Modelling in Autonomous Driving via Counterfactual Analysis

通过反事实分析在自动驾驶中进行自监督相关性建模

Lusvarghi, Luca, Gozalvez, Javier, Hidalgo, Pablo Urbano

Abstract

Autonomous driving relies on computationally intensive perception pipelines to continuously detect and track objects in the surrounding environment. While some objects are key to plan safe and effective maneuvers, others may not be relevant and have no impact on the autonomous vehicle's driving decisions. Focusing on relevant objects allows a more efficient usage of available computational resources, reduces processing latencies, and limits the downstream propagation of perception noise. In this work, we propose a novel self-supervised approach based on counterfactual analysis to develop a relevance model - an AI-based tool that quantifies the relevance of objects for an autonomous vehicle. To demonstrate the potential of the proposed approach, we train a relevance model on a synthetic causal dataset generated in a selected urban scenario. Results show that the relevance model is able to accurately estimate the objects' relevance with millisecond-level latency, enabling real-time relevance estimation also in high-density scenarios. We also show that the relevance model can be used to build relevance heatmaps that offer valuable insights into the autonomous vehicle's driving policy and can be used to proactively inform perception and planning tasks. We openly release both the relevance model and the causal dataset.

Chinese Translation

自动驾驶依赖于计算密集型的感知管道，以持续检测和跟踪周围环境中的物体。虽然某些物体对于规划安全有效的驾驶操作至关重要，但其他物体可能并不相关，对自动驾驶车辆的驾驶决策没有影响。关注相关物体可以更有效地利用可用的计算资源，减少处理延迟，并限制感知噪声的下游传播。在本研究中，我们提出了一种基于反事实分析的新型自监督方法，以开发相关性模型——一种量化物体对自动驾驶车辆相关性的人工智能工具。为了展示所提方法的潜力，我们在一个选定城市场景中生成的合成因果数据集上训练了相关性模型。结果表明，该相关性模型能够以毫秒级延迟准确估计物体的相关性，从而在高密度场景中实现实时相关性估计。我们还展示了相关性模型可以用于构建相关性热图，提供对自动驾驶车辆驾驶策略的宝贵见解，并可用于主动告知感知和规划任务。我们将相关性模型和因果数据集公开发布。

View on arXiv Download PDF AI Translation

cs.RO / 32 / 2606.10732

Vehicle Prediction Model for Enhanced MPC Path Tracking in Formula Student Driverless

用于增强Formula Student Driverless中MPC路径跟踪的车辆预测模型

Baader, Sebastian, Bergerhoff, Tamara, Meißner, Pascal, Deinzer, Frank

Abstract

Autonomous race cars, such as in Formula Student Driverless, operate close to their physical handling limits. The resulting highly nonlinear vehicle behavior increases the path tracking complexity, especially on narrow tracks. Model Predictive Control (MPC) is commonly used to address this issue, a method whose performance is closely tied to the accuracy of the underlying prediction model. This paper presents a novel, real-time capable prediction model for autonomous race cars that adjusts to changing conditions by combining information from past runs and the current driving situation. Our model is divided into three consecutive submodels: a nominal Kinematic Bicycle Model, an offline Bayesian Linear Regression (BLR) model, and an online Sparse Gaussian Process Regression (SGPR) model. The proposed approach enables efficient integration of all available data without significantly increasing computational cost, ensuring high prediction accuracy and a quantitative uncertainty assessment right from the start of the run. Compared to existing approaches, an improvement in prediction accuracy of up to 57% was achieved. Further, we successfully demonstrated the practical applicability of the model within an MPC-based path tracking controller on a real Formula Student race car.

Chinese Translation

自主赛车，如Formula Student Driverless，操作接近其物理操控极限。由此产生的高度非线性车辆行为增加了路径跟踪的复杂性，尤其是在狭窄的赛道上。模型预测控制（Model Predictive Control, MPC）通常用于解决这一问题，其性能与基础预测模型的准确性密切相关。本文提出了一种新颖的、具备实时能力的预测模型，能够通过结合过去运行的信息和当前驾驶情况来适应变化的条件。我们的模型分为三个连续的子模型：一个名义运动学自行车模型（Kinematic Bicycle Model）、一个离线贝叶斯线性回归模型（Bayesian Linear Regression, BLR）和一个在线稀疏高斯过程回归模型（Sparse Gaussian Process Regression, SGPR）。所提出的方法能够高效整合所有可用数据，而不会显著增加计算成本，确保高预测准确性和从运行开始就进行定量的不确定性评估。与现有方法相比，预测准确性提高了多达57%。此外，我们成功地在一辆真实的Formula Student赛车上展示了该模型在基于MPC的路径跟踪控制器中的实际应用。

View on arXiv Download PDF AI Translation

cs.RO / 33 / 2606.10733

Pushing the Performance Limits in Autonomous Racing: Continuous Stability-Aware Adaptive Velocity Planning in Formula Student Driverless

推动自主赛车的性能极限：在Formula Student Driverless中进行连续稳定性感知的自适应速度规划

Bergerhoff, Tamara, Baader, Sebastian, Meißner, Pascal, Deinzer, Frank

Abstract

In autonomous racing, especially in competitions such as Formula Student Driverless, precise planning of the target velocity of a race car is crucial for competitive lap times and stable driving behavior. Especially at high speeds, Velocity Planning (VP) is a significant challenge as it has to be performed in real time, taking into account track layouts, environmental influences, mechanical tolerances, and the resulting control inaccuracies. In this paper, we present a novel approach to VP that dynamically adapts to such changing conditions. Instead of estimating the physical Tire-Road Friction Coefficient (TRFC), a continuous scaling factor is inferred indirectly from vehicle stability. This factor not only reflects the effective tire-road interaction but also captures effects of control inaccuracies. From this, we generate a continuous friction map, which serves as a robust, adaptive basis for computing the optimal target speed, accounting for both vehicle and environmental limits. Our proposed approach was evaluated on a real Formula Student race car, showing a lap time improvement of 35 % over ten laps and an average increase of 8 % compared to a non-adaptive approach.

Chinese Translation

在自主赛车中，特别是在Formula Student Driverless等比赛中，精确规划赛车的目标速度对于竞争性的圈速和稳定的驾驶行为至关重要。尤其是在高速行驶时，速度规划（Velocity Planning, VP）是一项重大挑战，因为它必须实时进行，同时考虑赛道布局、环境影响、机械公差以及由此产生的控制不准确性。本文提出了一种新颖的速度规划方法，该方法能够动态适应这些变化的条件。我们并不直接估计物理轮胎-路面摩擦系数（Tire-Road Friction Coefficient, TRFC），而是通过车辆稳定性间接推导出一个连续的缩放因子。该因子不仅反映了有效的轮胎-路面相互作用，还捕捉了控制不准确性的影响。基于此，我们生成了一个连续的摩擦图，为计算最佳目标速度提供了一个稳健的自适应基础，同时考虑了车辆和环境的限制。我们在一辆真实的Formula Student赛车上评估了所提出的方法，结果显示在十圈比赛中圈速提高了35%，与非自适应方法相比平均提高了8%。

View on arXiv Download PDF AI Translation

cs.RO / 34 / 2606.10743

Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Open-World Contact Localization

基于开放世界接触定位的手部中心人机轨迹转移方法

Shi, Yitian, Wen, Di, Han, Zhengqi, Guo, Zicheng, Hu, Yu, Welte, Edgar, Peng, Kunyu, Stiefelhagen, Rainer, Rayyes, Rania

Abstract

Learning from human video demonstrations remains challenging due to noisy hand-object interactions, unseen objects with partial observation, and cross-embodiment discrepancy. To address these challenges, we present \textit{HOWTransfer} (\emph{H}and-\emph{O}bject \emph{O}pen-\emph{W}orld Transfer), a hand-centric framework that distills human demonstrations into contact-aware, taxonomy-informed, and diverse robotic trajectories. Instead of relying on object-specific descriptions, vision-language queries, or explicit object-state tracking, \emph{HOWTransfer} recovers temporally consistent 3D hand motion and localizes temporal contact intervals by reasoning over observed hand-object interaction cues. The localized contact onsets are then used to retarget human grasp intent into multi-modal parallel-jaw grasp hypotheses, which are propagated along the recovered wrist trajectory to generate robot-executable motions. Finally, a trajectory editing stage refines contact alignment and produces diverse executable variants from a single demonstration. Experiments across diverse manipulation tasks show that \emph{HOWTransfer} enables accurate contact localization and high-quality robot motion retargeting with $86\%$ success, which is preferred over teleoperated trajectories in a blinded preference study.

Chinese Translation

从人类视频演示中学习仍然面临挑战，主要由于手部与物体的噪声交互、部分观察下的未见物体以及跨体现差异。为了解决这些问题，我们提出了 extit{HOWTransfer}（ extit{H}and- extit{O}bject extit{O}pen- extit{W}orld Transfer），这是一个以手部为中心的框架，将人类演示提炼为关注接触、基于分类的多样化机器人轨迹。与依赖于特定物体描述、视觉-语言查询或显式物体状态跟踪的方法不同， extit{HOWTransfer}通过推理观察到的手-物体交互线索，恢复时间一致的三维手部运动并定位时间接触区间。然后，定位的接触起始点用于将人类抓取意图重新定向为多模态平行夹持假设，这些假设沿着恢复的手腕轨迹传播，以生成机器人可执行的运动。最后，轨迹编辑阶段精细化接触对齐，并从单一演示中生成多样的可执行变体。跨多种操作任务的实验表明， extit{HOWTransfer}能够实现准确的接触定位和高质量的机器人运动重新定向，成功率达到86%，在盲偏好研究中优于遥控轨迹。

View on arXiv Download PDF AI Translation

cs.RO / 35 / 2606.10746

ros2probe: Non-intrusive, Kernel-selective Observability for Robot Operating System 2 Middleware

ros2probe：针对机器人操作系统2中间件的非侵入式、内核选择性可观察性

Yu, Jisang, Lee, Sanghoon, Choi, Yeonwoo, Park, Kyung-Joon

Abstract

Robot Operating System 2 (ROS 2), the de facto standard middleware framework for robots, runs each robot as a graph of nodes communicating over the Data Distribution Service (DDS), a publish/subscribe substrate. Observing this inter-node communication in real time is essential to robot development, yet it has a price. A tool can receive data only by joining the DDS domain as a subscriber that discovery has matched to the publisher, so observing folds the tool into the system it measures and perturbs it. We define this protocol-inherent perturbation as the observer's probe effect. It inflates the discovery plane, adds deserialization cost on the observer, makes the loss it reports diverge from what the subscriber actually received, and near saturation displaces the subscriber's messages. The only escape, capturing all wire traffic passively, discards ROS 2 message semantics and scales with total traffic, not what is observed. We present ros2probe, a non-intrusive observation framework that removes the probe effect. It reconstructs the full ROS 2 communication state from the domain's discovery packets at no bandwidth cost, then drives an in-kernel filter restricted to the topics the user asks for, lifting only those packets at minimal cost and observing what the real subscriber receives. Its interfaces and recordings match the standard ROS 2 tools. Across three hardware platforms (laptop, Jetson, and Raspberry Pi), two DDS implementations, and seven robot-operation workloads, ros2probe holds the discovery graph within 0.5% of an unobserved system, whereas domain-joining tools inflate discovery up to 2.6$\times$ and drop 38.5% of the subscriber's messages at saturation while ros2probe drops none. It reports loss with a recall of 1.0, cuts observer CPU and memory by up to 7$\times$ and 28$\times$, and stays practical on the embedded robots where existing tools overload the system.

Chinese Translation

机器人操作系统2（ROS 2）是机器人领域的事实标准中间件框架，它将每个机器人作为一个节点图运行，节点之间通过数据分发服务（DDS）进行通信，这是一个发布/订阅的基础设施。实时观察这种节点间的通信对于机器人开发至关重要，但这也带来了代价。工具只能通过作为订阅者加入DDS域来接收数据，而这个订阅者必须与发布者匹配，因此观察工具会融入其测量的系统并对其造成干扰。我们将这种协议固有的干扰定义为观察者的探测效应。它膨胀了发现平面，增加了观察者的反序列化成本，使其报告的丢包情况与订阅者实际接收到的内容偏离，并且在接近饱和时会替代订阅者的消息。唯一的解决方案是被动捕获所有网络流量，但这会丢弃ROS 2消息的语义，并且与总流量成比例，而不是与所观察的内容成比例。我们提出了ros2probe，一个非侵入式观察框架，消除了探测效应。它从域的发现数据包中重建完整的ROS 2通信状态，而没有带宽成本，然后驱动一个限制在用户请求主题上的内核过滤器，仅以最小成本提取那些数据包，并观察真实订阅者接收到的内容。它的接口和记录与标准的ROS 2工具相匹配。在三种硬件平台（笔记本电脑、Jetson和树莓派）、两个DDS实现和七种机器人操作工作负载中，ros2probe将发现图保持在未观察系统的0.5%以内，而加入域的工具则将发现膨胀至2.6倍，并在饱和时丢失38.5%的订阅者消息，而ros2probe没有丢失任何消息。它以1.0的召回率报告丢包情况，将观察者的CPU和内存使用量分别减少了多达7倍和28倍，并在现有工具使系统超负荷的嵌入式机器人上保持实用性。

View on arXiv Download PDF AI Translation

cs.RO / 36 / 2606.10808

Bridging Semantics and Physical Execution: A Neuro-Symbolic Framework for Multi-Pair Robotic Assembly

弥合语义与物理执行之间的鸿沟：一种用于多对机器人组装的神经符号框架

Li, Xinyi, Song, Aiguo, Wei, Linhu, Li, Huijun

Abstract

Multi-pair robotic assembly in unstructured environments faces spatial interference and contact uncertainties. Existing paradigms fail to bridge cognitive decision-making and physical execution, as they either encounter state-space explosion and knowledge bottlenecks or suffer from logical hallucinations and topological conflicts. We propose an end-to-end neuro-symbolic framework that solves the challenge hierarchically: generating optimal subgraphs for each pair, decoupling generality from edge cases, and then resolving cross-pair interferences. Given an eye-on-hand RGB-D assembly scene, the framework extracts semantic instance identity and state while quantifying the scene for divergence calculation. For each pair, optimal subgraph is generated via LLM using barely basic actions to mitigate hallucinations. Supportive actions for edge cases are reasoned and inserted with a lightweight discriminator. Driven by the divergence between the quantified baseline and current scene, it is easily extensible at low cost. Augmented subgraphs are topologically coordinated into global sequences while preserving internal behavioral coherence. Dynamic behavior trees embedding atomic skills close the force-aware execution loop. Offline evaluation on 100 real-world scenes achieves 97.00% global executability, outperforming classical and state-of-the-art planners. Real-robot deployment on a UR3 arm attains 90% success rate with 0.5 mm tolerance under strong interference, demonstrating a unified and verifiable solution for complex autonomous assembly.

Chinese Translation

在非结构化环境中，多对机器人组装面临空间干扰和接触不确定性。现有范式未能有效连接认知决策与物理执行，因为它们要么遭遇状态空间爆炸和知识瓶颈，要么受到逻辑幻觉和拓扑冲突的困扰。我们提出了一种端到端的神经符号框架，分层解决这一挑战：为每对生成最佳子图，解耦一般性与边缘案例，然后解决跨对干扰。在一个眼在手的 RGB-D 组装场景中，该框架提取语义实例身份和状态，同时量化场景以进行偏差计算。对于每对，通过使用基本动作的 LLM 生成最佳子图，以减轻幻觉。边缘案例的支持性动作经过推理并与轻量级鉴别器插入。该框架通过量化基线与当前场景之间的偏差驱动，具有低成本的易扩展性。增强的子图在拓扑上协调成全局序列，同时保持内部行为的一致性。嵌入原子技能的动态行为树闭合了力感知执行循环。在 100 个真实场景上的离线评估实现了 97.00% 的全局可执行性，超越了经典和最先进的规划器。在 UR3 臂上的真实机器人部署在强干扰下达到了 90% 的成功率，容忍度为 0.5 毫米，展示了一种统一且可验证的复杂自主组装解决方案。

View on arXiv Download PDF AI Translation

cs.RO / 37 / 2606.10818

IMPACT: Learning Internal-Model Predictive Control for Forceful Robotic Manipulation

IMPACT：学习内部模型预测控制以实现强力机器人操作

Gao, Jiawei, Liu, Chaoqi, Wu, Peilin, Chen, Haonan, Du, Yilun

Abstract

Real-world robotic manipulation tasks often involve forceful interactions with the environment, such as using tools of varying weights, transporting objects with different masses, and performing contact-rich tasks like table wiping. Previous learning-based approaches typically employ imitation learning policies that output target end-effector poses tracked by low-level impedance controllers. In these systems, forceful interactions are either implicitly realized through steady-state tracking errors or explicitly commanded using wrist force/torque or tactile sensors. However, implicit approaches generalize poorly across object weights, while explicit approaches require specialized hardware and increase system complexity. In this work, we propose IMPACT, a framework that decouples these forceful tasks into task-planning and internal-model-based predictive control. Extensive simulation and real-world experiments demonstrate that the proposed framework achieves higher success rates and improved generalization to unseen object weights, as well as better safety and energy efficiency.

Chinese Translation

现实世界中的机器人操作任务通常涉及与环境的强力交互，例如使用不同重量的工具、运输不同质量的物体以及执行如擦桌子等接触丰富的任务。以往基于学习的方法通常采用模仿学习策略，这些策略输出由低级阻抗控制器跟踪的目标末端执行器姿态。在这些系统中，强力交互要么通过稳态跟踪误差隐式实现，要么通过手腕力/扭矩或触觉传感器显式指令。然而，隐式方法在不同物体重量之间的泛化能力较差，而显式方法则需要专用硬件并增加系统复杂性。在本研究中，我们提出了IMPACT，一个将这些强力任务解耦为任务规划和基于内部模型的预测控制的框架。大量的仿真和现实世界实验表明，所提出的框架在成功率和对未见物体重量的泛化能力上均表现更佳，同时在安全性和能效方面也有所改善。

View on arXiv Download PDF AI Translation

cs.RO / 38 / 2606.10832

GUIDE: Goal-Initialized Directional Understanding for End-to-End Visual Navigation

GUIDE：用于端到端视觉导航的目标初始化方向理解

Wang, Liang, Jin, Jin, Yao, KanZhong, Wu, YiBin, Ding, Fangqiang, Wang, Jin, Wu, Jun, Sun, Zhe, Zhu, Qiuguo

Abstract

Learning-based visual navigation for legged robots typically relies on continuous goal updates from hierarchical state estimation to provide a persistent directional reference. This reliance incurs additional sensory and computational overhead and deviates from fully end-to-end mobile autonomy. Furthermore, under partial observability, policies are prone to learn myopic behaviors, easily becoming trapped in dead ends and complex structural layouts. To address these limitations, we investigate a goal-initialized navigation setting, where the target is provided only once at the beginning of an episode, requiring the robot to operate based on intrinsic spatial memory without subsequent goal updates from external modules. In this work, we propose GUIDE, a fully end-to-end reinforcement learning framework designed to cultivate internal directional awareness. Specifically, GUIDE incorporates a spatial anchor predictor that leverages multi-frequency proprioceptive history to extract egomotion representations, thereby maintaining a persistent long-horizon spatial context for navigation. Concurrently, it utilizes raw depth streams to perceive local environmental geometry. We evaluate the proposed framework across both simulation and real-world scenarios on a quadruped robot. Experiments show that GUIDE learns reliable egomotion and directional awareness, enabling a fully end-to-end deployed policy to safely navigate through dense clutter and structured mazes without subsequent goal guidance or prior maps.

Chinese Translation

基于学习的四足机器人视觉导航通常依赖于来自层次状态估计的连续目标更新，以提供持久的方向参考。这种依赖带来了额外的感知和计算开销，并偏离了完全的端到端移动自主性。此外，在部分可观测性下，策略容易学习到短视行为，容易陷入死胡同和复杂的结构布局。为了解决这些限制，我们研究了一种目标初始化导航设置，在该设置中，目标仅在每个回合开始时提供一次，要求机器人基于内在空间记忆进行操作，而不依赖于外部模块的后续目标更新。在这项工作中，我们提出了GUIDE，一个完全端到端的强化学习框架，旨在培养内部方向意识。具体而言，GUIDE结合了一个空间锚点预测器，利用多频率的本体感觉历史提取自我运动表示，从而为导航保持持久的长时间空间上下文。同时，它利用原始深度流感知局部环境几何。我们在四足机器人上对所提出的框架进行了模拟和现实场景的评估。实验表明，GUIDE能够学习可靠的自我运动和方向意识，使得完全端到端部署的策略能够在没有后续目标指导或先前地图的情况下安全地穿越密集的杂物和结构迷宫。

View on arXiv Download PDF AI Translation

cs.RO / 39 / 2606.10841

Gradient based Bilevel for Inverse Optimal Control, a Riemannian approach

基于梯度的双层逆最优控制：一种黎曼方法

Dahmani, Ahmed-Manaf, Bonnet, Vincent, Daney, David, Charpillet, François

Abstract

Inverse Optimal Control (IOC) aims to recover the cost function that explains observed trajectories as solutions of an optimal control problem. Classical IOC formulations rely on bilevel optimization, which repeatedly solves a nested optimal control problem and quickly becomes computationally prohibitive for realistic systems. Recent projection-based approaches offer a promising alternative but suffer from numerical instability when solved with gradient-based methods due to violations of standard constraint qualifications. In this paper, we show that these difficulties stem from the geometric structure of the IOC feasible set. We demonstrate that the set of trajectories satisfying the optimality conditions naturally forms a manifold and reformulate IOC as an optimization problem on this manifold. Based on this insight, we propose a Riemannian Inverse Optimal Control (RIOC) method that projects observed trajectories onto the manifold of optimal solutions while preserving feasibility by construction. Experiments on real human arm trajectories show that the proposed method achieves comparable or better reconstruction accuracy than classical bilevel IOC while reducing computation time by about a factor of four. These results highlight the potential of geometric optimization methods to improve the scalability and reliability of IOC for robotics and human motion analysis.

Chinese Translation

逆最优控制（IOC）旨在恢复解释观察到的轨迹的成本函数，这些轨迹作为最优控制问题的解。经典的IOC公式依赖于双层优化，这种方法反复解决嵌套的最优控制问题，对于现实系统来说，计算成本迅速变得不可承受。最近的基于投影的方法提供了一种有前景的替代方案，但在使用基于梯度的方法求解时，由于违反标准约束资格，导致数值不稳定。本文表明，这些困难源于IOC可行集的几何结构。我们证明，满足最优性条件的轨迹集合自然形成一个流形，并将IOC重新表述为在该流形上的优化问题。基于这一见解，我们提出了一种黎曼逆最优控制（RIOC）方法，该方法在构造上通过将观察到的轨迹投影到最优解的流形上来保持可行性。对真实人类手臂轨迹的实验表明，所提方法在重建精度上与经典双层IOC相当或更好，同时将计算时间减少了约四倍。这些结果突显了几何优化方法在提高机器人技术和人类运动分析中IOC的可扩展性和可靠性方面的潜力。

View on arXiv Download PDF AI Translation

cs.RO / 40 / 2606.10856

An Exposure-Time-Aligned Primary-Path Architecture for Autonomous-Driving ECUs

一种用于自动驾驶电子控制单元的曝光时间对齐主路径架构

Saito, Toru, Hagura, Yuki, Konishi, Tatsuya, Mizusawa, Satoru, Yajima, Takumi

Abstract

While end-to-end (E2E) autonomous driving has become the dominant research direction, production vehicles continue to rely on modular multi-NN pipelines for a non-trivial transitional period. The subject of this paper is the design of an architecture that, during this phase, supports a modular pipeline and an E2E path side by side and embeds a path for staged migration. Transplanted to a production SoC, egalitarian late fusion is compute-inefficient and offers no natural unit for staged E2E substitution. As an alternative, we propose three design principles: (i) Primary-Path, which explicitly selects a primary perception chain and prioritizes its enclosure within a single SoC pair over the non-critical paths (ii) Exposure-Time-Aligned, which propagates the primary sensor's exposure time $\tau_{\rm exp}$ as a tag along the chain and event-drives the fusion node on matched $\tau_{\rm exp}$ rather than a fixed cycle and (iii) Co-Path Coexistence, which, building on (i) and (ii), lets an E2E output path co-run with the modular pipeline within the same $\tau_{\rm exp}$ cycle. On a Dual-SoC production AD-ECU, the implementation closes camera-shutter to planner-output latency at a mean of 296 ms within the 350 ms design budget. Under (iii), the modular pipeline is primary at production launch and the E2E path runs as shadow on real vehicles, and the E2E scope is expanded as evaluation evidence accumulates.

Chinese Translation

尽管端到端（E2E）自动驾驶已成为主流研究方向，生产车辆仍在经历一个非平凡的过渡期，依赖于模块化的多神经网络（multi-NN）管道。本文的主题是设计一种架构，在这一阶段支持模块化管道与E2E路径并行，并嵌入一个分阶段迁移的路径。移植到生产系统单芯片（SoC）后，平等的后期融合在计算上效率低下，并且没有自然的单位用于分阶段的E2E替代。作为替代方案，我们提出了三个设计原则：（i）主路径（Primary-Path），明确选择一个主要感知链，并优先将其封装在一对单一SoC中，而非关键路径；（ii）曝光时间对齐（Exposure-Time-Aligned），将主要传感器的曝光时间 $ au_{ m exp}$ 作为标签沿链传播，并在匹配的 $ au_{ m exp}$ 上驱动融合节点，而不是固定周期；（iii）共路径共存（Co-Path Coexistence），在（i）和（ii）的基础上，让E2E输出路径与模块化管道在同一 $ au_{ m exp}$ 周期内共同运行。在一个双SoC生产自动驾驶电子控制单元（AD-ECU）上，实施将相机快门到规划输出的延迟缩短至平均296毫秒，符合350毫秒的设计预算。在（iii）下，模块化管道在生产启动时为主，而E2E路径作为影子在真实车辆上运行，随着评估证据的积累，E2E范围逐步扩大。

View on arXiv Download PDF AI Translation

cs.RO / 41 / 2606.10857

Embodiment-conditioned Generalist Control for Multirotor Aerial Robots

基于体现条件的多旋翼空中机器人通用控制

Konstantaropoulos, Orestis, Rehberg, Welf, Kulkarni, Mihir, Alexis, Kostas

Abstract

We present a generalist position control policy capable of controlling arbitrary multirotor configurations of a certain rotor count (e.g., hexarotors or quadrotors) with a single set of network weights. The policy is conditioned on a physics-grounded embodiment descriptor: a mass and inertia-normalized control allocation matrix that captures how mass-normalized motor thrusts generate linear and angular accelerations in the body-frame. To train the policy, we sample from a broad distribution of arbitrary multirotor configurations, including non-planar and asymmetric systems, and optimize a single, compact network using Proximal Policy Optimization. Training requires only five minutes on an RTX 3090 GPU using a custom NVIDIA Warp-based dynamics simulator. Through extensive simulation experiments, we show that embodiment conditioning enables robust generalist control across arbitrary morphologies. We demonstrate zero-shot real-world transfer of this generalist policy on three diverse hexarotor systems, including a planar robot, a partially symmetric non-planar system, and a random asymmetric, non-planar configuration.

Chinese Translation

我们提出了一种通用位置控制策略，能够使用一组网络权重控制特定转子数量（例如六旋翼或四旋翼）的任意多旋翼配置。该策略基于物理基础的体现描述符进行条件化：一个质量和惯性归一化的控制分配矩阵，捕捉质量归一化的电机推力如何在机体坐标系中产生线性和角加速度。为了训练该策略，我们从广泛的任意多旋翼配置分布中进行采样，包括非平面和不对称系统，并使用近端策略优化（Proximal Policy Optimization）优化一个紧凑的单一网络。训练仅需在RTX 3090 GPU上使用自定义的基于NVIDIA Warp的动力学模拟器进行五分钟。通过广泛的仿真实验，我们展示了体现条件化使得在任意形态下的稳健通用控制成为可能。我们在三个不同的六旋翼系统上展示了该通用策略的零样本真实世界转移，包括一个平面机器人、一个部分对称的非平面系统和一个随机不对称的非平面配置。

View on arXiv Download PDF AI Translation

cs.RO / 42 / 2606.10899

MV-Actor: Aligning Multi-View Semantics and Spatial Awareness for Bimanual Manipulation

MV-Actor：对齐多视角语义与空间意识以实现双手操作

Tian, Yinchen, Li, Huan, Peng, Muyao, Wang, Xi, Wang, Yan, Yang, You

Abstract

Robotic manipulation has been widely applied in industrial scenarios. Compared with single-arm manipulation, bimanual manipulation is equipped with multiple cameras to capture information from different viewpoints. However, existing multi-view policies encode each view independently or fuse view features shallowly, resulting in limited sharing semantic perception and unreliable spatial awareness. In this paper, we propose \textbf{MV-Actor}, a multi-view perception framework that builds a unified semantic-spatial representation for bimanual manipulation. First, MV-Actor performs Multi-view Semantic Interaction to share semantic perception across views. Then it uses Semantic-Spatial Token Interaction to ground visual semantics with feed-forward reconstruction model features and acquire reliable spatial awareness. Finally, a Guided Metric Depth Repair module refines degraded sensor depth to provide more reliable metric anchors under consumer-grade depth noise. In simulation experiments conducted on the PerAct2 bimanual benchmark, MV-Actor achieves a state-of-the-art average success rate of 87.8\%. In real-world evaluations with more frequent viewpoint changes and unstable consumer-grade depth, MV-Actor outperforms both RGB and RGB-D baselines, further demonstrating the benefit of sharing semantic perception and reliable spatial awareness for bimanual manipulation.

Chinese Translation

机器人操作已广泛应用于工业场景。与单臂操作相比，双手操作配备了多个摄像头，以从不同视角捕获信息。然而，现有的多视角策略独立编码每个视角或浅层融合视角特征，导致语义感知共享有限和空间意识不可靠。本文提出了 extbf{MV-Actor}，一个为双手操作构建统一语义-空间表示的多视角感知框架。首先，MV-Actor执行多视角语义交互，以在视角之间共享语义感知。然后，它使用语义-空间令牌交互，将视觉语义与前馈重建模型特征结合，从而获得可靠的空间意识。最后，一个引导度量深度修复模块对退化的传感器深度进行精细化处理，以在消费级深度噪声下提供更可靠的度量锚点。在针对PerAct2双手基准的仿真实验中，MV-Actor实现了87.8\%的先进平均成功率。在实际评估中，面对更频繁的视角变化和不稳定的消费级深度，MV-Actor的表现优于RGB和RGB-D基线，进一步证明了共享语义感知和可靠空间意识对双手操作的益处。

View on arXiv Download PDF AI Translation

cs.RO / 43 / 2606.10903

AgniNav: Configuration-Driven Cross-Embodiment Local Planning for Robot Navigation

AgniNav：基于配置驱动的跨体现机器人导航局部规划

Zang, Tianhao, Cheng, Siwei, Huang, Haidong, Wang, Shanze, Zhang, Wei

Abstract

Monocular local navigation is attractive for lightweight robots, but existing vision-based policies often couple perception to a specific body, camera height, and footprint, making transfer from wheeled bases to legged platforms dependent on retraining or active depth hardware. This paper introduces AgniNav, a configuration-driven local navigation framework that standardizes cross-embodiment transfer at the collision-envelope level. Each robot is specified by a measurable four-parameter safety envelope: collision-relevant height, front length, rear length, and half width. The height parameter conditions an image-to-scan network to predict a one-dimensional, collision-relevant pseudo-laserscan from a monocular color image, while the remaining footprint parameters configure a dimension-aware local planner for collision checking. Training uses height-conditioned column-minimum scan labels generated from paired color-depth data, allowing the same image to supervise different safety envelopes without collecting robot-specific data. To the best of our knowledge, AgniNav is the first monocular local-navigation framework that jointly conditions perception and planning on a shared collision-envelope configuration for zero-retraining deployment across wheeled, quadruped, and humanoid platforms. Real-robot experiments on a Turtlebot2, Unitree Go2, and Accelerated Evolution K1 achieve 39/40, 18/20, and 18/20 successes with 0/40, 1/20, and 2/20 collisions, respectively, while running at 30 Hz on Jetson Orin.

Chinese Translation

单目局部导航对于轻量级机器人具有吸引力，但现有的基于视觉的策略通常将感知与特定的机体、相机高度和足迹耦合，使得从轮式底盘到四足平台的迁移依赖于重新训练或主动深度硬件。本文介绍了AgniNav，一种基于配置驱动的局部导航框架，在碰撞包络层面上标准化跨体现转移。每个机器人由一个可测量的四参数安全包络指定：与碰撞相关的高度、前部长度、后部长度和半宽度。高度参数条件化一个图像到扫描的网络，从单目彩色图像预测一维的与碰撞相关的伪激光扫描，而其余的足迹参数则配置一个维度感知的局部规划器以进行碰撞检测。训练使用从配对的彩色-深度数据生成的高度条件化列最小扫描标签，允许同一图像监督不同的安全包络，而无需收集特定于机器人的数据。据我们所知，AgniNav是第一个在共享碰撞包络配置上共同条件化感知和规划的单目局部导航框架，能够在轮式、四足和类人平台上实现零重新训练部署。在Turtlebot2、Unitree Go2和Accelerated Evolution K1上的真实机器人实验中，分别取得39/40、18/20和18/20的成功率，碰撞次数为0/40、1/20和2/20，且在Jetson Orin上以30 Hz的频率运行。

View on arXiv Download PDF AI Translation

cs.RO / 44 / 2606.10918

Task Robustness via Re-Labelling Vision-Action Robot Data

通过重新标注视觉-动作机器人数据实现任务鲁棒性

Kuramshin, Artur, Aslan, Özgür, Neary, Cyrus, Berseth, Glen

Abstract

The recent trend in scaling models for robot learning has resulted in impressive policies that can perform various manipulation tasks and generalize to novel scenarios. However, these policies continue to struggle with following instructions, likely due to the limited linguistic and action sequence diversity in existing robotics datasets. This paper introduces Task Robustness via Re-Labelling Vision-Action Robot Data (TREAD), a scalable framework that leverages large Vision-Language Models (VLMs) to augment existing robotics datasets without additional data collection, harnessing the transferable knowledge embedded in these models. Our approach leverages a pretrained VLM through three stages: generating semantic sub-tasks from original instruction labels and initial scenes, segmenting demonstration videos conditioned on these sub-tasks, and producing diverse instructions that incorporate object properties, effectively decomposing longer demonstrations into grounded language-action pairs. We further enhance robustness by augmenting the data with linguistically diverse versions of the text goals. Evaluations on LIBERO demonstrate that policies trained on our augmented datasets exhibit improved performance on novel, unseen tasks and goals. Our results show that TREAD enhances both planning generalization through trajectory decomposition and language-conditioned policy generalization through increased linguistic diversity.

Chinese Translation

近年来，机器人学习模型的规模化趋势产生了令人印象深刻的策略，这些策略能够执行各种操作任务并在新场景中进行泛化。然而，这些策略在遵循指令方面仍然面临挑战，这可能是由于现有机器人数据集中语言和动作序列的多样性有限。本文提出了通过重新标注视觉-动作机器人数据实现任务鲁棒性（Task Robustness via Re-Labelling Vision-Action Robot Data，TREAD），这是一个可扩展的框架，利用大型视觉-语言模型（Vision-Language Models，VLMs）来增强现有的机器人数据集，而无需额外的数据收集，充分利用这些模型中嵌入的可转移知识。我们的方法通过三个阶段利用预训练的VLM：从原始指令标签和初始场景生成语义子任务、基于这些子任务对演示视频进行分割，以及生成包含物体属性的多样化指令，有效地将较长的演示分解为有依据的语言-动作对。我们进一步通过用语言多样性版本的文本目标增强数据来提高鲁棒性。在LIBERO上的评估表明，基于我们增强的数据集训练的策略在新颖的、未见过的任务和目标上表现出更好的性能。我们的结果表明，TREAD通过轨迹分解增强了规划泛化，并通过增加语言多样性增强了基于语言的策略泛化。

View on arXiv Download PDF AI Translation

cs.RO / 45 / 2606.10927

AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning

AllDayNav：通过现实世界强化学习实现终身导航

Yin, Hang, Liang, Yinan, Zhang, Jiazhao, Liu, Jiahang, Li, Minghan, Zhang, Zhizheng, Wang, He

Abstract

Lifelong embodied navigation in dynamic environments requires robots to form persistent scene understanding from fragmentary observations, which remains difficult for existing methods that rely on explicit maps or scene graphs and struggle to generalize beyond structured settings. We propose AllDayNav, a lifelong self-learning navigation framework that implicitly encodes scene dynamics into the billion-scale parameters of a large model via reinforcement learning, powered by a self-evolving multimodal memory that maintains and updates visual keyframes, semantic descriptions, and temporal context while autonomously generating open-vocabulary instructions, image goals, and structured rewards. Experiments in both synthetic and real-world environments across cross-room, cross-episode, and cross-task scenarios show that AllDayNav achieves success rates approaching $100\%$ and consistently surpasses strong map-based, VLM, and RL baselines in path efficiency and robustness, demonstrating implicit, memory-driven reinforcement learning as a scalable alternative to explicit mapping for reliable lifelong navigation.

Chinese Translation

在动态环境中进行终身具身导航要求机器人从零散的观察中形成持久的场景理解，这对于依赖于显式地图或场景图的现有方法来说仍然是一个挑战，因为它们在结构化环境之外的泛化能力较弱。我们提出了AllDayNav，一种终身自学习导航框架，通过强化学习将场景动态隐式编码到一个大模型的数十亿参数中，该框架由一个自我演化的多模态记忆驱动，能够维护和更新视觉关键帧、语义描述和时间上下文，同时自主生成开放词汇指令、图像目标和结构化奖励。在跨房间、跨情节和跨任务场景下的合成和真实环境实验表明，AllDayNav的成功率接近$100 ext{%}$，并在路径效率和鲁棒性方面始终超越强大的基于地图的方法、视觉语言模型（VLM）和强化学习（RL）基线，展示了隐式、基于记忆的强化学习作为可靠的终身导航的可扩展替代方案。

View on arXiv Download PDF AI Translation

cs.RO / 46 / 2606.10971

Resilient Navigation for Autonomous Farm Robots by Leveraging Jerk-Augmented Models with IMU-Only Disturbance Rejection

通过利用加速增强模型和仅使用IMU的干扰抑制实现自主农场机器人韧性导航

Candan, Batu, Atallah, Mohammed, Servadio, Simone, Arabi, Saeed

Abstract

Precise state estimation for navigation of autonomous agricultural robots is often compromised by sensor outages (GNSS/LiDAR/Visual) and high-frequency vibrations inherent in off-road environments. This paper proposes a robust navigation algorithm based on a jerk-augmented Extended Kalman Filter (EKF) integrated with a Multiple Tuning Factor (MTF) adaptation method. Unlike standard EKF approaches that assume constant measurement noise, our method dynamically adjusts the measurement covariance matrix in real-time, allowing the system to cope with sudden disturbances and sensor outliers. We evaluate the algorithm using real-world data from a Salin247 autonomous robot. Results demonstrate that jerk-augmentation combined with MTF adaptation significantly reduces 3D position Root Mean Square Error (RMSE) compared to baseline EKF models, providing superior dead-reckoning capabilities.

Chinese Translation

自主农业机器人的导航精确状态估计常常受到传感器故障（GNSS/LiDAR/视觉）和越野环境中固有的高频振动的影响。本文提出了一种基于加速增强扩展卡尔曼滤波器（EKF）并结合多调节因子（MTF）适应方法的鲁棒导航算法。与假设测量噪声恒定的标准EKF方法不同，我们的方法实时动态调整测量协方差矩阵，使系统能够应对突发干扰和传感器异常值。我们使用Salin247自主机器人收集的真实数据对该算法进行了评估。结果表明，加速增强结合MTF适应显著降低了与基线EKF模型相比的3D位置均方根误差（RMSE），提供了更优的推算能力。

View on arXiv Download PDF AI Translation

cs.RO / 47 / 2606.10974

Language-Driven Cost Optimization for Autonomous Driving

基于语言的自主驾驶成本优化

Martinez-Baselga, Diego, Mustafa, Khaled, Alonso-Mora, Javier

Abstract

The driving behavior of autonomous vehicles is typically governed by the cost function of their motion planner, which encodes objectives such as speed tracking, smoothness, lane keeping, and collision avoidance. However, tuning the parameters that shape this cost function is a challenging task that requires technical expertise, limiting the vehicle's ability to adapt to evolving traffic scenarios or end-user preferences. This work presents a language-driven framework for adaptive cost design in autonomous driving. A Large Language Model (LLM) interprets structured scenario descriptions and natural language user queries to generate the parameters applied to a risk-aware Model Predictive Path Integral (MPPI) controller. The system incorporates a human-in-the-loop validation stage in which the proposed behavioral changes are described in non-technical language and confirmed prior to deployment. Users may additionally provide feedback either before or after deployment, enabling iterative refinement of the vehicle's motion behavior. The framework is evaluated across multiple queries in realistic driving scenarios to assess its effectiveness. Simulation results demonstrate that the method successfully induces behavioral changes that align with the intended requirements in an intuitive manner, thereby bridging the gap between intelligent vehicle control systems and end users.

Chinese Translation

自主车辆的驾驶行为通常由其运动规划器的成本函数决定，该函数编码了诸如速度跟踪、平滑性、车道保持和避免碰撞等目标。然而，调整塑造该成本函数的参数是一项具有挑战性的任务，需要技术专长，这限制了车辆适应不断变化的交通场景或最终用户偏好的能力。本研究提出了一种基于语言的自主驾驶适应性成本设计框架。大型语言模型（Large Language Model, LLM）解读结构化场景描述和自然语言用户查询，以生成应用于风险感知模型预测路径积分（Model Predictive Path Integral, MPPI）控制器的参数。该系统包含一个人机交互验证阶段，在该阶段，提出的行为变化以非技术性语言描述，并在部署前得到确认。用户还可以在部署前或后提供反馈，从而实现对车辆运动行为的迭代优化。该框架在现实驾驶场景中对多个查询进行了评估，以评估其有效性。仿真结果表明，该方法成功地以直观的方式引导行为变化，使其与预期要求相一致，从而弥合智能车辆控制系统与最终用户之间的差距。

View on arXiv Download PDF AI Translation

cs.RO / 48 / 2606.10986

Multi-UAV Active Sensing with Information Gain-based Planning and Belief Fusion

基于信息增益规划和信念融合的多无人机主动感知

Habibi, S., Marques, L.

Abstract

Unmanned aerial vehicles (UAVs) are increasingly used for active sensing and information gathering in spatially distributed environments. Their performance, however, is constrained by limited flight time, sensing uncertainty, and the trade-off between spatial coverage and observation accuracy. This paper presents a real-world validation of a multi-UAV active sensing framework for probabilistic binary terrain mapping, with precision agriculture used as the application case. The environment is represented as a probabilistic belief map, where spatial dependencies are modeled through a factor-graph formulation. UAV decision making is guided by Information Gain based Informative Path Planning (IGbIPP), and the approach is compared with Random Walk and Sweep coverage path planning baselines using both synthetic terrains and real UAV-derived agricultural imagery. The study also evaluates spatial correlation weights and several probabilistic belief-fusion rules for multi-UAV information sharing. Results show that IGbIPP reduces entropy and mapping error more effectively than the baselines, while a wider field of view improves real-world coverage and map accuracy. The results further show that simple equal or biased spatial weights can be more robust than adaptive weights, and that Bayesian, log-odds, and Dempster--Shafer fusion achieve the best cooperative mapping performance. These findings highlight the importance of uncertainty-driven planning, sensing geometry, spatial modeling, and probabilistic fusion for real-world UAV-based active sensing.

Chinese Translation

无人机（UAV）在空间分布环境中的主动感知和信息收集方面的应用日益增加。然而，它们的性能受到有限飞行时间、感知不确定性以及空间覆盖与观察精度之间权衡的限制。本文展示了一种多无人机主动感知框架在概率二元地形映射中的实际验证，以精准农业作为应用案例。环境被表示为概率信念图，其中空间依赖性通过因子图形式建模。无人机决策由基于信息增益的有效路径规划（Information Gain based Informative Path Planning, IGbIPP）指导，并将该方法与随机游走和扫掠覆盖路径规划基线进行比较，使用合成地形和真实无人机获取的农业影像。研究还评估了空间相关权重和几种多无人机信息共享的概率信念融合规则。结果表明，IGbIPP在降低熵和映射误差方面比基线方法更有效，而更广的视野则改善了实际覆盖和地图精度。结果进一步显示，简单的均等或偏置空间权重可能比自适应权重更具鲁棒性，而贝叶斯、对数几率和邓普斯特-谢弗融合实现了最佳的协作映射性能。这些发现突显了不确定性驱动的规划、感知几何、空间建模和概率融合在基于无人机的实际主动感知中的重要性。

View on arXiv Download PDF AI Translation

cs.RO / 49 / 2606.11019

Diffusion Forcing Planner: History-Annealed Planning with Time-Dependent Guidance for Autonomous Driving

扩散强制规划器：基于历史退火的时间依赖引导自主驾驶规划

Zhang, Zehan, Zhang, Neng, Li, Yaoyi, Cai, Jia, Wang, Zhiling

Abstract

Learning-based motion planners, despite recent progress, often suffer from temporal inconsistency. Small perturbations across frames can accumulate into unstable trajectories, degrading comfort and safety in closed-loop driving. Several methods attempt to inject history as a static conditioning signal to stabilize outputs, only to induce the planner to copy historical patterns instead of adapting to environment contexts. To address this limitation, we propose Diffusion Forcing Planner (DFP), a diffusion-based planning framework driven by history-guided control. Specifically, DFP decomposes the full trajectory into history, current and future segments, and assign independent noise levels to each segment. The model jointly denoises the historical and the future segments, enforcing a heterogeneous joint diffusion process. At inference, classifier-free guidance (CFG) is applied to steer future sampling using annealed history in a controllable manner. Closed-loop evaluation and comprehensive ablations on nuPlan show that DFP achieves competitive performance while producing continuous, stable, and controllable motion plans in complex driving scenarios.

Chinese Translation

尽管基于学习的运动规划器近年来取得了进展，但仍然常常面临时间不一致性的问题。帧间的小扰动可能会累积成不稳定的轨迹，从而降低闭环驾驶的舒适性和安全性。一些方法试图将历史作为静态条件信号注入，以稳定输出，但这往往导致规划器复制历史模式，而不是适应环境上下文。为了解决这一限制，我们提出了扩散强制规划器（Diffusion Forcing Planner, DFP），这是一个基于扩散的规划框架，由历史引导控制驱动。具体而言，DFP将完整轨迹分解为历史、当前和未来段，并为每个段分配独立的噪声水平。该模型联合去噪历史段和未来段，强制执行异质联合扩散过程。在推理时，采用无分类器引导（classifier-free guidance, CFG）以可控的方式使用退火历史引导未来采样。在nuPlan上的闭环评估和全面消融实验表明，DFP在复杂驾驶场景中实现了具有竞争力的性能，同时生成连续、稳定且可控的运动规划。

View on arXiv Download PDF AI Translation

cs.RO / 50 / 2606.11034

A Spiking Neural Architecture for Coordinating Arm and Locomotor Control

一种用于协调手臂和运动控制的脉冲神经网络架构

Steffen, Lea, Simone, Kathryn, Damberger, Graeme, DeWolf, Travis, Ly, Hudson, Eliasmith, Chris

Abstract

Spiking Neural Networks (SNNs) coupled with neuromorphic hardware offer energy-efficient solutions for humanoid robot control. However, existing SNN-based motor control systems address bipedal locomotion and arm control in isolation, leaving integrated control of both unaddressed. We present a spiking architecture that coordinates force-based arm control and bipedal locomotion in a simulated humanoid, using the Neural Engineering Framework (NEF) and Semantic Pointer Architecture (SPA). High-level action selection between locomotor and arm control is mediated by a biologically grounded spiking basal ganglia model. We validate the system through co-simulation of Nengo, for the neural control, and Isaac Sim, demonstrating successful target reaching, continuous digit drawing, path-following locomotion, and finally, switching between walking and arm control via basal ganglia disinhibition. To our knowledge, this is the first integrated spiking controller to combine bipedal locomotion and arm control on a full-scale humanoid platform. The full spike-based implementation enables future deployment on low-power neuromorphic hardware.

Chinese Translation

脉冲神经网络（SNN）与类脑硬件相结合，为类人机器人控制提供了能效高的解决方案。然而，现有的基于SNN的运动控制系统通常将双足运动和手臂控制孤立处理，未能实现两者的综合控制。我们提出了一种脉冲架构，能够在模拟的类人机器人中协调基于力的手臂控制和双足运动，使用神经工程框架（NEF）和语义指针架构（SPA）。在运动和手臂控制之间的高层次动作选择由一个生物学基础的脉冲基底神经节模型介导。我们通过Nengo进行神经控制的共同仿真和Isaac Sim进行验证，展示了成功的目标到达、连续的数字绘制、路径跟随运动，以及最终通过基底神经节的去抑制在行走和手臂控制之间的切换。据我们所知，这是第一个在全尺度类人平台上结合双足运动和手臂控制的集成脉冲控制器。完整的基于脉冲的实现为未来在低功耗类脑硬件上的部署奠定了基础。

View on arXiv Download PDF AI Translation

cs.RO / 51 / 2606.11037

Generation of Diverse and Functional Robot Designs using Superquadrics Parametrisation and Quality-Diversity

利用超二次体参数化和质量多样性生成多样化和功能性机器人设计

Goff, Leni Le, Smith, Simon, Hart, Emma

Abstract

Generative design of robots requires navigating a vast search-space, encompassing physical configurations and behavioural parameters. Evolutionary Algorithms (EAs) have shown promising results, but often converge prematurely to a small set of sub-optimal designs. Most EAs fail to maintain sufficient diversity in the population that would allow the discovery of distinct functional robots. To counter premature convergence, we introduce a superquadrics-based representation (SQs) for robot bodies. SQs are interpretable, compact and computationally efficient mathematical representations of 3D geometrical shapes that can be tuned to specific design-spaces. To encourage morphological diversity, we combine this representation with a quality-diversity (QD) algorithm (MAP-Elites). We compare SQs and Compositional Pattern Producing Networks representations as generators of morphologies, combining them with standard EAs and MAP-Elites. In two test environments, we find that using SQs to generate morphology in conjunction with the MAP-Elites algorithm reaches the highest QD-score across both environments, maximising diversity of design and functionality of generated robots. The findings highlight the benefits of using a compact and interpretable geometric representation for exploring a complex design-space and suggest that combining SQs with an explicit diversity mechanism increases the quality and number of designs generated.

Chinese Translation

机器人的生成设计需要在广阔的搜索空间中进行探索，包括物理配置和行为参数。进化算法（EAs）已显示出良好的结果，但往往会过早收敛于一小组次优设计。大多数EAs未能在种群中保持足够的多样性，从而无法发现不同功能的机器人。为了应对过早收敛的问题，我们引入了一种基于超二次体（superquadrics）的机器人身体表示。超二次体是可解释的、紧凑的且计算效率高的三维几何形状的数学表示，可以针对特定设计空间进行调节。为了鼓励形态多样性，我们将这种表示与质量多样性（quality-diversity, QD）算法（MAP-Elites）结合。我们比较了超二次体和组合模式生成网络（Compositional Pattern Producing Networks）作为形态生成器的表现，并将它们与标准的EAs和MAP-Elites结合。在两个测试环境中，我们发现使用超二次体生成形态并结合MAP-Elites算法能够在两个环境中达到最高的QD评分，最大化生成机器人设计和功能的多样性。研究结果突显了使用紧凑且可解释的几何表示在探索复杂设计空间中的优势，并表明将超二次体与显式多样性机制结合可以提高生成设计的质量和数量。

View on arXiv Download PDF AI Translation

cs.RO / 52 / 2606.11088

A Distributed Multi-UGV Exploration Framework With Loop-Aware Planning and Descriptor-Aided Localization in Resource-Limited Environments

一种在资源有限环境中具有循环感知规划和描述符辅助定位的分布式多无人地面车辆探索框架

Li, Zhiwei, Liu, Haiou, Zhao, Xijun, Li, Ji, Wang, Yingze, Wang, Boyang

Abstract

Robust and efficient cooperative exploration with multiple unmanned ground vehicles (UGVs) in unknown, GPSdenied, and bandwidth-limited environments without prior maps remains challenging, as localization drift degrades map consistency and induces redundant coverage. This paper presents a fully distributed exploration framework that couples descriptoraided inter-UGV loop closure with loop-aware hierarchical planning while enabling autonomous localization and exploration. We develop a lightweight LiDAR global descriptor with range-image prealignment to enable robust cross-UGV place recognition under large yaw and lateral variations, and use verified loop closures to maintain globally consistent trajectories and a sparse topological representation. We further introduce an uncertainty-aware crossUGV loop-closure selection module that scores candidate loop closures under pose uncertainty and retains high-utility loop closures as planning anchors for global task allocation and local route refinement. Simulations and real-UGV experiments show that the loop-closure module achieves AR@1/AR@1% of 89.9%/95.5%, distributed optimization reduces absolute trajectory error, the system substantially reduces two-way communication volume, and the overall framework reduces exploration time and travel distance by 15% and 14%, respectively, compared with an mTSP baseline.

Chinese Translation

在未知、无GPS和带宽受限的环境中进行多无人地面车辆（UGV）的稳健高效协作探索仍然面临挑战，因为定位漂移会降低地图一致性并导致冗余覆盖。本文提出了一种完全分布式的探索框架，该框架将描述符辅助的UGV间循环闭合与循环感知的分层规划相结合，同时实现自主定位和探索。我们开发了一种轻量级的激光雷达全局描述符，并通过范围图像预对齐来实现UGV间在大偏航和横向变化下的稳健位置识别，并利用经过验证的循环闭合来维护全局一致的轨迹和稀疏的拓扑表示。我们进一步引入了一种不确定性感知的UGV间循环闭合选择模块，该模块在姿态不确定性下对候选循环闭合进行评分，并保留高效用的循环闭合作为全局任务分配和局部路径优化的规划锚点。仿真和真实UGV实验表明，循环闭合模块的AR@1/AR@1%达到了89.9%/95.5%，分布式优化减少了绝对轨迹误差，系统显著降低了双向通信量，并且与mTSP基线相比，整体框架分别减少了15%和14%的探索时间和行程距离。

View on arXiv Download PDF AI Translation

cs.RO / 53 / 2606.11092

RoboNaldo: Accurate, Stable and Powerful Humanoid Soccer Shooting via Motion-Guided Curriculum Reinforcement Learning

RoboNaldo：通过运动引导的课程强化学习实现准确、稳定且强大的类人足球射门

Zhong, Yichao, Lu, Yidan, Lu, Yuhang, Tang, Tianyang, Mai, Haoguang, Pan, Yixuan, Li, Tianyu, Chen, Li, Wang, Jingbo, Li, Zhongyu, Lu, Peng, Li, Hongyang

Abstract

Elite humanoid soccer shooting requires whole-body stability, high-impulse whole-body interactions, and accuracy to targets. Motion tracking-driven reinforcement learning (RL) provides stability in whole-body movement coordination, but a fixed reference makes it hard to adapt to varied ball positions and strike timings; in contrast, task reward-driven RL struggles to explore and discover valid kicks from scratch. We therefore introduce RoboNaldo, a three-stage motion-guided curriculum RL framework for high-impulse humanoid interaction. A single human-kick reference is used as a scaffold and progressively shifts optimization towards shooting performance. The curriculum first learns a stable whole-body kicking prior, then adapts the kick to free-kick settings where the ball is stationary at random positions, and finally extends it to moving-ball shooting through a locomotion-command and kick-trigger interface. A high-level heuristic planner controls this interface during training, while alternative high-level controllers can drive the same low-level policy at inference. In simulation, RoboNaldo demonstrates free-kick shot error 48.6% lower and shoot velocity 2.96x than prior work baselines. In real world on a Unitree G1 with onboard perception, RoboNaldo attains 0.73 m and 0.86 m average target shooting error from 3 m away in free-kick and moving-ball cases, accordingly. And the post-contact ball velocity reaches 13.10 m/s, which is 59-71% of reported professional open-play shot speed. Project page: $\href{https://opendrivelab.com/RoboNaldo}{\text{opendrivelab.com/RoboNaldo}}$.

Chinese Translation

精英类人足球射门需要全身稳定性、高冲击力的全身交互以及对目标的准确性。基于运动跟踪的强化学习（RL）提供了全身运动协调的稳定性，但固定的参考使其难以适应不同的球位置和击球时机；相比之下，基于任务奖励的RL在从零开始探索和发现有效射门方面面临挑战。因此，我们提出了RoboNaldo，一个三阶段的运动引导课程RL框架，旨在实现高冲击力的类人交互。该框架使用单一的人类踢球参考作为支架，并逐步将优化转向射门表现。课程首先学习稳定的全身踢球先验，然后将踢球适应于自由踢的设置，即球在随机位置静止，最后通过运动指令和击球触发接口扩展到移动球射门。在训练过程中，高层启发式规划器控制该接口，而在推理时，其他高层控制器可以驱动相同的低层策略。在仿真中，RoboNaldo的自由踢射门误差比之前的工作基准低48.6%，射门速度提高了2.96倍。在实际应用中，RoboNaldo在Unitree G1上搭载感知系统，在自由踢和移动球情况下，分别从3米外达到0.73米和0.86米的平均目标射门误差。同时，接触后的球速达到13.10米/秒，约为报告的职业比赛开放式射门速度的59-71%。项目页面：$ ext{opendrivelab.com/RoboNaldo}$.

View on arXiv Download PDF AI Translation

cs.RO / 54 / 2606.11109

EM-Fall: Embodied mmWave Sensing for Day-and-Night Fall Detection on Humanoid Robots

EM-Fall：用于类人机器人昼夜跌倒检测的具身毫米波传感

Lu, Yanshuo, Hu, Yuxuan, Yuan, Shenghai, Zhou, Xinyu, Zuo, Kuangji, Lyu, Bofan, Yuan, XiChen, Yang, Jianfei

Abstract

Falls are one of the leading causes of injury and hospitalization among elderly individuals, making reliable fall awareness an essential capability for safety monitoring in residential environments. However, existing fall detection systems often rely on wearable devices or fixed sensing installations, which may suffer from low user compliance, limited spatial coverage, or degraded performance under occlusion and poor lighting conditions. In this work, we propose \textbf{EM-Fall}, an embodied fall detection framework deployed on a mobile humanoid robot. The system integrates millimeter-wave (mmWave) sensing with robotic mobility, allowing the robot to actively adjust its sensing viewpoint and maintain target observability across rooms and under occlusion. To address interference in complex residential environments, including pet motion and multipath artifacts, we design a human-centered perception pipeline combined with lightweight temporal modeling to capture motion evolution before, during, and after fall events. We evaluate the proposed system across eight real indoor environments with four participants and construct an in-home mmWave fall detection dataset. Experimental results show that the embodied mobile sensing paradigm improves monitoring continuity and maintains robust fall detection performance under diverse environmental conditions. The proposed framework provides a practical solution for robot-assisted safety monitoring in home environments.

Chinese Translation

跌倒是老年人受伤和住院的主要原因之一，因此可靠的跌倒意识成为居住环境安全监测的重要能力。然而，现有的跌倒检测系统通常依赖于可穿戴设备或固定传感器安装，这可能导致用户遵从性低、空间覆盖有限，或在遮挡和光照条件差的情况下性能下降。在本研究中，我们提出了 extbf{EM-Fall}，一个部署在移动类人机器人上的具身跌倒检测框架。该系统将毫米波（mmWave）传感与机器人移动性相结合，使机器人能够主动调整其传感视角，并在房间内和遮挡情况下保持目标可观察性。为了解决复杂居住环境中的干扰问题，包括宠物运动和多径伪影，我们设计了一个以人为中心的感知管道，并结合轻量级时间建模，以捕捉跌倒事件发生前、期间和之后的运动演变。我们在八个真实的室内环境中对所提出的系统进行了评估，参与者共有四人，并构建了一个家庭毫米波跌倒检测数据集。实验结果表明，具身移动传感范式提高了监测的连续性，并在多样的环境条件下保持了强健的跌倒检测性能。所提出的框架为家庭环境中的机器人辅助安全监测提供了一个实用的解决方案。

View on arXiv Download PDF AI Translation

cs.RO / 55 / 2606.11151

JOIN: Anchor-Grasp-Conditioned Joining via Opposition, Inference, and Navigation for Bimanual Assistive Manipulation

JOIN：通过对抗、推理和导航实现的锚定抓取条件下的双手辅助操作

Moore, Drake, Cheng, Matt, Tan, Xiang Zhi, Padır, Taşkın

Abstract

Assistive mobility and manipulation platforms have received increasing attention as a means of restoring independence to individuals with disabilities. While effective for many basic activities of daily living (ADLs), a significant percentage of everyday tasks such as opening a jar, pouring a liquid, lifting a tray, or basic meal preparation, is fundamentally bimanual and remains out of reach for any single-arm system. Adding a second arm to a wheelchair is impractical, due to the additional power draw, cost, and the loss of space required for transfers and mobility. We instead propose a heterogeneous, on-demand bimanual system, in which a wheelchair-mounted anchor arm is joined when needed by a summoned mobile manipulator that serves as a complement arm. The central technical problem, which we call bimanual joining, is conditional: the anchor has already committed to a grasp, and the complement arm must choose where to stand and what to grasp to complete the task. We formulate bimanual joining as a three-phase decomposition (plan, drive, grasp) and show that a vision-language model (VLM), coupled with standard geometric tools, provides task-level knowledge sufficient to solve a representative class of bimanual ADLs. Our system JOIN, contributes (i) a wheelchair-referenced opposition score, and (ii) task-conditioned directional manipulability. We evaluate JOIN on a Kinova Gen3 anchor and a Hello Robot Stretch~3 complement on representative same-object and different-object tasks. JOIN accomplished more attempts (19/20) than state-of-the-art methods (14/20) and required markedly less correction by the operator.

Chinese Translation

辅助移动和操作平台作为恢复残疾人士独立性的手段，受到越来越多的关注。尽管对于许多基本日常生活活动（ADLs）有效，但许多日常任务，如打开罐子、倒液体、提托盘或基本的餐食准备，基本上是双手操作的，仍然超出了任何单臂系统的能力。由于额外的功耗、成本以及转移和移动所需的空间损失，将第二只手臂添加到轮椅上是不切实际的。我们提出了一种异构的按需双手系统，其中在需要时，轮椅上安装的锚定手臂由召唤的移动操控器作为补充手臂连接。我们称之为双手连接的核心技术问题是条件性的：锚定手臂已经承诺进行抓取，而补充手臂必须选择站立的位置和抓取的物体以完成任务。我们将双手连接形式化为三个阶段的分解（规划、驱动、抓取），并展示了视觉-语言模型（VLM）结合标准几何工具提供的任务级知识足以解决一类典型的双手ADLs。我们的系统JOIN贡献了（i）基于轮椅的对抗评分，以及（ii）任务条件下的方向可操作性。我们在Kinova Gen3锚定手臂和Hello Robot Stretch~3补充手臂上评估JOIN，针对代表性的同物体和异物体任务。JOIN的尝试次数（19/20）超过了最先进的方法（14/20），并且操作员所需的纠正明显减少。

View on arXiv Download PDF AI Translation

cs.RO / 56 / 2606.11184

TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation

TacForeSight：基于力引导的触觉世界模型用于接触丰富的操作

Zang, Yujie, Zheng, Yuhang, Nie, Xian, Zheng, Yupeng, Tian, Shuai, Gu, Songen, Gao, Chen, Wang, Zining, Yan, Shuicheng, Ding, Wenchao

Abstract

Contact-rich manipulation requires robots to continuously perceive and regulate evolving physical interactions under dynamic contact transitions or complex surface geometries. Recent imitation learning methods improve contact-aware control by incorporating tactile or force feedback, but they rarely model the asymmetric spatiotemporal roles of global force and local tactile sensing. To address this, we propose TacForeSight, a lightweight force-conditioned tactile foresight framework for real-time manipulation. The core component is TacForceWM, a tactile world model that predicts short-horizon tactile latent dynamics from dual-finger tactile observations conditioned on high-frequency wrist force and torque signals. Another key component, the Predictive Tactile-Conditioned Policy, leverages the predicted latents as anticipatory contact priors, models the current-to-future tactile evolution via cross-attention, and adaptively fuses visuo-tactile features through a tactile-guided gating module. By forecasting purely within a compact latent space, TacForeSight enables proactive contact reasoning with efficient real-time inference suitable for high-frequency manipulation control. Real-robot experiments on five representative tasks and three in-process perturbation settings show that TacForeSight consistently outperforms existing baselines, particularly under dynamic contact disturbances. All models and datasets will be made publicly available on the project website at https://tacforesight.github.io/ProjectPage.

Chinese Translation

接触丰富的操作要求机器人在动态接触转变或复杂表面几何形状下持续感知和调节不断变化的物理交互。近期的模仿学习方法通过结合触觉或力反馈来改善接触感知控制，但很少对全局力和局部触觉感知的非对称时空角色进行建模。为了解决这一问题，我们提出了TacForeSight，一个轻量级的基于力条件的触觉前瞻框架，用于实时操作。其核心组件是TacForceWM，一个触觉世界模型，能够从双指触觉观察中预测短期触觉潜在动态，这些观察是基于高频手腕力和扭矩信号进行条件化的。另一个关键组件是预测性触觉条件策略，它利用预测的潜在变量作为预期接触先验，通过交叉注意力建模当前到未来的触觉演变，并通过触觉引导的门控模块自适应融合视觉-触觉特征。通过在紧凑的潜在空间内进行预测，TacForeSight实现了主动接触推理，并提供适合高频操作控制的高效实时推断。在五个代表性任务和三种过程扰动设置上的真实机器人实验表明，TacForeSight在动态接触干扰下始终优于现有基线。所有模型和数据集将在项目网站 https://tacforesight.github.io/ProjectPage 上公开发布。

View on arXiv Download PDF AI Translation

计算机视觉 (Computer Vision)

cs.CV / 1 / 2606.09871

SD-GRPO: Verifiable Segment Decomposition for Long-Form Vision-Language Generation

SD-GRPO：可验证的长形式视觉-语言生成的分段分解

Kim, Hyunwoong, Lee, Seongeun, Yun, Hannah, Park, Junhyun, Park, Jonggwon

Abstract

Group Relative Policy Optimization (GRPO) and its variants, originally developed for Large Language Models (LLMs), have recently been applied to Multimodal LLMs and produced strong results. However, their coarse-grained holistic credit assignment from a single scalar advantage underfits vision-language (VL) tasks, where outputs are often long-form responses grounded in semantically rich images. To address this limitation, we exploit a structured signal that single-scalar formulations discard: the natural segmentation of long-form VL outputs. Concretely, we propose Segment-Decomposed GRPO (SD-GRPO), which z-normalizes verifiable per-segment rewards across the rollout group, yielding a vector of per-segment advantages in place of a single scalar. We evaluate SD-GRPO across three settings spanning controlled and real-world long-form VL generation, organized by increasing semantic entanglement across segments. On a controlled multi-panel dense-captioning task constructed from DOCCI, where segments are semantically independent, SD-GRPO consistently outperforms the GRPO baseline, with larger gains at higher segment counts. Extending to a controlled multi-chart long-form VQA task constructed from MultiChartQA, we show both theoretically and empirically that rollout-level rewards suffer from cross-segment credit misattribution that scales with output length. On a real-world scientific figure captioning task on the MMSci dataset, where subfigure captions share context across the figure, blending holistic and per-segment rewards further improves on both, suggesting per-segment normalization alone is insufficient when segments are semantically entangled. Finally, by integrating SD-GRPO into Dr. GRPO, we confirm that it can be applied to any GRPO framework with minimal implementation overhead to enhance long-form VL generation.

Chinese Translation

群体相对策略优化（Group Relative Policy Optimization, GRPO）及其变体最初为大型语言模型（Large Language Models, LLMs）开发，最近已被应用于多模态大型语言模型，并取得了良好的效果。然而，它们基于单一标量优势的粗粒度整体信用分配在视觉-语言（Vision-Language, VL）任务中表现不足，因为这些任务的输出通常是基于语义丰富图像的长形式响应。为了解决这一局限性，我们利用了单标量公式所忽视的结构化信号：长形式 VL 输出的自然分段。具体而言，我们提出了分段分解的 GRPO（Segment-Decomposed GRPO, SD-GRPO），该方法在回滚组中对可验证的每段奖励进行 z 标准化，从而生成每段优势的向量，而不是单一标量。我们在三个设置中评估 SD-GRPO，这些设置涵盖了受控和真实世界的长形式 VL 生成，按照段落之间的语义纠缠程度进行组织。在从 DOCCI 构建的受控多面板密集标注任务中，由于各段在语义上相互独立，SD-GRPO 始终优于 GRPO 基线，且在段数较高时增益更大。扩展到从 MultiChartQA 构建的受控多图表长形式视觉问答（VQA）任务中，我们理论和实证表明，回滚级别的奖励受到跨段信用误归因的影响，且这种影响与输出长度成比例。在 MMSci 数据集上的真实世界科学图形标注任务中，子图标题在整个图形中共享上下文，将整体奖励与每段奖励结合进一步改善了两者的效果，表明当段落在语义上纠缠时，仅依靠每段标准化是不够的。最后，通过将 SD-GRPO 集成到 Dr. GRPO 中，我们确认它可以以最小的实现开销应用于任何 GRPO 框架，以增强长形式 VL 生成。

View on arXiv Download PDF AI Translation

cs.CV / 2 / 2606.09882

WHU-Infra3D: A Full-stack Multi-modal Dataset and Benchmark for 3D Roadside Infrastructure Inventory

WHU-Infra3D：一个全栈多模态数据集和3D路边基础设施清单的基准

Liu, Chong, Fu, Luxuan, Feng, Xuyu, Dong, Zhen, Yang, Bisheng

Abstract

The paradigm of digital twin cities is shifting from coarse visual mapping toward more precise and actionable digitization of urban assets. However, existing datasets predominantly focus on coarse visual perception, lacking the strict multi-modal alignment and attribute and status diagnosis required for automated infrastructure maintenance. To bridge this gap, we introduce WHU-Infra3D, a large-scale, multi-modal benchmark dataset dedicated to roadside infrastructure inventory. Covering 53.8 km across three cities, WHU-Infra3D uniquely integrates panoramic imagery and LiDAR point clouds with rigorous 2D-3D instance association and cross-frame tracking. Comprising over 175k multi-view 2D bounding boxes alongside thousands of 3D infrastructure instances, the dataset provides over 181k detailed attribute and status annotations (e.g., rust, occlusion) to empower operational health assessment. We establish comprehensive baselines across five core tasks: 2D detection, 2D cross-view matching, 3D geo-identification, 3D point cloud segmentation, and attribute recognition. Extensive evaluations expose significant cross-city domain gaps and inherent vulnerabilities of current models on long-tailed defective statuses, establishing WHU-Infra3D as an essential testbed for advancing scalable, AI-driven urban infrastructure inventory and lifecycle management. The WHU-Infra3D dataset is available at https://github.com/WHU-USI3DV/WHU-Infra3D.

Chinese Translation

数字双胞胎城市的范式正从粗略的视觉映射转向对城市资产更精确和可操作的数字化。然而，现有数据集主要集中于粗略的视觉感知，缺乏自动化基础设施维护所需的严格多模态对齐以及属性和状态诊断。为填补这一空白，我们推出了WHU-Infra3D，一个专注于路边基础设施清单的大规模多模态基准数据集。WHU-Infra3D覆盖三个城市的53.8公里，独特地将全景图像和激光雷达（LiDAR）点云与严格的2D-3D实例关联和跨帧跟踪相结合。该数据集包含超过175,000个多视角2D边界框以及数千个3D基础设施实例，提供超过181,000个详细的属性和状态注释（例如，锈蚀、遮挡），以支持操作健康评估。我们在五个核心任务上建立了全面的基准：2D检测、2D跨视图匹配、3D地理识别、3D点云分割和属性识别。广泛的评估揭示了跨城市领域的显著差距以及当前模型在长尾缺陷状态下的固有脆弱性，确立了WHU-Infra3D作为推动可扩展的人工智能驱动城市基础设施清单和生命周期管理的重要测试平台。WHU-Infra3D数据集可在 https://github.com/WHU-USI3DV/WHU-Infra3D 获取。

View on arXiv Download PDF AI Translation

cs.CV / 3 / 2606.09967

ABot-Earth 0.5: Generative 3D Earth Model

ABot-Earth 0.5：生成性三维地球模型

Qian, Ming, Ouyang, Tianjian, Sun, Mingchao, Wang, Zijian, Xiong, Jincheng, Han, Jiarong, Zhang, Yongchang, Zhang, Jiawei, Wang, Xu, Liu, Yu, Tang, Luyang, Yu, Fei, Ge, Zengye, Du, Mengmeng, Liu, Yuan, Fan, Nianfei, Wang, Song, Peng, Yingliang, Jia, Chunxue, Liu, Yang, Zeng, Shiying, Shi, Haozhe, Lai, Junnan, Pan, Hongyu, Wu, Zheng, Guo, Ning, Xu, Mu, Zhang, Hang

Abstract

We present ABot-Earth 0.5, a generative 3D framework designed to synthesize vast, seamless 3D environments from ubiquitous, geospatially referenced satellite imagery. To achieve this, we propose a novel generative model formulated directly with the 3D Gaussian Splatting (3DGS) representation. The model is trained on a diverse corpus of existing real-world urban reconstructions, learning to generate realistic geometry and textures. At inference, it synthesizes novel 3D scenes conditioned solely on satellite imagery at a scalable rate of under 10 minutes per square kilometer, while demonstrating exceptional realism. The framework is designed for accessibility, with integrated hierarchical level-of-detail (LOD) structures that permit real-time, interactive visualization on web-based map engines. This high-fidelity simulation sandbox effectively mitigates the sim-to-real domain gap, enabling critical downstream Embodied AI applications like closed-loop UAV navigation. By providing an ultra-low-cost and high-efficiency solution, ABot-Earth 0.5 significantly lowers the technical and financial barriers to large-scale 3D reconstruction and empowers the future of global digital earth visualization.

Chinese Translation

我们提出了ABot-Earth 0.5，这是一个生成性三维框架，旨在从普遍存在的地理空间参考卫星影像合成广阔、无缝的三维环境。为此，我们提出了一种新颖的生成模型，直接基于三维高斯点云（3D Gaussian Splatting，3DGS）表示进行构建。该模型在多样化的现有真实世界城市重建数据集上进行训练，学习生成逼真的几何形状和纹理。在推理阶段，它仅基于卫星影像合成新颖的三维场景，且每平方公里的可扩展处理时间低于10分钟，同时展现出卓越的真实感。该框架旨在实现可访问性，集成了分层细节等级（LOD）结构，允许在基于网络的地图引擎上进行实时交互式可视化。这个高保真模拟沙盒有效地减小了模拟与现实之间的领域差距，使得关键的下游具身人工智能应用，如闭环无人机导航成为可能。通过提供一种超低成本和高效率的解决方案，ABot-Earth 0.5显著降低了大规模三维重建的技术和财务门槛，推动了全球数字地球可视化的未来。

View on arXiv Download PDF AI Translation

cs.CV / 4 / 2606.10019

Generalized-CVO: Fast and Correspondence-Free Local Point Cloud Registration with Second Order Riemannian Optimization

广义-CVO：基于二阶黎曼优化的快速无对应局部点云配准

Zhang, Ray, Greiff, Marcus, Lew, Thomas, Subosits, John

Abstract

We propose a fast and correspondence-free local point cloud registration method that leverages geometric surface structure and reproducing kernel Hilbert space (RKHS) embeddings. The method represents point clouds as continuous functions with point-wise anisotropic kernels that encode local geometry. This formulation improves alignment along surface normals while relaxing alignment along tangential directions. To solve the resulting registration problem, we propose a second-order on-manifold optimization scheme with approximate Riemannian Hessians, achieving a speedup of up to 10x over the first-order solvers used in prior correspondence-free RKHS-based methods. We demonstrate improved frame-to-frame LiDAR and RGB-D tracking accuracy across diverse indoor and outdoor datasets. On a LiDAR tracking registration task in the driving domain, we achieve a reduction of $>55\%$ in both translational and rotational drift in challenging feature-sparse environments. On object registration benchmarks, we show improved robustness over ICP-based methods and further gains when refining global initialization, particularly under moderate misalignment.

Chinese Translation

我们提出了一种快速且无对应的局部点云配准方法，该方法利用几何表面结构和再生核希尔伯特空间（RKHS）嵌入。该方法将点云表示为具有点对点各向异性核的连续函数，从而编码局部几何信息。这种表述在表面法线方向上改善了对齐，同时放宽了切向方向上的对齐。为了解决所产生的配准问题，我们提出了一种基于近似黎曼黑塞矩阵的二阶流形优化方案，相较于先前基于RKHS的无对应方法中使用的一阶求解器，速度提升可达10倍。我们在多种室内和室外数据集上展示了改进的帧间激光雷达和RGB-D跟踪精度。在驾驶领域的激光雷达跟踪配准任务中，我们在特征稀疏的挑战性环境中实现了超过55%的平移和旋转漂移减少。在物体配准基准测试中，我们展示了相较于基于ICP的方法的更强鲁棒性，并在全局初始化精细化时，特别是在中等错位情况下，取得了进一步的提升。

View on arXiv Download PDF AI Translation

cs.CV / 5 / 2606.10021

SpineReport: Automated 3D Quantification and Reporting of Lumbar Spine Degeneration on MRI

SpineReport：基于MRI的腰椎退变自动化3D定量分析与报告

Molinier, Nathan, Marth, Adrian A., Sutter, Reto, Germann, Christoph, Connolly, Jacob A., Guay-Paquet, Mathieu, Schilaty, Nathan D., Weber II, Kenneth A., Cohen-Adad, Julien

Abstract

Lumbar spine conditions are a leading cause of disability worldwide, yet reliable quantification of degeneration from MRI remains challenging. In clinical practice, analysis is predominantly performed in two dimensions (2D), as manual three-dimensional (3D) assessment is time-consuming. However, 2D measurements suffer from limited reproducibility, particularly when anatomical structures are not aligned with the imaging plane. Existing automated approaches are often restricted to 2D, rely on discrete grading, or lack robustness and interpretability. We introduce SpineReport, an open-source, fully automated framework for comprehensive 3D morphometric analysis of lumbar spine MRI. Leveraging robust anatomical segmentations, the method extracts quantitative metrics from key structures, including the spinal canal, spinal cord, vertebrae, intervertebral discs, and foramina. These include both morphological and signal-based features, enabling cross-subject and longitudinal assessment. SpineReport further generates subject-specific reports that allow comparison with cohort distributions, improving interpretability and objective characterization of spinal morphology. Clinical relevance was evaluated against radiologist-reported severity grades for central canal, lateral recess, and foraminal stenosis. Metrics showed strong associations with central canal stenosis severity, with T2-weighted CSF signal providing the highest performance (AUC = 0.95). Canal AP diameter and area ratios also demonstrated strong correlations and high discriminative ability (AUC > 0.80). For lateral recess stenosis, associations were moderate, with lateral CSF signal being the most informative (AUC = 0.73). No significant associations were observed for foraminal stenosis despite robust region-of-interest extraction. SpineReport is released as an open-access tool: https://ivadomed.github.io/SpineReport/

Chinese Translation

腰椎疾病是全球残疾的主要原因之一，但从MRI中可靠地量化退变仍然具有挑战性。在临床实践中，分析主要在二维（2D）中进行，因为手动的三维（3D）评估耗时较长。然而，2D测量存在重复性有限的问题，特别是在解剖结构与成像平面不对齐时。现有的自动化方法通常仅限于2D，依赖于离散分级，或缺乏稳健性和可解释性。我们提出了SpineReport，一个开源的、完全自动化的框架，用于对腰椎MRI进行全面的3D形态测量分析。该方法利用稳健的解剖分割，从关键结构中提取定量指标，包括脊髓管、脊髓、椎骨、椎间盘和孔道。这些指标包括形态特征和信号特征，能够进行跨受试者和纵向评估。SpineReport进一步生成特定受试者的报告，允许与队列分布进行比较，从而提高了脊柱形态的可解释性和客观特征化。临床相关性通过与放射科医师报告的中央管、侧隐窝和孔道狭窄的严重程度等级进行评估。指标与中央管狭窄严重程度显示出强关联，T2加权的脑脊液信号提供了最高的性能（AUC = 0.95）。管道的前后径和面积比也显示出强相关性和高区分能力（AUC > 0.80）。对于侧隐窝狭窄，关联性为中等，侧脑脊液信号是最具信息量的（AUC = 0.73）。尽管进行了稳健的感兴趣区域提取，但对于孔道狭窄未观察到显著关联。SpineReport作为一个开放获取工具发布： https://ivadomed.github.io/SpineReport/

View on arXiv Download PDF AI Translation

cs.CV / 6 / 2606.10066

A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

公共医学视觉-语言基准中预训练污染的受控审计

Xu, Bruce Changlong, Wu, Lan, Ryu, Alexander

Abstract

Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years, yet reported accuracy assumes these examples were absent from pretraining. We audit open VLMs on SLAKE-En, PathVQA, VQA-RAD, and an auxiliary public OmniMedVQA mirror using four detector families: image-side near-neighbour overlap against PMC-OA-beta, canonical-order exchangeability, cohort-relative Min-K%++ tail enrichment, and cross-model top-K overlap. We find measurable image-side source overlap on SLAKE-En: 19.8% of images are flagged under SigLIP-B-16 and 4.2% under SigLIP-SO400M, while out-of-domain controls produce 0/2000 flags. Manual adjudication shows same-modality, same-projection matches to different patients rather than verified pixel-level duplicates, so we interpret this as source or distributional overlap rather than confirmed per-image memorization. On the text side, Qwen2.5-VL on SLAKE-En shows a canonical-order exchangeability signal that survives ordering ablation and external non-medical baselines. On the OmniMedVQA mirror, exchangeability fires for five medical and general VLMs while BLIP-2 remains clean. In contrast, cohort-relative Min-K%++ tail enrichment and cross-model top-K overlap collapse under an external pre-domain baseline: BLIP-2 reproduces the apparent positive signals despite lacking plausible medical-VQA exposure. We conclude that these cohort-relative detectors are unreliable as standalone membership-inference signals on small medical-VLM cohorts.

Chinese Translation

医学视觉-语言模型（VLMs）在公共基准上进行评估，这些基准的图像和问答对多年来一直可以自由下载，但报告的准确性假设这些示例在预训练中是缺失的。我们对开放的 VLMs 在 SLAKE-En、PathVQA、VQA-RAD 和一个辅助公共 OmniMedVQA 镜像上进行了审计，使用了四种检测器系列：图像侧近邻重叠对 PMC-OA-beta 的比较、规范顺序可交换性、队列相对 Min-K%++ 尾部富集，以及跨模型 top-K 重叠。我们发现 SLAKE-En 上存在可测量的图像侧源重叠：19.8% 的图像在 SigLIP-B-16 下被标记，4.2% 在 SigLIP-SO400M 下被标记，而域外控制产生 0/2000 的标记。人工裁定显示同模态、同投影的匹配是不同患者而非经过验证的像素级重复，因此我们将其解释为源或分布重叠，而非确认的逐图像记忆。在文本方面，SLAKE-En 上的 Qwen2.5-VL 显示出一种规范顺序可交换性信号，该信号在排序消融和外部非医学基准下仍然存在。在 OmniMedVQA 镜像上，五个医学和通用 VLMs 的可交换性被激活，而 BLIP-2 则保持干净。相比之下，队列相对 Min-K%++ 尾部富集和跨模型 top-K 重叠在外部预域基准下崩溃：BLIP-2 尽管缺乏合理的医学-VQA 曝露，但仍重现了明显的正信号。我们得出结论，这些队列相对检测器作为小型医学 VLM 队列的独立成员推断信号是不可靠的。

View on arXiv Download PDF AI Translation

cs.CV / 7 / 2606.10088

Interpretable Temporal Facial-Region Motion Analysis for In-the-Wild Parkinson's Disease Video Classification

可解释的时序面部区域运动分析用于野外帕金森病视频分类

Almushrafy, Riyadh

Abstract

Reduced facial expressivity is a common motor manifestation of Parkinson's disease (PD), often described as hypomimia or facial bradykinesia. This paper examines whether temporal motion descriptors extracted from facial-region keypoints can support in-the-wild PD-related video classification on the YouTubePD benchmark. Each video is represented using geometric descriptors from 14 predefined facial regions. Static geometry, normalized geometry, velocity-based descriptors, relative-velocity descriptors, and a GRU sequence baseline are compared under the same binary classification protocol. To assess stability and interpretability, the study includes seed-robustness analysis, region-level ablation, and permutation importance. The best result is obtained with normalized velocity descriptors and a Random Forest classifier, reaching a balanced accuracy of 0.826 and an AUROC of 0.855 on the held-out test split. Across 10 random seeds, this representation remains stable, with balanced accuracy of 0.810 +/- 0.018 and AUROC of 0.855 +/- 0.005. Overall, the results suggest that normalized facial-region motion is a lightweight and interpretable representation for YouTubePD video classification. The study is framed as a benchmark-level analysis and does not claim clinical severity assessment or MDS-UPDRS facial-expression scoring.

Chinese Translation

面部表情减少是帕金森病（PD）常见的运动表现，通常被描述为面部低表情或面部运动迟缓。本文研究了从面部区域关键点提取的时序运动描述符是否能够支持在YouTubePD基准上进行野外PD相关视频分类。每个视频使用来自14个预定义面部区域的几何描述符进行表示。在相同的二分类协议下，比较了静态几何、归一化几何、基于速度的描述符、相对速度描述符和GRU序列基线。为了评估稳定性和可解释性，本研究包括种子鲁棒性分析、区域级消融和置换重要性分析。使用归一化速度描述符和随机森林分类器获得最佳结果，在保留的测试集上达到0.826的平衡准确率和0.855的AUROC。在10个随机种子中，该表示保持稳定，平衡准确率为0.810 +/- 0.018，AUROC为0.855 +/- 0.005。总体而言，结果表明归一化的面部区域运动是一种轻量且可解释的表示方式，适用于YouTubePD视频分类。本研究被框定为基准级分析，并不声称进行临床严重性评估或MDS-UPDRS面部表情评分。

View on arXiv Download PDF AI Translation

cs.CV / 8 / 2606.10107

Maximum Matching Accuracy: An Instance Segmentation Evaluation Metric Utilizing Globally Optimal Matching

最大匹配准确度：一种利用全局最优匹配的实例分割评估指标

Stillwagon, Kaden, VandeLoo, Alexandra D., Forest, Craig R.

Abstract

Reliable evaluation of instance segmentation models requires metrics that accurately and consistently reflect segmentation quality. However, the metrics most widely used in biological imaging carry fundamental mathematical weaknesses: hard Intersection-over-Union (IoU) thresholds that produce discontinuous, low sensitivity scoring; per-object normalization that distorts scores under object size variation; and greedy or one-to-many matching procedures that yield non-optimal, order-dependent correspondences. Together, these properties produce unintuitive and unreliable model rankings under common failure modes such as split cells, merged cells, and cell boundary imprecision. We propose Maximum Matching Accuracy (MMA), a threshold-free continuous metric that finds a globally optimal one-to-one matching between predicted and ground truth objects and aggregates total overlap using per-pixel normalization. We evaluate MMA against AP@50, PQ, SEG, and AJI across three experiments: synthetic failure cases, progressive corruption tests, and a model ranking comparison. MMA produces scores that are more stable, more sensitive, and more interpretable than existing alternatives, providing a principled foundation for fair instance segmentation benchmarking in biological cell imaging.

Chinese Translation

可靠的实例分割模型评估需要能够准确且一致反映分割质量的指标。然而，生物成像中最广泛使用的指标存在基本的数学缺陷：硬性交并比（IoU）阈值导致不连续、低敏感度的评分；按对象归一化在对象大小变化下扭曲评分；贪婪或一对多匹配程序产生非最优、依赖顺序的对应关系。这些特性共同导致在常见失败模式（如细胞分裂、细胞合并和细胞边界不精确）下产生不直观且不可靠的模型排名。我们提出了最大匹配准确度（Maximum Matching Accuracy, MMA），这是一种无阈值的连续指标，能够在预测对象和真实对象之间找到全局最优的一对一匹配，并通过逐像素归一化聚合总重叠。我们在三个实验中评估MMA与AP@50、PQ、SEG和AJI的表现：合成失败案例、渐进性损坏测试和模型排名比较。MMA产生的评分比现有替代方案更稳定、更敏感且更易解释，为生物细胞成像中的公平实例分割基准测试提供了原则性基础。

View on arXiv Download PDF AI Translation

cs.CV / 9 / 2606.10115

Improving PET/CT-Based Whole-Body Lesion Segmentation Using Prediction Uncertainty-Augmented Models

基于PET/CT的全身病灶分割的改进：使用预测不确定性增强模型

Biswas, Bashirul Azam, Wagle, Biratal Raj, Yang, Zhihan, Seltzer, Marc A., Maeder, Matthew E., Yu, James B., Bhattacharya, Indrani

Abstract

Accurate lesion segmentation from whole-body Positron Emission Tomography (PET)/Computed Tomography (CT) scans is essential for cancer staging and treatment planning. PET provides functional metabolic information with different radiotracers, while CT offers anatomical localization. Lesion delineation from PET/CT imaging is clinically challenging due to subtle imaging features, confounders, and inter-reader variability. Existing deep learning approaches suffer from training-related stochasticity, inconsistent predictions, missed lesions in high tumor-burden cases, and lack uncertainty quantification, limiting their clinical reliability. Using nnU-Net as a baseline, we propose an uncertainty-aware framework for whole-body PET/CT lesion segmentation that integrates (1) Bayesian ensembling to reduce training stochasticity, (2) voxel-wise uncertainty quantification with epistemic and aleatoric decomposition, and (3) epistemic uncertainty-augmented training to improve lesion detection. Two public datasets, AutoPET-III (1,611 scans) and Deep-PSMA (200 scans), comprising FDG and PSMA studies across multiple cancer types, are used for training and evaluation. Bayesian ensembling improves robustness and performance over deterministic nnU-Net models on the unseen AutoPET-III test set. Uncertainty maps highlight regions of model disagreement and correlate with misclassifications, particularly false positives. Uncertainty-augmented training improves lesion recovery at the cost of increased FPVol, reflecting a precision-recall trade-off. A case-adaptive routing strategy further improves Dice by selecting between the base and augmented models. To our knowledge, this is the first study to systematically investigate uncertainty quantification in multi-tracer, pan-cancer PET/CT segmentation and to combine Bayesian ensembling with uncertainty-aware modeling for this task.

Chinese Translation

从全身正电子发射断层扫描（PET）/计算机断层扫描（CT）图像中准确分割病灶对于癌症分期和治疗计划至关重要。PET提供了不同放射性示踪剂的功能性代谢信息，而CT则提供了解剖定位。由于图像特征微妙、混杂因素和读者间变异，PET/CT成像中的病灶描绘在临床上具有挑战性。现有的深度学习方法受到训练相关随机性的影响，预测不一致、高肿瘤负担病例中漏检病灶以及缺乏不确定性量化，限制了其临床可靠性。以nnU-Net为基础，我们提出了一种不确定性感知框架，用于全身PET/CT病灶分割，该框架集成了（1）贝叶斯集成以减少训练随机性，（2）基于体素的不确定性量化，采用认知性和随机性分解，以及（3）增强认知性不确定性的训练以改善病灶检测。我们使用两个公共数据集，AutoPET-III（1,611个扫描）和Deep-PSMA（200个扫描），涵盖多种癌症类型的FDG和PSMA研究，用于训练和评估。贝叶斯集成在未见的AutoPET-III测试集上提高了相较于确定性nnU-Net模型的鲁棒性和性能。不确定性图突出显示了模型不一致的区域，并与误分类（特别是假阳性）相关。不确定性增强训练在提高病灶恢复的同时增加了假阳性体积（FPVol），反映了精确度与召回率之间的权衡。案例自适应路由策略通过在基础模型和增强模型之间进行选择进一步提高了Dice系数。据我们所知，这是首个系统性研究多示踪剂、全癌种PET/CT分割中的不确定性量化，并将贝叶斯集成与不确定性感知建模结合用于此任务的研究。

View on arXiv Download PDF AI Translation

cs.CV / 10 / 2606.10135

BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression

BiWM：通过双向自回归推进开源交互视频世界模型

Rui, Shaohao, Mao, Xiaofeng, Zhang, Zhanyu, Lin, Peijia, Zhu, Yansong, Zhang, Yibo, Wan, Haibin, Ma, Weijie

Abstract

Transitioning bidirectional video diffusion models into an autoregressive paradigm improves the interactivity of video world models, but existing causal pipelines need many stages (control fine-tuning, autoregressive training, causal initialization, few-step distillation) and still trail bidirectional models in quality due to error accumulation. Recent world models such as Yume-1.5 and Matrix-Game-3.0 instead adopt a bidirectional autoregressive approach, gaining fidelity and stable long-horizon rollout from self-correcting error propagation, yet open-source frameworks (e.g., minWM) support only causal models. We present BiWM, the first full-stack framework for interactive video world models under the bidirectional autoregressive paradigm, jointly optimizing generation quality and inference speed. From a pretrained video backbone, BiWM injects camera control by fine-tuning, then runs a few-step Distribution Matching Distillation (DMD) stage that turns the backbone into an action/camera-controllable world model: just two training stages instead of four in minWM, converging in a few hundred steps on 8xH200 GPUs. A single recipe spans Wan2.1-1.3B, Wan2.2-5B, HunyuanVideo-1.5-8B, and LTX-2.3-22B, and also supports secondary fine-tuning of existing bidirectional models. BiWM enables real-world camera control where minWM loses controllability, integrates pluggable history compression (FramePack-style and PackForcing-style) for long rollouts, and offers an optional NVFP4 4-bit training/inference pipeline. To counter DMD's mode-seeking degradation, we add GAN and mass-covering forward-KL objectives that preserve scene dynamics. We open-source BiWM for resource-constrained research and high-fidelity environment simulation.

Chinese Translation

将双向视频扩散模型转变为自回归范式可以提高视频世界模型的交互性，但现有的因果管道需要多个阶段（控制微调、自回归训练、因果初始化、少步蒸馏），并且由于误差累积，质量仍然落后于双向模型。最近的世界模型如 Yume-1.5 和 Matrix-Game-3.0 采用了双向自回归方法，通过自我修正的误差传播获得了更高的保真度和稳定的长时间滚动，但开源框架（如 minWM）仅支持因果模型。我们提出了 BiWM，这是第一个在双向自回归范式下的交互视频世界模型的全栈框架，联合优化生成质量和推理速度。BiWM 从一个预训练的视频主干网络开始，通过微调注入相机控制，然后运行一个少步的分布匹配蒸馏（DMD）阶段，将主干网络转变为一个可控的动作/相机世界模型：仅需两个训练阶段，而不是 minWM 中的四个，在 8xH200 GPU 上几百步内收敛。单一的配方涵盖了 Wan2.1-1.3B、Wan2.2-5B、HunyuanVideo-1.5-8B 和 LTX-2.3-22B，并且还支持现有双向模型的二次微调。BiWM 实现了在 minWM 失去可控性的情况下的真实世界相机控制，集成了可插拔的历史压缩（FramePack 风格和 PackForcing 风格）以支持长时间滚动，并提供可选的 NVFP4 4 位训练/推理管道。为了解决 DMD 的模式寻求降级问题，我们增加了 GAN 和大覆盖的前向 KL 目标，以保持场景动态。我们将 BiWM 开源，以支持资源受限的研究和高保真环境模拟。

View on arXiv Download PDF AI Translation

cs.CV / 11 / 2606.10136

iSAGE: A Human-in-the-Loop Framework for Remote Sensing Semantic Segmentation via Sparse Point Supervision

iSAGE：一种基于稀疏点监督的遥感语义分割人机协作框架

de Carvalho, Osmar Luiz Ferreira, Junior, Osmar Abilio de Carvalho, de Albuquerque, Anesmar Olino, Silva, Daniel Guerreiro e

Abstract

Semantic segmentation in remote sensing requires costly pixel-level annotations, and nearly every problem demands a new dataset since models rarely transfer across sensors, platforms, or geographies. Existing human-in-the-loop frameworks expand sparse clicks into dense supervision via auxiliary machinery (pseudo-labels, propagation, CRFs, foundation-model prompts, auxiliary heads), all operating on the model's predictive distribution. A confidently wrong pixel is indistinguishable from a confidently correct one in that distribution by construction, so no rule reading it can separate the two; the distinguishing signal is external to the model. This paper hypothesizes that expert clicks targeting confident model errors, not arbitrary pixels, suffice to match dense supervision, with no expansion machinery. iSAGE (Iterative Sparse Annotation Guided by Expert) realizes this hypothesis on an integrated open-source platform, where an error-weighted loss amplifies the gradient at each click and the annotation record itself is the dataset, extensible, correctable, and auditable. Experiments use a minimum-effort regime: at most one labeled pixel per class per frame. On BsB Aerial, iSAGE recovers 97.2% of dense supervision (74.79% mIoU on 0.040% of pixels) with contrasting class dynamics: amorphous classes (permeable areas) saturate from the seed, while small classes (cars) require late-iteration effort. On ISPRS Vaihingen (external benchmark), iSAGE reaches 76.78% mIoU with 0.011% of pixels, matching the dense baseline (76.65%) and exceeding all published methods. Under the same pipeline, four output-reading mechanisms (oracle entropy across budgets 1--100x, pseudo-labels across thresholds 0.90--0.99, CRF-based propagation, uniform random) plateau 7.4 to 14.5 pp below iSAGE. Across 31 surveyed methods, iSAGE is the only iterative human-in-the-loop framework operating without auxiliary machinery.

Chinese Translation

遥感中的语义分割需要昂贵的像素级标注，几乎每个问题都需要一个新的数据集，因为模型在不同传感器、平台或地理区域之间很少能够迁移。现有的人机协作框架通过辅助机制（伪标签、传播、条件随机场（CRFs）、基础模型提示、辅助头）将稀疏点击扩展为密集监督，所有这些都在模型的预测分布上运行。根据构造，一个自信错误的像素与一个自信正确的像素在该分布中是无法区分的，因此没有任何规则可以将两者分开；区分信号是外部于模型的。本文假设，针对自信模型错误的专家点击，而非任意像素，足以匹配密集监督，而无需扩展机制。iSAGE（专家引导的迭代稀疏标注）在一个集成的开源平台上实现了这一假设，其中一个错误加权损失在每次点击时放大梯度，标注记录本身就是数据集，具有可扩展性、可修正性和可审计性。实验采用最低努力模式：每个类每帧最多一个标注像素。在 BsB Aerial 数据集上，iSAGE 恢复了 97.2% 的密集监督（在 0.040% 的像素上达到 74.79% 的 mIoU），并表现出对比类动态：无定形类（可渗透区域）从种子处饱和，而小类（汽车）则需要后期迭代的努力。在 ISPRS Vaihingen（外部基准）上，iSAGE 在 0.011% 的像素上达到了 76.78% 的 mIoU，匹配密集基线（76.65%）并超越了所有已发布的方法。在相同的流程下，四种输出读取机制（预算 1--100 倍的 oracle 熵、阈值 0.90--0.99 的伪标签、基于 CRF 的传播、均匀随机）均低于 iSAGE 7.4 到 14.5 个百分点。在调查的 31 种方法中，iSAGE 是唯一一个不依赖辅助机制的迭代人机协作框架。

View on arXiv Download PDF AI Translation

cs.CV / 12 / 2606.10142

DB-3DME: From Dataset to Benchmark for Human-aligned Automatic 3D Mesh Evaluation

DB-3DME：从数据集到基准的人类对齐自动3D网格评估

Jia, Nanshan, Zhao, Zhenyu, Huang, Sui, Wang, Jingshen, Zheng, Zeyu

Abstract

Recent advances in 3D generation have led to substantial improvements in realism, controllability, and efficiency, yet the evaluation of 3D assets remains underexplored. Existing evaluation paradigms, including human evaluation, learned metrics, and vision-language models (VLMs) as judges, suffer from limitations in cost, scalability, resolution handling, or task-specific alignment. In this work, we focus on 3D mesh evaluation and introduce DB-3DME, the Dataset and Benchmark for 3D Mesh Evaluation. DB-3DME contains 2,619 synthetic 3D meshes paired with human ratings on Geometry and Prompt Adherence. Using this dataset, we systematically benchmark state-of-the-art VLMs and identify visual encoding of 3D representations as a key factor for human-aligned evaluation performance. Motivated by this finding, we fine-tune an open-weight VLM, Qwen-2.5-VL-7B, for 3D mesh evaluation by adapting the visual encoder while freezing the language model. The fine-tuned model substantially outperforms existing pre-trained VLMs across multiple evaluation dimensions, establishing a new benchmark for automatic 3D mesh evaluation. We publicly release the benchmark dataset on GitHub and Hugging Face to facilitate future research.

Chinese Translation

近年来，3D生成技术的进步在现实感、可控性和效率方面取得了显著提升，但3D资产的评估仍然未得到充分探索。现有的评估范式，包括人类评估、学习度量和作为评判者的视觉-语言模型（VLMs），在成本、可扩展性、分辨率处理或任务特定对齐方面存在局限性。在本研究中，我们专注于3D网格评估，并引入DB-3DME，即3D网格评估的数据集和基准。DB-3DME包含2619个合成3D网格，并与人类在几何形状和提示遵循上的评分配对。利用该数据集，我们系统性地对最先进的VLM进行基准测试，并确定3D表示的视觉编码是人类对齐评估性能的关键因素。基于这一发现，我们对开放权重的VLM Qwen-2.5-VL-7B进行了微调，以适应3D网格评估，通过调整视觉编码器而冻结语言模型。微调后的模型在多个评估维度上显著优于现有的预训练VLM，建立了自动3D网格评估的新基准。我们将在GitHub和Hugging Face上公开发布基准数据集，以促进未来的研究。

View on arXiv Download PDF AI Translation

cs.CV / 13 / 2606.10166

Fusing Satellite Imagery and Planimetric Maps for Cross-View Localization

融合卫星影像与平面图以实现跨视角定位

Ngo, Quang Long Ho, Xia, Zimin, Alahi, Alexandre

Abstract

Current cross-view localization methods predominantly rely on satellite imagery as the aerial modality. Although recent work explores planimetric maps (e.g., OpenStreetMap tiles), these approaches often lag in performance. Yet both modalities are widely available and possess complementary properties. Satellite images are closer to ground-level camera imagery, offering finer detail, whereas planimetric maps contain annotated objects (e.g., streetlamps) and remain informative in areas where the ground is occluded, such as by foliage. Despite this, only one prior work provides an end-to-end method to fuse the two modalities, and it does not demonstrate their potential within state-of-the-art methods. To combine the strengths of both modalities, we propose a new fusion module that augments standard encoders and demonstrates that integrating satellite imagery with planimetric maps improves state-of-the-art single-modality methods. The module comprises (i) cross-modal conditioning, which processes each modality's encoding with awareness of the other, and (ii) a patch-level fusion rule that controls the granularity of information exchange. We achieve state-of-the-art results, reducing the mean localization error by 30.13\%. Qualitatively, the fusion adaptively selects the more informative modality, improving overall accuracy.

Chinese Translation

当前的跨视角定位方法主要依赖于卫星影像作为空中模式。尽管近期的研究探索了平面图（例如，OpenStreetMap 瓦片），但这些方法的性能往往滞后。然而，这两种模式都广泛可用，并具有互补特性。卫星图像更接近地面摄像头影像，提供更精细的细节，而平面图则包含标注的物体（例如，路灯），在地面被遮挡（如树叶）时仍然保持信息丰富。尽管如此，之前仅有一项工作提供了融合这两种模式的端到端方法，并且未能展示其在最先进方法中的潜力。为了结合这两种模式的优势，我们提出了一种新的融合模块，该模块增强了标准编码器，并证明将卫星影像与平面图结合能够改善最先进的单一模式方法。该模块包括（i）跨模式条件处理，它在处理每种模式的编码时考虑到另一种模式，以及（ii）一个补丁级别的融合规则，控制信息交换的粒度。我们取得了最先进的结果，将平均定位误差降低了 30.13%。在定性方面，融合模块自适应地选择更具信息量的模式，提高了整体准确性。

View on arXiv Download PDF AI Translation

cs.CV / 14 / 2606.10167

FlexPath: Learned Semantic Path Priors for Image-Based Planning

FlexPath：基于图像规划的学习语义路径先验

Kim, Taehyoung, Schoenbrod, Tim, Eckel, David, Meeß, Henri

Abstract

Recent learning-based path planners use neural networks to process visual map representations and approximate heuristics for classical search algorithms, yielding near-optimal paths with reduced search effort. However, these methods are tied to the shortest-path objective implicit in their supervision, which limits their flexibility to accommodate alternative criteria. We introduce FlexPath, a two-stage framework that decouples feasibility from preference. In Stage 1, we use imitation learning to acquire a task-independent spatial prior over feasible paths from visual map inputs. In Stage 2, differentiable Path Shape Objectives (PSOs) adapt this prior toward task-specific criteria without relearning path structure, requiring only efficient objective-level adaptation. A single pretrained model can be adapted to multiple objectives. For shortest-path planning, FlexPath reduces search effort on TMP by 14.3% compared to the state-of-the-art TransPath, while also finding lower-cost paths on average and demonstrating strong zero-shot generalization across three unseen domains. For obstacle clearance with minimum clearance distance 2, it achieves 96.8% full obstacle avoidance while maintaining low search cost. The framework further extends to semantic-aware avoidance and waypoint guidance via objective-level adaptation, and remains compatible with classical planners at inference time. Data and code are available at https://github.com/FraunhoferIVI/FlexPath.

Chinese Translation

近期的基于学习的路径规划器利用神经网络处理视觉地图表示，并为经典搜索算法近似启发式，从而以较低的搜索成本获得近似最优路径。然而，这些方法受限于其监督中隐含的最短路径目标，限制了其适应替代标准的灵活性。我们提出了FlexPath，一个将可行性与偏好解耦的两阶段框架。在第一阶段，我们使用模仿学习从视觉地图输入中获取与任务无关的可行路径空间先验。在第二阶段，差分路径形状目标（Path Shape Objectives, PSOs）将这一先验调整为特定任务的标准，而无需重新学习路径结构，仅需高效的目标级适应。一个单一的预训练模型可以适应多个目标。在最短路径规划中，FlexPath在搜索努力上比最先进的TransPath减少了14.3%，同时平均找到更低成本的路径，并在三个未见领域展示了强大的零样本泛化能力。对于最小间隙距离为2的障碍物清除任务，它实现了96.8%的完全障碍物规避，同时保持低搜索成本。该框架进一步扩展到语义感知规避和途经点引导，通过目标级适应，且在推理时与经典规划器兼容。数据和代码可在 https://github.com/FraunhoferIVI/FlexPath 获取。

View on arXiv Download PDF AI Translation

cs.CV / 15 / 2606.10174

A Large Scale Open-Source Image and Video Dataset for Robust Wildfire Detection and Classification

用于鲁棒性野火检测和分类的大规模开源图像和视频数据集

Hamdan, Emadeldeen, Luo, Yingyi, Toreyin, B. Ugur, Koyuncu, Erdem, Watts, Adam J., Gudukbay, Ugur, Cetin, Ahmet Enis

Abstract

Wildfire detection and monitoring are critical for mitigating fire spread and reducing environmental and infrastructural damage. In this work, we introduce GWFP (Global Wildfire Prevention Dataset), a large-scale, open-source dataset of wildfire images and videos designed to support early fire and smoke detection research. GWFP contains geographically diverse wildfire scenes, including flames, smoke, Waterdog/Fog environmental conditions, Near Infrared (NIR) imagery, Ember, and challenging negative samples collected from real-world scenarios worldwide. To evaluate dataset robustness and cross-domain generalization, we benchmark multiple convolutional and transformer-based architectures across both in-domain and cross-dataset settings. Additionally, we explore lightweight frequency--spatial feature interaction using Hadamard-enhanced residual connections (HTE-ResNet) to analyze representation robustness under domain-shift conditions. Experimental results demonstrate strong cross-dataset generalization and practical utility for real-world wildfire monitoring applications. The dataset and source code will be publicly released upon acceptance.

Chinese Translation

野火的检测和监测对于减缓火势蔓延以及减少环境和基础设施损害至关重要。在本研究中，我们介绍了GWFP（全球野火预防数据集），这是一个大规模的开源野火图像和视频数据集，旨在支持早期火灾和烟雾检测研究。GWFP包含地理多样的野火场景，包括火焰、烟雾、水狗/雾环境条件、近红外（NIR）图像、火星和从全球真实场景中收集的具有挑战性的负样本。为了评估数据集的鲁棒性和跨领域泛化能力，我们在领域内和跨数据集设置下对多种卷积和基于变换器的架构进行了基准测试。此外，我们利用Hadamard增强残差连接（HTE-ResNet）探索轻量级频率-空间特征交互，以分析在领域转移条件下的表示鲁棒性。实验结果表明，数据集在跨数据集泛化和实际应用于真实世界野火监测方面具有良好的实用性。数据集和源代码将在接受后公开发布。

View on arXiv Download PDF AI Translation

cs.CV / 16 / 2606.10183

Making Time Editable in Video Diffusion Transformers

在视频扩散变换器中实现时间可编辑性

Kuklev, Konstantin, Vasilev, Viacheslav, Kunitsyn, Alexander, Ivaniuta, Andrei, Dimitrov, Denis

Abstract

Modern Diffusion Transformers for video generation provide limited control over the progression of time and the editing of temporal dynamics. We propose a temporal-control methodology that extends a pretrained DiT with explicit time editing, allowing control over motion speed and temporal structure without redesigning the backbone. Its core implementation augments the pretrained model with a lightweight temporal module, preserving the original generative prior while expanding its controllable dynamic range.

Chinese Translation

现代视频生成的扩散变换器对时间进程和时间动态的编辑提供了有限的控制。我们提出了一种时间控制方法，该方法扩展了预训练的 DiT（Diffusion Transformer）并实现了显式的时间编辑，允许在不重新设计主干网络的情况下控制运动速度和时间结构。其核心实现通过轻量级时间模块增强预训练模型，保留原始生成先验，同时扩展其可控动态范围。

View on arXiv Download PDF AI Translation

cs.CV / 17 / 2606.10196

Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning

基于Fisher引导的渐进参数选择用于自适应微调

Rostami, Ghodsiyeh, Chen, Po-Han, Hosseini, Mahdi S.

Abstract

Parameter-efficient fine-tuning (PEFT) aims to adapt pretrained models with a small trainable parameter subset, however, most existing methods choose this subset from fixed architectural heuristics rather than using dynamic, task-aware criteria. We introduce \textbf{FisherAdapTune}, a Fisher-guided Adaptive Fine-Tuning framework that progressively selects parameter groups by tracking the temporal drift of their Fisher geometry. Starting from a PAC-Bayesian view of fine-tuning, we decompose the generalization error bound into Fisher-weighted update costs and show that parameter groups whose curvature contribution has stabilized can be frozen to reduce the error bound without interrupting the remaining adaptation dynamics. FisherAdapTune formulates this criterion with a scale-invariant Jensen-Shannon distance between consecutive Fisher distributions, yielding an adaptive active parameter set. We evaluate our approach on a downstream segmentation task, and results show FisherAdapTune improves the in-distribution performance and zero-shot transfer in multiple settings, validating that Fisher structural drift is a useful signal for efficient, task-aware adaptation. We release our \href{https://github.com/AtlasAnalyticsLab/FisherAdapTune}{code} publicly to enable further application of our proposed approach.

Chinese Translation

参数高效微调（PEFT）旨在通过少量可训练参数子集来适应预训练模型，然而，大多数现有方法是从固定的架构启发式中选择这一子集，而不是使用动态的、任务感知的标准。我们提出了 extbf{FisherAdapTune}，一个基于Fisher引导的自适应微调框架，通过跟踪参数组的Fisher几何的时间漂移，逐步选择参数组。基于PAC-Bayesian的微调视角，我们将泛化误差界限分解为Fisher加权的更新成本，并表明那些曲率贡献已稳定的参数组可以被冻结，以减少误差界限，而不干扰其余的适应动态。FisherAdapTune使用连续Fisher分布之间的尺度不变的Jensen-Shannon距离来公式化这一标准，从而产生一个自适应的活跃参数集。我们在下游分割任务上评估了我们的方法，结果表明FisherAdapTune在多个设置中提高了分布内性能和零样本迁移，验证了Fisher结构漂移是高效、任务感知适应的有用信号。我们公开发布了我们的 exthref{https://github.com/AtlasAnalyticsLab/FisherAdapTune}{代码}，以便进一步应用我们提出的方法。

View on arXiv Download PDF AI Translation

cs.CV / 18 / 2606.10200

An Improved Generative Adversarial Network for Micro-Resistivity Imaging Logging Restoration

一种改进的生成对抗网络用于微电阻率成像测井图像恢复

Haque, Ahmed Faizul, Antu, S. M. Riaz Rahman, Ahmed, Saif, Galib, Asadullah Hil, Pramanik, Souvik, Khan, Mohammad Ashrafuzzaman, Qayum, Mohammad Abdul, Sajjad, Mohsin

Abstract

An improved GAN-based imaging logging image restoration method is presented in this paper for solving the problem of partially missing micro-resistivity imaging logging images. The method uses FCN as the generative network infrastructure and adds a depth-separable convolutional residual block to learn and retain more effective pixel and semantic information; an Inception module is added to increase the multi-scale perceptual field of the network and reduce the number of parameters in the network; and a multi-scale feature extraction module and a spatial attention residual block are added to combine the channel attention. The multi-scale module adds a multi-scale feature extraction module and a spatial attention residual block, which combine the channel attention mechanism and the residual block to achieve multi-scale feature extraction. The global discriminative network and the local discriminative network are designed to gradually improve the content and semantic structure coherence between the restored parts and the whole image by playing off each other and the generative network. According to the experimental results, the average structural similarity measure of the five sets of imaged logging images with different sizes of missing regions in the test set is 0.903, which is an improvement of about 0.3 compared with other similar methods. It is shown that the method in this study can be used for the restoration of micro-resistivity imaging log images with good improvement in semantic structural coherence and texture details, thus providing a new deep learning method to ensure the smooth advancement of the subsequent interpretation of micro-resistivity imaging log images.

Chinese Translation

本文提出了一种基于改进生成对抗网络（GAN）的成像测井图像恢复方法，以解决部分缺失的微电阻率成像测井图像的问题。该方法使用全卷积网络（FCN）作为生成网络基础，并添加深度可分离卷积残差块，以学习和保留更有效的像素和语义信息；同时增加了Inception模块，以提高网络的多尺度感知能力并减少网络中的参数数量；此外，添加了多尺度特征提取模块和空间注意力残差块，以结合通道注意力机制。多尺度模块通过结合通道注意力机制和残差块，实现多尺度特征提取。设计了全局判别网络和局部判别网络，通过相互作用和生成网络逐步提高恢复部分与整个图像之间的内容和语义结构一致性。根据实验结果，测试集中五组不同缺失区域大小的成像测井图像的平均结构相似性度量为0.903，相较于其他类似方法提高了约0.3。研究表明，本研究的方法能够有效恢复微电阻率成像测井图像，在语义结构一致性和纹理细节方面有良好改善，从而为后续微电阻率成像测井图像的解释提供了一种新的深度学习方法。

View on arXiv Download PDF AI Translation

cs.CV / 19 / 2606.10275

FoA-SR: Faithful or Aesthetic? Profile-Aware Preference Optimization for Real-World Image Super-Resolution

FoA-SR：忠实还是美学？面向特征的偏好优化用于现实世界图像超分辨率

Alqarni, Amjad Mahdi, Ju, Peizhong

Abstract

Real-world image super-resolution (SR) is often designed with a single restoration objective, despite the current capacity of generative models to produce multiple high-quality reconstructions for the same input. In this paper, we argue that the best restoration strategy is subject to the specific restoration profile: a Faithful restoration prioritizes reference consistency, structure preservation, and hallucination suppression, whereas an Aesthetic restoration prioritizes visually pleasing and natural-looking details. We propose FoA-SR, a novel preference optimization approach to real-world SR based on profiles. To achieve this goal, FoA-SR starts with our supervised FLUX.2-based SR adapter (Flux2SR) trained with LR latent conditioning, flow matching, and image-space reconstruction losses for paired LR-to-HR image super-resolution. Following the development of the shared supervised super-resolution adapter, FoA-SR generates a shared stochastic candidate pool for each input image and ranks the same candidates using profile-specific Faithful and Aesthetic rewards to mine winner-loser pairs. These pairs are used to fine-tune separate LoRA adapters while keeping the base model frozen. Experiments on RealSR and DIV2K show that FoA-SR can steer the same SR adapter towards distinct restoration objectives: a Faithful adapter improves reference-consistent metrics while an Aesthetic adapter boosts metrics that measure perceptual quality without reference. Our candidate-pool analysis shows that Faithful and Aesthetic rewards frequently select different winners, and a Hybrid-LoRA ablation shows that collapsing both profiles into one reward yields an implicit compromise rather than explicit profile control.

Chinese Translation

现实世界图像超分辨率（SR）通常以单一的恢复目标进行设计，尽管当前生成模型能够为同一输入生成多种高质量重建。在本文中，我们认为最佳的恢复策略取决于特定的恢复特征：忠实恢复优先考虑参考一致性、结构保留和幻觉抑制，而美学恢复则优先考虑视觉上令人愉悦和自然的细节。我们提出了FoA-SR，这是一种基于特征的现实世界超分辨率的新型偏好优化方法。为了实现这一目标，FoA-SR以我们的监督FLUX.2基础的SR适配器（Flux2SR）为起点，该适配器经过低分辨率（LR）潜在条件、流匹配和图像空间重建损失的训练，以实现配对的LR到高分辨率（HR）图像超分辨率。在开发共享的监督超分辨率适配器后，FoA-SR为每个输入图像生成一个共享的随机候选池，并使用特征特定的忠实和美学奖励对相同候选进行排名，以挖掘胜者-败者对。这些对用于微调独立的LoRA适配器，同时保持基础模型不变。在RealSR和DIV2K上的实验表明，FoA-SR能够引导相同的SR适配器朝向不同的恢复目标：忠实适配器改善了参考一致性指标，而美学适配器提升了无参考的感知质量指标。我们的候选池分析表明，忠实和美学奖励经常选择不同的胜者，而混合LoRA消融实验显示，将两个特征合并为一个奖励会导致隐式妥协，而非显式的特征控制。

View on arXiv Download PDF AI Translation

cs.CV / 20 / 2606.10309

Dissect and Prune: Enhancing Robustness in AI-Generated Image Detection

剖析与修剪：增强AI生成图像检测的鲁棒性

Kim, Dahye, Choi, Jaehyun, Seong, Hyun Seok, Kim, Seongho, Lee, Donghun, Yi, Sungwon, Choi, Jang-Ho

Abstract

While existing AI-generated image detectors report high performance, we identify that this is largely driven by a critical prediction asymmetry: a bias toward the real class that severely limits sensitivity to generated content, especially under standard post-processing operations such as compression and resizing. We hypothesize that this stems from the model's reliance on spurious features, distracting signals that obscure true generative artifacts. To address this, we propose DEAR (Dissect and Prune), which leverages inpainted images to identify and prune these interfering components. Specifically, we find that features strongly aligned to either inpainted or non-inpainted regions are less robust to post-processing. By measuring the alignment between channel activations and inpaint masks, DEAR removes features at both extremes, retaining only those that capture genuine generative artifacts. Experimental results demonstrate that our approach significantly enhances robustness against unseen generators and post-processing, effectively mitigating the prediction asymmetry. Our code is available at https://github.com/dahyedahye/dear.

Chinese Translation

尽管现有的AI生成图像检测器报告了高性能，但我们发现这在很大程度上是由一种关键的预测不对称性驱动的：对真实类别的偏向严重限制了对生成内容的敏感性，尤其是在标准后处理操作（如压缩和调整大小）下。我们假设这源于模型对虚假特征的依赖，这些干扰信号掩盖了真实的生成伪影。为了解决这个问题，我们提出了DEAR（Dissect and Prune），该方法利用修复图像来识别和修剪这些干扰成分。具体而言，我们发现与修复区域或非修复区域强相关的特征在后处理下的鲁棒性较差。通过测量通道激活与修复掩码之间的对齐程度，DEAR去除两端的特征，仅保留那些捕捉真实生成伪影的特征。实验结果表明，我们的方法显著增强了对未见生成器和后处理的鲁棒性，有效缓解了预测不对称性。我们的代码可在 https://github.com/dahyedahye/dear 获取。

View on arXiv Download PDF AI Translation

cs.CV / 21 / 2606.10328

Content-Induced Spatial-Spectral Aggregation Network for Change Detection in Remote Sensing Images

基于内容诱导的空间-光谱聚合网络用于遥感图像变化检测

Liu, Yunlong, Zhang, Zekai

Abstract

The integration of spatial and spectral information is beneficial to the improvement of change detection performance. However, existing methods cannot efficiently suppress the influences of spatial and spectral differences in unchanged areas. To address these issues, in this paper we propose a content-guided spatial-spectral integration network (CSI-Net) for the fusion of global spatial details and spectral difference information. Specifically, the proposed CSI-Net is composed of a spatial reasoning (SR) module, a spectral difference (SD) module, and a content-guided integration (CGI) module. In the SR module, the spatial information is learned by cascaded graph convolution blocks for global modeling. The SD module is responsible for the extraction of spectral features, by calculating the means and variances of features to reduce the impact of spectral differences in unchanged regions. In addition, in order to integrate the spatial-spectral features efficiently, we design a CGI module to further take advantage of their complementary information. In this module, high-level content information is introduced as a guide for a proper interaction. Due to the efficient spatial-spectral fusion, the proposed CSI-Net can learn the changed features better while achieving a suppression of spectral differences. Experimental results on LEVIR-CD, WHU-CD, and CLCD datasets demonstrate that the proposed CSI-Net produces better performance compared to state-of-the-art methods, and is applicable to different scenarios

Chinese Translation

空间与光谱信息的融合有助于提高变化检测的性能。然而，现有方法无法有效抑制未变化区域中空间和光谱差异的影响。为了解决这些问题，本文提出了一种基于内容引导的空间-光谱融合网络（CSI-Net），用于全球空间细节和光谱差异信息的融合。具体而言，所提出的CSI-Net由空间推理（SR）模块、光谱差异（SD）模块和内容引导融合（CGI）模块组成。在SR模块中，通过级联图卷积块学习空间信息以进行全局建模。SD模块负责光谱特征的提取，通过计算特征的均值和方差来减少未变化区域中光谱差异的影响。此外，为了有效整合空间-光谱特征，我们设计了CGI模块，以进一步利用它们的互补信息。在该模块中，引入高级内容信息作为适当交互的指导。由于高效的空间-光谱融合，所提出的CSI-Net能够更好地学习变化特征，同时抑制光谱差异。在LEVIR-CD、WHU-CD和CLCD数据集上的实验结果表明，所提出的CSI-Net相较于最先进的方法表现出更好的性能，并适用于不同场景。

View on arXiv Download PDF AI Translation

cs.CV / 22 / 2606.10329

Building Change Detection in Earthquake: A Multi-Scale Interaction Network and A Change Detection Dataset

地震中的建筑变化检测：多尺度交互网络及变化检测数据集

Liu, Yunlong, Zhang, Zekai

Abstract

As one of the most destructive natural disasters, earthquakes have struck many countries around the world in recent years, causing serious economic losses. Change detection (CD) can be applied to post-earthquake damage assessment as it can infer destroyed change regions from multi-temporal remote sensing images. Furthermore, the CD with short imaging interval will better satisfy the needs of the emergency rescues after earthquakes. However, the capability of current methods built on deep neural networks is limited because the dataset with short imaging interval is absent. To meet post-disaster immediate relief, we create a CD dataset, Turkey earthquake CD dataset (TUE-CD), for the evaluation of building damage in the short term after an earthquake. Because of the short acquisition interval of the post-event images, the imaging angle is different for different temporal images, which leads to some side-looking problems. To deal with these challenges, we present a multi-scale feature interaction network (MSI-Net) for efficient interaction between bi-temporal features, as well as mitigating the effect of side-looking problems. Specifically, the proposed MSI-Net consists of joint cross-attention (JCA) modules, multi-scale offset calibration (MOC) modules, and feature integration (FeI) modules. The JCA module unifies channel cross-attention and spatial joint attention for sufficient feature interaction. The MOC module further estimates the offsets to align the bi-temporal image with the multi-scale features. Finally, calibrated features and multi-scale features are fused by FeI modules for the prediction of changed areas. Experiments on the WHU-CD, CLCD, and the constructed TUE-CD dataset indicate that the proposed MSI-Net provides better results than considered state-of-the-art CD methods.

Chinese Translation

作为最具破坏性的自然灾害之一，地震近年来在全球多个国家造成了严重的经济损失。变化检测（Change Detection, CD）可以应用于地震后的损害评估，因为它能够从多时相遥感图像中推断出受损变化区域。此外，短时间间隔的变化检测能够更好地满足地震后紧急救援的需求。然而，目前基于深度神经网络的方法能力有限，因为缺乏短时间间隔的数据集。为了满足灾后即时救援的需求，我们创建了一个变化检测数据集，即土耳其地震变化检测数据集（Turkey earthquake CD dataset, TUE-CD），用于评估地震后短期内的建筑损害。由于事件后图像的获取间隔较短，不同时间的图像成像角度不同，这导致了一些侧视问题。为了解决这些挑战，我们提出了一种多尺度特征交互网络（Multi-Scale Feature Interaction Network, MSI-Net），以实现双时相特征之间的高效交互，并减轻侧视问题的影响。具体而言，所提出的MSI-Net由联合交叉注意力（Joint Cross-Attention, JCA）模块、多尺度偏移校准（Multi-Scale Offset Calibration, MOC）模块和特征融合（Feature Integration, FeI）模块组成。JCA模块统一了通道交叉注意力和空间联合注意力，以实现充分的特征交互。MOC模块进一步估计偏移量，以对齐双时相图像与多尺度特征。最后，通过FeI模块将校准后的特征和多尺度特征融合，以预测变化区域。在WHU-CD、CLCD和构建的TUE-CD数据集上的实验表明，所提出的MSI-Net提供了比现有最先进的CD方法更好的结果。

View on arXiv Download PDF AI Translation

cs.CV / 23 / 2606.10350

Multi-Angular Reflectance Anisotropy Observed from UAV Multispectral Imagery

从无人机多光谱影像中观察到的多角度反射各向异性

Qin, Zhenqiang, Dai, Chenguang, Wang, Min, Li, Xian

Abstract

UAV multispectral imagery naturally contains multi-angular observations due to low flight altitude and wide field-of-view imaging, which may introduce geometry-driven radiometric variability. This study proposes a geometry-aware multi-angular observation extraction workflow to quantify observation-geometry effects from a BRDF perspective. Specifically, camera intrinsics and extrinsics are refined via structure-from-motion (SFM), and homogeneous regions annotated on an orthomosaic are reprojected onto multiple raw sub-images acquired from different viewpoints. This enables joint extraction of multi-band reflectance and observation geometry parameters for the same ground targets under varying viewing directions. The extracted observations are further analyzed using band-wise polar visualization in the (VZA, RAA) domain. Results on a grassland target show clear reflectance anisotropy across ten bands, with red-edge and nearinfrared bands exhibiting 119-137% variability between maximum and minimum reflectance, indicating non-negligible observation-geometry effects on radiometric consistency.

Chinese Translation

无人机多光谱影像由于低飞行高度和广视场成像，自然包含多角度观测，这可能引入几何驱动的辐射度变化。本研究提出了一种基于几何的多角度观测提取工作流程，以从BRDF（双向反射分布函数）角度量化观测几何效应。具体而言，通过运动结构（SFM）方法对相机内外参数进行精细化，并将标注在正射影像上的均匀区域重新投影到从不同视点获取的多个原始子图像上。这使得能够在不同观察方向下，对相同地面目标的多波段反射率和观测几何参数进行联合提取。提取的观测结果进一步通过(VZA, RAA)域中的波段极性可视化进行分析。针对草地目标的结果显示，在十个波段中存在明显的反射各向异性，红边和近红外波段的最大和最小反射率之间的变化率达到119-137%，表明观测几何效应对辐射一致性具有不可忽视的影响。

View on arXiv Download PDF AI Translation

cs.CV / 24 / 2606.10364

Benchmarking stereo reconstruction for 3D printable Martian terrain models

火星可打印地形模型的立体重建基准测试

Wang, Josephine

Abstract

Reconstructing printable 3D models from Mars rover imagery is challenging because Martian terrain is low-texture, irregular, and partially observed. We evaluate a pipeline that estimates stereo depth from NASA Curiosity images, completes geometry, and exports watertight OBJ meshes. On Middlebury, RAFT-Stereo outperforms semi-global block matching (SGBM), reducing disparity MAE from 3.22px to 0.73px and increasing valid prediction coverage from 76.3% to 100.0%. On Curiosity imagery, however, RAFT's denser disparities show weaker edge alignment and higher photometric reprojection error, suggesting that benchmark accuracy does not directly transfer to Martian terrain reconstruction. Geometry completion demonstrates a tradeoff between local fidelity and global connectivity. We find that alpha shapes preserve accurate but fragmented structure, Poisson reconstruction produces more coherent meshes but adds unsupported surfaces, and a deterministic diffusion-fill baseline is intermediate but sensitive to stereo quality. Overall, standard stereo and completion methods can produce printable approximations of Martian terrain, but reliable reconstruction requires stronger domain-specific validation.

Chinese Translation

从火星探测器图像重建可打印的3D模型具有挑战性，因为火星地形低纹理、不规则且部分可见。我们评估了一种管道，该管道从NASA好奇号（Curiosity）图像中估计立体深度，完成几何形状，并导出密闭的OBJ网格。在Middlebury数据集上，RAFT-Stereo的表现优于半全局块匹配（SGBM），使得视差的平均绝对误差（MAE）从3.22像素降低到0.73像素，有效预测覆盖率从76.3%提高到100.0%。然而，在好奇号图像上，RAFT的更密集视差显示出较弱的边缘对齐和更高的光度重投影误差，这表明基准精度并不能直接转移到火星地形重建上。几何补全展示了局部保真度与全局连通性之间的权衡。我们发现，α形状（alpha shapes）保留了准确但支离破碎的结构，泊松重建（Poisson reconstruction）生成了更连贯的网格，但增加了不支持的表面，而确定性扩散填充基线则处于中间状态，但对立体质量敏感。总体而言，标准的立体和补全方法可以生成火星地形的可打印近似，但可靠的重建需要更强的领域特定验证。

View on arXiv Download PDF AI Translation

cs.CV / 25 / 2606.10372

ClinReadNet: A clinical reading-inspired network for low-dose abdominal CT image quality assessment

ClinReadNet：一种受临床阅读启发的低剂量腹部CT图像质量评估网络

Xiao, Xianye, Zou, Yulong, Luo, Yujie, Yu, Taihui, Zheng, Cun-Jing, Geng, Yuan-ming, Wang, Shuihua, Zhang, Yudong, Hong, Jin

Abstract

In abdominal CT imaging, developing a low-dose, no-reference image quality assessment (No-reference IQA) model that mimics doctors' reading habits for evaluating CT image quality has significant practical value. This paper proposes a novel deep learning-based framework, ClinReadNet, whose design aligns with the clinical reading logic of radiologists: first, it introduces the Sobel ordinal quality network (SOQN) module, which can simultaneously focus on edge details highly relevant to image quality and the quality distribution pattern of the entire image, accurately matching the clinical image-reading judgment habit of "considering both local details and overall context"; second, the framework integrates the (shifted) window multi-scale temperature multi-head self-attention ((S)W-MTMSA) module, which further replicates the radiologists' image-reading process of shifting from overall scanning to local focusing, and accurately locks in regions of interest through multi-sharpness attention; third, it designs the hierarchical ranked probability score (HRPS) loss function, which combines the dual logics of coarse classification and fine classification, while paying attention to the distance information between grading labels, effectively improving the performance of image quality assessment. Experiments conducted on the LDCTIQAG2023 dataset show that the proposed method achieves the current state-of-the-art (SOTA) performance: the values of Pearson's linear correlation coefficient (PLCC), Spearman's rank-order correlation coefficient (SROCC), and Kendall's rank-order correlation coefficient (KROCC) reach 0.9507, 0.9554, and 0.8629 respectively, with the sum of their absolute values (Score) being 2.7690, outperforming existing methods.

Chinese Translation

在腹部CT成像中，开发一种低剂量、无参考的图像质量评估（No-reference IQA）模型，以模拟医生的阅读习惯来评估CT图像质量，具有重要的实际价值。本文提出了一种新颖的基于深度学习的框架ClinReadNet，其设计与放射科医师的临床阅读逻辑相一致：首先，引入了Sobel序数质量网络（SOQN）模块，该模块能够同时关注与图像质量高度相关的边缘细节和整个图像的质量分布模式，准确匹配临床图像阅读判断习惯，即“兼顾局部细节与整体背景”；其次，框架集成了（偏移）窗口多尺度温度多头自注意力（(S)W-MTMSA）模块，进一步复制放射科医师从整体扫描到局部聚焦的图像阅读过程，并通过多锐度注意力准确锁定感兴趣区域；第三，设计了分层排名概率评分（HRPS）损失函数，该函数结合了粗分类和细分类的双重逻辑，同时关注分级标签之间的距离信息，有效提高了图像质量评估的性能。在LDCTIQAG2023数据集上进行的实验表明，所提方法达到了当前的最先进水平（SOTA）：Pearson线性相关系数（PLCC）、Spearman等级相关系数（SROCC）和Kendall等级相关系数（KROCC）的值分别达到0.9507、0.9554和0.8629，其绝对值之和（Score）为2.7690，超越了现有方法。

View on arXiv Download PDF AI Translation

cs.CV / 26 / 2606.10373

PF-Trans: Physics-Embedded Frequency-Aware Transformer for Spectral Reconstruction

PF-Trans：物理嵌入频率感知变换器用于光谱重建

Gui, Yuzhe, Liu, Tianzhu, Gu, Yanfeng, Li, Xian

Abstract

Snapshot Broadband Filter Array (BFA) imaging provides high light throughput for spectral reconstruction but introduces severe spectral aliasing due to complex modulation. Current deep learning approaches, limited to spatial denoising, often fail to address the global frequency-specific degradations caused by the mask structure. To address this, we propose a Physics-embedded Frequency-aware Transformer (PF-Trans) for high-fidelity remote sensing spectral reconstruction. Our method explicitly integrates the physical sensing model through mask injection and a gray-scale consistency loss to ensure physical fidelity. Furthermore, we introduce a Dual-domain Block with a parallel Fast Fourier Transform (FFT) branch, enabling the network to perceive and suppress aliasing artifacts in the frequency domain. Extensive experiments on multiple datasets demonstrate that PF-Trans achieves state-of-the-art performance, achieving a Peak Signal-to-Noise Ratio (PSNR) of up to 48.50 dB on the GF-5 Shanghai dataset, significantly outperforming comparison methods.

Chinese Translation

快照宽带滤波器阵列（BFA）成像为光谱重建提供了高光通量，但由于复杂的调制引入了严重的光谱混叠。目前的深度学习方法仅限于空间去噪，往往无法解决由掩模结构引起的全局频率特定退化。为了解决这个问题，我们提出了一种物理嵌入频率感知变换器（PF-Trans），用于高保真遥感光谱重建。我们的方法通过掩模注入和灰度一致性损失显式整合物理传感模型，以确保物理保真性。此外，我们引入了一个双域块，配备并行快速傅里叶变换（FFT）分支，使网络能够感知并抑制频域中的混叠伪影。在多个数据集上的广泛实验表明，PF-Trans实现了最先进的性能，在GF-5上海数据集上达到了高达48.50 dB的峰值信噪比（PSNR），显著优于对比方法。

View on arXiv Download PDF AI Translation

cs.CV / 27 / 2606.10378

FSS-Net: Frequency-Spatial Synergy Network with Wavelet Attention for Carotid Artery Ultrasound Segmentation

FSS-Net：具有小波注意力的频率-空间协同网络用于颈动脉超声分割

Liu, Jiawei, Wan, Zhijiang, Hu, Junhua, Zhang, Rongli, Xu, Zhongbiao, Cao, Yankun, Chen, Yuan, Hong, Jin

Abstract

Accurate segmentation of carotid arteries in ultrasound imaging is critical for stroke risk assessment. However, speckle noise, low contrast, and blurred boundaries remain major challenges. In this paper, we propose a Frequency-Spatial Synergy Network (FSS-Net) to achieve noise-robust and high-precision carotid artery segmentation. The network integrates wavelet transform, multi-domain attention, and edge enhancement into a unified encoder-decoder architecture. Specifically, a Channel-Spatial-Wavelet Attention (CSWA) module is designed to suppress noise and purify semantic features in the frequency domain. A Wavelet-Enhanced Bottleneck (WEB) module is introduced to capture long-range global dependencies efficiently. Furthermore, a Laplacian-Guided Adaptive Edge Fusion (LAEF) module compensates high-frequency details and maintains boundary continuity. Extensive experiments on carotid ultrasound datasets show that FSS-Net achieves a Dice score (DSC) of 96.46% and strong robustness under low SNR conditions, outperforming several state-of-the-art methods. This method realizes accurate segmentation of carotid artery in ultrasonic imaging, effectively identifies carotid atherosclerotic plaque, and is verified by other task (such as segmentation of breast cancer), suggesting that it has good clinical application potential in identifying abnormal tissue masses in ultrasonic images.

Chinese Translation

在超声成像中，准确分割颈动脉对中风风险评估至关重要。然而，斑点噪声、低对比度和模糊边界仍然是主要挑战。本文提出了一种频率-空间协同网络（FSS-Net），以实现抗噪声和高精度的颈动脉分割。该网络将小波变换、多域注意力和边缘增强集成到一个统一的编码-解码架构中。具体而言，设计了一个通道-空间-小波注意力（CSWA）模块，以抑制噪声并净化频域中的语义特征。引入了一个小波增强瓶颈（WEB）模块，以高效捕获长距离全局依赖。此外，拉普拉斯引导自适应边缘融合（LAEF）模块补偿高频细节并保持边界连续性。在颈动脉超声数据集上的大量实验表明，FSS-Net在低信噪比条件下实现了96.46%的Dice系数（DSC）和强大的鲁棒性，超越了几种最先进的方法。该方法实现了超声成像中颈动脉的准确分割，有效识别颈动脉粥样硬化斑块，并通过其他任务（如乳腺癌分割）进行了验证，表明其在超声图像中识别异常组织肿块方面具有良好的临床应用潜力。

View on arXiv Download PDF AI Translation

cs.CV / 28 / 2606.10395

Efficient RWKV-based Representation Learning for 3D Point Clouds

基于RWKV的高效3D点云表示学习

Liu, Yun, Yan, Xuefeng, Nan, Liangliang, Li, Xianzhi, Li, Peng, Zhu, Zhe, Chen, Honghua, Wei, Mingqiang

Abstract

The recent receptance weighted key value (RWKV) model combines RNN-style recurrence, offering a linear-complexity alternative to Transformers' quadratic self-attention for modeling global dependencies. However, when directly applied to point clouds, RWKV, originally developed for sequential text, struggles to capture local geometric structures and model spatial dependencies effectively. To address this, we propose the \textbf{P-RWKV} block, which bridges the gap between sequence modeling and irregular 3D geometry while preserving the efficiency advantages of RWKV. It consists of a Local Perception Expansion (LPE) component to expand contextual perception along the spatio-temporal sequence and a Spatial Context Enhancement (SCE) component to strengthen spatial awareness. To validate the effectiveness of P-RWKV for point cloud understanding, we construct PointER, a single-modality self-supervised representation learning framework whose encoder is composed of stacked P-RWKV blocks. Furthermore, we extend P-RWKV to a cross-modality setting and integrate the proposed core sub-modules into multiple architectures, demonstrating strong plug-and-play flexibility and architectural generality. Extensive experiments show that the P-RWKV block and its key sub-modules achieve competitive performance across various tasks with lower computational cost and inference latency. Code will be released upon acceptance.

Chinese Translation

最近的接收加权关键值（RWKV）模型结合了RNN风格的递归，为建模全局依赖关系提供了一种线性复杂度的替代方案，取代了Transformers的二次自注意力。然而，当RWKV直接应用于点云时，这一最初为序列文本开发的模型在捕捉局部几何结构和有效建模空间依赖性方面面临挑战。为了解决这一问题，我们提出了 extbf{P-RWKV}模块，它在序列建模与不规则3D几何之间架起了桥梁，同时保留了RWKV的效率优势。该模块由局部感知扩展（Local Perception Expansion, LPE）组件组成，用于沿时空序列扩展上下文感知，以及空间上下文增强（Spatial Context Enhancement, SCE）组件，用于增强空间意识。为了验证P-RWKV在点云理解中的有效性，我们构建了PointER，这是一个单模态自监督表示学习框架，其编码器由堆叠的P-RWKV模块组成。此外，我们将P-RWKV扩展到跨模态设置，并将所提出的核心子模块集成到多个架构中，展示了强大的即插即用灵活性和架构通用性。大量实验表明，P-RWKV模块及其关键子模块在各种任务中以更低的计算成本和推理延迟实现了竞争性能。代码将在接受后发布。

View on arXiv Download PDF AI Translation

cs.CV / 29 / 2606.10401

CoCoSI: Collaborative Cognitive Map Construction for Spatial Intelligence

CoCoSI：用于空间智能的协作认知地图构建

Zhang, Yiming, Cao, Ruoxuan, Zhong, Zhihang

Abstract

Spatial intelligence is a key frontier for multimodal large language models (MLLMs), enabling them to reason about the physical world from visual experience. Inspired by human spatial cognition, recent approaches construct grid-based cognitive maps from multi-frame visual inputs to maintain coherent spatial representations over time. However, limited context lengths still challenge spatial understanding, while existing methods, such as long-context modeling and external memory, often require architectural changes, memory modules, or finetuning, limiting their applicability to off-the-shelf pretrained MLLMs. This motivates a lightweight, model-agnostic method for preserving spatial information beyond the native context window. To this end, we propose a plug-and-play multi-agent framework that collaboratively constructs cognitive maps as structured spatial memory, enhancing the spatial understanding of arbitrary pretrained MLLMs without architectural modification or additional training. Our framework features local-global agent coordination, cognitive map construction with atomic commits, and cross-agent verification. Extensive experiments demonstrate that our method achieves superior performance on spatial understanding tasks while remaining fully training-free. Code will be released.

Chinese Translation

空间智能是多模态大型语言模型（MLLMs）的一个关键前沿，使其能够从视觉经验中推理物理世界。受人类空间认知的启发，最近的方法通过多帧视觉输入构建基于网格的认知地图，以在时间上保持一致的空间表征。然而，有限的上下文长度仍然对空间理解构成挑战，而现有的方法，如长上下文建模和外部记忆，通常需要架构更改、记忆模块或微调，这限制了它们在现成预训练的 MLLMs 中的适用性。这促使我们提出一种轻量级、模型无关的方法，以在超出原生上下文窗口的情况下保留空间信息。为此，我们提出了一个即插即用的多智能体框架，协同构建作为结构化空间记忆的认知地图，增强任意预训练 MLLMs 的空间理解，而无需架构修改或额外训练。我们的框架具有局部-全局智能体协调、通过原子提交构建认知地图以及跨智能体验证的特点。大量实验表明，我们的方法在空间理解任务上实现了优越的性能，同时保持完全无训练。代码将会发布。

View on arXiv Download PDF AI Translation

cs.CV / 30 / 2606.10431

Vision-Assisted Foundation Model for Solving Multi-Task Vehicle Routing Problems

视觉辅助基础模型用于解决多任务车辆路径问题

Gui, Shuangchun, Cao, Zhiguang, Song, Wen, Ong, Yew-Soon

Abstract

Multi-task vehicle routing problems play a critical role in enhancing efficiency across various industries and service sectors. These problems consist of multiple variants that optimize routing costs while meeting diverse customer constraints. Existing multi-task VRP solvers solely utilize a graph-based modality, limiting their ability to address variants with multiple constraints. As a format to represent complex semantics, vision modality shows great potential for encoding diverse VRP constraints. This motivates us to learn patch-level semantics from the vision images, and then integrate them into a graph-based model to solve various VRP variants simultaneously. However, directly applying this approach to multi-task VRPs presents three challenges: 1) existing VRP images lack constraint representations, which are essential for multi-task VRPs, 2) the fixed receptive field of individual patches cannot effectively accommodate varying requirements across tasks, and 3) imbalanced pixel distribution among constraints may cause the model to overlook constraints with fewer pixels. In this paper, we propose a vision-assisted foundation model (VaFM) to address these challenges. In the vision modality, input images tailored to all constraints are encoded by a convolutional neural network. The obtained patch embeddings are fused with graph-based nodes to generate solutions, with an auxiliary task designed to address the pixel-imbalanced issue. The performance of VaFM is evaluated across 16 different VRP variants. The experimental results demonstrate the superiority of VaFM over state-of-the-art methods, especially for variants with complex constraints.

Chinese Translation

多任务车辆路径问题在提升各行业和服务领域的效率方面发挥着关键作用。这些问题由多个变体组成，旨在优化路径成本，同时满足多样化的客户约束。现有的多任务车辆路径问题求解器仅利用基于图的模式，这限制了它们处理具有多重约束的变体的能力。作为表示复杂语义的一种形式，视觉模式在编码多样化的车辆路径问题约束方面显示出巨大的潜力。这激励我们从视觉图像中学习补丁级别的语义，然后将其整合到基于图的模型中，以同时解决各种车辆路径问题变体。然而，直接将这种方法应用于多任务车辆路径问题面临三个挑战：1）现有的车辆路径问题图像缺乏约束表示，而这些表示对于多任务车辆路径问题至关重要；2）单个补丁的固定感受野无法有效适应跨任务的不同需求；3）约束之间的不平衡像素分布可能导致模型忽视像素较少的约束。在本文中，我们提出了一种视觉辅助基础模型（VaFM）来应对这些挑战。在视觉模式中，针对所有约束定制的输入图像通过卷积神经网络进行编码。获得的补丁嵌入与基于图的节点融合以生成解决方案，并设计了一个辅助任务以解决像素不平衡问题。我们在16种不同的车辆路径问题变体上评估了VaFM的性能。实验结果表明，VaFM在与最先进的方法相比时表现出优越性，尤其是在具有复杂约束的变体中。

View on arXiv Download PDF AI Translation

cs.CV / 31 / 2606.10450

Few-step Generative Models as Lossy Compression

少步生成模型作为有损压缩

Kimishima, Fuma, Zhou, Jinjia

Abstract

DiffC provides a principled way to reuse pre-trained diffusion models for lossy compression, but its encoding and decoding procedures remain slow because they require many discretized forward and reverse steps. We study whether few-step generative models -- Rectified Flow, Consistency Trajectory Models (CTM), and MeanFlow -- can be cast as codecs within the same reverse channel coding (RCC) framework. The main challenge is that RCC requires posterior and shared distribution parameters, whereas these models do not explicitly parameterize intermediate conditional distributions. For Rectified Flow and MeanFlow, we use the equivalence between velocity parameterization and diffusion-style denoising parameterization to derive the quantities required by RCC. For CTM, which is distilled from EDM, we adopt the EDM noise parameterization together with local Gaussian approximations of the sender and shared distributions at intermediate states. This yields a proof-of-concept probabilistic formulation that enables compression with pre-trained few-step generative models without retraining. On low-resolution benchmarks, the resulting codecs reduce encoding and decoding time and improve realism in the low-bit-rate regime.

Chinese Translation

DiffC 提供了一种原则性的方法来重用预训练的扩散模型进行有损压缩，但其编码和解码过程仍然较慢，因为需要多个离散的前向和反向步骤。我们研究少步生成模型——修正流（Rectified Flow）、一致性轨迹模型（Consistency Trajectory Models, CTM）和均值流（MeanFlow）——是否可以在相同的反向信道编码（Reverse Channel Coding, RCC）框架内被视为编解码器。主要挑战在于 RCC 需要后验和共享分布参数，而这些模型并未明确参数化中间条件分布。对于修正流和均值流，我们利用速度参数化与扩散风格去噪参数化之间的等价性，推导出 RCC 所需的量。对于从 EDM 中提炼出的 CTM，我们采用 EDM 噪声参数化以及在中间状态下发送者和共享分布的局部高斯近似。这产生了一种概念验证的概率性公式，使得可以在不重新训练的情况下，使用预训练的少步生成模型进行压缩。在低分辨率基准测试中，所得到的编解码器减少了编码和解码时间，并在低比特率范围内提高了真实感。

View on arXiv Download PDF AI Translation

cs.CV / 32 / 2606.10468

Geometric Coastline Localization using Vision-Language Models

基于视觉-语言模型的几何海岸线定位

Malik, Rafia, Pfahringer, Bernhard, Bryan, Karin, Dickson, Mark, Frank, Eibe

Abstract

Coastline detection in remote sensing imagery is commonly formulated as a pixel-wise segmentation problem, where the final coastline is extracted from a predicted mask through post-processing. This formulation relegates coastline geometry, the primary representation used in coastal change analysis, to a secondary artifact rather than the learning objective. In practice, coastlines are defined by geomorphic proxies such as vegetation lines, dune toes, or cliff edges, rather than an instantaneous land-water boundary often used in pixel-based segmentation approaches. In this work, we revisit coastline extraction from a representation perspective and formulate the task as geometric boundary localization. We use the New Zealand Coastal Change Dataset (NZCCD) and high-resolution aerial imagery from Land Information New Zealand (LINZ) to develop CoastlineVLM-7B, a vision-language model (VLM) built on the GeoChat-7B/LLaVA-1.5 architecture that jointly performs coastline presence detection, proxy-type classification, and coastline grounding. The model directly predicts a coastline as a polyline rather than a dense segmentation mask. We evaluate CoastlineVLM-7B against segmentation baselines under strict one-pixel boundary supervision. Results show that geometry-based metrics are more suitable for assessing coastline localization quality than pixel-overlap metrics such as Intersection over Union (IoU). CoastlineVLM-7B improves global geometric alignment with reference coastlines, reducing Hausdorff distance from 37.74 m to 31.84 m and Earth Mover's Distance from 21.12 m to 17.32 m. These results indicate that output representation is a critical design choice in coastline extraction, and that geometry-oriented learning, combined with the semantic reasoning capabilities of vision-language models, aligns well with how coastlines are defined and evaluated in operational coastal monitoring.

Chinese Translation

遥感图像中的海岸线检测通常被表述为像素级分割问题，最终的海岸线通过后处理从预测的掩膜中提取。这种表述将海岸线几何形状——在海岸变化分析中使用的主要表示—— relegated为次要的产物，而非学习目标。在实践中，海岸线是通过地貌代理（如植被线、沙丘底部或悬崖边缘）来定义的，而不是像素级分割方法中常用的瞬时陆水边界。在本研究中，我们从表示的角度重新审视海岸线提取，并将任务表述为几何边界定位。我们使用新西兰海岸变化数据集（NZCCD）和新西兰土地信息局（LINZ）的高分辨率航空影像，开发了CoastlineVLM-7B，这是一个基于GeoChat-7B/LLaVA-1.5架构的视觉-语言模型（VLM），能够联合执行海岸线存在检测、代理类型分类和海岸线定位。该模型直接将海岸线预测为多线段，而不是密集的分割掩膜。我们在严格的一像素边界监督下评估CoastlineVLM-7B与分割基线的表现。结果表明，基于几何的度量更适合评估海岸线定位质量，而不是像素重叠度量（如交并比IoU）。CoastlineVLM-7B在与参考海岸线的全局几何对齐上有所改善，将Hausdorff距离从37.74米减少到31.84米，将地球搬运者距离从21.12米减少到17.32米。这些结果表明，输出表示是海岸线提取中的一个关键设计选择，而几何导向的学习结合视觉-语言模型的语义推理能力，与海岸线在实际海岸监测中的定义和评估方式高度一致。

View on arXiv Download PDF AI Translation

cs.CV / 33 / 2606.10478

3D-CoS: A New 3D Reconstruction Paradigm Based on VLM Code Synthesis

3D-CoS：基于VLM代码合成的新型3D重建范式

Wang, Yuhao, Wang, Puyi, Li, Linjie, Yang, Zhengyuan, Lin, Kevin Qinghong, Cheng, Yu

Abstract

Most recent 3D reconstruction and editing systems operate on implicit and explicit representations such as NeRF, point clouds, or meshes. While these representations enable high-fidelity rendering, they are fundamentally low-level and hard to control programmatically. In contrast, we propose and systematically evaluate a new 3D reconstruction paradigm, 3D Code Synthesis (3D-CoS), where 3D assets are constructed as executable Blender code, a programmatic and interpretable medium. To assess how well current VLMs can use code to represent 3D objects, we evaluate representative open-source and closed-source VLMs in code-based reconstruction under a unified protocol. We further introduce a suite of structured code-synthesis workflows, including blueprint-based planning, Retrieval-Augmented Generation (RAG) over Blender API documentation, few-shot geometric demonstrations, and a component-level Agent workflow for part-wise code generation. To demonstrate the unique advantages of this representation, we further evaluate localized text-driven modifications and compare our code-based edits with a point-cloud-based 3D editing baseline. Our study shows that code as a 3D representation offers strong controllability and locality, yielding stronger edit fidelity and better preservation of unedited regions in our targeted editing evaluation. Our work also analyzes the potential of this paradigm, delineates the current capability frontier of VLMs for programmatic 3D modeling, and highlights code synthesis as a promising direction for editable 3D reconstruction.

Chinese Translation

最近的3D重建和编辑系统主要基于隐式和显式表示，如NeRF、点云或网格。尽管这些表示能够实现高保真渲染，但它们本质上是低级的，且难以通过编程进行控制。相比之下，我们提出并系统评估了一种新的3D重建范式——3D代码合成（3D-CoS），在该范式中，3D资产被构建为可执行的Blender代码，这是一种程序化且可解释的媒介。为了评估当前的VLM（视觉语言模型）在使用代码表示3D对象方面的能力，我们在统一协议下评估了具有代表性的开源和闭源VLM在基于代码的重建中的表现。我们进一步引入了一套结构化的代码合成工作流程，包括基于蓝图的规划、对Blender API文档的检索增强生成（RAG）、少量示例几何演示，以及用于逐部分代码生成的组件级代理工作流程。为了展示这种表示的独特优势，我们进一步评估了局部文本驱动的修改，并将我们的基于代码的编辑与基于点云的3D编辑基线进行了比较。我们的研究表明，作为3D表示的代码提供了强大的可控性和局部性，能够在我们的目标编辑评估中实现更强的编辑保真度和更好地保留未编辑区域。我们的工作还分析了这一范式的潜力，划定了VLM在程序化3D建模方面的当前能力边界，并强调了代码合成作为可编辑3D重建的一个有前景的方向。

View on arXiv Download PDF AI Translation

cs.CV / 34 / 2606.10488

5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning

5% > 100%：平坦性偏好是多模态参数高效微调所需的一切

Zhu, Yifan, Lin, Can, Yuan, Hangjie, Zhao, Zixiang, Zhang, Pengfei, Feng, Tao, Ou, Zhonghong

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods provide a streamlined and efficient tool for adapting large models to domain-specific multimodal downstream tasks. Although these methods proved their tangible effects in practice, their principal aspects remain under-explored. Therefore we remain curious about the underlying generalization mechanisms in various PEFT methods and how they can be further enhanced. In this paper, we reveal the flatness preference widely present in various PEFTs, where a small fraction of sharp dimensions dominates the generalization of PEFT. This finding suggests an appealing possibility: we may be satisfied with a better generalization by merely attending to this small fraction of sharp dimensions instead of all of them. Furthermore, we propose Flatness Preference Optimization (FlatPO) to flatten these key sharpness dimensions, leading various PEFTs toward better generalization. Extensive experiments demonstrate the effectiveness of our findings and the proposed method. Code is available at https://github.com/Can-Lin/FlatPO.

Chinese Translation

参数高效微调（Parameter-Efficient Fine-Tuning, PEFT）方法为将大型模型适应于特定领域的多模态下游任务提供了一种简化且高效的工具。尽管这些方法在实践中证明了其显著效果，但其主要方面仍然未得到充分探索。因此，我们对各种PEFT方法中潜在的泛化机制以及如何进一步增强这些机制保持好奇。在本文中，我们揭示了在多种PEFT中普遍存在的平坦性偏好，其中少量的尖锐维度主导了PEFT的泛化。这一发现提出了一个引人注目的可能性：我们可能只需关注这一小部分尖锐维度，而不是所有维度，就能获得更好的泛化。此外，我们提出了平坦性偏好优化（Flatness Preference Optimization, FlatPO）方法，以平坦化这些关键的尖锐维度，从而推动各种PEFT实现更好的泛化。大量实验验证了我们的发现和所提方法的有效性。代码可在 https://github.com/Can-Lin/FlatPO 获取。

View on arXiv Download PDF AI Translation

cs.CV / 35 / 2606.10492

PathRelax: Parallel-Path Relaxed Speculative Jacobi Decoding for Accelerating Auto-Regressive Text-to-Image Generation

PathRelax：用于加速自回归文本到图像生成的并行路径放松推测雅可比解码

Lei, Haodong, Wang, Hongsong, Dai, Bingxuan, Zhou, Pan

Abstract

The growing need for high-resolution image generation in autoregressive text-to-image models has resulted in extended token sequences, significantly increasing computational costs and inference times. However, existing state-of-the-art methods for accelerating autoregressive text-to-image models rely on chain-structured draft token sequences, leading to inefficient draft token search and limited acceptance lengths. To address this, we propose parallel-path cross-relaxed speculative Jacobi decoding (\textbf{PathSpec}), a novel framework that enhances efficiency through a multi-sequence draft tree structure. Our parallel-path speculative Jacobi decoding (\textbf{PathExplore}) expands the token search space, achieving a higher speedup ratio without sacrificing image quality. Additionally, we introduce cross-path relaxed verification (\textbf{PathRelax}) that exploits semantic similarities across sequences to further boost token acceptance rates. Evaluated on the Parti-Prompts, MSCOCO2017, and T2ICompBench datasets, our method achieves a speedup ratio of 4.14 $\times$, 3.95$\times$, and 4.18$\times$, respectively. Remarkably, PathExplore, without any relaxed sampling, outperforms relaxed sampling methods in the speedup ratio, such as GSD and LANTERN. Moreover, PathRelax's relaxation mechanism can be seamlessly integrated with other relaxation techniques, enabling further acceleration and providing an efficient solution for real-time text-to-image generation. Our code is available at https://github.com/Haodong-Lei-Ray/PathSpec.

Chinese Translation

自回归文本到图像模型对高分辨率图像生成的日益需求导致了扩展的令牌序列，显著增加了计算成本和推理时间。然而，现有的加速自回归文本到图像模型的最先进方法依赖于链式结构的草稿令牌序列，导致草稿令牌搜索效率低下和接受长度有限。为了解决这个问题，我们提出了并行路径交叉放松推测雅可比解码（PathSpec），这是一个通过多序列草稿树结构提升效率的新框架。我们的并行路径推测雅可比解码（PathExplore）扩展了令牌搜索空间，实现了更高的加速比而不牺牲图像质量。此外，我们引入了交叉路径放松验证（PathRelax），利用序列间的语义相似性进一步提高令牌接受率。在Parti-Prompts、MSCOCO2017和T2ICompBench数据集上的评估显示，我们的方法分别实现了4.14倍、3.95倍和4.18倍的加速比。值得注意的是，PathExplore在没有任何放松采样的情况下，超越了如GSD和LANTERN等放松采样方法的加速比。此外，PathRelax的放松机制可以与其他放松技术无缝集成，从而实现进一步加速，并为实时文本到图像生成提供高效解决方案。我们的代码可在https://github.com/Haodong-Lei-Ray/PathSpec获取。

View on arXiv Download PDF AI Translation

cs.CV / 36 / 2606.10517

LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching

LAFP：通过流匹配在潜在策略学习中保持潜在动作结构

Lyu, Jiexi, Bu, Xizhou, Huang, Qingqiu, Tang, Chufeng, Hao, Xiaoshuai, Wang, Hongbo, Li, Wei

Abstract

Learning high-quality latent actions from large-scale unlabeled videos, coupled with limited real-world interaction data for training an action decoder, has emerged as a promising paradigm for scalable latent policy learning. However, existing approaches typically rely on behavior cloning, which tends to collapse inherently multimodal action distributions into unimodal ones, thereby degrading the pretrained latent action structure. While flow matching provides a potential alternative, directly applying it leads to a misalignment between latent actions and physical actions during action decoder training, due to the stochastic nature of the learned policy. To address these, we propose Latent Action Flow Policy (LAFP), which leverages flow matching for latent policy learning and introduces an inference-time interpolation mechanism to mitigate stochasticity-induced misalignment. Experimental results demonstrate that LAFP consistently outperforms prior methods on downstream imitation learning tasks, achieving up to 10-15% improvement in success rate while incurring less than 1x additional inference overhead.

Chinese Translation

从大规模未标记视频中学习高质量的潜在动作，同时结合有限的真实世界交互数据来训练动作解码器，已成为可扩展潜在策略学习的一个有前景的范式。然而，现有的方法通常依赖于行为克隆，这往往将固有的多模态动作分布压缩为单模态，从而降低了预训练的潜在动作结构。尽管流匹配提供了一种潜在的替代方案，但直接应用它会导致在动作解码器训练过程中潜在动作与物理动作之间的错位，这是由于学习策略的随机性所致。为了解决这些问题，我们提出了潜在动作流策略（Latent Action Flow Policy, LAFP），该策略利用流匹配进行潜在策略学习，并引入了一种推理时插值机制，以减轻随机性引起的错位。实验结果表明，LAFP在下游模仿学习任务中始终优于先前的方法，成功率提高了10-15%，同时额外的推理开销不超过1倍。

View on arXiv Download PDF AI Translation

cs.CV / 37 / 2606.10522

GUI-AC: Enhancing Continual Learning in GUI Agents

GUI-AC：提升图形用户界面代理的持续学习能力

Lin, Can, Feng, Tao, Yuan, Hangjie, Zhang, Dan, Zhu, Yifan, Ou, Zhonghong

Abstract

Graphical User Interfaces (GUIs) serve as the dominant medium for human-computer interaction, yet building GUI agents that generalize across the vast diversity of real-world interface environments, with the same flexibility and robustness that humans naturally exhibit, remains unsolved. Notably, GUI data are inherently non-stationary: the continual emergence of previously unseen interface instances (e.g., novel domains and resolutions) induces persistent distribution shifts, significantly impeding the continual learning of existing GUI agents. Reinforcement fine-tuning (RFT) has attracted considerable attention as a promising approach. Nevertheless, RFT exhibits pronounced instability in its grounding capability, manifested as sharp reward discontinuities and high-variance oscillations. The imbalanced distribution of rollout outcomes introduces substantial noise into advantage estimation, leading to policy overconfidence. The fixed clipping bound suppresses the increase in policy probabilities needed to adapt to new distributions, leading to a collapse in exploration capacity. To address these challenges, we propose GUI-AC, a method that enhances the continual learning capability of GUI agents. GUI-AC introduces grounding certainty to support two core mechanisms: (i) Adaptive Advantage, which down-weights noisy advantage estimates to prevent policy overconfidence; and (ii) Dynamic Clipping, which relaxes the clipping bound to encourage exploration range. Extensive experiments show that these mechanisms jointly improve performance, enabling our method to surpass state-of-the-art baselines. Code is available anonymously at https://anonymous.4open.science/r/GUI-AC.

Chinese Translation

图形用户界面（GUIs）作为人机交互的主要媒介，然而构建能够在广泛多样的现实界面环境中进行泛化的GUI代理，且具备人类自然展现的灵活性和鲁棒性，仍然是一个未解决的问题。值得注意的是，GUI数据本质上是非平稳的：以前未见的界面实例（例如新领域和分辨率）的持续出现引发了持续的分布变化，显著阻碍了现有GUI代理的持续学习。强化微调（Reinforcement Fine-Tuning, RFT）作为一种有前景的方法受到了广泛关注。然而，RFT在其基础能力上表现出明显的不稳定性，表现为奖励的不连续性和高方差的振荡。回放结果的不平衡分布为优势估计引入了显著噪声，导致策略过于自信。固定的裁剪界限抑制了适应新分布所需的策略概率的增加，导致探索能力的崩溃。为了解决这些挑战，我们提出了GUI-AC，一种增强GUI代理持续学习能力的方法。GUI-AC引入了基础确定性以支持两个核心机制：（i）自适应优势（Adaptive Advantage），通过降低噪声优势估计的权重来防止策略过于自信；（ii）动态裁剪（Dynamic Clipping），放宽裁剪界限以鼓励探索范围。大量实验表明，这些机制共同改善了性能，使我们的方法超越了最先进的基线。代码可匿名访问，网址为 https://anonymous.4open.science/r/GUI-AC。

View on arXiv Download PDF AI Translation

cs.CV / 38 / 2606.10533

Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioning

音视频交换感知的令牌剪枝用于高效的音视频字幕生成

Meng, Zihan, Hong, Dexiang, Chen, Weidong, Zhou, Ziyu, Hu, Bo, Mao, Zhendong

Abstract

Audio-visual captioning generates natural language descriptions from video and audio content. Multimodal LLMs have advanced this task, but both modalities contribute many tokens to the LLM input, where prefill self-attention scales quadratically. Existing token-pruning methods usually retain tokens by attention, saliency, or cross-entropy loss, yet the hard threshold selection makes it difficult to retain tokens that are truly valuable, especially for high-confusing tokens near the decision boundary. To this end, we propose a AVEX-Prune, an RL-based audio-visual dynamic token pruning method in this work. In our AVEX-Prune, an audio-visual token exchange strategy is proposed to select truly valuable tokens by replacing low-confidence retained tokens with high-confidence candidate tokens from the same or the other modality, and measuring the differences in caption generation from token swaps. AVEX-Prune preserves full-token quality at a 40% retention ratio on both VILA 1.5-8B (54.5 vs. 54.6) and VideoLLaMA 2 (57.0 vs. 56.8).

Chinese Translation

音视频字幕生成从视频和音频内容中生成自然语言描述。多模态大语言模型（LLMs）推动了这一任务的发展，但两种模态为LLM输入贡献了大量令牌，而预填充自注意力的计算复杂度呈平方级增长。现有的令牌剪枝方法通常通过注意力、显著性或交叉熵损失来保留令牌，但硬阈值选择使得保留真正有价值的令牌变得困难，尤其是对于接近决策边界的高混淆令牌。为此，我们在本研究中提出了一种基于强化学习的音视频动态令牌剪枝方法AVEX-Prune。在我们的AVEX-Prune中，提出了一种音视频令牌交换策略，通过用来自同一模态或另一模态的高置信度候选令牌替换低置信度的保留令牌，从而选择真正有价值的令牌，并测量令牌交换对字幕生成的影响。AVEX-Prune在VILA 1.5-8B（54.5 vs. 54.6）和VideoLLaMA 2（57.0 vs. 56.8）上以40%的保留比例保持了完整令牌的质量。

View on arXiv Download PDF AI Translation

cs.CV / 39 / 2606.10541

GRAR: Glass-induced Reflection Artifact Removal in LiDAR Point Clouds

GRAR：激光雷达点云中的玻璃诱导反射伪影去除

Shao, Wanpeng, Guo, Zeyi, Zhang, Bo, Xue, Yifei, Ji, Tie, Lao, Yizhen

Abstract

Terrestrial Laser Scanning (TLS) point clouds captured in urban environments frequently suffer from glass-induced reflection artifacts, severely degrading downstream applications. Existing reflection artifact removal methods generally rely on ideal reflection symmetry assumptions, yet their performance is limited by inaccurate glass estimation and insufficient geometric representations. To address these issues, we propose a novel unified framework aimed at robust reflection artifact removal: In the first stage, we leverage a multi-modal vision foundation model to produce initial glass masks, which are then refined using geometric cues to achieve high-precision glass regions, followed by glass completion to recover missing regions caused by no-return measurements on transparent surfaces; In the second stage, we propose a physics-driven descriptor, termed Reflection-aware Local-Global Geometric Similarity (RE-LGGS), which is grounded in actual laser reflection geometry and jointly encodes multi-scale geometric structures and orientation consistency using PCA-based local shape representations, thereby significantly improving robustness against imperfect observations. Extensive experiments on multiple public TLS datasets demonstrate that our framework consistently outperforms state-of-the-art methods in reflection artifacts removal.

Chinese Translation

在城市环境中捕获的地面激光扫描（TLS）点云常常受到玻璃诱导的反射伪影的影响，这严重降低了后续应用的效果。现有的反射伪影去除方法通常依赖于理想的反射对称性假设，但其性能受到不准确的玻璃估计和不足的几何表示的限制。为了解决这些问题，我们提出了一种新颖的统一框架，旨在实现稳健的反射伪影去除：在第一阶段，我们利用多模态视觉基础模型生成初始玻璃掩膜，然后使用几何线索对其进行精细化，以实现高精度的玻璃区域，接着进行玻璃补全，以恢复因透明表面上的无返回测量而导致的缺失区域；在第二阶段，我们提出了一种基于物理驱动的描述符，称为反射感知局部-全局几何相似性（Reflection-aware Local-Global Geometric Similarity，RE-LGGS），该描述符基于实际的激光反射几何，联合编码多尺度几何结构和方向一致性，使用基于主成分分析（PCA）的局部形状表示，从而显著提高了对不完美观测的鲁棒性。在多个公共TLS数据集上的广泛实验表明，我们的框架在反射伪影去除方面始终优于最先进的方法。

View on arXiv Download PDF AI Translation

cs.CV / 40 / 2606.10550

PrismAvatar: Pseudo-Multiview Reconstruction and Subpixel Prism Rendering for Real-Time Stereoscopic Communication

PrismAvatar：伪多视图重建与亚像素棱镜渲染用于实时立体通信

Fang, Chufeng, Teng, Dongdong, Liu, Lilin

Abstract

Real-time stereoscopic video communication has long been a goal of immersive telepresence, yet practical systems still require specialized capture rigs or reduce remote users to a single portrait view. We present PrismAvatar, a Gaussian head-avatar system that connects monocular avatar capture with subpixel-encoded glasses-free lenticular display for real-time autostereoscopic communication. From a monocular portrait video, PrismAvatar reconstructs a controllable head avatar and optimizes it for the lateral viewing zones induced by the display. The method uses natural head turns as pseudo-multiview (PMV) supervision to constrain regions that are otherwise weakly observed in monocular training, including hair, ears, jaw contours, and neck boundaries. Reliable side frames are yaw-binned, aligned to virtual cameras, and supervised within a strict head-and-hair domain; contour-aware losses and staged regularization further suppress ghosting, alpha leakage, and depth instability while preserving lateral detail. At runtime, PrismAvatar renders 32 virtual views and encodes them into a 4K lenticular raster with calibrated subpixel-routing masks. The live-tracker prototype sustains 10.65 FPS, and a subject-specific distilled driver raises the same display pipeline to 38.49 FPS.

Chinese Translation

实时立体视频通信一直是沉浸式远程存在的目标，但实际系统仍需专门的捕捉设备，或将远程用户简化为单一的肖像视图。我们提出了PrismAvatar，一种高斯头部头像系统，将单目头像捕捉与亚像素编码的无眼镜透镜显示相结合，实现实时自立体通信。从单目肖像视频中，PrismAvatar重建可控的头部头像，并针对显示引起的侧向观察区域进行优化。该方法利用自然的头部转动作为伪多视图（PMV）监督，以约束在单目训练中观察较弱的区域，包括头发、耳朵、下颌轮廓和颈部边界。可靠的侧面帧经过偏航分箱，与虚拟相机对齐，并在严格的头部和头发领域内进行监督；轮廓感知损失和分阶段正则化进一步抑制了鬼影、透明度泄漏和深度不稳定，同时保留了侧向细节。在运行时，PrismAvatar渲染32个虚拟视图，并将其编码为带有校准亚像素路由掩码的4K透镜光栅。实时跟踪原型维持10.65帧每秒的帧率，而特定于受试者的精简驱动程序将相同的显示管道提升至38.49帧每秒。

View on arXiv Download PDF AI Translation

cs.CV / 41 / 2606.10571

Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction

通过替代特定偏差校正提高视觉-语言预训练模型的对抗可转移性

Yu, Lijia, Cao, Jiuxin, Qiang, Yuchen, Chen, Changhao, Huang, Yifei, Liu, Bo

Abstract

Adversarial examples reveal vulnerabilities in Vision-Language Pre-training (VLP) models and provide insights for improving robustness. A key property is cross-model transferability, which enables transfer-based black-box attacks. However, existing attacks often rely heavily on the surrogate model, causing cross-model performance drops. One reason is that adversarial optimization may follow surrogate model responses more than input semantics, making the update direction effective on the surrogate but less transferable to unseen targets. We refer to this dependency as surrogate-specific bias. Motivated by this observation, DeBias-Attack improves transferability by correcting surrogate-specific bias in adversarial optimization directions. It maintains two perturbation branches. The main branch optimizes a perturbation on the original image and obtains the adversarial gradient used to disrupt image-text alignment. The reference branch optimizes a perturbation on a weak-semantic image constructed from the dataset mean image with small Gaussian noise resampled at each iteration. Since this weak-semantic image contains little clear visual content, its optimization reflects surrogate responses more than image semantics, and its reference gradient estimates surrogate-specific bias. DeBias-Attack removes the aligned projection of the main gradient on the reference gradient before updating the adversarial image, then performs context-aware text substitution using the updated adversarial image. DeBias-Attack is the first transfer-based VLP attack that corrects surrogate-specific bias through gradient correction. Experiments show strong performance across VLP models, downstream tasks, and open-source and closed-source multimodal large language models.

Chinese Translation

对抗样本揭示了视觉-语言预训练（VLP）模型的脆弱性，并为提高其鲁棒性提供了见解。一个关键特性是跨模型可转移性，这使得基于转移的黑箱攻击成为可能。然而，现有攻击往往过于依赖替代模型，导致跨模型性能下降。其原因之一是对抗优化可能更多地遵循替代模型的响应，而非输入语义，从而使得更新方向在替代模型上有效，但在未见目标上可转移性较差。我们将这种依赖关系称为替代特定偏差。基于这一观察，DeBias-Attack通过校正对抗优化方向中的替代特定偏差来提高可转移性。它维持两个扰动分支。主分支在原始图像上优化扰动，并获得用于破坏图像-文本对齐的对抗梯度。参考分支在从数据集均值图像构建的弱语义图像上优化扰动，该图像在每次迭代中重新采样小的高斯噪声。由于这个弱语义图像包含较少清晰的视觉内容，其优化更多地反映替代响应而非图像语义，其参考梯度估计了替代特定偏差。DeBias-Attack在更新对抗图像之前，去除主梯度在参考梯度上的对齐投影，然后使用更新后的对抗图像进行上下文感知的文本替换。DeBias-Attack是第一个通过梯度校正来校正替代特定偏差的基于转移的VLP攻击。实验表明，在VLP模型、下游任务以及开源和闭源的多模态大型语言模型中表现出强大的性能。

View on arXiv Download PDF AI Translation

cs.CV / 42 / 2606.10594

Segment and Select: Vision-Language Segmentation in 3D Scenarios

分割与选择：3D场景中的视觉-语言分割

Chen, Yulin, Zhong, Zhihang, Hou, Yuenan

Abstract

3D vision-language segmentation aims to segment target objects in 3D scenarios according to the linguistic instructions and visual observations. Prior art heavily relies on the coarse superpoint representation to reduce the computation complexity, which suffers from poor segmentation quality and messy object boundaries. In this paper, we propose the SEGment-And-select (SEGA3D) paradigm for 3D visionlanguage segmentation that directly operates on the fine-grained visual information and is free from the superpoint dependency. Specifically, we first leverage a mask candidate generator to provide fine-grained categorical mask candidates, substantially improving the quality of candidate masks over the superpoint counterparts. Then, a Large Language Model (LLM) is utilized to generate the semantic and spatial information based on the linguistic description and visual features. The LLM output and visual features are fed to the Semantic-Spatial Selector (SSS) to produce the top-ranking mask candidates. Eventually, the Loopback Verification Module (LVM) is designed to yield the segmentation mask from the selected candidate masks. Our SEGA3D attains competitive performance on ScanRefer, ScanNet and Matterport3D benchmarks. Notably, our SEGA3D surpasses the top-performing counterpart by 8.3 mIoU and 5.3 mIoU on ScanNet and Matterport3D, respectively. Codes will be available upon publication.

Chinese Translation

3D视觉-语言分割旨在根据语言指令和视觉观察对3D场景中的目标物体进行分割。现有研究主要依赖粗糙的超点表示来降低计算复杂性，但这导致分割质量差和物体边界模糊。在本文中，我们提出了SEGment-And-select (SEGA3D)范式，用于3D视觉-语言分割，该范式直接在细粒度视觉信息上操作，且不依赖超点。具体而言，我们首先利用掩膜候选生成器提供细粒度的类别掩膜候选，大幅提高了候选掩膜的质量，相较于超点对应物。然后，利用大型语言模型（Large Language Model, LLM）根据语言描述和视觉特征生成语义和空间信息。LLM的输出和视觉特征被输入到语义-空间选择器（Semantic-Spatial Selector, SSS），以产生排名靠前的掩膜候选。最终，设计了回环验证模块（Loopback Verification Module, LVM）以从选定的候选掩膜中生成分割掩膜。我们的SEGA3D在ScanRefer、ScanNet和Matterport3D基准测试中表现出色。值得注意的是，我们的SEGA3D在ScanNet和Matterport3D上分别超越了表现最佳的对手8.3 mIoU和5.3 mIoU。代码将在发表时提供。

View on arXiv Download PDF AI Translation

cs.CV / 43 / 2606.10602

Globally Localizing Lunar Rover in Pixels via Graph Alignment

通过图对齐实现全球范围内的月球探测器像素级定位

Chen, Mao, Yang, Xu, Liu, Chuankai, Zhang, Xiangkai, Wang, Xiaoxue, Bo, Zheng, Zhang, Zuoyu, Liu, Zhiyong

Abstract

Precise rover localization is a prerequisite for autonomous lunar exploration, yet the absence of Global Navigation Satellite System (GNSS) signals and the cumulative drift of local localization methods severely constrain long-range missions. Cross-view localization provides a promising drift-free global solution by matching rover-view and satellite-view imagery. However, the lunar environment poses unique challenges for correspondence alignment, including inter-entity entanglement, inter-viewpoint divergence, and simulation-to-real domain shift. To address these challenges, we propose Warped Alignment of Reprojected Graphs (WARG), a framework that leverages unified graph learning and reprojected graph matching for robust cross-view alignment. Pretrained on the synthetic LuSNAR dataset, WARG achieves an average test error of 0.32 m and demonstrates robust zero-shot generalization to the synthetic lunar south pole region with an error of 3.63 m. More importantly, when validated on real-world data from the YuTu-2 rover, WARG achieves a localization error of 1.68 m within a 100 m x 100 m search area, corresponding to nearly one-pixel precision in low-resolution satellite imagery with a spatial resolution of 1.40 m/pixel. Beyond accuracy, WARG is computationally efficient, containing only 1.56M parameters, corresponding to 16.12% of previous lightweight models, and operating at 5.49 Hz on an NVIDIA RTX A6000 GPU, approaching GNSS-level update frequency. Finally, we observe that WARG naturally develops low-level spatial awareness, including semantic segmentation and structural reasoning, through cross-view localization learning, highlighting its potential as a promising paradigm for spatial intelligence with minimal annotation cost. The source code is available at https://github.com/maochen-casia/warg.

Chinese Translation

精确的探测器定位是自主月球探索的前提，但全球导航卫星系统（GNSS）信号的缺失以及局部定位方法的累积漂移严重限制了长距离任务的开展。跨视角定位通过匹配探测器视图和卫星视图图像提供了一种有前景的无漂移全球解决方案。然而，月球环境对对应对齐提出了独特的挑战，包括实体间纠缠、视点间发散以及模拟到真实域的转变。为了解决这些挑战，我们提出了重投影图的扭曲对齐（Warped Alignment of Reprojected Graphs, WARG）框架，该框架利用统一图学习和重投影图匹配实现稳健的跨视角对齐。在合成的LuSNAR数据集上进行预训练后，WARG实现了平均测试误差为0.32米，并在合成的月球南极区域展示了稳健的零样本泛化，误差为3.63米。更重要的是，当在来自玉兔二号探测器的真实数据上进行验证时，WARG在100米 x 100米的搜索区域内实现了1.68米的定位误差，相当于在空间分辨率为1.40米/像素的低分辨率卫星图像中接近一个像素的精度。除了准确性外，WARG在计算上也非常高效，仅包含1.56M参数，相当于之前轻量级模型的16.12%，并在NVIDIA RTX A6000 GPU上以5.49 Hz的频率运行，接近GNSS级别的更新频率。最后，我们观察到WARG通过跨视角定位学习自然发展出低级空间意识，包括语义分割和结构推理，突显其作为一种具有最小标注成本的空间智能有前景范式的潜力。源代码可在 https://github.com/maochen-casia/warg 获取。

View on arXiv Download PDF AI Translation

cs.CV / 44 / 2606.10612

GaussTrace: Provenance Analysis of 3D Gaussian Splatting Models with Evidence-based LLM Reasoning

GaussTrace：基于证据的LLM推理的3D高斯喷溅模型的来源分析

Han, Haoliang, Luo, Ziyuan, Wan, Renjie

Abstract

3D Gaussian Splatting (3DGS) is a powerful technique for creating high-fidelity 3D assets. However, the widespread sharing and iterative modification of 3DGS models across digital platforms create pressing challenges for intellectual property protection and forensic traceability. To address this, we propose GaussTrace, a novel framework for constructing directed provenance graphs for 3DGS models. GaussTrace formulates provenance analysis as an evidence-based reasoning problem. It builds upon attribute-wise statistical profiling of 3DGS parameters to capture intrinsic properties. Moreover, we introduce hypothesis-driven editing simulations of common operations to provide auxiliary evidence for plausible transformation pathways. These statistical and simulated cues jointly enable a Large Language Model (LLM) to perform structured Chain-of-Thought (CoT) reasoning, yielding directional provenance inferences and explainable edge reasons. Experimental results demonstrate that GaussTrace effectively constructs evolutionary relationships among diverse 3DGS models, delivering accurate, interpretable, and robust provenance graphs without requiring model training or access to editing histories. Project page: https://haolianghan.github.io/GaussTrace.

Chinese Translation

3D高斯喷溅（3DGS）是一种用于创建高保真3D资产的强大技术。然而，3DGS模型在数字平台上的广泛共享和迭代修改给知识产权保护和法医可追溯性带来了迫切挑战。为了解决这一问题，我们提出了GaussTrace，一个用于构建3DGS模型的有向来源图的新框架。GaussTrace将来源分析公式化为一个基于证据的推理问题。它基于3DGS参数的属性统计分析来捕捉内在特性。此外，我们引入了基于假设的常见操作编辑模拟，以提供合理变换路径的辅助证据。这些统计和模拟线索共同使大型语言模型（LLM）能够进行结构化的思维链（CoT）推理，从而产生方向性的来源推断和可解释的边缘原因。实验结果表明，GaussTrace有效构建了多样化3DGS模型之间的演变关系，提供了准确、可解释且稳健的来源图，而无需模型训练或访问编辑历史。项目页面：https://haolianghan.github.io/GaussTrace。

View on arXiv Download PDF AI Translation

cs.CV / 45 / 2606.10617

SSR-Merge: Subspace Signal Routing for Training-Free LoRA Merging in Diffusion Models

SSR-Merge：用于无训练LoRA合并的子空间信号路由在扩散模型中的应用

Wei, Zhengxuan, Dong, Yi, Li, Zonghui, Lin, Xianhui, Liu, Xing, Gu, Hong, Zhang, Shaofeng, Li, Wenbin, Fan, Qi

Abstract

Low-Rank Adaptation (LoRA) merging can efficiently combine diverse generative capabilities from multiple trained LoRAs for a diffusion model. However, existing LoRA merging techniques often suffer from severe parameter interference, causing destructive collisions in the shared parameter space. To address this, we propose Subspace Signal Routing (SSR), which resolves interference by routing internal signals instead of performing parameter-space merge. Specifically, SSR first constructs a unified subspace by concatenating candidate LoRAs along the rank dimension. Next, SSR employs an inverse correlation matrix to decorrelate mixed signals within this space. Finally, a directional guide matrix steers these purified signals into their respective task-specific subspaces. We provide a rigorous theoretical analysis proving that SSR aligns with the Ordinary Least Squares (OLS) solution, thereby ensuring mathematical optimality. We utilize the additivity of sufficient statistics to design a streaming algorithm. This enables on-the-fly updates that significantly reduce memory overhead and computation time. Extensive experiments validate that SSR significantly outperforms state-of-the-art methods while maintaining comparable efficiency. Code is available at https://github.com/nagara214/SSR-Merge.

Chinese Translation

低秩适应（LoRA）合并可以有效地结合来自多个训练LoRA的多样化生成能力，以用于扩散模型。然而，现有的LoRA合并技术往往遭受严重的参数干扰，导致共享参数空间中的破坏性冲突。为了解决这个问题，我们提出了子空间信号路由（SSR），通过路由内部信号而不是执行参数空间合并来解决干扰。具体而言，SSR首先通过沿秩维度连接候选LoRA构建统一的子空间。接下来，SSR采用逆相关矩阵来去相关化该空间内的混合信号。最后，方向引导矩阵将这些净化的信号引导到各自的任务特定子空间。我们提供了严格的理论分析，证明SSR与普通最小二乘（OLS）解一致，从而确保数学最优性。我们利用充分统计量的可加性设计了一种流式算法。这使得即时更新成为可能，显著减少了内存开销和计算时间。大量实验验证了SSR在保持相当效率的同时，显著优于最先进的方法。代码可在 https://github.com/nagara214/SSR-Merge 获取。

View on arXiv Download PDF AI Translation

cs.CV / 46 / 2606.10620

Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency

图像模型能否想象时间？ImageTime：通过时空一致性探测视觉世界建模的新基准

Wu, Xinrui, Huang, Lichen

Abstract

Image generation models now produce high-quality static images, yet their ability to represent how a visual world changes over time remains poorly understood. Practical workflows such as storyboarding, step-by-step illustration, reference-guided editing, and video previsualization require models to preserve identities, objects, spatial relations, and causal order across multiple visual states. Existing evaluations largely measure single-image correctness, compositional alignment, or video quality, leaving open whether an image model can coherently imagine a temporally ordered process. We introduce ImageTime, a diagnostic benchmark that uses spatiotemporal consistency as a behavioral probe of visual world modeling in image generation. Given an action instruction, and optionally a reference image specifying the initial state, a model must generate one image containing four ordered key states: initial state, action onset, transition state, and final state. This four-keyframe protocol is more temporally demanding than single-image generation while avoiding the confounds of dense video dynamics. ImageTime organizes tasks with a progressive capability hierarchy and decomposes each scenario into stage-wise state predicates, cross-frame temporal constraints, and forbidden causal violations. GPT-5.5 scores all generated images under a structured VLM-as-judge protocol, producing interpretable capability scores, diagnostic subscores, and failure labels. Through multi-family benchmarking, ImageTime reveals where current image generation systems succeed, fail, and drift when asked to maintain coherent visual world states over time.

Chinese Translation

图像生成模型现在能够生成高质量的静态图像，但它们在表示视觉世界如何随时间变化方面的能力仍然不够明确。实际工作流程如故事板制作、逐步插图、参考引导编辑和视频预可视化要求模型在多个视觉状态之间保持身份、物体、空间关系和因果顺序。现有评估主要测量单幅图像的正确性、组合对齐或视频质量，尚未探讨图像模型是否能够连贯地想象一个有时间顺序的过程。我们引入了ImageTime，一个使用时空一致性作为图像生成中视觉世界建模行为探针的诊断基准。在给定一个动作指令和可选的指定初始状态的参考图像的情况下，模型必须生成一幅包含四个有序关键状态的图像：初始状态、动作开始、过渡状态和最终状态。这种四关键帧协议在时间上比单幅图像生成要求更高，同时避免了密集视频动态的混淆。ImageTime以渐进的能力层次组织任务，并将每个场景分解为阶段性状态谓词、跨帧时间约束和禁止的因果违反。GPT-5.5在结构化的VLM作为评判者协议下对所有生成的图像进行评分，产生可解释的能力分数、诊断子分数和失败标签。通过多家庭基准测试，ImageTime揭示了当前图像生成系统在被要求在时间上保持连贯的视觉世界状态时的成功、失败和偏差之处。

View on arXiv Download PDF AI Translation

cs.CV / 47 / 2606.10628

Leveraging Metric Depth for Relative Depth Prediction

利用度量深度进行相对深度预测

Bi, Xiaoyang, Liu, Shuaikun, Liu, Zhaohong, Yang, Yuxin, Zhao, Zhe, Qi, Mengshi, Liu, Liang, Ma, Huadong

Abstract

We present our solution to the 2025 SoccerNet Monocular Depth Estimation Competition Challenge. Predicting the relative depth in football scenarios is challenging, especially with only thousands of training samples available. To address this issue, our method leverages the powerful zero-shot capabilities of models pretrained on large-scale datasets to learn metric depth for effective relative depth prediction, achieving a score of $2.68 \times 10^{-3}$ on the challenge set.

Chinese Translation

我们提出了针对2025年SoccerNet单目深度估计竞赛挑战的解决方案。在足球场景中预测相对深度具有挑战性，尤其是在仅有数千个训练样本可用的情况下。为了解决这个问题，我们的方法利用在大规模数据集上预训练模型的强大零样本能力，学习度量深度以有效进行相对深度预测，在挑战集上取得了$2.68 imes 10^{-3}$的得分。

View on arXiv Download PDF AI Translation

cs.CV / 48 / 2606.10640

ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinement

ChartLens：一种用于图表数据修正和事实摘要精炼的双分支框架

Liu, Hao, Cao, Ruping, Wang, Kun, Li, Zhiran, Liu, Fan, Hu, Yupeng, Nie, Liqiang

Abstract

In this report, we present our champion solution for the DataMFM Challenge Track 2: Chart Understanding. This track requires models to recover structured chart data and generate faithful natural-language summaries from chart images. To address the complementary requirements of accurate data extraction and factual narration, we propose ChartLens, a dual-branch framework for chart data correction and summary refinement. ChartLens consists of two key modules: Structure-Aware CSV Verification and Correction (SAVC) and Text-Retention-Guided Summary Refinement (TRSR). SAVC improves the reliability of structured data extraction through verification and correction, while TRSR enhances summary generation by preserving critical textual and numerical evidence from charts. By combining model adaptation, correction-based generation, and OCR-assisted evidence grounding, ChartLens improves both structured data recovery and summary factuality. On the test set, our final system achieves an overall score of 69.10 and ranks first in Track 2, demonstrating its effectiveness for accurate chart understanding. Our code will be released at: https://github.com/iLearn-Lab/CVPRW26-ChartLens.

Chinese Translation

在本报告中，我们提出了针对DataMFM挑战赛第二赛道：图表理解的冠军解决方案。该赛道要求模型从图表图像中恢复结构化图表数据并生成真实的自然语言摘要。为了满足准确数据提取和事实叙述的互补需求，我们提出了ChartLens，一种用于图表数据修正和摘要精炼的双分支框架。ChartLens由两个关键模块组成：结构感知的CSV验证与修正（Structure-Aware CSV Verification and Correction, SAVC）和文本保留引导的摘要精炼（Text-Retention-Guided Summary Refinement, TRSR）。SAVC通过验证和修正提高结构化数据提取的可靠性，而TRSR通过保留图表中的关键文本和数值证据来增强摘要生成。通过结合模型适应、基于修正的生成和OCR辅助的证据基础，ChartLens改善了结构化数据的恢复和摘要的事实性。在测试集上，我们的最终系统获得了69.10的总体得分，并在第二赛道中排名第一，证明了其在准确图表理解方面的有效性。我们的代码将发布在：https://github.com/iLearn-Lab/CVPRW26-ChartLens。

View on arXiv Download PDF AI Translation

cs.CV / 49 / 2606.10645

ManiSplat: Manipulation Trajectory Synthesis from Monocular Video via Decoupled 3D Gaussian Splatting

ManiSplat：通过解耦的3D高斯点云从单目视频合成操控轨迹

Hu, Wenhao, Zhou, Haonan, Liu, Liu, Du, Yun, Wang, Xinjie, Li, Ziang, Su, Zhizhong, Wang, Gaoang

Abstract

Reconstructing dynamic and interactive 3D scenes from real-world observations remains a fundamental challenge in computer vision and robotics. While recent advances in 3D Gaussian Splatting have enabled high-fidelity static reconstruction, extending it to interactive environments with articulated robots and manipulable objects remains difficult due to complex contact interactions and abrupt pose changes. To address these challenges, we introduce ManiSplat, a unified framework that reconstructs controllable and decoupled Gaussian digital twins directly from monocular ego-view robotic videos. Our method introduces a Graph-Structured Disentangled Representation that separates the robot, objects, and background into independently optimizable Gaussian subfields organized within a scene graph. To ensure stability, we propose a Task-Oriented Spatio-Temporal Alignment module that leverages the inherent logic of manipulation tasks-alternating between Motion and Skill phases-to construct accurate pseudo-ground-truth trajectories. Finally, a joint photometric-geometric optimization ensures the reconstructed scenes are temporally coherent, physically consistent, and simulation-ready. Extensive experiments demonstrate that our approach reconstructs interaction-driven dynamic scenes with high fidelity and controllability, effectively supporting downstream robotic tasks and policy learning.

Chinese Translation

从现实世界观察中重建动态和交互式3D场景仍然是计算机视觉和机器人领域的一项基本挑战。尽管最近在3D高斯点云方面的进展使得高保真静态重建成为可能，但由于复杂的接触交互和突变的姿态变化，将其扩展到具有关节机器人和可操控物体的交互环境中仍然困难。为了解决这些挑战，我们提出了ManiSplat，一个统一框架，能够直接从单目自视角机器人视频中重建可控和解耦的高斯数字双胞胎。我们的方法引入了一种图结构解耦表示，将机器人、物体和背景分离为可以独立优化的高斯子域，并在场景图中组织。为了确保稳定性，我们提出了一种任务导向的时空对齐模块，利用操控任务的内在逻辑——在运动和技能阶段之间交替——构建准确的伪真实轨迹。最后，联合光度-几何优化确保重建的场景在时间上连贯、物理上一致，并且适合模拟。大量实验表明，我们的方法能够以高保真度和可控性重建以交互为驱动的动态场景，有效支持下游机器人任务和策略学习。

View on arXiv Download PDF AI Translation

cs.CV / 50 / 2606.10651

Kwai Keye-VL-2.0 Technical Report

Kwai Keye-VL-2.0 技术报告

Kwai Keye Team, Wen, Bin, Liu, Changyi, Song, Chengru, Rao, Chongling, Zhang, Guowang, Li, Han, Fan, Haonan, Ju, Hengrui, Chen, Jiankang, Chen, Jiapeng, Yuan, Jiawei, Yang, Kaixuan, Jiang, Kaiyu, Gai, Kun, Zhou, Lingzhi, Nie, Na, Na, Sen, Zhang, Tianke, Gao, Tingting, Zheng, Xuanyu, Chen, Yulong, Yang, Fan, Gao, Haixuan, Yang, Lele, Liu, Mingqiao, Diao, Muxi, Zhang, Qi, Su, Qile, Chen, Wei, Hong, Wentao, Lu, Xingyu, Long, Yancheng, Yang, Yankai, Li, Yingxin, Fan, Yiyang, Xia, Yu, Chen, Yuzhe, Lai, Ziliang, Yi, Chuan, Jia, Haonan, Liang, Tianming, Xu, Weixin, Ma, Xiaoxiao, Tian, Yang, Han, Yufei, Han, Feng, Li, Hang, Wang, Jing, Jia, Jinghui, Chen, Junmin, Shi, Junyu, Zhang, Ruilin

Abstract

We introduce Kwai Keye-VL-2.0-30B-A3B, an open-source Mixture-of-Experts (MoE) multimodal foundation model designed to advance long-video understanding and agentic intelligence. To address the challenges of ultra-long contexts, information redundancy, and prohibitive computational costs inherent in hour-level videos, Keye-VL-2.0 is the first to adapt DeepSeek Sparse Attention (DSA) to GQA-based multimodal architectures, enabling lossless 256K context processing while capturing critical frames and long-range temporal dependencies. This architecture is underpinned by a highly optimized training and inference infrastructure, including scalable video I/O, heterogeneous ViT-LM parallelism, and custom DSA kernels that significantly maximize throughput and minimize computational overhead. Furthermore, to overcome the algorithmic dilemma of catastrophic forgetting during multi-task alignment, we introduce Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) paired with Context-RL and Video-RL. By distilling dense token-level teacher feedback from on-policy rollouts back into the MoE backbone, which activates only 3B parameters, Keye-VL-2.0 natively empowers advanced agent collaboration across Code, Tool, and Search scenarios with multimodal self-correction. Extensive evaluations across video understanding, temporal grounding, reasoning, STEM, and agent benchmarks demonstrate that Keye-VL-2.0-30B-A3B achieves state-of-the-art performance among models of similar scale, particularly excelling in fine-grained temporal localization on TimeLens and long-video comprehension on Video-MME-v2 and LongVideoBench. We release our model checkpoints to accelerate community progress toward scalable and robust multimodal agentic applications.

Chinese Translation

我们介绍了Kwai Keye-VL-2.0-30B-A3B，这是一个开源的混合专家（Mixture-of-Experts, MoE）多模态基础模型，旨在推动长视频理解和智能体智能的发展。为了应对超长上下文、信息冗余和小时级视频固有的高计算成本等挑战，Keye-VL-2.0首次将深度寻址稀疏注意力（DeepSeek Sparse Attention, DSA）应用于基于GQA的多模态架构，使得能够在捕捉关键帧和长距离时间依赖的同时，实现无损的256K上下文处理。该架构建立在高度优化的训练和推理基础设施之上，包括可扩展的视频输入/输出、异构ViT-LM并行处理以及自定义的DSA内核，这些都显著提高了吞吐量并最小化了计算开销。此外，为了克服多任务对齐过程中灾难性遗忘的算法困境，我们引入了跨模态多教师在线蒸馏（Cross-Modal Multi-Teacher On-Policy Distillation, MOPD），并结合上下文强化学习（Context-RL）和视频强化学习（Video-RL）。通过将来自在线策略回放的密集令牌级教师反馈蒸馏回MoE主干网络，该网络仅激活3B参数，Keye-VL-2.0原生地增强了在代码、工具和搜索场景中的高级智能体协作，具备多模态自我修正能力。针对视频理解、时间定位、推理、STEM和智能体基准的广泛评估表明，Keye-VL-2.0-30B-A3B在同规模模型中实现了最先进的性能，特别是在TimeLens上的细粒度时间定位和Video-MME-v2及LongVideoBench上的长视频理解方面表现优异。我们发布了模型检查点，以加速社区在可扩展和稳健的多模态智能体应用方面的进展。

View on arXiv Download PDF AI Translation

cs.CV / 51 / 2606.10653

STEDiff: Strengthening Text Embedding for Text-to-Image Alignment in Diffusion Model

STEDiff：在扩散模型中增强文本嵌入以实现文本与图像的对齐

Zhang, Hailan, Liu, Haipeng, Fu, Bo, Wang, Yang

Abstract

Although pretrained text-to-image (T2I) generation models can produce high-quality images, they often fail to faithfully reflect the semantic intent of complex prompts due to stochastic noise and inherent model limitations. This issue frequently manifests as the model overlooking specific objects or failing to correctly bind attributes to their corresponding entities, a challenge referred to as semantic alignment. Unlike existing approaches that rely on computationally expensive fine-tuning or labor-intensive layout priors, we propose STEDiff, a training-free method designed to enhance semantic representations directly within the text-embedding space. Specifically, we introduce a method that primarily leverages the [EOT] token to strengthen the relevant semantics of sub-sentences and then replaces the corresponding tokens in the original prompt. Furthermore, a novel semantic enhancement loss is incorporated to enforce spatial constraints, ensuring that the semantics of each entity are precisely mapped to their respective image regions. Extensive quantitative and qualitative evaluations on the T2I-CompBench demonstrate that our method notably improves semantic consistency and generation integrity in complex scenarios.

Chinese Translation

尽管预训练的文本到图像（T2I）生成模型能够生成高质量的图像，但由于随机噪声和模型固有的局限性，它们往往无法忠实地反映复杂提示的语义意图。这个问题通常表现为模型忽视特定对象或未能正确将属性绑定到相应实体，这一挑战被称为语义对齐。与依赖计算成本高昂的微调或劳动密集型布局先验的现有方法不同，我们提出了STEDiff，这是一种无训练的方法，旨在直接增强文本嵌入空间中的语义表示。具体而言，我们引入了一种主要利用[EOT]标记来增强子句相关语义的方法，然后在原始提示中替换相应的标记。此外，结合了一种新颖的语义增强损失，以强制施加空间约束，确保每个实体的语义精确映射到其各自的图像区域。在T2I-CompBench上的广泛定量和定性评估表明，我们的方法显著提高了复杂场景中的语义一致性和生成完整性。

View on arXiv Download PDF AI Translation

cs.CV / 52 / 2606.10656

Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Driving

Envision4D：通过前馈4D高斯散射实现自主驾驶的视觉未来展望

Song, Qi, He, Yifei, Zhang, Chi, Fu, Zheng, Zhao, Xuhe, Yang, Mengmeng, Jiang, Kun, Huang, Rui, Yang, Diange

Abstract

Forecasting the future evolution of dynamic scenes is crucial in autonomous driving. However, existing feed-forward paradigms are primarily designed for interpolation. When extended to future extrapolation, they suffer from ghosting artifacts under large displacements and are constrained by simplified motion assumptions or strict future priors. To overcome these challenges, we propose Envision4D, a fully self-supervised feed-forward framework for pose-free future extrapolation. Specifically, we introduce a Future Pose Prediction module that infers future camera parameters via an iterative denoising process. Furthermore, to capture non-linear dynamics, we propose In-layer Temporal Attention and employ Conditioned Motion Lifting, which transforms the highly uncertain extrapolation process into robust relational mappings. Finally, a Progressive Training Strategy is utilized to stabilize unsupervised motion learning against error accumulation. Extensive experiments demonstrate that Envision4D achieves state-of-the-art performance, significantly outperforming existing methods in future view synthesis.

Chinese Translation

预测动态场景的未来演变在自主驾驶中至关重要。然而，现有的前馈范式主要设计用于插值。当扩展到未来外推时，它们在大位移下会出现鬼影伪影，并受到简化运动假设或严格未来先验的限制。为了解决这些挑战，我们提出了Envision4D，一个完全自监督的前馈框架，用于无姿态的未来外推。具体而言，我们引入了一个未来姿态预测模块，通过迭代去噪过程推断未来相机参数。此外，为了捕捉非线性动态，我们提出了层内时间注意力（In-layer Temporal Attention），并采用条件运动提升（Conditioned Motion Lifting），将高度不确定的外推过程转化为稳健的关系映射。最后，采用渐进训练策略（Progressive Training Strategy）来稳定无监督运动学习，防止误差累积。大量实验表明，Envision4D实现了最先进的性能，在未来视图合成方面显著超越现有方法。

View on arXiv Download PDF AI Translation

cs.CV / 53 / 2606.10666

Analyzing Training-Free Corruption Detection for Object Detection Datasets

分析无训练腐败检测在目标检测数据集中的应用

Sieberichs, Christian, Geerkens, Simon, Waschulzik, Thomas, Ramesh, Viswanathan, Braun, Alexander

Abstract

Annotation errors are widespread in computer vision datasets and can significantly degrade the performance of systems trained on them, particularly in complex tasks such as object detection. Several approaches exist to identify annotation errors, including training-free feature-space methods which provide a fast and interpretable way to analyze annotations. However, the behavior on object detection annotations, which include semantic and spatial information, remains largely unexplored. In this work we analyze the applicability of feature-space-based approaches for detecting annotation errors in object detection datasets. By adapting an existing feature-space method, we show that such approaches reliably expose semantic mislabel, while positional errors remain difficult to detect. We evaluate this behavior across multiple pretrained embedding models, synthetic noise types (symmetric, asymmetric, and positional), and real-world annotation errors using VOC2012 and KITTI. All code and real-world corruptions are publicly available at the following repository: https://github.com/ ChristianSieberichs/BoundingBox\_corruption\_detection

Chinese Translation

注释错误在计算机视觉数据集中普遍存在，可能显著降低基于这些数据集训练的系统的性能，尤其是在目标检测等复杂任务中。现有多种方法可用于识别注释错误，包括无训练的特征空间方法，这些方法提供了一种快速且可解释的分析注释的方式。然而，对于包含语义和空间信息的目标检测注释的行为，仍然很大程度上未被探索。在本研究中，我们分析了基于特征空间的方法在目标检测数据集中检测注释错误的适用性。通过改编现有的特征空间方法，我们展示了此类方法能够可靠地揭示语义错误标注，而位置错误则仍然难以检测。我们在多个预训练嵌入模型、合成噪声类型（对称、非对称和位置）以及使用 VOC2012 和 KITTI 的真实世界注释错误上评估了这种行为。所有代码和真实世界的腐败数据均可在以下代码库中公开获取：https://github.com/ChristianSieberichs/BoundingBox_corruption_detection

View on arXiv Download PDF AI Translation

cs.CV / 54 / 2606.10671

FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion

FadeMem：基于距离感知的自回归视频扩散记忆整合

Lu, Yu, Yang, Junjie, Koniusz, Piotr, Song, YuXin, Yang, Yi

Abstract

Autoregressive video generators synthesize long videos by generating successive temporal segments, but their historical KV cache grows with video length. Existing bounded-cache methods reduce this cost with local windows, sink tokens, or compressed memory states, yet they usually assign fixed roles to different parts of the history. We propose FadeMem, a distance-aware KV memory consolidation mechanism that organizes historical KV blocks into a temporal hierarchy under a fixed cache budget. This design is motivated by frequency-dependent temporal decay: fine details decorrelate quickly, while coarse scene structure and identity remain useful over longer horizons. During generation, new history is inserted as fine-grained entries, while older adjacent entries are progressively merged under a power-law temporal allocation schedule, yielding a dense-near, sparse-far memory within one cache. Without architectural changes, FadeMem preserves recent context for short-term dynamics and compact long-range anchors for identity and scene coherence. Experiments show improved subject consistency, background stability, and temporal coherence over existing bounded-cache strategies.

Chinese Translation

自回归视频生成器通过生成连续的时间段合成长视频，但其历史键值（KV）缓存随着视频长度的增加而增长。现有的有界缓存方法通过局部窗口、沉没令牌或压缩内存状态来降低这一成本，但它们通常为历史的不同部分分配固定角色。我们提出了FadeMem，一种基于距离感知的键值内存整合机制，在固定的缓存预算下将历史KV块组织成时间层次结构。该设计受到频率依赖的时间衰减的启发：细节快速去相关，而粗略的场景结构和身份在更长的时间范围内仍然有用。在生成过程中，新历史作为细粒度条目插入，而较旧的相邻条目在幂律时间分配计划下逐步合并，从而在一个缓存中形成密集-近、稀疏-远的内存。没有架构上的变化，FadeMem为短期动态保留了最近的上下文，并为身份和场景一致性提供了紧凑的长程锚点。实验表明，与现有的有界缓存策略相比，FadeMem在主体一致性、背景稳定性和时间一致性方面有所改善。

View on arXiv Download PDF AI Translation

cs.CV / 55 / 2606.10696

Don't waste SAM

不要浪费SAM

Baker, Nermeen Abou, Handmann, Uwe

Abstract

Meta AI has recently released the Segment Anything Model (SAM), which demonstrates exceptional zero-shot image segmentation performance across various tasks with remarkable accuracy. Despite its inability to provide accurate segmentation across multiple research fields, SAM still serves as a valuable starting point for supporting the segmentation pipeline process, particularly for tasks that require extensive and senior skills annotations. This study aims to evaluate the generalization of SAM and fine-tuning SAM models using three waste segmentation datasets. Although they are captured from real scenes as SAM was pretrained on, these datasets present several challenges, including occlusions, deformable objects, transparency, and objects easily confused with backgrounds. In our findings, the fine-tuned SAM-ViT-H model outperforms the state-ofthe-art Zerowaste, and TACO datasets with a significant increase of +30 in IoU, and it closely approaches performance levels of TrashCan 1.0, with only a -1.44 difference. After evaluating these popular waste datasets, it became evident that fine-tuning SAM as a foundational model is a crucial step for providing better generalization for downstream waste segmentation tasks. Therefore, SAM should not be disregarded or wasted.

Chinese Translation

Meta AI最近发布了Segment Anything Model (SAM)，该模型在各种任务中展示了卓越的零样本图像分割性能，具有显著的准确性。尽管在多个研究领域中无法提供准确的分割，SAM仍然作为支持分割流程的宝贵起点，特别是对于需要广泛和高级技能注释的任务。本研究旨在评估SAM的泛化能力，并使用三个废物分割数据集对SAM模型进行微调。尽管这些数据集是从真实场景中捕获的，正如SAM的预训练所基于的，但它们仍然面临诸多挑战，包括遮挡、可变形物体、透明度以及容易与背景混淆的物体。我们的研究结果表明，微调后的SAM-ViT-H模型在IoU上显著提高了+30，超越了最先进的Zerowaste和TACO数据集，并且与TrashCan 1.0的性能水平接近，仅有-1.44的差距。在评估这些流行的废物数据集后，显然将SAM微调为基础模型是为下游废物分割任务提供更好泛化能力的重要步骤。因此，SAM不应被忽视或浪费。

View on arXiv Download PDF AI Translation

cs.CV / 56 / 2606.10699

Using the YOLOv12 Model for Verifying the Correct Color Sequence of Wires in Network Cables (Patch Cords) on the Production Line

在生产线上使用YOLOv12模型验证网络电缆（跳线）中电线正确的颜色顺序

Doroodchi, Amin, Soleimany, Danial

Abstract

In the production process of network cables, ensuring the correct color sequence of wire pairs inside the standard connector plays a critical role in the final performance of the cable, as any misplacement or color-ordering error can lead to defective products and impose significant costs. Traditional inspection methods based on visual examination through digital microscopes are typically time-consuming, tedious, and prone to human error. In this study, an intelligent system based on the twelfth version of the YOLO1 object detection model was developed to identify the position and verify the correct color sequence of wires in patch cords. The dataset used consisted of 2,500 images captured from microscopic views of network connectors, which were divided into 70% for training, 15% for validation, and 15% for testing. The proposed model, leveraging a single-stage architecture and attention mechanisms during learning, achieved highly accurate wire detection with approximately 98% precision. Additionally, the overall mean accuracy, classification precision, and recall were around 95%, 99%, and 98%, respectively. The results demonstrate that this system can reliably and in real time verify the correctness of wire color sequencing on the production line without the need for human intervention, thereby reducing human error and enhancing efficiency in the manufacturing process.

Chinese Translation

在网络电缆的生产过程中，确保标准连接器内部电线对的正确颜色顺序对电缆的最终性能至关重要，因为任何错误的放置或颜色顺序错误都可能导致缺陷产品并产生显著成本。基于数字显微镜的传统检查方法通常耗时、繁琐且容易出现人为错误。在本研究中，开发了一种基于YOLO1目标检测模型第十二版的智能系统，用于识别跳线中电线的位置并验证其正确的颜色顺序。所使用的数据集由2500张从网络连接器的显微视图中捕获的图像组成，这些图像被分为70%用于训练，15%用于验证，15%用于测试。所提出的模型利用单阶段架构和学习过程中的注意力机制，实现了约98%的高精度电线检测。此外，整体平均准确率、分类精度和召回率分别约为95%、99%和98%。结果表明，该系统能够可靠且实时地验证生产线电线颜色顺序的正确性，无需人工干预，从而减少人为错误并提高制造过程的效率。

View on arXiv Download PDF AI Translation

cs.CV / 57 / 2606.10701

Vector Map as Language: Toward Unified Remote Sensing Vector Mapping

向量地图作为语言：统一遥感向量制图的探索

Yan, Yinglong, Yang, Yunkai, Wang, Haoyi, Fu, Wei, Wu, Linshan, Pan, Honghu, Xia, Shaobo, Zhang, Shanghang, Chen, Hao, Fang, Leyuan

Abstract

Remote sensing vector mapping aims to generate structured maps of geospatial entities, such as buildings, roads, and water bodies, from remote sensing imagery. In practice, vector maps usually contain multiple category layers and heterogeneous entity structures, requiring a unified model for diverse mapping needs. However, existing methods typically represent vector objects as polygons or graphs, making them suitable only for specific categories: polygons poorly capture topological relations, while graphs often blur instance boundaries. We observe that language, as a natural medium for human communication, offers a flexible and expressive representation that can accommodate heterogeneous map elements, including geometry, semantics, and topolog. Motivated by this insight, we propose Vector Map as Language (VecLang), a unified paradigm that reformulates multiclass vector mapping as structured text generation. VecLang encodes the common elements of different geospatial entities into a GeoJSON-like vector language, enabling cross-category modeling within a shared textual format. To generate this language reliably, we design a progressive vision-language mapping framework that first localizes vectorization units and then generates structured map elements. We further introduce Hierarchical Vector Language Optimization, which uses reinforcement learning to improve syntax validity, content fidelity, and map executability. We also build VecMap-Bench with 54K images and 800K instances, supporting training and evaluation across standard and generalization settings. Extensive experiments demonstrate that VecLang handles both single-class and multiclass vector mapping while achieving strong cross-dataset and open-vocabulary generalization. The model and dataset are publicly available at https://github.com/yyyyll0ss/VecLang.

Chinese Translation

遥感向量制图旨在从遥感影像中生成地理空间实体的结构化地图，例如建筑物、道路和水体。在实际应用中，向量地图通常包含多个类别层和异构实体结构，因此需要一个统一的模型以满足多样化的制图需求。然而，现有方法通常将向量对象表示为多边形或图形，这使得它们仅适用于特定类别：多边形难以捕捉拓扑关系，而图形往往模糊实例边界。我们观察到，语言作为人类交流的自然媒介，提供了一种灵活且富有表现力的表示方式，可以容纳异构地图元素，包括几何、语义和拓扑。基于这一洞察，我们提出了向量地图作为语言（Vector Map as Language，简称 VecLang），这是一个将多类别向量制图重新表述为结构化文本生成的统一范式。VecLang将不同地理空间实体的共同元素编码为类似 GeoJSON 的向量语言，从而在共享文本格式中实现跨类别建模。为了可靠地生成这种语言，我们设计了一个渐进式视觉-语言映射框架，该框架首先定位向量化单元，然后生成结构化地图元素。我们进一步引入了层次向量语言优化（Hierarchical Vector Language Optimization），该方法利用强化学习来提高语法有效性、内容保真度和地图可执行性。我们还构建了 VecMap-Bench，包含 54K 图像和 800K 实例，支持在标准和泛化设置下的训练和评估。大量实验表明，VecLang 能够处理单类别和多类别向量制图，同时实现强大的跨数据集和开放词汇泛化。模型和数据集可在 https://github.com/yyyyll0ss/VecLang 上公开获取。

View on arXiv Download PDF AI Translation

cs.CV / 58 / 2606.10735

Patient-Level Diagnosis of Acute Myeloid Leukemia via Deep Learning Analysis of Bone Marrow Smear

通过深度学习分析骨髓涂片实现急性髓性白血病的患者级诊断

Ma, Yuqi, Wang, Tianyi, Meng, Weihua, Chen, Hongru, Tao, Fajin, Lu, Qunxian, An, Lin, Mo, Xiaodong, Yang, Gen

Abstract

Bone marrow smear review remains important for acute myeloid leukemia (AML) assessment, but manual single-cell interpretation is labor-intensive and patient-level diagnosis requires aggregation of many cellular observations. We present a cell-to-patient deep learning pipeline for AML-assisted diagnosis from bone marrow smear images. The study included 258 patients from six anonymized centers, including a main cohort of 169 patients from Centers 1-3 and an external validation cohort of 89 patients from Centers 4-6. A 16-category cell annotation vocabulary was used to describe the global cellular composition, including granulocytic, monocytic, erythroid, lymphoid, eosinophilic, and other cells. Rather than identifying strict AML blasts or leukemic blasts, the model targets an expert-defined composite category termed Composite Blast-like Cells (CBLC), comprising N, N1, M, M1, R, R1, J, and J1 according to the project-wide morphological standard. A fixed YOLO-based segmentation module detected cells, predicted contours were matched to expert polygon annotations by contour IoU, and standardized single-cell crops were generated. An EfficientNet-B0 classifier was trained through a two-stage GT-to-YOLO and YOLO-to-YOLO strategy with class-imbalance correction, center-border regularization, and morphology-assisted supervision. Cell-level predictions were aggregated into patient-level CBLC ratios for AML-oriented diagnostic support. The pipeline achieved stable internal validation and maintained external generalization, with ensemble weighted F1-scores of 0.9076, 0.8696, and 0.9124 on Centers 4, 5, and 6, respectively.

Chinese Translation

骨髓涂片的审查在急性髓性白血病（AML）的评估中仍然重要，但手动单细胞解释劳动强度大，患者级诊断需要聚合许多细胞观察。我们提出了一种基于细胞到患者的深度学习管道，用于从骨髓涂片图像中辅助诊断AML。本研究包括来自六个匿名中心的258名患者，其中主队列为来自中心1-3的169名患者，外部验证队列为来自中心4-6的89名患者。使用16类细胞注释词汇描述全球细胞组成，包括粒细胞、单核细胞、红细胞、淋巴细胞、嗜酸性细胞及其他细胞。该模型并非严格识别AML芽细胞或白血病芽细胞，而是针对一个专家定义的复合类别，称为复合芽细胞样细胞（Composite Blast-like Cells, CBLC），根据项目范围内的形态学标准，包括N、N1、M、M1、R、R1、J和J1。一个固定的基于YOLO的分割模块检测细胞，预测的轮廓通过轮廓IoU与专家的多边形注释进行匹配，并生成标准化的单细胞裁剪。通过两阶段的GT到YOLO和YOLO到YOLO策略，结合类别不平衡校正、中心-边界正则化和形态学辅助监督，训练了一个EfficientNet-B0分类器。细胞级预测被聚合为患者级CBLC比率，以支持AML导向的诊断。该管道实现了稳定的内部验证，并保持外部泛化，在中心4、5和6的集成加权F1分数分别为0.9076、0.8696和0.9124。

View on arXiv Download PDF AI Translation

cs.CV / 59 / 2606.10756

DD-INR: Dynamics-Driven Implicit Neural Representation for Accelerated Whole-Brain Functional MRI Reconstruction

DD-INR：基于动态驱动的隐式神经表示用于加速全脑功能性磁共振成像重建

Li, Qiaoxin, Pan, Caini, Comby, Pierre-Antoine, Giliyar, Chaithya, Ciuciu, Philippe

Abstract

Accelerated acquisition of fMRI enables enhanced detection of neurovascular (BOLD) activity in the brain, but image reconstruction becomes challenging with high k-space undersampling: Task-evoked BOLD signals are small in magnitude, which traditional anatomical MRI reconstruction methods fail to recover, as they favor spatial accuracy over temporal fidelity. We present DD-INR, a Dynamics-Driven Implicit Neural Representation framework tailored for accelerated fMRI that benefits from incoherent time-varying sampling and a tailored spatiotemporal prior, outperforming traditional methods, demonstrated in simulation and in-vivo acquisition, both in terms of image quality and retrieval of activation patterns. DD-INR achieves this by splitting the fMRI data into a static background and a temporally varying dynamic component, representing only the dynamics with a dedicated INR, thereby focusing the model's capacity on activation-relevant changes while remaining compact. In general, DD-INR provides a promising framework for accelerated fMRI reconstruction, with the potential to improve the sensitivity and robustness of fMRI studies within practical scan time limits. The source code is available at https://github.com/JoosenLi/DD-INR.

Chinese Translation

加速获取功能性磁共振成像（fMRI）能够增强对大脑神经血管（BOLD）活动的检测，但在高k空间欠采样的情况下，图像重建变得具有挑战性：任务诱发的BOLD信号幅度较小，传统的解剖结构MRI重建方法无法恢复这些信号，因为它们更注重空间精度而非时间保真度。我们提出了DD-INR，一种针对加速fMRI的基于动态驱动的隐式神经表示框架，该框架利用非相干的时间变化采样和量身定制的时空先验，超越了传统方法，在图像质量和激活模式检索方面在模拟和体内获取中均表现出色。DD-INR通过将fMRI数据分为静态背景和时间变化的动态成分，仅用专门的隐式神经表示（INR）表示动态，从而将模型的能力集中在与激活相关的变化上，同时保持紧凑性。总体而言，DD-INR为加速fMRI重建提供了一个有前景的框架，有潜力在实际扫描时间限制内提高fMRI研究的灵敏度和鲁棒性。源代码可在 https://github.com/JoosenLi/DD-INR 获取。

View on arXiv Download PDF AI Translation

cs.CV / 60 / 2606.10769

ZODS-RS -- Zero-training Oriented Detection & Segmentation for Remote Sensing

ZODS-RS -- 面向零训练的遥感检测与分割

Gu, Zuan, Gao, Tianhan, Zhao, Langxu

Abstract

Remote-sensing and UAV applications need models that generalize across platforms and viewpoints without task-specific training. Yet training-free pipelines often falter on oriented geometry, scale/rotation variation, and crowded ports or airfields, and rarely unify detection and segmentation. We introduce ZODS-RS, a training-free, closed-form pipeline that outputs horizontal boxes (HBB) and instance masks. Built on DINOv3 dense features and SAM-style proposals, ZODS-RS chains: PP (prototype purification via Tyler covariance), R-SEM (rotation-scale equivariant matching with separable kernels and global Hungarian assignment), and UAM (uncertainty-aware pixelwise merging with adaptive priors and optional negative prototypes). A lightweight CWLA fuses multiple DINOv3 layers. On FAIR1M (HBB) we obtain $\mathrm{mAP}_{0.50:0.95}=\mathbf{13.06}$ and $\mathrm{AP}_S=\mathbf{2.93}$ \emph{(class-averaged over ship/airplane)}; on xView (HBB) we report $\mathrm{mAP}=\mathbf{16.69}$. On our UAV dataset, ZODS-RS achieves mask $\mathrm{mIoU}=\mathbf{31.10}$ and improves small-object AP by $\mathbf{+30.70}$ over Grounded-SAM on a single 5090. This work offers a unified, \emph{no-training} solution for horizontal-box detection plus instance segmentation in aerial imagery; provides explicit closed-form formulations for PP/R-SEM/UAM tightly coupled with DINOv3; and demonstrates \emph{consistent} gains on small and crowded targets and under cross-domain shifts while keeping deployment simple.

Chinese Translation

遥感和无人机应用需要能够跨平台和视角进行泛化的模型，而无需特定任务的训练。然而，无训练管道在定向几何、尺度/旋转变化以及拥挤的港口或机场上往往表现不佳，并且很少能统一检测和分割。我们提出了ZODS-RS，一个无训练的闭式管道，能够输出水平框（HBB）和实例掩膜。ZODS-RS基于DINOv3的稠密特征和SAM风格的提议，链式组合了：PP（通过Tyler协方差进行原型净化）、R-SEM（使用可分离核和全局匈牙利分配的旋转-尺度等变匹配）和UAM（具有自适应先验和可选负原型的基于不确定性的逐像素合并）。一个轻量级的CWLA融合了多个DINOv3层。在FAIR1M（HBB）上，我们获得了$ ext{mAP}_{0.50:0.95}= extbf{13.06}$和$ ext{AP}_S= extbf{2.93}$ extit{（在船舶/飞机类别上平均）}；在xView（HBB）上，我们报告了$ ext{mAP}= extbf{16.69}$。在我们的无人机数据集上，ZODS-RS实现了掩膜$ ext{mIoU}= extbf{31.10}$，并在单个5090上将小物体AP提高了$ extbf{+30.70}$，相较于Grounded-SAM。该工作提供了一种统一的 extit{无训练}解决方案，用于航空图像中的水平框检测和实例分割；提供了与DINOv3紧密耦合的PP/R-SEM/UAM的明确闭式公式；并在小型和拥挤目标以及跨域迁移下展示了 extit{一致}的增益，同时保持了简单的部署。

View on arXiv Download PDF AI Translation

cs.CV / 61 / 2606.10775

Spatially Selective Self-Training for Unsupervised Building Change Detection

空间选择性自我训练用于无监督建筑变化检测

Hussin, Wafaa I. M., Lu, Zhi, Mohammed, Anas M. I., Zhou, Xiang, Abubaker, Ratiba A. H., Peng, Zhenming

Abstract

Unsupervised building change detection aims to learn building-change masks from unlabeled bi-temporal remote sensing images. Existing label-free methods often follow a discrepancy-to-mask paradigm, directly using temporal differences, frozen foundation-model responses, prompt-based outputs, or post-processing results as final change maps. Although these strategies provide annotation-free cues, they do not learn a task-specific building-change detector and remain vulnerable to the gap between generic temporal discrepancies and building-defined structural changes. In practice, such discrepancies are often noisy and task-irrelevant, as appearance shifts, registration errors, and non-building modifications can produce strong but misleading responses. To address this problem, we propose SST-CD, a spatially selective self-training framework that reformulates fully label-free building change detection as end-to-end detector learning under noisy pseudo supervision. SST-CD uses temporal discrepancies as candidate pseudo labels and trains the detector only on spatially reliable pixels, whose reliability is estimated by a local consistency criterion that filters inconsistent regions from supervision. To further stabilize noisy self-training, a lightweight feature adapter recalibrates bi-temporal features, while a prototype-based decoder produces compact change and no-change representations. Experiments on LEVIR-CD, WHU-CD, and DSIFN-CD show that SST-CD achieves F1 scores of 83.08\%, 91.69\%, and 86.60\%, respectively, outperforming existing unsupervised and label-free baselines. Code will be made publicly available.

Chinese Translation

无监督建筑变化检测旨在从未标记的双时相遥感图像中学习建筑变化掩膜。现有的无标签方法通常遵循差异到掩膜的范式，直接使用时间差异、冻结的基础模型响应、基于提示的输出或后处理结果作为最终变化图。尽管这些策略提供了无注释的线索，但它们并没有学习特定任务的建筑变化检测器，并且仍然容易受到通用时间差异与建筑定义的结构变化之间差距的影响。在实际应用中，这种差异通常是嘈杂且与任务无关的，因为外观变化、配准错误和非建筑修改可能产生强烈但误导性的响应。为了解决这个问题，我们提出了SST-CD，一个空间选择性自我训练框架，将完全无标签的建筑变化检测重新表述为在嘈杂伪监督下的端到端检测器学习。SST-CD使用时间差异作为候选伪标签，仅在空间可靠的像素上训练检测器，其可靠性通过局部一致性标准进行估计，以过滤掉不一致的监督区域。为了进一步稳定嘈杂的自我训练，一个轻量级特征适配器重新校准双时相特征，而基于原型的解码器生成紧凑的变化和无变化表示。在LEVIR-CD、WHU-CD和DSIFN-CD上的实验表明，SST-CD分别达到了83.08%、91.69%和86.60%的F1分数，超越了现有的无监督和无标签基线。代码将公开发布。

View on arXiv Download PDF AI Translation

cs.CV / 62 / 2606.10778

From Patches to Patients: A study of the tile-to-slide performance transferability in Digital Pathology

从切片到患者：数字病理学中瓦片到幻灯片性能可转移性的研究

Boutaj, Sofiène, Fillioux, Leo, Vakalopoulou, Maria, Christodoulidis, Stergios, Marza, Pierre

Abstract

Foundation Models (FMs) have recently redefined the state-of-the-art in histopathology by providing robust representations for whole-slide image (WSI) analysis. However, selecting the optimal foundation model (FM) for a specific clinical cohort currently requires multiple preprocessing steps, followed by computationally expensive feature extraction and the training of a Multiple Instance Learning (MIL) aggregator for every model. In this work, we investigate whether efficient tile-level linear probing can serve as a reliable proxy for slide-level performance, reducing the need to run full slide-level pipelines for every candidate encoder. We benchmark 19 state-of-the-art FMs on 42 slide-level and 16 tile-level tasks, comparing tile probing metrics against slide-level outcomes using ABMIL and Mean Pooling aggregations. We observe a high correlation between tile and slide performance across varying task difficulties, indicating that encoder representation quality is the primary determinant of WSI success. Sensitivity analyses show that transferability is stable across models and is more influenced by cohort sizes and numbers of tiles per slide than by average task difficulty. We also measure the agreement in best performing models between tile and slide-level tasks, showing tile benchmarks reliably shortlist strong candidates. Overall, our study indicates that tile-level benchmarking provides an efficient and practical first step for narrowing down candidate models, while slide-level evaluation remains essential for final validation on clinical tasks.

Chinese Translation

基础模型（Foundation Models, FMs）最近通过为全幻灯片图像（Whole-Slide Image, WSI）分析提供稳健的表示，重新定义了组织病理学的最新技术。然而，为特定临床队列选择最佳基础模型（FM）目前需要多个预处理步骤，随后是计算成本高昂的特征提取和为每个模型训练多个实例学习（Multiple Instance Learning, MIL）聚合器。在本研究中，我们探讨了高效的瓦片级线性探测是否可以作为幻灯片级性能的可靠代理，从而减少对每个候选编码器运行完整幻灯片级流程的需求。我们在42个幻灯片级和16个瓦片级任务上对19个最新的基础模型进行了基准测试，比较了瓦片探测指标与幻灯片级结果，使用了ABMIL和均值池化（Mean Pooling）聚合。我们观察到在不同任务难度下，瓦片和幻灯片性能之间存在高度相关性，表明编码器表示质量是WSI成功的主要决定因素。敏感性分析表明，模型间的可转移性是稳定的，受队列规模和每张幻灯片的瓦片数量影响大于平均任务难度。我们还测量了瓦片和幻灯片级任务之间最佳表现模型的一致性，显示瓦片基准能够可靠地筛选出强有力的候选者。总体而言，我们的研究表明，瓦片级基准测试为缩小候选模型范围提供了高效且实用的第一步，而幻灯片级评估仍然是临床任务最终验证的必要环节。

View on arXiv Download PDF AI Translation

cs.CV / 63 / 2606.10790

A Multimodal RGB and Events Dataset for Hand Detection in First-Person View

用于第一人称视角手部检测的多模态RGB与事件数据集

Kota, Bharghav, Sandamirskaya, Yulia

Abstract

Existing hand detection algorithms work on images and the detection rate is restricted by the frame rate of the camera. In hand detection applications for moving robotic systems, conventional cameras cause motion blur, especially in darker lighting conditions. We can leverage the use of event-based cameras which possess a high dynamic range, high temporal resolution, and low power consumption. Recent work has shown that using a stereo setup of an event-based and a frame-based camera improves detection accuracy and the bandwidth-latency tradeoff. The main bottleneck in using event-based cameras in object detection and recognition tasks is a relatively low amount of training data. In this work, we propose a methodology and an exemplary synthetic event-based hand dataset from an egocentric, first-person view perspective. The data is synthesized from the existing RGB Egohands dataset with the v2e toolbox. Parameters of the v2e toolbox are varied to provide versions of the dataset with different lighting conditions and scales. Ground truth detections are generated with a fine-tuned YOLOv8 model which is applied to the RGB images in the Egohands dataset and interpolated on the high-temporal resolution events. We use the multi-modal dataset to perform hand detection with existing object detection algorithms which use a multi-modal setup of event and RGB cameras and demonstrate performance comparable to the state-of-the-art.

Chinese Translation

现有的手部检测算法主要针对图像进行处理，其检测率受到相机帧率的限制。在移动机器人系统的手部检测应用中，传统相机会导致运动模糊，尤其是在较暗的光照条件下。我们可以利用事件驱动相机，这种相机具有高动态范围、高时间分辨率和低功耗的特点。最近的研究表明，使用事件驱动相机与帧驱动相机的立体设置可以提高检测精度和带宽-延迟的权衡。在物体检测和识别任务中，使用事件驱动相机的主要瓶颈是相对较少的训练数据。在本研究中，我们提出了一种方法论，并从自我中心的第一人称视角出发，构建了一个示例性的合成事件驱动手部数据集。该数据集是通过使用v2e工具箱从现有的RGB Egohands数据集中合成的。我们对v2e工具箱的参数进行了调整，以提供不同光照条件和尺度的多个数据集版本。通过对Egohands数据集中的RGB图像应用经过微调的YOLOv8模型生成真实检测结果，并在高时间分辨率事件上进行插值。我们利用这个多模态数据集，结合现有的物体检测算法，使用事件和RGB相机的多模态设置进行手部检测，并展示了与最先进技术相当的性能。

View on arXiv Download PDF AI Translation

cs.CV / 64 / 2606.10804

SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning

SCAIL-2：统一受控角色动画与端到端上下文条件

Yan, Wenhao, Guo, Fengjia, Yang, Zhuoyi, Tang, Jie

Abstract

Controlled character animation requires transferring motion from a driving sequence to a reference character. Prior works heavily rely on intermediate representations, including pose skeletons to represent motion or masked background to represent environment, which inevitably leads to information loss. To address this, we present SCAIL-2, an framework that bypasses those intermediates and achieves \textbf{end-to-end} character animation. By directly concatenating driving videos to the sequence, the model can obtain all the required visual information from the input video. To address lack of end-to-end data, we unify sub-tasks of character animation with decoupled conditions and then curate a pipeline to synthesize MotionPair-60K, an end-to-end motion transfer dataset containing heterogeneous tasks of character animation. To archive the unification, we utilize in-context mask conditioning and mode-specific RoPE as soft guidance beyond textual instructions and raw visual information. To address synthetic discrepancy in detailed regions, we propose Bias-Aware DPO to construct preference items to mitigate the errors. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches in various character animation tasks. A large subset of synthetic data as well as model weights will be released at our project page: https://teal024.github.io/SCAIL-2/.

Chinese Translation

受控角色动画需要将运动从驱动序列转移到参考角色。以往的研究严重依赖于中间表示，包括用于表示运动的姿势骨架或用于表示环境的遮罩背景，这不可避免地导致信息损失。为了解决这个问题，我们提出了 SCAIL-2，一个绕过这些中间环节并实现端到端角色动画的框架。通过直接将驱动视频与序列连接，模型可以从输入视频中获取所有所需的视觉信息。为了应对端到端数据的缺乏，我们将角色动画的子任务与解耦条件统一，并策划一个管道来合成 MotionPair-60K，这是一个包含异构角色动画任务的端到端运动转移数据集。为了实现统一，我们利用上下文遮罩条件和特定模式的 RoPE 作为超越文本指令和原始视觉信息的软指导。为了解决细节区域的合成差异，我们提出了偏见感知 DPO 来构建偏好项，以减轻错误。大量实验表明，我们的方法在各种角色动画任务中显著优于现有的最先进方法。我们将在项目页面 https://teal024.github.io/SCAIL-2/ 发布大量合成数据子集及模型权重。

View on arXiv Download PDF AI Translation

cs.CV / 65 / 2606.10811

Deep learning for echo sounder data

深度学习在回声探测器数据中的应用

Malde, Ketil

Abstract

There is no doubt that over the last decade, techniques from the field of machine learning have revolutionized how we process and interpret data, especially images and text. For underwater observations acoustics is a primary source of information, and naturally, deep learning methods have been applied to echograms and other acoustics data, but so far with rather modest results. Here, we argue that due to intrinsic properties of acoustic data, substantial advances will likely require research into deep learning methods beyond mere recycling of models and techniques from image processing. Currently, the potential for breakthroughs in method development is hindered by the lack of standard data formats and organization, and even more by the lack of readily available, high quality data sets with established performance goals. To advance the field, these shortcomings should be remedied

Chinese Translation

毫无疑问，在过去十年中，机器学习领域的技术彻底改变了我们处理和解释数据的方式，尤其是图像和文本。对于水下观测，声学是主要的信息来源，因此，深度学习方法已被应用于回声图和其他声学数据，但迄今为止，结果相对有限。在此，我们认为，由于声学数据的内在特性，实质性的进展可能需要对深度学习方法进行研究，而不仅仅是对图像处理模型和技术的简单重复。目前，方法开发的突破潜力受到标准数据格式和组织缺乏的限制，更重要的是，缺乏现成的高质量数据集和既定的性能目标。为了推动该领域的发展，必须解决这些不足之处。

View on arXiv Download PDF AI Translation

cs.CV / 66 / 2606.10819

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

Earth-OneVision：将遥感多模态大语言模型扩展至更多传感器模态和任务

Cai, Miaoxin, Wang, Guanqun, Zhang, Wei, Zhou, Guangyao, Zhuang, Yin, Zhang, Tong, Wang, Hao, Chen, He, Li, Jun

Abstract

RS-MLLMs enable natural-language understanding and spatial reasoning over earth observation imagery. However, existing models support only a narrow range of sensor types and tasks, yielding a fragmented view of the earth and leaving cross-modal geoscientific knowledge largely unexploited. This work presents Earth-OneVision, a 2B RS-MLLM that unifies six sensor modalities (i.e., optical, SAR, infrared, multispectral, temporal, and video) and cross-sensor fusion across 9 task categories within a single autoregressive framework. Three dedicated mechanisms address three bottlenecks. Full-Granularity Vision-Language Alignment (FGVLA) aligns multi-level visual features with the multi-dimensional language space. Spatial-Linguistic Isomorphic Serialization (SLIS) unifies heterogeneous spatial outputs as autoregressive tokens. Progressive Cross-Modality Adaptation (PCMA) decomposes the compound domain gap into sequential stages, tackling the viewpoint and imaging physics gaps in turn. To support joint training, MMRS-OneVision is constructed with ~34M QA pairs spanning all six sensor modalities and cross-sensor fusion across 9 task categories, substantially exceeding existing RS multimodal instruction datasets. With only 2B parameters, Earth-OneVision achieves competitive or state-of-the-art results across extensive benchmarks, consistently matching or outperforming 4B-72B RS-MLLMs. It achieves 87.52% [email protected] on the OPT-RSVG testset for optical visual grounding and 80.68% on the SAR VQA benchmark SARLANG-Bench, exceeding 7B models by over 7%. It further achieves 75.74% recall on the BigEarthNet-MS testset for multispectral classification, and 81.94% MCQ accuracy on EarthMind-Bench for cross-modality reasoning.

Chinese Translation

RS-MLLMs（遥感多模态大语言模型）使得对地球观测影像的自然语言理解和空间推理成为可能。然而，现有模型仅支持有限的传感器类型和任务，导致对地球的视角碎片化，并使得跨模态地球科学知识未得到充分利用。本研究提出了Earth-OneVision，一个统一六种传感器模态（即光学、合成孔径雷达（SAR）、红外、多光谱、时间序列和视频）的2B RS-MLLM，并在单一自回归框架内实现了跨传感器融合，涵盖9个任务类别。三个专门机制解决了三个瓶颈问题。全粒度视觉-语言对齐（FGVLA）将多层次视觉特征与多维语言空间对齐。空间-语言同构序列化（SLIS）将异构空间输出统一为自回归标记。渐进式跨模态适应（PCMA）将复合领域差距分解为顺序阶段，依次解决视角和成像物理差距。为了支持联合训练，MMRS-OneVision构建了约3400万个涵盖所有六种传感器模态和跨传感器融合的问答对，显著超过现有的遥感多模态指令数据集。仅用20亿参数，Earth-OneVision在广泛的基准测试中取得了具有竞争力或领先的结果，始终与40亿至720亿参数的RS-MLLMs相匹配或超越。在光学视觉定位的OPT-RSVG测试集上，其[email protected]达到了87.52%，在SAR VQA基准SARLANG-Bench上的表现为80.68%，超过了70亿模型超过7%。此外，在BigEarthNet-MS测试集上，其多光谱分类召回率达到了75.74%，在EarthMind-Bench上跨模态推理的多项选择题准确率为81.94%。

View on arXiv Download PDF AI Translation

cs.CV / 67 / 2606.10839

HarmoView: Harmonizing Multi-View Constraints for Identity-Consistent Video Generation

HarmoView：协调多视角约束以实现身份一致的视频生成

Wang, Cong, Yu, Zhentao, Wang, Hongmei, Liang, Weicong, Zhou, Zixiang, Yang, Zilin, Ou, Jiarong, Chen, Rui, Zhou, Yuan, Lu, Qinglin

Abstract

Current identity-consistent video generation methods struggle to preserve appearance fidelity under large viewpoint changes. While introducing multi-view reference input offers a natural solution, progress remains constrained by the lack of effective frameworks for multi-view inputs and the scarcity of multi-view data. We address these challenges by proposing HarmoView, a robust framework for identity-consistent video generation that effectively integrates multi-view cues through three architectural refinements complemented by a staged training curriculum. Specifically, we first introduce Multi-level Feature Injection to anchor identity fidelity; by injecting raw ViT features from frontal references alongside text tokens via cross-attention, MFI provides persistent low-level appearance anchors that complement the high-level identity features within DiT blocks, leading to enhanced identity preservation. Then, we employ learnable proxy tokens to unify heterogeneous reference layouts across single-/multi-view settings while simultaneously resolving the reference-view mismatch problem. Jump-RoPE is further developed for identity-wise feature isolation to reduce identity crosstalk. To activate these structural capabilities while preserving the original generative priors, we propose the Progressive View Curriculum. This four-stage training strategy employs view dropout to facilitate a stable transition from vanilla T2V generation to high-fidelity, identity-persistent spatial reasoning. Furthermore, we construct a large-scale multi-view dataset to address the issue of data scarcity. Extensive evaluation on our multi-view benchmark, comprising 100 manually-curated cases spanning 52 unique identities, demonstrates that HarmoView significantly outperforms open-source baselines and matches leading closed-source engines, achieving state-of-the-art performance in identity-consistent video generation.

Chinese Translation

当前的身份一致视频生成方法在大视角变化下难以保持外观保真度。虽然引入多视角参考输入提供了一种自然的解决方案，但由于缺乏有效的多视角输入框架和多视角数据的稀缺，进展仍然受到限制。我们通过提出HarmoView来解决这些挑战，这是一种强大的身份一致视频生成框架，通过三种架构改进有效整合多视角线索，并辅以分阶段的训练课程。具体而言，我们首先引入多层特征注入（Multi-level Feature Injection）以锚定身份保真度；通过通过交叉注意力注入来自正面参考的原始ViT特征和文本令牌，MFI提供了持久的低级外观锚点，补充了DiT块中的高级身份特征，从而增强了身份保留。然后，我们采用可学习的代理令牌统一单视角/多视角设置中的异构参考布局，同时解决参考视图不匹配问题。进一步开发的Jump-RoPE用于身份特征隔离，以减少身份串扰。为了激活这些结构能力，同时保持原始生成先验，我们提出了渐进视图课程（Progressive View Curriculum）。这一四阶段训练策略采用视图丢弃（view dropout）以促进从普通T2V生成到高保真、身份持久空间推理的稳定过渡。此外，我们构建了一个大规模多视角数据集以解决数据稀缺问题。在我们的多视角基准上进行了广泛评估，该基准包含100个手动策划的案例，涵盖52个独特身份，结果表明HarmoView显著优于开源基线，并与领先的闭源引擎相匹配，在身份一致视频生成中实现了最先进的性能。

View on arXiv Download PDF AI Translation

cs.CV / 68 / 2606.10862

LIBERO-Occ: Evaluating and Improving Vision-Language-Action Models under Scene-Induced Occlusion via Viewpoint Imagination

LIBERO-Occ：通过视角想象评估和改进场景诱导遮挡下的视觉-语言-动作模型

Li, Taishan, Zhang, Jiwen, Wang, Siyuan, Huang, Xuanjing, Wei, Zhongyu

Abstract

Vision-Language-Action (VLA) models achieve strong performance on standard manipulation benchmarks, but most evaluations assume that task-relevant objects are fully visible. This assumption often fails in realistic settings, where occlusion makes manipulation partially observable. In this paper, we study \textit{scene-induced occlusion} as a fundamental challenge for VLA models and introduce \textbf{LIBERO-Occ}, an occlusion-oriented extension of LIBERO. Experiments show that state-of-the-art VLAs suffer substantial performance degradation under occlusion. To address this issue, we propose \textbf{Viewpoint Imagination (VIM)}, which generates a complementary view from an occluded primary observation and conditions action prediction on both observed and imagined evidence. VIM improves robustness across task suites, occlusion types, and severity levels without requiring additional cameras at deployment time, suggesting that viewpoint imagination is an promising mechanism for perception completion in partially observable manipulation. Our benchmark and corresponding code are available at: \href{https://github.com/litsh/Libero-Occ}{https://github.com/litsh/Libero-Occ}.

Chinese Translation

视觉-语言-动作（VLA）模型在标准操作基准上表现出色，但大多数评估假设任务相关对象是完全可见的。这一假设在现实环境中往往不成立，因为遮挡使得操作部分可观察。在本文中，我们研究了 extit{场景诱导遮挡}作为VLA模型的一个基本挑战，并引入了 extbf{LIBERO-Occ}，这是LIBERO的一个以遮挡为导向的扩展。实验表明，最先进的VLA在遮挡情况下性能显著下降。为了解决这一问题，我们提出了 extbf{视角想象（Viewpoint Imagination, VIM）}，该方法从被遮挡的主要观察中生成一个补充视角，并在观察到的证据和想象的证据上对动作预测进行条件化。VIM在任务套件、遮挡类型和严重程度上提高了鲁棒性，而无需在部署时增加额外的摄像头，这表明视角想象是部分可观察操作中感知补全的一个有前景的机制。我们的基准和相应的代码可在以下链接获取： exttt{https://github.com/litsh/Libero-Occ}。

View on arXiv Download PDF AI Translation

cs.CV / 69 / 2606.10874

Schmidt Decomposition-Based Methods for Efficient Quantum Image Encoding

基于施密特分解的高效量子图像编码方法

Pangeva, Ana-Maria, Ferhi, Yassine, Geng, Alexander, Weinmann, Andreas, Ivanova, Desislava, Moghiseh, Ali

Abstract

In quantum image processing, a fundamental step is encoding classical image data into quantum states. This can be achieved using methods such as Flexible Representation of Quantum Images (FRQI), Quantum Probability Image Encoding (QPIE), and Novel Enhanced Quantum Representation (NEQR). However, on real quantum hardware, these encodings can quickly lead to circuits with many gates, large circuit depth, and high qubit usage, which is a problem for Noisy Intermediate-Scale Quantum (NISQ) devices. In this work, we investigate whether low-rank state approximation, formulated via Schmidt decomposition, can help reduce this complexity. The method keeps only the most significant parts of a quantum state's entanglement structure, making state preparation more efficient while preserving most of the image information. We compare the three encoding techniques in their original form and with low-rank approximation, evaluating metrics such as circuit depth, CNOT count, MSE, and visual quality of reconstructed images. The results reveal meaningful trade-offs between accuracy and resource efficiency, with the FRQI model achieving a 97 percent reduction in circuit depth while maintaining a near-perfect reconstruction (MSE of about 0.27). This demonstrates the potential of low-rank techniques for advancing practical quantum image processing on near-term hardware.

Chinese Translation

在量子图像处理过程中，一个基本步骤是将经典图像数据编码为量子态。这可以通过灵活的量子图像表示（Flexible Representation of Quantum Images, FRQI）、量子概率图像编码（Quantum Probability Image Encoding, QPIE）和新型增强量子表示（Novel Enhanced Quantum Representation, NEQR）等方法实现。然而，在实际的量子硬件上，这些编码方法很快会导致电路中有大量门、深度较大以及高量子比特使用，这对噪声中间规模量子（Noisy Intermediate-Scale Quantum, NISQ）设备来说是一个问题。在本研究中，我们探讨了通过施密特分解形成的低秩状态近似是否能够帮助减少这种复杂性。该方法仅保留量子态纠缠结构中最重要的部分，使得状态准备更加高效，同时保留大部分图像信息。我们比较了这三种编码技术在其原始形式和低秩近似下的表现，评估了电路深度、CNOT门数量、均方误差（MSE）和重构图像的视觉质量等指标。结果揭示了准确性与资源效率之间的有意义权衡，其中FRQI模型实现了97%的电路深度减少，同时保持了近乎完美的重构（MSE约为0.27）。这表明低秩技术在推动近期硬件上实用量子图像处理方面的潜力。

View on arXiv Download PDF AI Translation

cs.CV / 70 / 2606.10876

Advancing Wood Identification in the Philippines: Utilizing the Xylorix Platform for Efficient AI Model Development and Deployment for Five Key Species

推进菲律宾木材鉴定：利用Xylorix平台高效开发和部署五种关键物种的人工智能模型

Mendoza, Rosalie C., Daracan, Vivian C., Romano, Arlene D., Manalo, Ronniel D., Tang, Xin Jie, Wong, Yi Hong, Tay, Yong Haur

Abstract

Illegal logging and timber trade continue to pose significant challenges in the Philippines, where accurate wood species identification is essential for enforcement but limited by the need for specialised equipment and expertise. This study aims to evaluate whether AI models for macroscopic wood identification can be developed and deployed by wood scientists without programming expertise using the Xylorix platform, focusing on five Philippine hardwood species: Mangium (Acacia mangium Willd.), Rain Tree [Samanea saman (Jacq.) Merr.], Banuyo (Wallaceodendron celebicum Koord.), Tindalo [Afzelia rhomboidea (Blanco) Vidal], and Ipil [Intsia bijuga (Colebr.) O. Kuntze]. Binary classifiers were trained on 10,663 verified cross-section images from 260 specimens and evaluated using specimen-level mean scoring to mirror operational field conditions. Area Under the ROC Curve (AUC) values ranged from 0.969 (Ipil) to 1.000 (Mangium), and Average Precision (AP) values ranged from 0.589 (Samanea) to 1.000 (Mangium). Four of five species achieved AA grade (AUC and AP both \geq 0.90); Rain Tree received AE (AUC \geq 0.90, AP < 0.60) due to AP compression from its small positive test set (3 specimens). All five classifiers rank their target specimens above non-target specimens with near-perfect fidelity. Specimen-level error analysis revealed 9 false negatives from Ipil, primarily stemming from localized image artifacts and 3 false positives for Rain Tree and 1 false positive for Tindalo caused by shared tribal-level anatomical traits. These findings demonstrate that Xylorix non-programmers can leverage the Xylorix platform to construct operationally reliable wood identification models suitable for field deployment at supply chain checkpoints.

Chinese Translation

非法伐木和木材贸易在菲律宾持续构成重大挑战，准确的木材物种鉴定对于执法至关重要，但受到专业设备和专业知识需求的限制。本研究旨在评估木材科学家是否可以利用Xylorix平台在没有编程专业知识的情况下开发和部署宏观木材鉴定的人工智能模型，重点关注五种菲律宾硬木物种：Mangium (Acacia mangium Willd.)、Rain Tree [Samanea saman (Jacq.) Merr.]、Banuyo (Wallaceodendron celebicum Koord.)、Tindalo [Afzelia rhomboidea (Blanco) Vidal] 和Ipil [Intsia bijuga (Colebr.) O. Kuntze]。我们在260个标本的10,663张经过验证的横截面图像上训练了二元分类器，并使用标本级平均评分进行评估，以反映实际操作现场条件。ROC曲线下面积（AUC）值范围从0.969（Ipil）到1.000（Mangium），平均精度（AP）值范围从0.589（Samanea）到1.000（Mangium）。五种物种中有四种达到了AA等级（AUC和AP均≥0.90）；Rain Tree由于其小的正测试集（3个标本）导致AP压缩，获得了AE等级（AUC≥0.90，AP<0.60）。所有五个分类器均将其目标标本的排名置于非目标标本之上，且几乎完美的保真度。标本级错误分析显示，Ipil出现了9个假阴性，主要源于局部图像伪影，而Rain Tree和Tindalo分别出现了3个和1个假阳性，原因是共享的族群级解剖特征。这些发现表明，Xylorix非程序员可以利用Xylorix平台构建适合在供应链检查点现场部署的操作可靠的木材鉴定模型。

View on arXiv Download PDF AI Translation

cs.CV / 71 / 2606.10887

Listen, Look, and Learn: Learning Without Forgetting through SAM-Audio

倾听、观察与学习：通过SAM-Audio实现无遗忘学习

Gupta, Avi, Sinha, Nilotpal, Raj, Vishnu, Saha, Sambuddha, Joshi, Pratik, Jerripothula, Koteswar Rao, Tillo, Tammam

Abstract

Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. While recent CIL advances have spurred significant interest across various modalities, the audio-visual setting remains underexplored. Furthermore, although foundational multimodal models like SAM-Audio encapsulate rich static priors, our empirical analysis reveals that these representations struggle in incremental settings. This work bridges this gap by integrating SAM-Audio's audio-visual priors into the CIL setting. Specifically, we leverage its dense audio and visual representations and employ a novel guided attention strategy where the audio features contextually guide the visual representations. To further mitigate catastrophic forgetting, we introduce dual-level distillation objectives at both the feature and logit levels. Extensive evaluations on audio-visual CIL benchmarks demonstrate that our approach consistently outperforms state-of-the-art methods.

Chinese Translation

类增量学习（Class-Incremental Learning, CIL）旨在不断学习新类别而不遗忘先前获得的知识。尽管最近的CIL进展在各个模态中引发了显著的兴趣，但音视频设置仍然未被充分探索。此外，尽管基础的多模态模型如SAM-Audio封装了丰富的静态先验，我们的实证分析显示这些表示在增量设置中表现不佳。本研究通过将SAM-Audio的音视频先验整合到CIL设置中，填补了这一空白。具体而言，我们利用其密集的音频和视觉表示，并采用一种新颖的引导注意力策略，使音频特征在上下文中引导视觉表示。为了进一步减轻灾难性遗忘，我们在特征和对数几率层引入了双层蒸馏目标。在音视频CIL基准上的广泛评估表明，我们的方法始终优于最先进的技术。

View on arXiv Download PDF AI Translation

cs.CV / 72 / 2606.10892

Improving Text-Instance Alignment Of Foreground Conditioned Out-Painting Via Customized Concept Embedding

通过定制概念嵌入改善前景条件下的图像外扩文本实例对齐

Zhao, Yihao, Han, Xuan, He, Bin, You, Mingyu

Abstract

To showcase products, merchants often incur substantial costs creating high-quality display images. Foreground Conditioned Outpainting (FCO) meets this demand, allowing users to create desired backgrounds for foreground instances at a low cost by adjusting the text prompt. However, existing text-driven FCO methods exhibit critical flaws in their outputs, most notably the presence of artifacts, which refer to regions in the synthesized background that share the same semantics as the foreground instance. Such artifacts diminish the object's prominence and degrade image quality. We attribute the issue to the misalignment between the given instance and text-derived concept embeddings. To address this, we propose the Customized Concept Embedding Diffusion (CCE-Diffusion) framework. Its core is a CCE-Module to customize concept embeddings, bridging the gap between generic noun semantics and a specific visual instance. An Instance-Aware Loss guides the module's optimization, while a Semantic-Preserving Prompt Template prevents customized embeddings from distorting other words in the prompt. Both qualitative and quantitative evaluations demonstrate that CCE-Diffusion significantly reduces artifacts in the outputs. As a plug-and-play component, the CCE-Module can integrate with various FCO methods, enhancing their performance.

Chinese Translation

为了展示产品，商家通常需要投入大量成本来创建高质量的展示图像。前景条件下的图像外扩（Foreground Conditioned Outpainting, FCO）满足了这一需求，使用户能够通过调整文本提示以低成本为前景实例创建所需的背景。然而，现有的基于文本的FCO方法在输出中存在严重缺陷，最显著的是出现伪影，即合成背景中与前景实例具有相同语义的区域。这些伪影降低了物体的显著性并降低了图像质量。我们将这一问题归因于给定实例与文本派生概念嵌入之间的错位。为了解决这个问题，我们提出了定制概念嵌入扩散（Customized Concept Embedding Diffusion, CCE-Diffusion）框架。其核心是CCE模块，用于定制概念嵌入，弥合通用名词语义与特定视觉实例之间的差距。实例感知损失（Instance-Aware Loss）指导模块的优化，而语义保留提示模板（Semantic-Preserving Prompt Template）则防止定制嵌入扭曲提示中的其他词汇。定性和定量评估均表明，CCE-Diffusion显著减少了输出中的伪影。作为一个即插即用的组件，CCE模块可以与各种FCO方法集成，从而提升其性能。

View on arXiv Download PDF AI Translation

cs.CV / 73 / 2606.10894

The 1st PortraitCraft Challenge: A CVPR 2026 Workshop Competition on Portrait Composition Understanding and Generation

首届 PortraitCraft 挑战赛：CVPR 2026 研讨会关于肖像构图理解与生成的竞赛

Lou, Zijie, Tang, Youyun, Qu, Xiaochao, Li, Haoxiang, Liu, Ting, Liu, Luoqi, Zhu, Xun, Zhang, Zheng, Chen, Xi, Li, Miao, Wu, Ji, Zhang, Dizhe, Ge, Xian, Wang, Sujia, Zhang, Ruiyang, Wang, Jiaming, Wang, Xianshun, Qi, Lu, Kang, Boao, Zhou, Wei, Sun, Jinghui, Yan, Zhenyu, Zhao, Jiliang, Yang, Rui, Huang, Yipo, Liu, Boyuan, Li, Shanglin, Xie, Zifan, Zhang, Yichen, Wang, Anlan, Lin, Wenfeng, Guo, Mingyu, Li, Dong, Wang, Xinghao, Li, Yanting, Tong, Shanzhao, He, Shuai, Zhou, Qiu, Yang, Yongqi, Mu, Taoyang, Lei, Dianqiao, Ming, Anlong, Ma, Huadong

Abstract

This paper presents an overview of the inaugural PortraitCraft Challenge, held as one of the official competitions at CVPR 2026. The challenge focuses on portrait composition understanding and generation, aiming to advance AI research in portrait aesthetics analysis and controllable image synthesis. Unlike existing datasets and tasks that primarily focus on global aesthetic scoring, PortraitCraft introduces a unified evaluation framework comprising two complementary tracks. Track 1 requires models to perform structured portrait composition understanding, and Track 2 requires models to generate portrait images from structured composition descriptions under explicit compositional constraints. To support the challenge, we constructed and publicly released a large-scale portrait composition dataset consisting of approximately 50,000 curated real portrait images, providing multi-level supervision. This report describes the challenge setup, evaluation protocols, dataset composition, and final results, along with an analysis of the technical characteristics of the submitted solutions. The PortraitCraft Challenge provides a standardized and reproducible platform for research on portrait composition understanding and generation, and is expected to foster further progress in the fields of portrait aesthetics and controllable image generation.

Chinese Translation

本文概述了首届 PortraitCraft 挑战赛，该挑战赛作为 CVPR 2026 的官方竞赛之一举行。该挑战聚焦于肖像构图的理解与生成，旨在推动人工智能在肖像美学分析和可控图像合成方面的研究。与现有主要关注全球美学评分的数据集和任务不同，PortraitCraft 引入了一个统一的评估框架，包括两个互补的赛道。赛道 1 要求模型进行结构化的肖像构图理解，而赛道 2 要求模型在明确的构图约束下，从结构化的构图描述中生成肖像图像。为了支持该挑战，我们构建并公开发布了一个大规模的肖像构图数据集，包含约 50,000 张经过精心挑选的真实肖像图像，提供多层次的监督。本文报告描述了挑战的设置、评估协议、数据集构成及最终结果，并对提交解决方案的技术特征进行了分析。PortraitCraft 挑战赛为肖像构图理解与生成的研究提供了一个标准化和可重复的平台，预计将促进肖像美学和可控图像生成领域的进一步发展。

View on arXiv Download PDF AI Translation

cs.CV / 74 / 2606.10902

Pose-ICL: 3D-Aware In-Context Learning for Pose-Controllable Subject Customization

Pose-ICL：用于姿态可控主体定制的3D感知上下文学习

Han, Xuan, Zhao, Yihao, You, Mingyu

Abstract

Subject Customization is a foundational task in modern image generation. By providing a few reference images and a text prompt, users can generate images of a specific object in any desired scene. However, existing methods still struggle to achieve effective pose control for customized subjects. In practice, they often exhibit inaccurate poses or inconsistent cross-pose appearances. These limitations suggest that understanding objects in a volumetric manner remains a significant challenge for 2D-native backbones. To address this challenge, we propose Pose-ICL, a tuning-free framework that leverages 3D-aware In-Context Learning (ICL) to directly adapt to new subjects through multiple paired image-pose references. Its core mechanism,Surface-Anchored Position Embedding (SAPE), equips the model with explicit 3D awareness by anchoring image tokens to the surface coordinates of a volumetric bounding box. Dedicated refinements ensure its seamless compatibility with existing DiT models. Extensive evaluations on both 3D assets and real-world subjects demonstrate that Pose-ICL significantly outperforms current methods in both pose accuracy and identity consistency.

Chinese Translation

主体定制是现代图像生成中的一项基础任务。通过提供少量参考图像和文本提示，用户可以在任何期望的场景中生成特定对象的图像。然而，现有方法在实现定制主体的有效姿态控制方面仍然存在困难。在实际应用中，它们往往表现出不准确的姿态或不一致的跨姿态外观。这些局限性表明，以体积方式理解对象仍然是对2D原生骨干网络的一项重大挑战。为了解决这一挑战，我们提出了Pose-ICL，这是一种无调优框架，利用3D感知上下文学习（ICL）通过多个配对的图像-姿态参考直接适应新主体。其核心机制，表面锚定位置嵌入（Surface-Anchored Position Embedding, SAPE），通过将图像标记锚定到体积边界框的表面坐标，赋予模型明确的3D感知。专门的改进确保其与现有的DiT模型无缝兼容。对3D资产和真实世界主体的广泛评估表明，Pose-ICL在姿态准确性和身份一致性方面显著优于当前方法。

View on arXiv Download PDF AI Translation

cs.CV / 75 / 2606.10905

Beyond Model Size: Probing the Gaps in Visual in-Context Learning by Training a Tiny Model

超越模型规模：通过训练一个微型模型探究视觉上下文学习中的差距

Khatri, Sunil, Landgraf, Steven, Ulrich, Markus, Reiß, Simon

Abstract

Visual in-Context Learning (VICL) aims at making progress towards adaptive vision models, that can -- based on a few examples -- adapt to a new task at test-time. With the history of in-context learning in natural language processing research, where large, parameter-heavy models are in use, one pathway that current VICL methods take is model- and data-scaling as key ingredients. Yet, it is not clear, whether these ingredients are the key for in-context learning to take shape in vision models. To stress-test such large models, we challenge them with an extreme counterexample: we train a tiny visual in-context model with merely $1$ million parameters and a modest amount of $70,000$ images. We compare the results of this severely capacity capped tiny model to $7,000\times$ larger VICL models in different adaptive settings, (1) on image data with small distribution shifts, (2) on unseen task encodings and (3) on a completely new task, i.e., the setting VICL envisions. With the chasm of training resources between the tiny- and large models, our experiments showcase a lack in how adaptive capabilities are measured, with respect to how tasks are encoded, which tasks were used in pre-training and the choice of metrics. These gaps in current VICL benchmarking underscore a need for innovation in evaluation of adaptive capabilities.

Chinese Translation

视觉上下文学习（Visual in-Context Learning, VICL）旨在朝着自适应视觉模型的方向发展，这些模型能够基于少量示例在测试时适应新任务。在自然语言处理研究中，上下文学习的历史表明，当前使用的是大型、参数繁重的模型，而现有的VICL方法则将模型和数据扩展作为关键要素。然而，目前尚不清楚这些要素是否是视觉模型实现上下文学习的关键。为了对这些大型模型进行压力测试，我们提出了一个极端的反例：我们训练了一个仅有100万参数和7万张图像的微型视觉上下文模型。我们将这个严重限制容量的微型模型的结果与在不同自适应设置下的7000倍更大VICL模型进行比较，（1）在具有小分布偏移的图像数据上，（2）在未见过的任务编码上，以及（3）在完全新任务上，即VICL设想的设置。由于微型模型与大型模型之间的训练资源差距，我们的实验展示了在如何测量自适应能力方面的不足，这与任务编码、预训练中使用的任务以及指标选择有关。这些当前VICL基准中的差距强调了在评估自适应能力方面创新的必要性。

View on arXiv Download PDF AI Translation

cs.CV / 76 / 2606.10939

PENet+: A Lightweight Residual Transformer Framework for Efficient Image Steganalysis

PENet+: 一种轻量级残差变换器框架用于高效图像隐写分析

AN, Jincheol, Kim, Dongsu, Jang, Haneol, Yoo, YoungJoon

Abstract

Image steganalysis, the detection of hidden information embedded in digital images, is a core component of modern cybersecurity and digital forensics. Recent residual Transformer architectures, such as the Pixel-Difference-Convolution and Enhanced-Transformer-Network (PENet) [1], achieve strong detection accuracy, but their computational and memory demands hinder deployment in resource-constrained settings. We present PENet+, a lightweight steganalysis framework that preserves PENet's discriminative structure while substantially improving efficiency. Rather than redesigning or compressing the attention blocks, we retain PENet's self-attention topology for reproducibility and add a classifier-streamlining stage that progressively narrows the SPP-to-FC1 input channels (SPP: spatial pyramid pooling; FC1: first fully connected layer), yielding large reductions in parameters and FLOPs with negligible accuracy loss. We further refine the high-pass-filter (HPF) stem with an activation-aware mechanism that aggregates HPF responses early and selects a balanced SRM-Gabor top-K subset, and we replace PENet's backbone with a MobileNetV2-style inverted residual network. A balanced configuration with K=31 filters (16 Gabor + 15 SRM) matches or surpasses heavier settings at lower compute. Finally, we motivate PReLU from a steganalysis standpoint, arguing that preserving negative responses helps capture weak stego cues that ReLU suppresses. On a disjoint ALASKA2 JPEG QF90 protocol at 512x512 resolution (5,000 cover images for training, validation, and internal testing; a separate 19,000-cover evaluation set), PENet+ achieves up to 45.5% fewer parameters and about 97% fewer FLOPs than the re-evaluated PENet baseline, offering a computationally efficient direction for resource-constrained steganalysis. Device-level latency and power measurements remain future work.

Chinese Translation

图像隐写分析是检测嵌入数字图像中的隐藏信息的核心组成部分，属于现代网络安全和数字取证的重要领域。近年来的残差变换器架构，如像素差异卷积和增强变换器网络（PENet）[1]，在检测准确性上取得了显著成果，但其计算和内存需求限制了在资源受限环境中的应用。我们提出了PENet+，一种轻量级的隐写分析框架，保留了PENet的区分性结构，同时显著提高了效率。我们没有重新设计或压缩注意力模块，而是保留了PENet的自注意力拓扑以确保可重复性，并增加了一个分类器精简阶段，该阶段逐步缩小了SPP到FC1的输入通道（SPP：空间金字塔池化；FC1：第一个全连接层），在参数和FLOPs上实现了大幅减少，同时准确率损失微乎其微。我们进一步优化了高通滤波器（HPF）主干，采用了一种激活感知机制，早期聚合HPF响应并选择一个平衡的SRM-Gabor前K子集，并将PENet的主干替换为MobileNetV2风格的倒残差网络。配置为K=31个滤波器（16个Gabor + 15个SRM）的平衡设置在较低的计算需求下与更重的设置相匹配或超越。最后，我们从隐写分析的角度提出了PReLU，认为保留负响应有助于捕捉ReLU抑制的微弱隐写线索。在512x512分辨率的独立ALASKA2 JPEG QF90协议下（5,000张封面图像用于训练、验证和内部测试；另有19,000张封面评估集），PENet+在参数上减少了多达45.5%，FLOPs减少约97%，为资源受限的隐写分析提供了计算效率高的方向。设备级延迟和功耗测量仍需未来研究。

View on arXiv Download PDF AI Translation

cs.CV / 77 / 2606.10940

Democratising Camera Trap AI: An Open-Source Model for Detecting UK Mammals

民主化相机陷阱人工智能：用于检测英国哺乳动物的开源模型

Fergus, Paul, Stephens, Philip, Hill, Russell A., Oliver, Lee, Appleby, Katie, Beatham, Sarah, Walsh, Naomi Davies, Nixon, Stuart, Matthews, Naomi, Sutherland, Chris, Hitchcock, Kelly

Abstract

Camera traps have become a cornerstone of biodiversity monitoring, but the artificial intelligence that turns vast quantities of images into usable ecological data is often locked behind commercial platforms or trained on fauna that does not match that of the British Isles. In an attempt to remove barriers and increase uptake, we release an open-source object detection model for 31 classes, 28 common UK mammal and bird species, plus utility classes for humans, calibration poles, and vehicles, drawn from a curated dataset of 48,165 labelled instances assembled from multiple sites over a decade of operational deployment through Conservation AI and its successor, Trap Tracker. The model, a YOLO26x detector trained and tested on an 80/10/10 class-stratified split, achieves a mean Average Precision of 0.984 at Intersection over Union (IoU) of 0.5 (0.956 at IoU 0.5-0.95) on the held-out validation set, with precision 0.988 and recall 0.965. On an unseen held-out test split, mean per-species confidence ranged from 0.96 to 0.99 across the 31 classes, with a 0.17% false-negative rate concentrated in difficult night-time, distant, or occluded images. These metrics are from data from the same pool of sites and cameras as training, so performance at entirely new sites is left to future work. We release the trained weights in ONNX format under a non-commercial licence, with local desktop and real-time camera support, aimed explicitly at ecologists with no machine-learning experience. This release is a deliberate counterweight to the multiple paid for models that have developed over the last decade.

Chinese Translation

相机陷阱已成为生物多样性监测的基石，但将大量图像转化为可用生态数据的人工智能通常被锁定在商业平台上，或训练于与不符合英国群岛的动物群不匹配的物种。为了消除障碍并增加使用率，我们发布了一个开源目标检测模型，涵盖31个类别，包括28种常见的英国哺乳动物和鸟类物种，以及用于人类、校准杆和车辆的实用类别，数据来源于一个经过精心策划的数据集，该数据集包含48,165个标注实例，历经十年的运营部署，通过Conservation AI及其后续项目Trap Tracker收集而成。该模型是一个YOLO26x检测器，在80/10/10的类别分层拆分上进行训练和测试，在保留的验证集上，在交并比（IoU）为0.5时，平均精确度达到0.984（在IoU 0.5-0.95时为0.956），精确率为0.988，召回率为0.965。在一个未见的保留测试集上，31个类别的每种物种的平均置信度范围从0.96到0.99，假阴性率为0.17%，主要集中在困难的夜间、远距离或被遮挡的图像中。这些指标来自与训练相同的地点和相机的数据，因此在全新地点的表现留待未来工作。我们以非商业许可证发布训练权重，采用ONNX格式，支持本地桌面和实时相机，特别针对没有机器学习经验的生态学家。这一发布是对过去十年中发展起来的多个付费模型的有意反制。

View on arXiv Download PDF AI Translation

cs.CV / 78 / 2606.10967

Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks

视觉上下文学习何去何从？跨领域和任务的统一基准

Halady, Pradnya, Wei, Jiale, Marinov, Zdravko, Jaus, Alexander, Reiß, Simon

Abstract

Visual in-context learning has been proposed as a pathway towards dynamic models that can generate predictions based on a provided context and thereby can adapt to new vision tasks at test-time. Yet, the evaluation of the adaptation capabilities of these models has been limited to narrow setups that mainly mirror tasks or image domains from pre-training for which real adaptation is not required. We address this gap by constructing a broad Visual In-Context BEnchmark (VIBE) with a focus on diverse imaging domains and a wide range of tasks. With this, we are able to get a much clearer picture of the adaptive capabilities of visual in-context models when faced with new image- and task distributions. We stress test six models on $14$ datasets and $12$ tasks (in total, we explore $106$ dataset-task combinations) and compare them under a unified, reproducible evaluation protocol, in an one-shot setting. Our evaluation uncovers key insights on the state of visual in-context learning, including limitations, systematic failure modes and promising directions. To foster broader evaluation, we will openly release our VIBE toolkit.

Chinese Translation

视觉上下文学习被提出作为一种动态模型的路径，这些模型可以基于提供的上下文生成预测，从而能够在测试时适应新的视觉任务。然而，这些模型的适应能力评估一直局限于狭窄的设置，主要反映了预训练任务或图像领域，而不需要真正的适应。我们通过构建一个广泛的视觉上下文基准（Visual In-Context BEnchmark, VIBE）来填补这一空白，重点关注多样的成像领域和广泛的任务。通过这一基准，我们能够更清晰地了解视觉上下文模型在面对新的图像和任务分布时的适应能力。我们在$14$个数据集和$12$个任务上对六个模型进行了压力测试（总共探索了$106$个数据集-任务组合），并在统一、可重复的评估协议下进行比较，采用单次学习设置。我们的评估揭示了视觉上下文学习的状态，包括局限性、系统性失败模式和有前景的方向。为了促进更广泛的评估，我们将公开发布我们的VIBE工具包。

View on arXiv Download PDF AI Translation

cs.CV / 79 / 2606.10988

AnimaSpark: A Feed-Forward Method for Animating Arbitrary 3D Objects

AnimaSpark：一种用于动画任意3D对象的前馈方法

Zhao, Yiming, Sun, Haoyu, Wang, Aoyu

Abstract

While recent advancements in generative AI have substantially accelerated static 3D model creation workflows, the synthesis of category-agnostic 3D animations remains a significant bottleneck in 3D asset production. Current methods for category-agnostic animation generation exhibit critical limitations in inference speed, motion quality, and adherence to textual prompts, thereby leaving the process dependent on labor-intensive manual artistry. To address these challenges, this paper introduces AnimaSpark, a novel pipeline for category-agnostic 3D animation generation. Our approach is motivated by the key insight that for many fundamental motions in the 3D world, the corresponding joint transformations can often be effectively modeled within a two-dimensional subspace. The pipeline begins by rendering a rigged static 3D model into multi-layered image representations of its mesh and skeleton, which are subsequently fed into a video generation model. We then employ a keypoint tracking algorithm on the generated video to capture the motion of the skeletal joints projected onto the camera's viewing plane. In the final stage, we distill the planar translations and rotations from these tracked keypoints and lift them from the 2D domain into 3D space to animate the character. Comprehensive evaluations reveal that our method achieves superior performance over existing state-of-the-art techniques across key metrics, including text-motion alignment, quality of motion, and computational efficiency.

Chinese Translation

尽管近年来生成式人工智能的进步显著加速了静态3D模型的创建工作流程，但类别无关的3D动画合成仍然是3D资产生产中的一个重要瓶颈。目前的类别无关动画生成方法在推理速度、运动质量和对文本提示的遵循方面存在重大局限性，因此该过程仍然依赖于劳动密集型的手工艺术创作。为了解决这些挑战，本文提出了AnimaSpark，一种新颖的类别无关3D动画生成管道。我们的方法的动机在于一个关键见解：在3D世界中的许多基本运动中，相应的关节变换往往可以在二维子空间内有效建模。该管道首先将一个带骨骼的静态3D模型渲染为其网格和骨架的多层图像表示，随后将其输入到视频生成模型中。接着，我们在生成的视频上应用关键点跟踪算法，以捕捉投影到相机视平面上的骨骼关节的运动。在最后阶段，我们从这些跟踪的关键点中提取平面平移和旋转，并将其从二维域提升到三维空间，以实现角色动画。全面的评估表明，我们的方法在文本-运动对齐、运动质量和计算效率等关键指标上优于现有的最先进技术。

View on arXiv Download PDF AI Translation

cs.CV / 80 / 2606.11001

IPSM-Bench: A New Intermediate Phase Segmentation Benchmark in Microstructure Images of Zinc-Based Absorbable Biomaterials

IPSM-Bench：锌基可吸收生物材料微观结构图像中的新中间相分割基准

Xu, Jinglin, Zhao, Shangyan, Wang, Jiabo, Mu, Xinghong, Lei, Yulong, Zhang, Jiacheng, Sun, Hongbo, Li, Yageng

Abstract

Zinc-based alloys are indispensable emerging absorbable metallic biomaterials, and their macroscopic performance is governed by microstructural characteristics. Intermediate phases-key microstructural constituents-are pivotal in regulating mechanical and functional properties. However, intermediate phase segmentation in zinc alloy microstructures faces formidable challenges: scarce annotated datasets, low contrast, difficulty detecting small targets, and heterogeneous morphologies. To this end, we construct IPSM-Bench, the largest high-quality dataset for zinc-alloy intermediate phase segmentation. Furthermore, we propose SCoP-SAM, a new Spatial Context Prior-guided SAM method that leverages the gradient structure and grayscale properties of intermediate phases to capture spatial context priors and incorporates them into the entire SAM encoding-decoding process, improving segmentation performance. Based on the proposed IPSM-Bench, we establish a new benchmark for intermediate phase segmentation to systematically evaluate state-of-the-art (SOTA) methods and advance research on zinc alloy microstructure analysis. Extensive experiments on IPSM-Bench and additional public alloy benchmarks demonstrate that our SCoP-SAM not only achieves SOTA performance for zinc-alloy intermediate phase segmentation but also generalizes remarkably well to other alloy scenarios.

Chinese Translation

锌基合金是不可或缺的新兴可吸收金属生物材料，其宏观性能受到微观结构特征的影响。中间相作为关键的微观结构成分，在调节机械和功能特性方面至关重要。然而，锌合金微观结构中的中间相分割面临着严峻的挑战：标注数据集稀缺、对比度低、难以检测小目标以及形态异质性。为此，我们构建了IPSM-Bench，这是用于锌合金中间相分割的最大高质量数据集。此外，我们提出了SCoP-SAM，这是一种新的空间上下文先验引导的SAM方法，利用中间相的梯度结构和灰度特性来捕捉空间上下文先验，并将其融入整个SAM编码-解码过程中，从而提高分割性能。基于所提出的IPSM-Bench，我们建立了一个新的中间相分割基准，以系统评估最先进（SOTA）方法并推动锌合金微观结构分析的研究。在IPSM-Bench和其他公共合金基准上的广泛实验表明，我们的SCoP-SAM不仅在锌合金中间相分割中实现了SOTA性能，而且在其他合金场景中也表现出显著的泛化能力。

View on arXiv Download PDF AI Translation

cs.CV / 81 / 2606.11012

An Uncertainty Estimation Framework for Dose Accumulation in Adaptive Radiotherapy: Application to CBCT-Guided Radiotherapy for Cervical Cancer

自适应放疗中剂量累积的不确定性估计框架：应用于颈癌的CBCT引导放疗

Hemon, Cedric, Lebret, Delphine, Nunes, Jean-Claude, Boussot, Valentin, Peignaux, Karine, Mesgouez-Nebout, Nathalie, Hanzen, Chantal, Simon, Antoine, Barateau, Anaïs, de Crevoisier, Renaud, Lafond, Caroline

Abstract

Background and purpose: oART enables daily plan adaptation to interfraction anatomical variations, but cumulative dose estimation remains limited by DIR, segmentation, and anatomical uncertainties. We introduce IMPACT-DoseAcc, an uncertainty-aware dose accumulation framework, within IMPACT for semantic feature-driven image analysis. The framework is modality- and disease-agnostic and is applied to CBCT-guided oART for cervical cancer (LACC). Material and Methods: Nine LACC patients were retrospectively analyzed using daily CBCT-derived virtual CTs for dose recalculation. IMPACT-DoseAcc focuses on uncertainty from DIR, without modeling vCT-generation uncertainty. Two DIR uncertainty strategies were tested within IMPACT-Reg: a Bayesian segmentation-guided approach using one probabilistic model to quantify anatomical uncertainty, and an ensemble of segmentation models targeting structures to capture epistemic variability. Voxel-wise uncertainty maps were propagated through dose warping and accumulation to generate probabilistic dose-volume histograms. Ensemble uncertainty was quantified from voxel-wise standard deviation across deformation fields, and geometric error was assessed using surface distance between warped and validated contours. Anatomical-variability weighting refined aggregation. Results: Ensemble DIR uncertainty correlated with geometric error, with Pearson coefficients of 0.63 for CTVt and 0.66 for bladder. For CTVt, pDVHs achieved 96.3 +/- 3.9% coverage, showing calibration of propagated uncertainty. Weighting stabilized estimates across fractions and organs. Conclusions: IMPACT-DoseAcc propagates registration-driven uncertainty to cumulative dose metrics, improving interpretation of accumulated dose under anatomical variations. Its 3DSlicer integration supports reproducible, uncertainty-informed ART workflows.

Chinese Translation

背景与目的：oART使得每日计划能够适应分次间的解剖变化，但累积剂量估计仍然受到DIR、分割和解剖不确定性的限制。我们引入了IMPACT-DoseAcc，这是一个关注不确定性的剂量累积框架，属于IMPACT语义特征驱动的图像分析。该框架与成像方式和疾病无关，并应用于颈癌（LACC）的CBCT引导oART。材料与方法：对九名LACC患者进行了回顾性分析，使用每日CBCT衍生的虚拟CT进行剂量重新计算。IMPACT-DoseAcc专注于来自DIR的不确定性，而不建模vCT生成的不确定性。在IMPACT-Reg中测试了两种DIR不确定性策略：一种是基于贝叶斯分割的引导方法，使用一个概率模型量化解剖不确定性；另一种是针对特定结构的分割模型集成，以捕捉认知变异。体素级不确定性图通过剂量变形和累积传播，以生成概率剂量-体积直方图。集成不确定性通过变形场的体素级标准差进行量化，几何误差通过扭曲和验证轮廓之间的表面距离进行评估。解剖变异加权精炼了聚合。结果：集成DIR不确定性与几何误差相关，CTVt和膀胱的Pearson系数分别为0.63和0.66。对于CTVt，pDVHs达到了96.3 +/- 3.9%的覆盖率，显示了传播不确定性的校准。加权在各分次和器官之间稳定了估计。结论：IMPACT-DoseAcc将注册驱动的不确定性传播到累积剂量指标，改善了在解剖变化下对累积剂量的解释。其与3DSlicer的集成支持可重复的、不确定性知情的自适应放疗工作流程。

View on arXiv Download PDF AI Translation

cs.CV / 82 / 2606.11032

U-TTT: Towards Generalizable PET Image Denoising via Test-Time Training

U-TTT：通过测试时训练实现可泛化的正电子发射断层扫描图像去噪

Yang, Zhiwen, Li, Jiayin, Lu, Hao, Zhang, Hui, Wang, Zihua, Wei, Bingzheng, Xu, Yan

Abstract

Existing deep learning models for Positron Emission Tomography (PET) image denoising often suffer from severe performance degradation under distribution shifts, fundamentally restricting their robust clinical deployment. This lack of generalization stems from the conventional paradigm of fixed-parameter models that cannot adapt to variations in test data (e.g., dose levels or scanner types) after training. To overcome this limitation and achieve robust generalization, we introduce U-TTT, a novel U-shaped model that integrates Test-Time Training (TTT) layers to dynamically adjust model parameters during inference through self-supervision, thereby adapting to the specific characteristics of each test instance. Furthermore, to comprehensively capture the complex degradations of 3D PET data, U-TTT features a dual-domain adaptation mechanism comprising a Spatial Test-Time Training (S-TTT) layer and a Frequency Test-Time Training (F-TTT) layer. The S-TTT layer captures and corrects spatial structural degradations, while the F-TTT layer suppresses global noise spectra and restores delicate high-frequency details. Extensive experiments demonstrate that U-TTT achieves state-of-the-art PET denoising performance and exhibits superior generalization under challenging distribution shifts, including both unseen dose levels and unseen scanners. Our code will be available at https://github.com/Yaziwel/U-TTT.

Chinese Translation

现有的深度学习模型在正电子发射断层扫描（PET）图像去噪方面常常在分布变化下表现出严重的性能下降，根本限制了其在临床中的稳健应用。这种缺乏泛化能力源于传统的固定参数模型范式，这些模型在训练后无法适应测试数据（例如，剂量水平或扫描仪类型）的变化。为克服这一限制并实现稳健的泛化，我们提出了U-TTT，这是一种新颖的U型模型，集成了测试时训练（TTT）层，通过自我监督在推理过程中动态调整模型参数，从而适应每个测试实例的特定特征。此外，为了全面捕捉3D PET数据的复杂退化，U-TTT具有双域适应机制，包括空间测试时训练（S-TTT）层和频率测试时训练（F-TTT）层。S-TTT层捕捉并修正空间结构退化，而F-TTT层则抑制全局噪声谱并恢复细腻的高频细节。大量实验表明，U-TTT在PET去噪性能上达到了最先进水平，并在具有挑战性的分布变化下展现出优越的泛化能力，包括未见的剂量水平和未见的扫描仪。我们的代码将发布在 https://github.com/Yaziwel/U-TTT。

View on arXiv Download PDF AI Translation

cs.CV / 83 / 2606.11096

IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder

IDEAL：深度对齐使离散表示自编码器

Chen, Yitong, Diao, Zijie, Wang, Junke, Kong, Lingyu, Ren, Yixuan, He, Bo, Jiang, Yu-Gang, Wu, Zuxuan

Abstract

Built on pretrained vision foundation models (VFMs), representation autoencoders (RAEs) have recently emerged as a promising approach for constructing semantically rich latent spaces for image generation. However, their reconstruction quality often remains suboptimal, largely because deep VFM representations do not preserve sufficient fine-grained visual detail. This limitation becomes even more severe after discretization, where missing low-level information is difficult to recover. In fact, we observe that shallow VFM features retain considerably richer local appearance and structural detail, which complements the high-level semantics carried by deep features used in existing RAEs. Motivated by this complementary property, we propose Ideal, an In-depth Alignment framework for discrete representation autoencoding. By jointly aligning quantized tokens with both shallow and deep VFM features, Ideal enables the resulting discrete visual tokens to preserve both visual fidelity and rich semantics. Extensive experiments demonstrate that Ideal yields superior reconstruction performance, achieving 0.61 rFID on ImageNet and outperforming the previous best method by 0.28. When used for autoregressive image generation, Ideal further produces a gFID of 1.89, establishing a new state of the art for autoregressive image generation.

Chinese Translation

基于预训练的视觉基础模型（VFMs），表示自编码器（RAEs）最近作为构建语义丰富的潜在空间以进行图像生成的有前景的方法而出现。然而，它们的重建质量往往仍然不尽如人意，主要是因为深层VFM表示未能保留足够的细粒度视觉细节。这个限制在离散化后变得更加严重，因为缺失的低级信息难以恢复。事实上，我们观察到浅层VFM特征保留了更丰富的局部外观和结构细节，这补充了现有RAEs中使用的深层特征所携带的高级语义。受到这一互补特性的启发，我们提出了Ideal，一个用于离散表示自编码的深度对齐框架。通过将量化的标记与浅层和深层VFM特征共同对齐，Ideal使得生成的离散视觉标记能够同时保留视觉保真度和丰富的语义。大量实验表明，Ideal在重建性能上表现优越，在ImageNet上达到了0.61 rFID，超越了之前最佳方法0.28的成绩。在用于自回归图像生成时，Ideal进一步产生了1.89的gFID，确立了自回归图像生成的新状态。

View on arXiv Download PDF AI Translation

cs.CV / 84 / 2606.11106

FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language model

FADA：基于选择性蒸馏的统一视觉-语言模型实现可访问的胎儿超声解读与标注

Alzubaidi, Mahmood, Shah, Uzair, Muaz, Raden, Abbes, Ines, Mohammed, Nader, Magram, Abdullatif, Alyafei, Khalid, Househ, Mowafa, Agus, Marco

Abstract

A global shortage of trained sonographers limits prenatal ultrasound screening in low- and middle-income countries, where over half of pregnant women receive no skilled sonography. Current deep learning approaches address detection, segmentation, or classification in isolation, each demanding a separate model and expert-specified labels at inference. We present FADA, a unified vision-language model built on Qwen3.5-VL that performs clinical interpretation, classification, detection, and segmentation through a single interpretation-first pipeline without external labels. FADA distills knowledge from four domain-specific foundation models (FetalCLIP, UltraSAM, USF-MAE, UltraFedFM) via offline pre-computed feature caching. Selective distillation, which applies feature alignment only to annotation tasks while interpretation relies on standard fine-tuning, consistently outperforms full distillation across most evaluation axes. The recommended variant, FADA-SKD, achieves 0.8820 mean Dice for segmentation, 0.7671 [email protected] for detection, and 100% structured interpretation compliance. Expert sonographer validation across 237 images confirms clinically acceptable outputs in both autonomous and human-in-the-loop modes, with 73.5% of interpretations scoring perfectly under clinician guidance. The system is trainable on a single consumer GPU and deployable without cloud connectivity. We validate edge deployment by running the compressed 0.8B model on a commodity smartphone (Qualcomm Snapdragon 7 Gen 1, 12 GB RAM) using llama.cpp with GGUF quantization, completing the full 5-phase pipeline in approximately 60 seconds entirely offline. This establishes a practical pathway for integrating AI-assisted fetal assessment with portable ultrasound devices, directly addressing diagnostic access gaps in resource-constrained settings. Code, models, and data are available at https://github.com/mahmoodphd/FADA.

Chinese Translation

全球范围内训练有素的超声技师短缺限制了低收入和中等收入国家的产前超声筛查，其中超过一半的孕妇未接受专业超声检查。目前的深度学习方法在检测、分割或分类方面各自独立进行，每种方法都需要一个单独的模型和专家指定的标签进行推理。我们提出了FADA，这是一种基于Qwen3.5-VL构建的统一视觉-语言模型，通过单一的以解读为主的流程执行临床解读、分类、检测和分割，而无需外部标签。FADA通过离线预计算特征缓存从四个特定领域的基础模型（FetalCLIP、UltraSAM、USF-MAE、UltraFedFM）中提取知识。选择性蒸馏仅对标注任务应用特征对齐，而解读则依赖于标准微调，这在大多数评估维度上始终优于完全蒸馏。推荐的变体FADA-SKD在分割任务中达到0.8820的平均Dice，在检测任务中达到0.7671的[email protected]，并实现100%的结构化解读合规性。对237幅图像的专家超声技师验证确认了在自主和人机协作模式下均可接受的临床输出，其中73.5%的解读在临床医生指导下得分完美。该系统可在单个消费级GPU上进行训练，并可在没有云连接的情况下部署。我们通过在一款普通智能手机（高通Snapdragon 7 Gen 1，12 GB RAM）上运行压缩后的0.8B模型，使用llama.cpp和GGUF量化，验证了边缘部署，完整的五阶段流程在大约60秒内完全离线完成。这为将AI辅助的胎儿评估与便携式超声设备整合提供了切实可行的途径，直接解决了资源有限环境中的诊断可及性差距。代码、模型和数据可在https://github.com/mahmoodphd/FADA获取。

View on arXiv Download PDF AI Translation

cs.CV / 85 / 2606.11129

WorldOlympiad: Can Your World Model Survive a Triathlon?

世界奥林匹克：你的世界模型能否在三项全能中生存？

Zhao, Yuke, Zhao, Wangbo, Wang, Weijie, Zhang, Zeyu, An, Dakai, Liu, Akide, Yu, Yinghao, Tang, Jiasheng, Wang, Fan, Wang, Wei, Zhuang, Bohan

Abstract

We introduce WorldOlympiad, a benchmark for diagnosing video-based world models across physical faithfulness, geometric consistency, and interaction fidelity. While existing benchmarks often focus on visual quality, semantic alignment, or short-term temporal coherence, they provide limited insight into whether generated videos obey physical rules, preserve coherent 3D structure, and sustain controllable interactions over long horizons. To address this gap, WorldOlympiad decomposes world-model evaluation into three complementary dimensions. The physical track uses object segmentation and MLLM-as-judge to assess whether generated videos follow interpretable rules in mechanics, thermal phenomena, and material properties. The geometry track reconstructs generated videos with Gaussian splatting and evaluates structural consistency, cross-view coherence, and camera-trajectory alignment. The interaction track assesses whether generated rollouts follow complex action prompts and maintain smooth, coherent transitions across consecutive video chunks. WorldOlympiad further covers three major downstream scenarios, including gaming, robotics, and general real-world videos, capturing diverse challenges from interactive control and embodied manipulation to open-domain motion and camera dynamics. Together, these tracks and scenarios form a scalable and interpretable evaluation suite that exposes failure modes beyond generic video quality. Experiments on state-of-the-art models reveal substantial gaps in physical reasoning, 3D consistency, and long-horizon interaction, underscoring the need for more structured evaluation protocols for generative world models.

Chinese Translation

我们介绍了世界奥林匹克（WorldOlympiad），这是一个用于诊断基于视频的世界模型在物理真实性、几何一致性和交互保真度方面的基准。现有的基准通常关注视觉质量、语义对齐或短期时间一致性，但对生成的视频是否遵循物理规则、保持一致的三维结构以及在长时间范围内维持可控交互提供的洞察有限。为了解决这一问题，世界奥林匹克将世界模型评估分解为三个互补的维度。物理轨道使用物体分割和MLLM-as-judge来评估生成的视频是否遵循力学、热现象和材料属性中的可解释规则。几何轨道通过高斯点云重建生成的视频，并评估结构一致性、跨视角一致性和相机轨迹对齐。交互轨道评估生成的展开是否遵循复杂的动作提示，并在连续视频片段之间保持平滑、一致的过渡。世界奥林匹克还涵盖了三个主要的下游场景，包括游戏、机器人技术和一般现实世界视频，捕捉从交互控制和具身操控到开放域运动和相机动态的多样化挑战。这些轨道和场景共同构成了一个可扩展且可解释的评估套件，揭示了超越通用视频质量的失败模式。对最先进模型的实验揭示了在物理推理、三维一致性和长时间交互方面的显著差距，强调了对生成世界模型更结构化评估协议的需求。

View on arXiv Download PDF AI Translation

cs.CV / 86 / 2606.11131

UniPET: a universal network for high-quality PET image denoising across varied dose reduction factors

UniPET：一种适用于不同剂量减少因子的高质量PET图像去噪的通用网络

Yang, Zhiwen, Zhou, Yang, Chen, Haowei, Zhang, Hui, Zhao, Dan, Wei, Bingzheng, Xu, Yan

Abstract

Most existing deep learning-based PET image denoising methods assume a fixed and known dose reduction factor (DRF) for low-dose PET images. However, these methods encounter significant performance degradation when the DRF varies beyond the assumed one in practical applications. To address the challenge posed by varied DRFs, several preliminary studies focus on the task of universal PET image denoising, aiming to train a universal model over low-dose data across DRFs. Nonetheless, these vanilla universal models often struggle with misaligned styles present in different DRF data, leading to the \textit{style elimination issue} with a significant over-smoothing effect. To deal with this issue, we innovatively introduce domain generalization to PET image denoising and propose a universal PET image denoising network (UniPET) to achieve high-quality PET image denoising across diverse DRFs. UniPET comprises two primary innovations: a style alignment network (SAN) and a region-aware learning strategy (RALS). Specifically, SAN utilizes style alignment techniques derived from domain generalization to align and recover styles across different DRFs, ensuring the model's generalizability across various DRFs while effectively preserving styles. Furthermore, to enhance style recovery, RALS distinguishes between flat and stylized regions, exclusively conducting adversarial learning on the latter, thereby more effectively guiding the model's focus towards learning stylized regions. It is demonstrated that our proposed UniPET can adaptively recover different DRF styles and achieve high-quality PET image denoising across DRFs. Comprehensive experiments show that UniPET exhibits comparable performance to individual DRF-specific models at specific DRFs and realizes state-of-the-art performance in universal PET image denoising quantitatively, perceptually, and clinically.

Chinese Translation

大多数现有的基于深度学习的PET图像去噪方法假设低剂量PET图像具有固定且已知的剂量减少因子（DRF）。然而，当DRF在实际应用中超出假设值变化时，这些方法会遭遇显著的性能下降。为了解决不同DRF带来的挑战，一些初步研究集中于通用PET图像去噪任务，旨在训练一个能够在不同DRF的低剂量数据上通用的模型。然而，这些基础的通用模型往往在不同DRF数据中存在的风格不一致性方面表现不佳，导致了显著的过度平滑效应，即 extit{风格消除问题}。为了解决这一问题，我们创新性地将领域泛化引入PET图像去噪，并提出了一种通用PET图像去噪网络（UniPET），以实现跨不同DRF的高质量PET图像去噪。UniPET包含两个主要创新：风格对齐网络（SAN）和区域感知学习策略（RALS）。具体而言，SAN利用源自领域泛化的风格对齐技术，在不同DRF之间对齐和恢复风格，确保模型在各种DRF上的泛化能力，同时有效保留风格。此外，为了增强风格恢复，RALS区分平坦区域和风格化区域，仅在后者上进行对抗学习，从而更有效地引导模型关注学习风格化区域。实验表明，我们提出的UniPET能够自适应地恢复不同DRF的风格，并在不同DRF上实现高质量的PET图像去噪。全面的实验结果显示，UniPET在特定DRF上表现出与个别DRF特定模型相当的性能，并在通用PET图像去噪方面在定量、感知和临床上实现了最先进的性能。

View on arXiv Download PDF AI Translation

cs.CV / 87 / 2606.11148

MOFA-VTON: More Fashion Possibilities with Fine-Grained Adaptations in Virtual Try-On

MOFA-VTON：通过细粒度适应在虚拟试穿中实现更多时尚可能性

Han, Xiaoyu, Wang, Chenyang, Wang, Jing, Zheng, Shunyuan, Meng, Quanling, Zhang, Shengping

Abstract

Virtual try-on aims to fit an in-shop clothing image onto a specific human body. An optimal virtual try-on method should provide diverse and flexible dressing options, accurately reflecting the varied wearing styles encountered in real-life scenarios, tailored to individual preferences and fashion aspirations. However, current methods predominantly perform a direct replacement of the original clothing with the target clothing, following the same dressing pattern. This limited control over clothing adaptation may result in fixed and monotonous try-on outputs. To delve into More Fashion Possibilities with Fine-Grained Adaptations in Virtual Try-On, we propose a novel virtual try-on method, termed MOFA-VTON, which allows adjustment for clothing adaptations in try-on results through simple sketches by users. Specifically, we first design a mask construction strategy that transforms user-drawn curve sketches into a dual-region mask, replacing the traditional clothing-agnostic mask and providing fine-grained layout guidance for the subsequent generation process. Further, we propose layout adjustment blocks that utilize the cross-attention mechanism to independently learn layout correspondences for upper and lower regions of the human body, refining the spatial arrangement of the two regions. With these implementations, our method enables flexible and fine-grained adaptations of target clothing, overcoming the constraints of a fixed layout. Extensive experiments on VITON-HD and DressCode datasets demonstrate that our proposed MOFA-VTON outperforms previous state-of-the-art methods and provides more fashion possibilities for virtual try-on.

Chinese Translation

虚拟试穿旨在将店内服装图像适配到特定的人体上。一个理想的虚拟试穿方法应提供多样化和灵活的穿搭选项，准确反映现实场景中遇到的各种穿着风格，满足个体的偏好和时尚追求。然而，目前的方法主要是直接将原始服装替换为目标服装，遵循相同的穿搭模式。这种对服装适应的有限控制可能导致固定和单调的试穿输出。为了深入探讨在虚拟试穿中通过细粒度适应实现更多时尚可能性，我们提出了一种新颖的虚拟试穿方法，称为MOFA-VTON，该方法允许用户通过简单的草图调整试穿结果中的服装适应。具体而言，我们首先设计了一种掩模构建策略，将用户绘制的曲线草图转换为双区域掩模，替代传统的与服装无关的掩模，并为后续生成过程提供细粒度的布局指导。此外，我们提出了布局调整模块，利用交叉注意力机制独立学习人体上半身和下半身的布局对应关系，优化这两个区域的空间排列。通过这些实现，我们的方法能够灵活且细致地适应目标服装，克服固定布局的限制。在VITON-HD和DressCode数据集上的大量实验表明，我们提出的MOFA-VTON优于之前的最先进方法，并为虚拟试穿提供了更多时尚可能性。

View on arXiv Download PDF AI Translation

cs.CV / 88 / 2606.11152

P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning

P3D-Bench：参数化3D生成和结构推理的多模态大型语言模型基准测试

Yang, Yikang, Hu, Zhanpeng, Lin, Youtian, Zhou, Mengqi, Xu, Jingxi, Zhang, Feihu, Liu, Jiaheng, Yao, Yao

Abstract

Multimodal large language models can write code to produce complex programs as well as use programs to do 3D modeling, which opens up a new avenue for 3D generation powered by their priors, world knowledge and reasoning. Yet existing benchmarks rarely evaluate 3D modeling through code. Such modeling demands more than runnable code: from a text or visual specification, a model must generate a parametric 3D program that is geometrically precise, semantically aligned and assembly-consistent. We introduce P3D-Bench, a benchmark for parametric 3D generation. Unlike a 3D mesh, a parametric 3D program exposes explicit dimensions, construction operations and part relations, revealing whether a model recovers a design's structure, not just its appearance. Under a unified protocol, P3D-Bench covers three task families (Text-to-3D, Image-to-3D and Assembly-3D) and scores each output for executability, geometric fidelity, topology, text-grounded constraints, multiview semantic alignment and part-level structure. We evaluate frontier MLLMs and text-only LLMs on 400 text cases, 400 image cases and 203 annotated assemblies, with domain-specific models as reference points. Our extensive evaluation yields three findings. First, assemblies are the hardest setting, where models still fail to compose multiple parts into a coherent structure. Second, models can often recover the global shape and semantic identity of the target object, yet fail to reproduce the precise parametric geometry specified by the input. Third, part-level modeling remains weak on assemblies, where models recover neither the geometry of each part nor the right number of parts. These results position P3D-Bench as a benchmark for evaluating precise parametric geometry and part-level structure in parametric 3D generation.

Chinese Translation

多模态大型语言模型能够编写代码以生成复杂程序，并利用这些程序进行3D建模，这为基于其先验知识、世界知识和推理能力的3D生成开辟了新的途径。然而，现有基准测试很少通过代码评估3D建模。这种建模不仅需要可运行的代码：模型必须从文本或视觉规范中生成一个在几何上精确、语义上对齐且组装一致的参数化3D程序。我们引入了P3D-Bench，这是一个用于参数化3D生成的基准测试。与3D网格不同，参数化3D程序暴露了显式的维度、构造操作和部件关系，揭示了模型是否能够恢复设计的结构，而不仅仅是其外观。在统一的协议下，P3D-Bench涵盖了三个任务类别（文本到3D、图像到3D和组装到3D），并对每个输出进行可执行性、几何保真度、拓扑、文本基础约束、多视图语义对齐和部件级结构的评分。我们在400个文本案例、400个图像案例和203个注释组装上评估了前沿的多模态大型语言模型和仅文本的大型语言模型，并以领域特定模型作为参考点。我们的广泛评估得出了三个发现。首先，组装是最具挑战性的设置，模型仍然无法将多个部件组合成一个连贯的结构。其次，模型通常能够恢复目标对象的整体形状和语义身份，但未能重现输入所指定的精确参数化几何。第三，部件级建模在组装上仍然较弱，模型既未能恢复每个部件的几何形状，也未能恢复正确数量的部件。这些结果使P3D-Bench成为评估参数化3D生成中精确参数化几何和部件级结构的基准测试。

View on arXiv Download PDF AI Translation

cs.CV / 89 / 2606.11155

Mean Flow Distillation: Robust and Stable Distillation for Flow Matching Models

均流蒸馏：用于流匹配模型的稳健与稳定蒸馏

Zhao, An, Zhang, Shengyuan, Sun, Zhongjian, Zhou, Yixiang, Li, Zejian, Yang, Ling, Chen, Tianrun, Sun, Lingyun

Abstract

Flow Matching models have demonstrated strong performance across a wide range of generative tasks. However, their reliance on ODE-based iterative sampling incurs substantial computational overhead in inference, which limits their applicability in real-time scenes. While distillation is a promising solution, existing approaches largely borrow from diffusion-based score matching, often failing to exploit the intrinsic geometric structure of flows and suffering from training instability, high variance, and degraded generation quality. In this paper, we propose Mean Flow Distillation (MFD), a novel distillation framework tailored for flow matching models. We theoretically demonstrate that MFD acts as a temporal low-pass filter, effectively suppressing the high-frequency optimization noise inherent in variational score distillation (VSD) while ensuring global trajectory consistency. We further prove the Mean Flow Matching Theorem, establishing that matching expected average velocities is sufficient for strict distribution alignment. Empirically, on challenging tasks of high-dimensional manifolds including 4D occupancy forecasting and text-to-image generation, MFD achieves state-of-the-art performance, enabling high-fidelity single-step generation.

Chinese Translation

流匹配模型在广泛的生成任务中表现出色。然而，它们对基于常微分方程（ODE）的迭代采样的依赖在推理时带来了巨大的计算开销，这限制了它们在实时场景中的应用。尽管蒸馏是一种有前景的解决方案，但现有方法大多借鉴于基于扩散的评分匹配，往往未能充分利用流的内在几何结构，并且存在训练不稳定、高方差和生成质量下降的问题。本文提出了均流蒸馏（Mean Flow Distillation, MFD），这是一个专为流匹配模型量身定制的新型蒸馏框架。我们从理论上证明，MFD充当了一个时间低通滤波器，有效抑制了变分评分蒸馏（Variational Score Distillation, VSD）中固有的高频优化噪声，同时确保全局轨迹一致性。我们进一步证明了均流匹配定理，确立了匹配期望平均速度对于严格分布对齐的充分性。在包括4D占用预测和文本到图像生成等高维流形的挑战性任务中，MFD实现了最先进的性能，能够实现高保真度的单步生成。

View on arXiv Download PDF AI Translation

cs.CV / 90 / 2606.11176

Data Journalist Agent: Transforming Data into Verifiable Multimodal Stories

数据记者代理：将数据转化为可验证的多模态故事

Lin, Kevin Qinghong, EI, Batu, Shi, Yuhong, Lu, Pan, Torr, Philip, Zou, James

Abstract

Data tells stories that shape society; the data journalist's job is to turn raw information into stories non-experts can trust. A high-quality news feature takes a newsroom team weeks: hunting for context, running statistics, choosing an angle, and designing visuals. Recent agents handle individual steps well: data-science agents close the analysis loop, while design agents synthesize beautiful websites. But can an agent serve as a data journalist end to end? We introduce Data Journalist Agent (Data2Story), a multi-agent framework that orchestrates specialized roles into a single virtual newsroom. Data2Story contributes two innovations. (i) Claims are evidence-grounded: an Inspector links every number, angle, and asset back to data, code, or an external reference. (ii) Articles are multimodally generative: rather than defaulting to plain text and static charts, Data2Story reasons about what readers will want to see, then deploys multimodal tools, such as interactive maps for geography and audio for music. We evaluate Data2Story on 18 articles, each paired with the originally published expert piece, along four axes: (a) human-agent angle coverage; (b) rubric evaluation with 53 participants across five dimensions; (c) computer-use agents as judges, a cost-saving proxy for how readers navigate interactive articles; and (d) verifiability, where a coding verifier re-executes statements against the data and checks claims against references. Data2Story produces competitive, evidence-traceable multimedia stories, with particular strength in transparency and auditability. Human articles retain an edge in editorial angle, creative design, and presentation. We position Data2Story as a collaborator for journalists, enabling more evidence-based, transparent, and verifiable reporting. Code and demos are available at https://data2story.github.io.

Chinese Translation

数据讲述塑造社会的故事；数据记者的工作是将原始信息转化为非专家可以信任的故事。一篇高质量的新闻特写需要新闻团队数周的时间：寻找背景、进行统计、选择角度和设计视觉效果。最近的代理在处理各个步骤方面表现良好：数据科学代理完成分析闭环，而设计代理则合成美观的网站。但一个代理能否作为数据记者完成整个过程？我们介绍了数据记者代理（Data Journalist Agent，Data2Story），这是一个将专业角色协调成单一虚拟新闻室的多代理框架。Data2Story贡献了两个创新点。(i) 论点是基于证据的：一个检查员将每个数字、角度和资产链接回数据、代码或外部参考。(ii) 文章是多模态生成的：Data2Story不仅仅默认使用纯文本和静态图表，而是推理读者希望看到的内容，然后部署多模态工具，例如用于地理的互动地图和用于音乐的音频。我们在18篇文章上评估Data2Story，每篇文章与原始发布的专家作品配对，从四个方面进行评估：(a) 人工-代理角度覆盖；(b) 53名参与者在五个维度上的评分评估；(c) 计算机使用代理作为评审，作为读者如何浏览互动文章的成本节约代理；(d) 可验证性，其中编码验证器重新执行语句以验证数据，并检查论点与参考文献的一致性。Data2Story生成具有竞争力、可追溯证据的多媒体故事，在透明度和可审计性方面表现尤为突出。人类撰写的文章在编辑角度、创意设计和呈现上仍具有优势。我们将Data2Story定位为记者的合作伙伴，使其能够进行更具证据基础、透明和可验证的报道。代码和演示可在 https://data2story.github.io 获取。

View on arXiv Download PDF AI Translation

cs.CV / 91 / 2606.11180

Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization

唇部强制：用于实时唇部同步的少步自回归扩散

Cho, Paul Hyunbin, Jang, Jinhyuk, Lee, SeokYoung, Lee, Joungbin, Jin, Siyoon, Shin, Heeseong, Yi, Jung, Park, Yunjin, Park, Chulmin, Kim, Seungryong

Abstract

Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, $17.6\times$ faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs $39.8\times$ faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.

Chinese Translation

基于扩散的唇部同步模型在视觉质量和音视频对齐方面表现出色，但全序列双向注意力和多个去噪步骤使其在实时推理中不够实用。我们提出了唇部强制（Lip Forcing），据我们所知，这是首个用于视频到视频（V2V）唇部同步的自回归扩散方法，它将一个14B音频条件的双向视频扩散教师提炼为因果学生。在推理时，学生仅需两个去噪步骤生成每个片段，无需推理时的条件生成（CFG），从而实现实时唇部同步。一项针对唇部同步特定教师轨迹的分析揭示了CFG保真度与同步之间的权衡：无CFG预测更倾向于参考保真度，而CFG引导的预测则更倾向于在中间轨迹带内实现同步。唇部强制将这一发现转化为三个基于分析的组件：同步窗口动态模式分解（Sync-Window DMD）、两步推理调度和基于SyncNet的奖励。我们在两个学生规模上验证了唇部强制，这两个学生均来自14B教师。1.3B学生在31 FPS下实现实时流媒体，速度比同规模的双向模型快$17.6 imes$。14B学生是目前报告的用于V2V唇部同步的最大扩散模型，其运行速度比教师快$39.8 imes$，且参考保真度相当。在两个规模下，首次帧的时间均低于毫秒，远低于所有扩散基线。

View on arXiv Download PDF AI Translation

cs.CV / 92 / 2606.11186

AnyMod-LLVE: Low-Light Video Enhancement with Modality-Agnostic Inference

AnyMod-LLVE：基于模态无关推理的低光视频增强

Liang, Hangfeng, Hu, Yutao, Hu, Yanhan, Wu, Xiaohan, Shao, Wenqi, Fu, Ying

Abstract

Low-light video enhancement (LLVE) remains a challenging task due to severe information degradation under low-illumination conditions. Recent multimodal approaches have significantly improved enhancement performance by incorporating auxiliary modalities, such as event streams and infrared images. However, these methods typically assume the availability of these modalities at inference, which is often not feasible in real-world scenarios. To solve this problem, in this work, we propose AMNet, a unified multimodal framework for LLVE, to support flexible modality-agnostic inference, where auxiliary modalities may be unavailable. To address the issue of modality absence, we introduce a Spatial-Spectral Dual-Gated Translator that learns the correspondence between auxiliary modalities and RGB inputs, producing implicit auxiliary representations to support the robust enhancement. Additionally, to fully facilitate the learning of cross-modal correspondence, we conduct large-scale multimodal pretraining based on the RGB-only dataset with synthetic auxiliary modalities. Extensive experiments demonstrate that AMNet could handle arbitrary inference-time modality combinations and exhibits superior performance for LLVE under modality absence conditions. Code and models are available on the project page.

Chinese Translation

低光视频增强（LLVE）由于在低照明条件下信息严重退化而仍然是一项具有挑战性的任务。最近的多模态方法通过结合辅助模态（如事件流和红外图像）显著提高了增强性能。然而，这些方法通常假设在推理时可以获得这些模态，而这在现实场景中往往不可行。为了解决这个问题，在本研究中，我们提出了AMNet，一个统一的多模态框架，用于LLVE，支持灵活的模态无关推理，即使辅助模态可能不可用。为了解决模态缺失的问题，我们引入了一种空间-光谱双门翻译器（Spatial-Spectral Dual-Gated Translator），该翻译器学习辅助模态与RGB输入之间的对应关系，生成隐式辅助表示以支持稳健的增强。此外，为了充分促进跨模态对应关系的学习，我们基于仅包含RGB的数据集进行大规模多模态预训练，并使用合成辅助模态。大量实验表明，AMNet能够处理任意推理时模态组合，并在模态缺失条件下展现出卓越的LLVE性能。代码和模型可在项目页面获取。

View on arXiv Download PDF AI Translation

cs.CV / 93 / 2606.11187

Next Forcing: Causal World Modeling with Multi-Chunk Prediction

下一步强制：基于多块预测的因果世界建模

Xu, Gangwei, Zhang, Qihang, Zhou, Jiaming, Zhu, Xing, Shen, Yujun, Yang, Xin, Xu, Yinghao

Abstract

Autoregressive video generation has emerged as a powerful paradigm for World Action Models (WAMs). However, existing approaches suffer from slow training convergence and limited converged accuracy, particularly at high frame rates, as the training supervision is confined to the current chunk without explicit signals about future dynamics; they also suffer from slow inference due to iterative video denoising. In this paper, we present Next Forcing, a multi-chunk prediction (MCP) framework for causal world modeling that enables faster training, higher accuracy, and accelerated inference. Inspired by multi-token prediction in large language models, Next Forcing introduces an MCP training objective that augments the main model with lightweight auxiliary MCP modules to simultaneously denoise video chunks at multiple future temporal horizons (next$^1$, next$^2$, next$^3$ chunks). These MCP modules form a causal chain across prediction depths, where intermediate features fused from multiple layers of the main model are leveraged to predict future dynamics, allowing near-future predictions to inform farther-future ones and providing dense multi-scale temporal supervision back to the main model. During training, the MCP modules significantly accelerate convergence and improve converged accuracy, especially at high frame rates: at 50 fps, Next Forcing achieves a 93.1% relative improvement over LingBot-VA at 5k training steps and 2.3x faster convergence, and establishes new state-of-the-art results on the RoboTwin benchmark (94.1/93.5% on Clean/Random). At inference, the MCP modules can be retained to predict the next video chunk in parallel with the current one, achieving 2x inference acceleration. Next Forcing also demonstrates significant improvements on PhyWorld, a benchmark evaluating adherence to physical laws in video generation, and over 50% FVD reduction on general video pretraining.

Chinese Translation

自回归视频生成已成为世界行动模型（World Action Models, WAMs）的一种强大范式。然而，现有方法在训练收敛速度和收敛准确性方面存在不足，特别是在高帧率下，因为训练监督仅限于当前块，而没有关于未来动态的明确信号；此外，由于迭代视频去噪，它们在推理时也面临缓慢的问题。本文提出了下一步强制（Next Forcing），一种用于因果世界建模的多块预测（Multi-Chunk Prediction, MCP）框架，能够实现更快的训练、更高的准确性和加速的推理。受到大型语言模型中多标记预测的启发，下一步强制引入了一种MCP训练目标，通过轻量级辅助MCP模块增强主模型，以同时在多个未来时间视域（next$^1$、next$^2$、next$^3$块）去噪视频块。这些MCP模块在预测深度上形成因果链，利用从主模型多个层融合的中间特征来预测未来动态，使得近未来的预测能够为更远未来的预测提供信息，并为主模型提供密集的多尺度时间监督。在训练过程中，MCP模块显著加速了收敛并提高了收敛准确性，尤其是在高帧率下：在50 fps时，下一步强制在5k训练步骤上相较于LingBot-VA实现了93.1%的相对提升和2.3倍的收敛速度，并在RoboTwin基准测试上建立了新的最先进结果（在Clean/Random上分别为94.1%/93.5%）。在推理时，MCP模块可以保留以并行预测下一个视频块与当前块，从而实现2倍的推理加速。下一步强制在PhyWorld基准测试中也展示了显著的改进，该基准评估视频生成中对物理法则的遵循，并在一般视频预训练中实现了超过50%的FVD减少。

View on arXiv Download PDF AI Translation

cs.CV / 94 / 2606.11188

ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations

ARM：一种具有统一离散表示的自回归大型多模态模型

Wang, Junke, Wang, Xiao, Pan, Jiacheng, Hu, Xuefeng, Li, Feng, Sun, Jingxiang, Deng, Chaorui, Chen, Zilong, Chen, Yunpeng, Tian, Kaibin, Gwilliam, Matthew, Chen, Hao, Guan, Danhui, Xu, Kun, Huang, Weilin, Wu, Zuxuan, Fan, Haoqi, Jiang, Yu-Gang, Yang, Zhenheng

Abstract

This paper introduces ARM, a discrete representation-based AutoRegressive Model that unifies image understanding, generation, and editing within a next-token prediction framework. ARM is built on three efforts: first, we train a discrete semantic visual tokenizer that maps images into compact token sequences. Our tokenizer is supervised with multiple objectives that jointly promote semantic discriminability, language alignment and faithful reconstruction, thereby supporting diverse tasks in a shared latent space. With this, we train a 7B autoregressive model over large-scale text and image token sequences, seamlessly developing vision-language perception and generation capabilities. Finally, to further improve preference-aligned behavior for text-to-image generation and instruction-guided editing, ARM applies reinforcement learning (RL) to optimize task-level objectives such as visual quality, instruction adherence, and edit consistency. Surprisingly, the results show that RL not only substantially improves performance on the target tasks (e.g., raising WISE overall from 0.50 to 0.56, GEdit-Bench-EN G_O from 5.75 to 6.68), but also induces cross-task synergy between text-to-image generation and editing. Collectively, these findings highlight autoregressive modeling, when paired with strong representations and preference optimization, as a scalable foundation for multimodal intelligence. Code: https://github.com/wdrink/ARM.

Chinese Translation

本文介绍了ARM，一种基于离散表示的自回归模型，该模型在下一个标记预测框架内统一了图像理解、生成和编辑。ARM的构建基于三个方面的努力：首先，我们训练了一个离散语义视觉标记器，将图像映射为紧凑的标记序列。我们的标记器在多个目标的监督下进行训练，这些目标共同促进了语义可区分性、语言对齐和忠实重建，从而支持在共享潜在空间中的多样化任务。基于此，我们在大规模文本和图像标记序列上训练了一个7B自回归模型，顺利发展了视觉-语言感知和生成能力。最后，为了进一步改善文本到图像生成和指令引导编辑的偏好对齐行为，ARM应用强化学习（RL）来优化任务级目标，例如视觉质量、指令遵循和编辑一致性。令人惊讶的是，结果表明，RL不仅显著提高了目标任务的性能（例如，将WISE整体从0.50提高到0.56，GEdit-Bench-EN G_O从5.75提高到6.68），还在文本到图像生成和编辑之间引发了跨任务协同。总体而言，这些发现强调了自回归建模与强表示和偏好优化相结合时，作为多模态智能的可扩展基础。代码：https://github.com/wdrink/ARM。

View on arXiv Download PDF AI Translation

人工智能 (Artificial Intelligence)

计算语言学 (Computation and Language)

arXiv Papers

Uncertainty-Aware Motion Planning for Autonomous Driving in Mixed Traffic Environment

GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation

Robotic Nonprehensile Object Transportation with a Hanging Tray

Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imagination

Flow Control: Steering Vision-Language-Action Models with Simple Real-Time Inputs

Exploration of Foundation Model-Based Robots in Patient and Elderly Care

What Demonstration Curation Metrics Do to Your Policy

YUBI: Yielding Universal Bidigital Interface for Bimanual Dexterous Manipulation at Scale

What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agents

Locomotion analysis of a quadruped interacting with the lunar granular surface

Hierarchical Policies from Verbal and Egocentric Human Signals for Natural Human-Robot Interaction

MARCH: Model-Assisted Reinforcement Learning for the Perceptive Control of Humanoids over Sparse Footholds

Improved Representation of Matrix Lie Group Operations through Tensor Notation

SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation

OMG: Omni-Modal Motion Generation for Generalist Humanoid Control

Rethinking Embodied Navigation via Relational Inductive Bias

HiMem-WAM: Hierarchical Memory-Gated World Action Models for Robotic Manipulation

A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation

Test-time Adversarial Takeover: A Real-time Hijacking Interface against Robotic Diffusion Policies

UMI-Bench 1.0: An Open and Reproducible Real-World Benchmark for Tabletop Robotic Manipulation with UMI Data

Information-Preserving Continuous Occupancy Mapping with Variance-Weighted Submap Joining

GuideWalk: Learning Unified Autonomous Navigation and Locomotion for Humanoid Robots across Versatile Terrains

Act on What You See: Unlocking Safe Social Navigation in Vision-Language-Action Models

Uncovering Vulnerability of Vision-Language-Action Models under Joint-Level Physical Faults

VeriSpace: Spatially Grounded Action Verification for Vision-Language-Action Models

AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness

LieIPM: Lie Group Interior Point Method for Direct Trajectory Optimization of Rigid Bodies

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations

Planar-Sector LOS Guidance for Interception of Agile Targets with Lifting-Wing Quadcopters

UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data

Self-Supervised Relevance Modelling in Autonomous Driving via Counterfactual Analysis

Vehicle Prediction Model for Enhanced MPC Path Tracking in Formula Student Driverless

Pushing the Performance Limits in Autonomous Racing: Continuous Stability-Aware Adaptive Velocity Planning in Formula Student Driverless

Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Open-World Contact Localization

ros2probe: Non-intrusive, Kernel-selective Observability for Robot Operating System 2 Middleware

Bridging Semantics and Physical Execution: A Neuro-Symbolic Framework for Multi-Pair Robotic Assembly

IMPACT: Learning Internal-Model Predictive Control for Forceful Robotic Manipulation

GUIDE: Goal-Initialized Directional Understanding for End-to-End Visual Navigation

Gradient based Bilevel for Inverse Optimal Control, a Riemannian approach

An Exposure-Time-Aligned Primary-Path Architecture for Autonomous-Driving ECUs

Embodiment-conditioned Generalist Control for Multirotor Aerial Robots

MV-Actor: Aligning Multi-View Semantics and Spatial Awareness for Bimanual Manipulation

AgniNav: Configuration-Driven Cross-Embodiment Local Planning for Robot Navigation

Task Robustness via Re-Labelling Vision-Action Robot Data

AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning

Resilient Navigation for Autonomous Farm Robots by Leveraging Jerk-Augmented Models with IMU-Only Disturbance Rejection

Language-Driven Cost Optimization for Autonomous Driving

Multi-UAV Active Sensing with Information Gain-based Planning and Belief Fusion

Diffusion Forcing Planner: History-Annealed Planning with Time-Dependent Guidance for Autonomous Driving

A Spiking Neural Architecture for Coordinating Arm and Locomotor Control

Generation of Diverse and Functional Robot Designs using Superquadrics Parametrisation and Quality-Diversity

A Distributed Multi-UGV Exploration Framework With Loop-Aware Planning and Descriptor-Aided Localization in Resource-Limited Environments

RoboNaldo: Accurate, Stable and Powerful Humanoid Soccer Shooting via Motion-Guided Curriculum Reinforcement Learning

EM-Fall: Embodied mmWave Sensing for Day-and-Night Fall Detection on Humanoid Robots

JOIN: Anchor-Grasp-Conditioned Joining via Opposition, Inference, and Navigation for Bimanual Assistive Manipulation

TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation

SD-GRPO: Verifiable Segment Decomposition for Long-Form Vision-Language Generation

WHU-Infra3D: A Full-stack Multi-modal Dataset and Benchmark for 3D Roadside Infrastructure Inventory

ABot-Earth 0.5: Generative 3D Earth Model

Generalized-CVO: Fast and Correspondence-Free Local Point Cloud Registration with Second Order Riemannian Optimization

SpineReport: Automated 3D Quantification and Reporting of Lumbar Spine Degeneration on MRI

A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

Interpretable Temporal Facial-Region Motion Analysis for In-the-Wild Parkinson's Disease Video Classification

Maximum Matching Accuracy: An Instance Segmentation Evaluation Metric Utilizing Globally Optimal Matching

Improving PET/CT-Based Whole-Body Lesion Segmentation Using Prediction Uncertainty-Augmented Models

BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression

iSAGE: A Human-in-the-Loop Framework for Remote Sensing Semantic Segmentation via Sparse Point Supervision

DB-3DME: From Dataset to Benchmark for Human-aligned Automatic 3D Mesh Evaluation

Fusing Satellite Imagery and Planimetric Maps for Cross-View Localization

FlexPath: Learned Semantic Path Priors for Image-Based Planning

A Large Scale Open-Source Image and Video Dataset for Robust Wildfire Detection and Classification

Making Time Editable in Video Diffusion Transformers

Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning

An Improved Generative Adversarial Network for Micro-Resistivity Imaging Logging Restoration

FoA-SR: Faithful or Aesthetic? Profile-Aware Preference Optimization for Real-World Image Super-Resolution

Dissect and Prune: Enhancing Robustness in AI-Generated Image Detection

Content-Induced Spatial-Spectral Aggregation Network for Change Detection in Remote Sensing Images

Building Change Detection in Earthquake: A Multi-Scale Interaction Network and A Change Detection Dataset

Multi-Angular Reflectance Anisotropy Observed from UAV Multispectral Imagery