Daily papers related to Image/Video/Multimodal Generation from cs.CV
January 02, 2026
We present SpaceTimePilot, a video diffusion model that disentangles space and time for controllable generative rendering. Given a monocular video, SpaceTimePilot can independently alter the camera viewpoint and the motion sequence within the generative process, re-rendering the scene for continuous and arbitrary exploration across space and time. To achieve this, we introduce an effective animation time-embedding mechanism in the diffusion process, allowing explicit control of the output video's motion sequence with respect to that of the source video. As no datasets provide paired videos of the same dynamic scene with continuous temporal variations, we propose a simple yet effective temporal-warping training scheme that repurposes existing multi-view datasets to mimic temporal differences. This strategy effectively supervises the model to learn temporal control and achieve robust space-time disentanglement. To further enhance the precision of dual control, we introduce two additional components: an improved camera-conditioning mechanism that allows altering the camera from the first frame, and CamxTime, the first synthetic space-and-time full-coverage rendering dataset that provides fully free space-time video trajectories within a scene. Joint training on the temporal-warping scheme and the CamxTime dataset yields more precise temporal control. We evaluate SpaceTimePilot on both real-world and synthetic data, demonstrating clear space-time disentanglement and strong results compared to prior work. Project page: https://zheninghuang.github.io/Space-Time-Pilot/ Code: https://github.com/ZheningHuang/spacetimepilot
TLDR: SpaceTimePilot is a video diffusion model that disentangles space and time, enabling controllable generative rendering with independent camera viewpoint and motion sequence manipulation. It introduces a novel animation time-embedding and temporal-warping training scheme.
TLDR: SpaceTimePilot 是一个视频扩散模型，能够解耦空间和时间，从而实现可控的生成渲染，并独立操纵相机视角和运动序列。该模型引入了一种新颖的动画时间嵌入和时间扭曲训练方案。
Read Paper (PDF)We present Edit3r, a feed-forward framework that reconstructs and edits 3D scenes in a single pass from unposed, view-inconsistent, instruction-edited images. Unlike prior methods requiring per-scene optimization, Edit3r directly predicts instruction-aligned 3D edits, enabling fast and photorealistic rendering without optimization or pose estimation. A key challenge in training such a model lies in the absence of multi-view consistent edited images for supervision. We address this with (i) a SAM2-based recoloring strategy that generates reliable, cross-view-consistent supervision, and (ii) an asymmetric input strategy that pairs a recolored reference view with raw auxiliary views, encouraging the network to fuse and align disparate observations. At inference, our model effectively handles images edited by 2D methods such as InstructPix2Pix, despite not being exposed to such edits during training. For large-scale quantitative evaluation, we introduce DL3DV-Edit-Bench, a benchmark built on the DL3DV test split, featuring 20 diverse scenes, 4 edit types and 100 edits in total. Comprehensive quantitative and qualitative results show that Edit3r achieves superior semantic alignment and enhanced 3D consistency compared to recent baselines, while operating at significantly higher inference speed, making it promising for real-time 3D editing applications.
TLDR: Edit3r is a feed-forward framework for fast 3D scene reconstruction and editing from unposed images, trained with a novel cross-view-consistent supervision strategy and asymmetric input approach, demonstrating superior performance and speed compared to existing methods.
TLDR: Edit3r是一个前馈框架，用于从无姿态图像快速进行三维场景重建和编辑，它采用了一种新颖的跨视角一致性监督策略和非对称输入方法进行训练，与现有方法相比，展现出更优越的性能和速度。
Read Paper (PDF)Audio-driven visual dubbing aims to synchronize a video's lip movements with new speech, but is fundamentally challenged by the lack of ideal training data: paired videos where only a subject's lip movements differ while all other visual conditions are identical. Existing methods circumvent this with a mask-based inpainting paradigm, where an incomplete visual conditioning forces models to simultaneously hallucinate missing content and sync lips, leading to visual artifacts, identity drift, and poor synchronization. In this work, we propose a novel self-bootstrapping framework that reframes visual dubbing from an ill-posed inpainting task into a well-conditioned video-to-video editing problem. Our approach employs a Diffusion Transformer, first as a data generator, to synthesize ideal training data: a lip-altered companion video for each real sample, forming visually aligned video pairs. A DiT-based audio-driven editor is then trained on these pairs end-to-end, leveraging the complete and aligned input video frames to focus solely on precise, audio-driven lip modifications. This complete, frame-aligned input conditioning forms a rich visual context for the editor, providing it with complete identity cues, scene interactions, and continuous spatiotemporal dynamics. Leveraging this rich context fundamentally enables our method to achieve highly accurate lip sync, faithful identity preservation, and exceptional robustness against challenging in-the-wild scenarios. We further introduce a timestep-adaptive multi-phase learning strategy as a necessary component to disentangle conflicting editing objectives across diffusion timesteps, thereby facilitating stable training and yielding enhanced lip synchronization and visual fidelity. Additionally, we propose ContextDubBench, a comprehensive benchmark dataset for robust evaluation in diverse and challenging practical application scenarios.
TLDR: The paper introduces a self-bootstrapping framework for audio-driven visual dubbing, transforming it from an inpainting problem to a video editing task using a Diffusion Transformer for data generation and a DiT-based audio-driven editor for precise lip synchronization, robust against in-the-wild scenarios. They also introduce a new benchmark dataset.
TLDR: 该论文介绍了一种用于音频驱动的视觉配音的自引导框架，利用扩散变换器生成数据，将配音从图像修复问题转变为视频编辑任务，并使用基于DiT的音频驱动编辑器进行精确的唇部同步，对实际场景具有鲁棒性。他们还介绍了一个新的基准数据集。
Read Paper (PDF)