Daily papers related to Image/Video/Multimodal Generation from cs.CV
January 08, 2026
Recent text-to-video diffusion models can generate compelling video sequences, yet they remain silent -- missing the semantic, emotional, and atmospheric cues that audio provides. We introduce LTX-2, an open-source foundational model capable of generating high-quality, temporally synchronized audiovisual content in a unified manner. LTX-2 consists of an asymmetric dual-stream transformer with a 14B-parameter video stream and a 5B-parameter audio stream, coupled through bidirectional audio-video cross-attention layers with temporal positional embeddings and cross-modality AdaLN for shared timestep conditioning. This architecture enables efficient training and inference of a unified audiovisual model while allocating more capacity for video generation than audio generation. We employ a multilingual text encoder for broader prompt understanding and introduce a modality-aware classifier-free guidance (modality-CFG) mechanism for improved audiovisual alignment and controllability. Beyond generating speech, LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene -- complete with natural background and foley elements. In our evaluations, the model achieves state-of-the-art audiovisual quality and prompt adherence among open-source systems, while delivering results comparable to proprietary models at a fraction of their computational cost and inference time. All model weights and code are publicly released.
TLDR: LTX-2 is a new open-source audiovisual foundation model that generates high-quality, synchronized video and audio, achieving state-of-the-art results compared to open-source systems while being computationally efficient.
TLDR: LTX-2是一个新的开源视听基础模型，可以生成高质量、同步的视频和音频，与开源系统相比，它实现了最先进的结果，同时具有计算效率。
Read Paper (PDF)As world models gain momentum in Embodied AI, an increasing number of works explore using video foundation models as predictive world models for downstream embodied tasks like 3D prediction or interactive generation. However, before exploring these downstream tasks, video foundation models still have two critical questions unanswered: (1) whether their generative generalization is sufficient to maintain perceptual fidelity in the eyes of human observers, and (2) whether they are robust enough to serve as a universal prior for real-world embodied agents. To provide a standardized framework for answering these questions, we introduce the Embodied Turing Test benchmark: WoW-World-Eval (Wow,wo,val). Building upon 609 robot manipulation data, Wow-wo-val examines five core abilities, including perception, planning, prediction, generalization, and execution. We propose a comprehensive evaluation protocol with 22 metrics to assess the models' generation ability, which achieves a high Pearson Correlation between the overall score and human preference (>0.93) and establishes a reliable foundation for the Human Turing Test. On Wow-wo-val, models achieve only 17.27 on long-horizon planning and at best 68.02 on physical consistency, indicating limited spatiotemporal consistency and physical reasoning. For the Inverse Dynamic Model Turing Test, we first use an IDM to evaluate the video foundation models' execution accuracy in the real world. However, most models collapse to $\approx$ 0% success, while WoW maintains a 40.74% success rate. These findings point to a noticeable gap between the generated videos and the real world, highlighting the urgency and necessity of benchmarking World Model in Embodied AI.
TLDR: The paper introduces WoW-World-Eval, a new benchmark for evaluating world models in embodied AI, focusing on their generative generalization and robustness, and finds significant gaps between generated videos and real-world performance.
TLDR: 本文介绍了一个名为 WoW-World-Eval 的新基准，用于评估具身人工智能中的世界模型，重点关注其生成泛化能力和鲁棒性。研究发现，生成的视频与真实世界的表现之间存在显著差距。
Read Paper (PDF)We present Gen3R, a method that bridges the strong priors of foundational reconstruction models and video diffusion models for scene-level 3D generation. We repurpose the VGGT reconstruction model to produce geometric latents by training an adapter on its tokens, which are regularized to align with the appearance latents of pre-trained video diffusion models. By jointly generating these disentangled yet aligned latents, Gen3R produces both RGB videos and corresponding 3D geometry, including camera poses, depth maps, and global point clouds. Experiments demonstrate that our approach achieves state-of-the-art results in single- and multi-image conditioned 3D scene generation. Additionally, our method can enhance the robustness of reconstruction by leveraging generative priors, demonstrating the mutual benefit of tightly coupling reconstruction and generative models.
TLDR: Gen3R generates 3D scenes by bridging reconstruction and video diffusion models, achieving state-of-the-art results in conditioned scene generation and improving reconstruction robustness.
TLDR: Gen3R通过桥接重建和视频扩散模型生成3D场景，在条件场景生成方面实现了最先进的结果，并提高了重建的鲁棒性。
Read Paper (PDF)Aligning text-to-video diffusion models with human preferences is crucial for generating high-quality videos. Existing Direct Preference Otimization (DPO) methods rely on multi-sample ranking and task-specific critic models, which is inefficient and often yields ambiguous global supervision. To address these limitations, we propose LocalDPO, a novel post-training framework that constructs localized preference pairs from real videos and optimizes alignment at the spatio-temporal region level. We design an automated pipeline to efficiently collect preference pair data that generates preference pairs with a single inference per prompt, eliminating the need for external critic models or manual annotation. Specifically, we treat high-quality real videos as positive samples and generate corresponding negatives by locally corrupting them with random spatio-temporal masks and restoring only the masked regions using the frozen base model. During training, we introduce a region-aware DPO loss that restricts preference learning to corrupted areas for rapid convergence. Experiments on Wan2.1 and CogVideoX demonstrate that LocalDPO consistently improves video fidelity, temporal coherence and human preference scores over other post-training approaches, establishing a more efficient and fine-grained paradigm for video generator alignment.
TLDR: The paper introduces LocalDPO, a novel post-training framework for video diffusion models that uses localized preference pairs from real videos to optimize alignment at the spatio-temporal region level, improving video fidelity and temporal coherence.
TLDR: 该论文介绍了一种名为LocalDPO的新型视频扩散模型后训练框架，该框架使用来自真实视频的局部偏好对，以在时空区域级别优化对齐，从而提高视频保真度和时间连贯性。
Read Paper (PDF)Existing 1D visual tokenizers for autoregressive (AR) generation largely follow the design principles of language modeling, as they are built directly upon transformers whose priors originate in language, yielding single-hierarchy latent tokens and treating visual data as flat sequential token streams. However, this language-like formulation overlooks key properties of vision, particularly the hierarchical and residual network designs that have long been essential for convergence and efficiency in visual models. To bring "vision" back to vision, we propose the Residual Tokenizer (ResTok), a 1D visual tokenizer that builds hierarchical residuals for both image tokens and latent tokens. The hierarchical representations obtained through progressively merging enable cross-level feature fusion at each layer, substantially enhancing representational capacity. Meanwhile, the semantic residuals between hierarchies prevent information overlap, yielding more concentrated latent distributions that are easier for AR modeling. Cross-level bindings consequently emerge without any explicit constraints. To accelerate the generation process, we further introduce a hierarchical AR generator that substantially reduces sampling steps by predicting an entire level of latent tokens at once rather than generating them strictly token-by-token. Extensive experiments demonstrate that restoring hierarchical residual priors in visual tokenization significantly improves AR image generation, achieving a gFID of 2.34 on ImageNet-256 with only 9 sampling steps. Code is available at https://github.com/Kwai-Kolors/ResTok.
TLDR: The paper introduces ResTok, a 1D visual tokenizer for autoregressive image generation that incorporates hierarchical residuals into both image and latent tokens, leading to improved performance and faster sampling. It achieves a gFID of 2.34 on ImageNet-256 with only 9 sampling steps.
TLDR: 该论文介绍了ResTok，一种用于自回归图像生成的1D视觉分词器，它将分层残差融入到图像和潜在令牌中，从而提高了性能并加快了采样速度。在ImageNet-256上仅用9个采样步骤就实现了2.34的gFID。
Read Paper (PDF)Existing text-guided image editing methods primarily rely on end-to-end pixel-level inpainting paradigm. Despite its success in simple scenarios, this paradigm still significantly struggles with compositional editing tasks that require precise local control and complex multi-object spatial reasoning. This paradigm is severely limited by 1) the implicit coupling of planning and execution, 2) the lack of object-level control granularity, and 3) the reliance on unstructured, pixel-centric modeling. To address these limitations, we propose I2E, a novel "Decompose-then-Action" paradigm that revisits image editing as an actionable interaction process within a structured environment. I2E utilizes a Decomposer to transform unstructured images into discrete, manipulable object layers and then introduces a physics-aware Vision-Language-Action Agent to parse complex instructions into a series of atomic actions via Chain-of-Thought reasoning. Further, we also construct I2E-Bench, a benchmark designed for multi-instance spatial reasoning and high-precision editing. Experimental results on I2E-Bench and multiple public benchmarks demonstrate that I2E significantly outperforms state-of-the-art methods in handling complex compositional instructions, maintaining physical plausibility, and ensuring multi-turn editing stability.
TLDR: The paper introduces I2E, a novel paradigm for text-guided image editing that decomposes images into manipulable object layers and uses a physics-aware agent for complex compositional editing, outperforming existing methods on a new benchmark.
TLDR: 本文介绍了一种新的文本引导图像编辑范例 I2E，该范例将图像分解为可操作的对象层，并使用具有物理感知能力的代理进行复杂的构图编辑，在新基准测试中优于现有方法。
Read Paper (PDF)Current video generation models produce high-quality aesthetic videos but often struggle to learn representations of real-world physics dynamics, resulting in artifacts such as unnatural object collisions, inconsistent gravity, and temporal flickering. In this work, we propose PhysVideoGenerator, a proof-of-concept framework that explicitly embeds a learnable physics prior into the video generation process. We introduce a lightweight predictor network, PredictorP, which regresses high-level physical features extracted from a pre-trained Video Joint Embedding Predictive Architecture (V-JEPA 2) directly from noisy diffusion latents. These predicted physics tokens are injected into the temporal attention layers of a DiT-based generator (Latte) via a dedicated cross-attention mechanism. Our primary contribution is demonstrating the technical feasibility of this joint training paradigm: we show that diffusion latents contain sufficient information to recover V-JEPA 2 physical representations, and that multi-task optimization remains stable over training. This report documents the architectural design, technical challenges, and validation of training stability, establishing a foundation for future large-scale evaluation of physics-aware generative models.
TLDR: The paper introduces PhysVideoGenerator, a framework that incorporates a learnable physics prior into video generation by predicting physics features from diffusion latents and injecting them into the generator, demonstrating the feasibility of physics-aware video generation.
TLDR: 该论文介绍了PhysVideoGenerator，一个将可学习的物理先验融入视频生成的框架，通过从扩散潜在空间预测物理特征，并将它们注入生成器中，展示了物理感知视频生成的可行性。
Read Paper (PDF)Maintaining consistent characters, props, and environments across multiple shots is a central challenge in narrative video generation. Existing models can produce high-quality short clips but often fail to preserve entity identity and appearance when scenes change or when entities reappear after long temporal gaps. We present VideoMemory, an entity-centric framework that integrates narrative planning with visual generation through a Dynamic Memory Bank. Given a structured script, a multi-agent system decomposes the narrative into shots, retrieves entity representations from memory, and synthesizes keyframes and videos conditioned on these retrieved states. The Dynamic Memory Bank stores explicit visual and semantic descriptors for characters, props, and backgrounds, and is updated after each shot to reflect story-driven changes while preserving identity. This retrieval-update mechanism enables consistent portrayal of entities across distant shots and supports coherent long-form generation. To evaluate this setting, we construct a 54-case multi-shot consistency benchmark covering character-, prop-, and background-persistent scenarios. Extensive experiments show that VideoMemory achieves strong entity-level coherence and high perceptual quality across diverse narrative sequences.
TLDR: The paper introduces VideoMemory, a framework for generating consistent long-form videos by integrating narrative planning with visual generation through a dynamic memory bank that stores and updates visual and semantic descriptors of entities.
TLDR: 该论文介绍了VideoMemory，一个用于生成一致长视频的框架，通过动态记忆库将叙事规划与视觉生成相结合，该记忆库存储和更新实体的视觉和语义描述。
Read Paper (PDF)Instruction-driven image editing with unified multimodal generative models has advanced rapidly, yet their underlying visual reasoning remains limited, leading to suboptimal performance on reasoning-centric edits. Reinforcement learning (RL) has been investigated for improving the quality of image editing, but it faces three key challenges: (1) limited reasoning exploration confined to denoising stochasticity, (2) biased reward fusion, and (3) unstable VLM-based instruction rewards. In this work, we propose ThinkRL-Edit, a reasoning-centric RL framework that decouples visual reasoning from image synthesis and expands reasoning exploration beyond denoising. To the end, we introduce Chain-of-Thought (CoT)-based reasoning sampling with planning and reflection stages prior to generation in online sampling, compelling the model to explore multiple semantic hypotheses and validate their plausibility before committing to a visual outcome. To avoid the failures of weighted aggregation, we propose an unbiased chain preference grouping strategy across multiple reward dimensions. Moreover, we replace interval-based VLM scores with a binary checklist, yielding more precise, lower-variance, and interpretable rewards for complex reasoning. Experiments show our method significantly outperforms prior work on reasoning-centric image editing, producing instruction-faithful, visually coherent, and semantically grounded edits.
TLDR: ThinkRL-Edit introduces a reinforcement learning framework to improve reasoning in instruction-driven image editing by using chain-of-thought reasoning, unbiased reward fusion, and a binary checklist for VLM rewards, resulting in improved performance on reasoning-centric edits.
TLDR: ThinkRL-Edit 引入了一种强化学习框架，通过使用思维链推理、无偏奖励融合和 VLM 奖励的二元清单来提高指令驱动图像编辑中的推理能力，从而提高了以推理为中心的编辑性能。
Read Paper (PDF)Image restoration has traditionally required training specialized models on thousands of paired examples per degradation type. We challenge this paradigm by demonstrating that powerful pre-trained text-conditioned image editing models can be efficiently adapted for multiple restoration tasks through parameter-efficient fine-tuning with remarkably few examples. Our approach fine-tunes LoRA adapters on FLUX.1 Kontext, a state-of-the-art 12B parameter flow matching model for image-to-image translation, using only 16-128 paired images per task, guided by simple text prompts that specify the restoration operation. Unlike existing methods that train specialized restoration networks from scratch with thousands of samples, we leverage the rich visual priors already encoded in large-scale pre-trained editing models, dramatically reducing data requirements while maintaining high perceptual quality. A single unified LoRA adapter, conditioned on task-specific text prompts, effectively handles multiple degradations including denoising, deraining, and dehazing. Through comprehensive ablation studies, we analyze: (i) the impact of training set size on restoration quality, (ii) trade-offs between task-specific versus unified multi-task adapters, (iii) the role of text encoder fine-tuning, and (iv) zero-shot baseline performance. While our method prioritizes perceptual quality over pixel-perfect reconstruction metrics like PSNR/SSIM, our results demonstrate that pre-trained image editing models, when properly adapted, offer a compelling and data-efficient alternative to traditional image restoration approaches, opening new avenues for few-shot, prompt-guided image enhancement. The code to reproduce our results are available at: https://github.com/makinyilmaz/Edit2Restore
TLDR: The paper presents Edit2Restore, a method that leverages pre-trained text-conditioned image editing models for few-shot image restoration by fine-tuning LoRA adapters, achieving high perceptual quality with significantly fewer training examples than traditional methods. It uses text prompts to guide the restoration.
TLDR: 该论文提出了Edit2Restore，一种通过微调LoRA适配器，利用预训练的文本条件图像编辑模型进行少样本图像恢复的方法，以显著少于传统方法的训练样本实现了高感知质量，并使用文本提示来指导恢复。
Read Paper (PDF)We present Muses, the first training-free method for fantastic 3D creature generation in a feed-forward paradigm. Previous methods, which rely on part-aware optimization, manual assembly, or 2D image generation, often produce unrealistic or incoherent 3D assets due to the challenges of intricate part-level manipulation and limited out-of-domain generation. In contrast, Muses leverages the 3D skeleton, a fundamental representation of biological forms, to explicitly and rationally compose diverse elements. This skeletal foundation formalizes 3D content creation as a structure-aware pipeline of design, composition, and generation. Muses begins by constructing a creatively composed 3D skeleton with coherent layout and scale through graph-constrained reasoning. This skeleton then guides a voxel-based assembly process within a structured latent space, integrating regions from different objects. Finally, image-guided appearance modeling under skeletal conditions is applied to generate a style-consistent and harmonious texture for the assembled shape. Extensive experiments establish Muses' state-of-the-art performance in terms of visual fidelity and alignment with textual descriptions, and potential on flexible 3D object editing. Project page: https://luhexiao.github.io/Muses.github.io/.
TLDR: Muses is a training-free method for generating fantasy 3D creatures by using a structured skeleton-guided approach, achieving state-of-the-art performance without part-aware optimization or 2D image generation.
TLDR: Muses是一种无需训练的方法，通过结构化的骨架引导方法生成奇幻3D生物，无需进行部分感知优化或2D图像生成，即可实现最先进的性能。
Read Paper (PDF)With the advancement of AIGC (AI-generated content) technologies, an increasing number of generative models are revolutionizing fields such as video editing, music generation, and even film production. However, due to the limitations of current AIGC models, most models can only serve as individual components within specific application scenarios and are not capable of completing tasks end-to-end in real-world applications. In real-world applications, editing experts often work with a wide variety of images and video inputs, producing multimodal outputs -- a video typically includes audio, text, and other elements. This level of integration across multiple modalities is something current models are unable to achieve effectively. However, the rise of agent-based systems has made it possible to use AI tools to tackle complex content generation tasks. To deal with the complex scenarios, in this paper, we propose a MultiMedia-Agent designed to automate complex content creation. Our agent system includes a data generation pipeline, a tool library for content creation, and a set of metrics for evaluating preference alignment. Notably, we introduce the skill acquisition theory to model the training data curation and agent training. We designed a two-stage correlation strategy for plan optimization, including self-correlation and model preference correlation. Additionally, we utilized the generated plans to train the MultiMedia-Agent via a three stage approach including base/success plan finetune and preference optimization. The comparison results demonstrate that the our approaches are effective and the MultiMedia-Agent can generate better multimedia content compared to novel models.
TLDR: The paper proposes a MultiMedia-Agent for automating complex multimedia content creation, addressing the limitations of current AIGC models in handling multimodal inputs and outputs by incorporating a data generation pipeline, a tool library, and preference alignment metrics, trained via a novel skill acquisition theory and plan optimization strategy.
TLDR: 该论文提出了一个用于自动化复杂多媒体内容创作的MultiMedia-Agent，解决了当前AIGC模型在处理多模态输入和输出方面的局限性，方法是结合数据生成管道、工具库和偏好对齐指标，并通过新颖的技能获取理论和计划优化策略进行训练。
Read Paper (PDF)Synthetic Aperture Radar (SAR) imaging results are highly sensitive to observation geometries and the geometric parameters of targets. However, existing generative methods primarily operate within the image domain, neglecting explicit geometric information. This limitation often leads to unsatisfactory generation quality and the inability to precisely control critical parameters such as azimuth angles. To address these challenges, we propose GeoDiff-SAR, a geometric prior guided diffusion model for high-fidelity SAR image generation. Specifically, GeoDiff-SAR first efficiently simulates the geometric structures and scattering relationships inherent in real SAR imaging by calculating SAR point clouds at specific azimuths, which serves as a robust physical guidance. Secondly, to effectively fuse multi-modal information, we employ a feature fusion gating network based on Feature-wise Linear Modulation (FiLM) to dynamically regulate the weight distribution of 3D physical information, image control parameters, and textual description parameters. Thirdly, we utilize the Low-Rank Adaptation (LoRA) architecture to perform lightweight fine-tuning on the advanced Stable Diffusion 3.5 (SD3.5) model, enabling it to rapidly adapt to the distribution characteristics of the SAR domain. To validate the effectiveness of GeoDiff-SAR, extensive comparative experiments were conducted on real-world SAR datasets. The results demonstrate that data generated by GeoDiff-SAR exhibits high fidelity and effectively enhances the accuracy of downstream classification tasks. In particular, it significantly improves recognition performance across different azimuth angles, thereby underscoring the superiority of physics-guided generation.
TLDR: GeoDiff-SAR introduces a geometric prior-guided diffusion model for high-fidelity SAR image generation by incorporating SAR point clouds and feature fusion to improve control and accuracy, especially across varying azimuth angles, using a LoRA-tuned Stable Diffusion model.
TLDR: GeoDiff-SAR 提出了一种几何先验引导的扩散模型，用于生成高保真 SAR 图像，通过结合 SAR 点云和特征融合来提高控制和精度，尤其是在不同的方位角下，使用 LoRA 调优的 Stable Diffusion 模型。
Read Paper (PDF)In the manufacturing industry, computer vision systems based on artificial intelligence (AI) are widely used to reduce costs and increase production. Training these AI models requires a large amount of training data that is costly to acquire and annotate, especially in high-variance, low-volume manufacturing environments. A popular approach to reduce the need for real data is the use of synthetic data that is generated by leveraging computer-aided design (CAD) models available in the industry. However, in the agricultural industry these models are not readily available, increasing the difficulty in leveraging synthetic data. In this paper, we present different techniques for substituting CAD files to create synthetic datasets. We measure their relative performance when used to train an AI object detection model to separate stones and potatoes in a bin picking environment. We demonstrate that using highly representative 3D models acquired by scanning or using image-to-3D approaches can be used to generate synthetic data for training object detection models. Finetuning on a small real dataset can significantly improve the performance of the models and even get similar performance when less representative models are used.
TLDR: The paper explores methods for generating synthetic 3D models of agricultural products to train AI object detection models, particularly for separating stones and potatoes, finding that scanning and image-to-3D approaches are effective with fine-tuning on real data.
TLDR: 该论文探讨了生成农业产品合成3D模型的方法，用于训练AI物体检测模型，特别是用于区分石头和土豆。研究发现扫描和图像到3D的方法有效，并可通过在真实数据上进行微调来提高性能。
Read Paper (PDF)