Daily papers related to Image/Video/Multimodal Generation from cs.CV
March 05, 2026
Generating long-form storytelling videos with consistent visual narratives remains a significant challenge in video synthesis. We present a novel framework, dataset, and a model that address three critical limitations: background consistency across shots, seamless multi-subject shot-to-shot transitions, and scalability to hour-long narratives. Our approach introduces a background-consistent generation pipeline that maintains visual coherence across scenes while preserving character identity and spatial relationships. We further propose a transition-aware video synthesis module that generates smooth shot transitions for complex scenarios involving multiple subjects entering or exiting frames, going beyond the single-subject limitations of prior work. To support this, we contribute with a synthetic dataset of 10,000 multi-subject transition sequences covering underrepresented dynamic scene compositions. On VBench, InfinityStory achieves the highest Background Consistency (88.94), highest Subject Consistency (82.11), and the best overall average rank (2.80), showing improved stability, smoother transitions, and better temporal coherence.
TLDR: The paper introduces InfinityStory, a new framework, dataset, and model for generating hour-long storytelling videos with consistent backgrounds, character identities, and smooth multi-subject shot transitions, outperforming existing methods on VBench.
TLDR: 该论文介绍了 InfinityStory，一个新的框架、数据集和模型，用于生成具有一致背景、角色身份和流畅的多主体镜头过渡的长达一小时的故事叙述视频，在 VBench 上优于现有方法。
Read Paper (PDF)State-of-the-art text-to-video (T2V) generators frequently violate physical laws despite high visual quality. We show this stems from insufficient physical constraints in prompts rather than model limitations: manually adding physics details reliably produces physically plausible videos, but requires expertise and does not scale. We present PhyPrompt, a two-stage reinforcement learning framework that automatically refines prompts for physically realistic generation. First, we fine-tune a large language model on a physics-focused Chain-of-Thought dataset to integrate principles like object motion and force interactions while preserving user intent. Second, we apply Group Relative Policy Optimization with a dynamic reward curriculum that initially prioritizes semantic fidelity, then progressively shifts toward physical commonsense. This curriculum achieves synergistic optimization: PhyPrompt-7B reaches 40.8\% joint success on VideoPhy2 (8.6pp gain), improving physical commonsense by 11pp (55.8\% to 66.8\%) while simultaneously increasing semantic adherence by 4.4pp (43.4\% to 47.8\%). Remarkably, our curriculum exceeds single-objective training on both metrics, demonstrating compositional prompt discovery beyond conventional multi-objective trade-offs. PhyPrompt outperforms GPT-4o (+3.8\% joint) and DeepSeek-V3 (+2.2\%, 100$\times$ larger) using only 7B parameters. The approach transfers zero-shot across diverse T2V architectures (Lavie, VideoCrafter2, CogVideoX-5B) with up to 16.8\% improvement, establishing that domain-specialized reinforcement learning with compositional curricula surpasses general-purpose scaling for physics-aware generation.
TLDR: The paper introduces PhyPrompt, a reinforcement learning framework that refines text prompts to improve the physical plausibility of text-to-video generation, achieving state-of-the-art results with a 7B parameter model.
TLDR: 该论文介绍了PhyPrompt，一个强化学习框架，通过优化文本提示来提高文本到视频生成的物理合理性，并使用一个7B参数的模型实现了最先进的结果。
Read Paper (PDF)Generating high-fidelity 3D avatars from text or image prompts is highly sought after in virtual reality and human-computer interaction. However, existing text-driven methods often rely on iterative Score Distillation Sampling (SDS) or CLIP optimization, which struggle with fine-grained semantic control and suffer from excessively slow inference. Meanwhile, image-driven approaches are severely bottlenecked by the scarcity and high acquisition cost of high-quality 3D facial scans, limiting model generalization. To address these challenges, we first construct a novel, large-scale dataset comprising over 100,000 pairs across four modalities: fine-grained textual descriptions, in-the-wild face images, high-quality light-normalized texture UV maps, and 3D geometric shapes. Leveraging this comprehensive dataset, we propose PromptAvatar, a framework featuring dual diffusion models. Specifically, it integrates a Texture Diffusion Model (TDM) that supports flexible multi-condition guidance from text and/or image prompts, alongside a Geometry Diffusion Model (GDM) guided by text prompts. By learning the direct mapping from multi-modal prompts to 3D representations, PromptAvatar eliminates the need for time-consuming iterative optimization, successfully generating high-fidelity, shading-free 3D avatars in under 10 seconds. Extensive quantitative and qualitative experiments demonstrate that our method significantly outperforms existing state-of-the-art approaches in generation quality, fine-grained detail alignment, and computational efficiency.
TLDR: This paper introduces PromptAvatar, a dual diffusion model framework that leverages a large-scale multi-modal dataset to generate high-fidelity 3D avatars from text or image prompts significantly faster than existing methods.
TLDR: 本文介绍了 PromptAvatar，一个双扩散模型框架，它利用大规模多模态数据集，从文本或图像提示中生成高保真 3D 头像，速度比现有方法快得多。
Read Paper (PDF)Generating high-quality 360° panoramic videos from perspective input is one of the crucial applications for virtual reality (VR), whereby high-resolution videos are especially important for immersive experience. Existing methods are constrained by computational limitations of vanilla diffusion models, only supporting $\leq$ 1K resolution native generation and relying on suboptimal post super-resolution to increase resolution. We introduce CubeComposer, a novel spatio-temporal autoregressive diffusion model that natively generates 4K-resolution 360° videos. By decomposing videos into cubemap representations with six faces, CubeComposer autoregressively synthesizes content in a well-planned spatio-temporal order, reducing memory demands while enabling high-resolution output. Specifically, to address challenges in multi-dimensional autoregression, we propose: (1) a spatio-temporal autoregressive strategy that orchestrates 360° video generation across cube faces and time windows for coherent synthesis; (2) a cube face context management mechanism, equipped with a sparse context attention design to improve efficiency; and (3) continuity-aware techniques, including cube-aware positional encoding, padding, and blending to eliminate boundary seams. Extensive experiments on benchmark datasets demonstrate that CubeComposer outperforms state-of-the-art methods in native resolution and visual quality, supporting practical VR application scenarios. Project page: https://lg-li.github.io/project/cubecomposer
TLDR: CubeComposer is a novel spatio-temporal autoregressive diffusion model that natively generates 4K-resolution 360° videos from perspective video by decomposing videos into cubemap representations and employing a spatio-temporal autoregressive strategy.
TLDR: CubeComposer 是一种新颖的时空自回归扩散模型，通过将视频分解为立方体贴图表示并采用时空自回归策略，原生生成基于透视视频的 4K 分辨率 360° 视频。
Read Paper (PDF)Recent breakthroughs in Diffusion Transformers (DiTs) have revolutionized the field of visual synthesis due to their superior scalability. To facilitate DiTs' capability of capturing meaningful internal representations, recent works such as REPA incorporate external pretrained encoders for representation alignment. However, the underlying mechanisms governing representation learning within DiTs are not well understood. To this end, we first systematically investigate the representation dynamics of DiTs. Through analyzing the evolution and influence of internal representations under various settings, we reveal that representation diversity across blocks is a crucial factor for effective learning. Based on this key insight, we propose DiverseDiT, a novel framework that explicitly promotes representation diversity. DiverseDiT incorporates long residual connections to diversify input representations across blocks and a representation diversity loss to encourage blocks to learn distinct features. Extensive experiments on ImageNet 256x256 and 512x512 demonstrate that our DiverseDiT yields consistent performance gains and convergence acceleration when applied to different backbones with various sizes, even when tested on the challenging one-step generation setting. Furthermore, we show that DiverseDiT is complementary to existing representation learning techniques, leading to further performance gains. Our work provides valuable insights into the representation learning dynamics of DiTs and offers a practical approach for enhancing their performance.
TLDR: The paper investigates representation learning in Diffusion Transformers (DiTs), identifies representation diversity as a crucial factor, and proposes DiverseDiT, a framework promoting diversity leading to performance gains and faster convergence.
TLDR: 该论文研究了扩散Transformer (DiT)中的表征学习，发现表征多样性是关键因素，并提出了DiverseDiT，该框架通过促进多样性来提高性能和加速收敛。
Read Paper (PDF)Diffusion models achieve state-of-the-art image quality. However, sampling is costly at inference time because it requires a large number of function evaluations (NFEs). To reduce NFEs, classical ODE numerical methods have been adopted. Yet, the choice of prediction type and integration domain leads to different sampling behaviors. To address these issues, we introduce Dual-Solver, which generalizes multistep samplers through learnable parameters that continuously (i) interpolate among prediction types, (ii) select the integration domain, and (iii) adjust the residual terms. It retains the standard predictor-corrector structure while preserving second-order local accuracy. These parameters are learned via a classification-based objective using a frozen pretrained classifier (e.g., MobileNet or CLIP). For ImageNet class-conditional generation (DiT, GM-DiT) and text-to-image generation (SANA, PixArt-$α$), Dual-Solver improves FID and CLIP scores in the low-NFE regime ($3 \le$ NFE $\le 9$) across backbones.
TLDR: The paper introduces Dual-Solver, a generalized ODE solver for diffusion models that learns to optimize prediction types, integration domains, and residual terms, improving image generation FID and CLIP scores, especially in low-NFE regimes.
TLDR: 该论文介绍了Dual-Solver，一种用于扩散模型的通用ODE求解器，它学习优化预测类型、积分域和残差项，从而提高图像生成的FID和CLIP分数，尤其是在低NFE情况下。
Read Paper (PDF)Diffusion models achieve strong generative performance but remain slow at inference due to the need for repeated full-model denoising passes. We present Token-Adaptive Predictor (TAP), a training-free, probe-driven framework that adaptively selects a predictor for each token at every sampling step. TAP uses a single full evaluation of the model's first layer as a low-cost probe to compute proxy losses for a compact family of candidate predictors (instantiated primarily with Taylor expansions of varying order and horizon), then assigns each token the predictor with the smallest proxy error. This per-token "probe-then-select" strategy exploits heterogeneous temporal dynamics, requires no additional training, and is compatible with various predictor designs. TAP incurs negligible overhead while enabling large speedups with little or no perceptual quality loss. Extensive experiments across multiple diffusion architectures and generation tasks show that TAP substantially improves the accuracy-efficiency frontier compared to fixed global predictors and caching-only baselines.
TLDR: This paper introduces TAP, a training-free method for accelerating diffusion model inference by adaptively selecting token-specific predictors based on a low-cost initial layer probe, achieving significant speedups with minimal quality loss.
TLDR: 这篇论文介绍了TAP，一种无需训练的加速扩散模型推理的方法，它通过基于低成本初始层探测自适应地选择token特定的预测器，从而在最小化质量损失的情况下实现显著的加速。
Read Paper (PDF)Classifier-Free Guidance (CFG) has established the foundation for guidance mechanisms in diffusion models, showing that well-designed guidance proxies significantly improve conditional generation and sample quality. Autoguidance (AG) has extended this idea, but it relies on an auxiliary network and leaves solver-induced errors unaddressed. In stiff regions, the ODE trajectory changes sharply, where local truncation error (LTE) becomes a critical factor that deteriorates sample quality. Our key observation is that these errors align with the dominant eigenvector, motivating us to leverage the solver-induced error as a guidance signal. We propose Embedded Runge-Kutta Guidance (ERK-Guid), which exploits detected stiffness to reduce LTE and stabilize sampling. We theoretically and empirically analyze stiffness and eigenvector estimators with solver errors to motivate the design of ERK-Guid. Our experiments on both synthetic datasets and the popular benchmark dataset, ImageNet, demonstrate that ERK-Guid consistently outperforms state-of-the-art methods. Code is available at https://github.com/mlvlab/ERK-Guid.
TLDR: This paper introduces Embedded Runge-Kutta Guidance (ERK-Guid), a novel approach for improving diffusion model sampling by leveraging solver-induced error as a guidance signal, particularly in stiff regions where ODE trajectories change rapidly, leading to improved sample quality.
TLDR: 本文介绍了嵌入式龙格-库塔引导（ERK-Guid），这是一种新的方法，通过利用求解器引起的误差作为引导信号来改进扩散模型采样，特别是在刚性区域，其中ODE轨迹变化迅速，从而提高样本质量。
Read Paper (PDF)Recent advances in large vision models (LVMs) have shifted from modality-specific designs toward unified architectures that jointly process images, videos, and 3D data. However, existing unified LVMs primarily pursue functional integration, while overlooking the deeper goal of cross-vision synergy: the ability to reason over complementary priors across visual modalities. To address this, we present PolyV, a unified LVM that achieves cross-vision synergy at both the architectural and training levels. Architecturally, PolyV adopts a sparse Mixture-of-Experts LVM coordinated by a dynamic modality router, allowing each expert to specialize in modality-specific priors while enabling bidirectional interaction and mutual refinement across modalities. Training-wise, a synergy-aware paradigm combines modality-specific pretraining with coarse-to-fine synergy tuning via knowledge distillation and object-/relation-level alignment. Extensive experiments on 10 benchmarks spanning image, video, and 3D understanding, including synergy-focused datasets requiring spatial or temporal priors, demonstrate that PolyV consistently outperforms existing models, achieving over 10% average improvement over its backbone. Overall, PolyV establishes a unified framework for synesthetic visual reasoning, advancing toward truly synergistic LVMs. Project page: https://sqwu.top/PolyV.
TLDR: The paper introduces PolyV, a unified Large Vision Model (LVM) that achieves cross-vision synergy by using a Mixture-of-Experts architecture and a synergy-aware training paradigm, outperforming existing models on image, video, and 3D understanding tasks.
TLDR: 该论文介绍了PolyV，一种统一的大型视觉模型（LVM），通过使用混合专家架构和协同感知训练模式实现了跨视觉协同作用，在图像、视频和3D理解任务上优于现有模型。
Read Paper (PDF)Interactive world models continually generate video by responding to a user's actions, enabling open-ended generation capabilities. However, existing models typically lack a 3D representation of the environment, meaning 3D consistency must be implicitly learned from data, and spatial memory is restricted to limited temporal context windows. This results in an unrealistic user experience and presents significant obstacles to down-stream tasks such as training agents. To address this, we present PERSIST, a new paradigm of world model which simulates the evolution of a latent 3D scene: environment, camera, and renderer. This allows us to synthesize new frames with persistent spatial memory and consistent geometry. Both quantitative metrics and a qualitative user study show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods, enabling coherent, evolving 3D worlds. We further demonstrate novel capabilities, including synthesising diverse 3D environments from a single image, as well as enabling fine-grained, geometry-aware control over generated experiences by supporting environment editing and specification directly in 3D space. Project page: https://francelico.github.io/persist.github.io
TLDR: The paper introduces PERSIST, a world model that incorporates a latent 3D scene representation for improved spatial memory, 3D consistency, and long-horizon stability in interactive video generation, enabling novel editing capabilities.
TLDR: 该论文介绍了一种名为PERSIST的世界模型，它结合了潜在的3D场景表示，以提高交互式视频生成中的空间记忆、3D一致性和长期稳定性，并实现了新的编辑功能。
Read Paper (PDF)Classifier-Free Guidance (CFG) has emerged as a central approach for enhancing semantic alignment in flow-based diffusion models. In this paper, we explore a unified framework called CFG-Ctrl, which reinterprets CFG as a control applied to the first-order continuous-time generative flow, using the conditional-unconditional discrepancy as an error signal to adjust the velocity field. From this perspective, we summarize vanilla CFG as a proportional controller (P-control) with fixed gain, and typical follow-up variants develop extended control-law designs derived from it. However, existing methods mainly rely on linear control, inherently leading to instability, overshooting, and degraded semantic fidelity especially on large guidance scales. To address this, we introduce Sliding Mode Control CFG (SMC-CFG), which enforces the generative flow toward a rapidly convergent sliding manifold. Specifically, we define an exponential sliding mode surface over the semantic prediction error and introduce a switching control term to establish nonlinear feedback-guided correction. Moreover, we provide a Lyapunov stability analysis to theoretically support finite-time convergence. Experiments across text-to-image generation models including Stable Diffusion 3.5, Flux, and Qwen-Image demonstrate that SMC-CFG outperforms standard CFG in semantic alignment and enhances robustness across a wide range of guidance scales. Project Page: https://hanyang-21.github.io/CFG-Ctrl
TLDR: The paper introduces CFG-Ctrl, a novel Classifier-Free Guidance framework using Sliding Mode Control (SMC) to improve semantic alignment and robustness in text-to-image diffusion models, addressing instability issues in existing linear control methods.
TLDR: 该论文介绍了一种名为CFG-Ctrl的新型无分类器引导框架，该框架采用滑模控制（SMC）来改善文本到图像扩散模型中的语义对齐和鲁棒性，解决了现有线性控制方法中的不稳定性问题。
Read Paper (PDF)The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.
TLDR: This paper explores multimodal pretraining using a Transfusion framework with next-token prediction for language and diffusion for vision, revealing insights about optimal visual representations, modality synergy, world modeling, and scaling laws, especially addressing vision's higher data needs.
TLDR: 该论文探索了使用Transfusion框架进行多模态预训练，通过语言的next-token预测和视觉的扩散，揭示了关于最优视觉表示、模态协同、世界建模和缩放规律的见解，尤其关注了视觉对更高数据需求的问题。
Read Paper (PDF)Earth observation applications increasingly rely on data from multiple sensors, including optical, radar, elevation, and land-cover products. Relationships between these modalities are fundamental for data integration but are inherently non-injective: identical conditioning information can correspond to multiple physically plausible observations. Thus, such conditional mappings should be parametrised as data distributions. As a result, deterministic models tend to collapse toward conditional means and fail to represent the uncertainty and variability required for tasks such as data completion and cross-sensor translation. We introduce COP-GEN, a multimodal latent diffusion transformer that models the joint distribution of heterogeneous Earth Observation modalities at their native spatial resolutions. By parameterising cross-modal mappings as conditional distributions, COP-GEN enables flexible any-to-any conditional generation, including zero-shot modality translation, spectral band infilling, and generation under partial or missing inputs, without task-specific retraining. Experiments on a large-scale global multimodal dataset show that COP-GEN generates diverse yet physically consistent realisations while maintaining strong peak fidelity across optical, radar, and elevation modalities. Qualitative and quantitative analyses demonstrate that the model captures meaningful cross-modal structure and systematically adapts its output uncertainty as conditioning information increases. These results highlight the practical importance of stochastic generative modeling for Earth observation and motivate evaluation protocols that move beyond single-reference, pointwise metrics. Website: https:// miquel-espinosa.github.io/cop-gen
TLDR: This paper introduces COP-GEN, a latent diffusion transformer for generating diverse and physically consistent Earth observation data across multiple modalities (optical, radar, elevation) at native resolutions, addressing limitations of deterministic models in capturing uncertainty.
TLDR: 本文介绍了COP-GEN，一种潜在扩散Transformer，用于生成多样且物理上一致的地球观测数据，涵盖多种模态（光学、雷达、海拔）的原生分辨率，解决了确定性模型在捕捉不确定性方面的局限性。
Read Paper (PDF)Deep learning in cardiac MRI (CMR) is fundamentally constrained by both data scarcity and privacy regulations. This study systematically benchmarks three generative architectures: Denoising Diffusion Probabilistic Models (DDPM), Latent Diffusion Models (LDM), and Flow Matching (FM) for synthetic CMR generation. Utilizing a two-stage pipeline where anatomical masks condition image synthesis, we evaluate generated data across three critical axes: fidelity, utility, and privacy. Our results show that diffusion-based models, particularly DDPM, provide the most effective balance between downstream segmentation utility, image fidelity, and privacy preservation under limited-data conditions, while FM demonstrates promising privacy characteristics with slightly lower task-level performance. These findings quantify the trade-offs between cross-domain generalization and patient confidentiality, establishing a framework for safe and effective synthetic data augmentation in medical imaging.
TLDR: This paper benchmarks DDPM, LDM, and Flow Matching for generating synthetic cardiac MRI, finding that DDPM strikes the best balance between fidelity, utility, and privacy, particularly in limited-data scenarios.
TLDR: 本文对DDPM、LDM和Flow Matching三种生成模型在合成心脏MRI方面的性能进行了基准测试，发现DDPM在保真度、效用和隐私之间取得了最佳平衡，尤其是在数据有限的情况下。
Read Paper (PDF)Proactive and real-time interactive experiences are essential for human-like AI companions, yet face three key challenges: (1) achieving low-latency inference under continuous streaming inputs, (2) autonomously deciding when to respond, and (3) controlling both quality and quantity of generated content to meet real-time constraints. In this work, we instantiate AI companions through two gaming scenarios, commentator and guide, selected for their suitability for automatic evaluation. We introduce the Live Gaming Benchmark, a large-scale dataset with three representative scenarios: solo commentary, co-commentary, and user guidance, and present Proact-VL, a general framework that shapes multimodal language models into proactive, real-time interactive agents capable of human-like environment perception and interaction. Extensive experiments show Proact-VL achieves superior response latency and quality while maintaining strong video understanding capabilities, demonstrating its practicality for real-time interactive applications.
TLDR: The paper introduces Proact-VL, a framework for proactive and real-time interactive AI companions using multimodal language models, along with a new Live Gaming Benchmark dataset, demonstrating improved latency and quality in gaming scenarios.
TLDR: 本文介绍了 Proact-VL，一个用于主动和实时交互的 AI 伴侣框架，它使用多模态语言模型，并提出了一个新的 Live Gaming Benchmark 数据集， 并在游戏场景中展示了改进的延迟和质量。
Read Paper (PDF)