Last 7 Days (July 29 – August 04, 2026)
Modern vision language models (VLMs) turn high-resolution images into long sequences of visual tokens. Every token traverses the language decoder and persists in its prompt KV cache, inflating inference cost and motivating aggressive visual compression. Existing score-based methods assign each token an independent importance score and retain the Top-K. However, text queries consume collective, signed attention messages from the visual population, not isolated patches. Consequently, equally sized Top-K sets can repeatedly cover one salient region, omit sparse but complementary evidence and discard information carried by the removed population. We therefore formulate faithful visual compression as constructing a compact coreset for decoder messages, and introduce our training-free Grounded Message Coreset Pruning (GMC) which jointly allocates support across query-grounded, appearance, and coordinate-aware evidence, then transports discarded states into selected representatives at their original multimodal positions before physical compaction and native attention resume. This decomposes faithful compression into two coupled components, including selecting carriers that cover the required message modes and realizing the signed population message on those carriers. We further derive bounds connecting their errors to signed-message distortion, visual innovation, and candidate-margin stability. Experiments across multiple VLM families and diverse benchmarks demonstrate strong performance, with GMC-H2 retaining 97.78% Full-relative mean capability on Qwen2.5-VL-7B using 80.2% fewer visual tokens, while GMC-L16 reaches 100.36%. Controlled interventions verify that collective support and population realization jointly drive these gains.
Primary: Cardiff University
All Institutions: Cardiff University, Chinese Academy of Sciences, Foundation Model Research Center, Institute of Automation, School of Engineering, School of Future Technology, University of Chinese Academy of Sciences, Wuhan AI Research
[One sentence main contribution]. [The paper introduces GMC, a training-free visual token compression method that formulates faithful compression as constructing a coreset for decoder messages, jointly optimizing support allocation and population realization to preserve signed, complementary evidence, achieving state-of-the-art fidelity with significantly reduced token counts across multiple VLM families.]
The paper proposes "Grounded Message Coreset Pruning" (GMC), a training-free method for compressing visual tokens in Vision-Language Models (VLMs). The core theoretical contribution is reframing token selection not as independent importance scoring, but as constructing a "coreset" for the collective, signed attention messages received by the language decoder. The method involves two coupled steps: (1) Support Allocation: Using a facility-location objective with query-grounded, appearance, and spatial clients to select a subset of tokens that cover complementary evidence modes; (2) Population Realization: Transporting the hidden states of discarded tokens to the selected representatives, preserving their original multimodal coordinates and signed contributions before native attention resumes. The authors provide theoretical bounds connecting the compression error to signed-message distortion and visual innovation. The approach is technically sophisticated, moving beyond simple redundancy reduction to address the specific mechanics of cross-attention in VLMs.
The evaluation is extensive, covering multiple VLM families (Qwen2.5-VL-7B, LLaVA-1.5-7B) and diverse benchmarks (POPE, AMBER, HallusionBench, CHAIR, TextVQA, ChartQA, MME, MMBench, GQA). The results demonstrate that GMC retains significantly higher fidelity than state-of-the-art baselines (VisionZip, MMTok) at aggressive compression ratios (e.g., retaining ~98% capability with 80% fewer tokens). The paper includes rigorous ablation studies isolating the effects of support allocation vs. population realization, and controlled interventions verifying the theoretical claims. The performance gains are consistent across discriminative and generative tasks, and the method shows strong transferability across different model architectures without retraining.
The paper provides detailed descriptions of the algorithm, including the facility-location solver, the transport mechanism, and the coordinate preservation strategy. It mentions frozen probes and specific settings in the appendix (referenced but not fully visible in the provided text snippet, though standard for such submissions). The training-free nature of the method enhances reproducibility as it requires no additional training data or parameter updates. The use of standard benchmarks and open-source base models facilitates independent verification.
The method introduces computational overhead during the pruning phase due to the facility-location optimization and state transport, although this is a one-time prefill cost. The theoretical bounds, while insightful, rely on assumptions about Lipschitz continuity and submodularity that may not hold perfectly in deep non-linear transformers. The paper focuses on static image inputs; the behavior with video or highly dynamic sequences is less explored. Additionally, the "appearance" and "spatial" clients rely on fixed heuristics or banks which might require tuning for specific domains (e.g., dense text vs. open scenes).
This work addresses a critical bottleneck in deploying large VLMs: inference cost and memory usage. By enabling faithful compression without retraining, it lowers the barrier for running powerful multimodal models on edge devices or in high-throughput applications. The theoretical framework of "message coresets" could inspire similar approaches for other sequence-to-sequence or multimodal architectures where collective context matters. It also highlights the importance of preserving signed, complementary information in attention mechanisms, which may influence future model design and pruning strategies. [One sentence main contribution]. [The paper introduces GMC, a training-free visual token compression method that formulates faithful compression as constructing a coreset for decoder messages, jointly optimizing support allocation and population realization to preserve signed, complementary evidence, achieving state-of-the-art fidelity with significantly reduced token counts across multiple VLM families.]
Vision-language models (VLMs) are expected to revise their reasoning when visual evidence changes. Failures to do so are often attributed to insufficient visual attention or contextual inertia, leaving unclear what models reuse instead of recomputing from the current image. We show that evidence-bearing reasoning in a prior chain of thought (CoT) can form a textual shortcut that competes behaviorally with visual recomputation. Across 16 VLMs, a matched counterfactual analysis identifies evidence-bearing content as the most robust carrier of prior-CoT influence. Removing this evidence-bearing content shifts answer preference more than removing length-matched non-evidence context or the final-answer span, with prior control weakening progressively as more stale evidence is removed. Reordering this evidence also weakens prior control, showing that its organization modulates shortcut strength. Beyond the immediate answer, the shortcut can retain residual influence after answer correction: weakening current-image support shifts preference back toward the prior answer, while repeated prior answers and reused premises arise mainly when the shortcut remains active. To limit this influence, we introduce Fresh-State Attention Firewall (FSAF), a training-free intervention that isolates fresh computation from the prior CoT. Across five VLMs, FSAF raises visual update rate from 35.28% to 53.61% and reduces prior-answer rate from 39.22% to 3.67%. Reliable VLM self-reflection therefore requires more than looking again: fresh visual recomputation must be protected from stale textual reuse.
Primary: JD.com
All Institutions: JD.com
The paper makes a significant contribution by diagnosing the "textual shortcut" problem in VLM self-reflection and proposing a novel, training-free attention masking intervention (FSAF) that effectively mitigates this issue, leading to more reliable visual recomputation across a wide range of models.
The paper proposes a rigorous diagnostic framework to isolate "textual shortcuts" in Vision-Language Models (VLMs) during self-reflection. The core methodological contribution is the "Fresh-State Attention Firewall" (FSAF), a training-free intervention that uses attention masking to prevent the model's fresh reasoning trace from attending to the prior, potentially stale, chain-of-thought. The diagnostic methodology involves a matched counterfactual analysis where evidence-bearing content is systematically removed or reordered to measure its causal influence on answer preference. This approach is technically sound, leveraging standard transformer attention mechanisms but applying them in a novel, interventionist manner to diagnose internal model behavior. The distinction between "evidence-bearing" and "non-evidence" context is well-defined and operationally implemented.
The evaluation is extensive, covering 16 different VLMs across multiple families (Qwen, Gemma, InternVL, Kimi). The experiments are carefully controlled, using paired comparisons to isolate the effect of the prior CoT. The results are robust: removing evidence-bearing content consistently shifts answer preference toward the current image, and FSAF significantly improves the "visual update rate" while reducing "prior-answer rate." The inclusion of "support withdrawal" tests to show residual dependence even after correct answers is a strong empirical addition. The use of a large language model as a semantic judge is noted, which is a standard but imperfect practice; however, the paired design mitigates some of this noise. The scale of evaluation (16 models) provides high confidence in the generalizability of the findings.
The paper provides detailed implementation specifics, including the exact attention masking logic, the conversation templates, and the evaluation protocols. The use of vLLM and Transformers backends is standard. The code for FSAF is described in sufficient detail for replication. The dataset (VS-Bench) is referenced from prior work, ensuring consistency. The paper includes an appendix with extensive ablation studies and protocol maps, enhancing reproducibility.
The primary limitation is that FSAF is a training-free intervention that modifies the inference-time attention pattern. While effective, it may not be compatible with all model architectures or inference engines that do not expose low-level attention hooks. Furthermore, the "semantic judge" introduces a potential bias, although the paired design helps. The paper focuses on Qwen models for the FSAF evaluation, which limits the generalizability of the *intervention's* effectiveness to other architectures, although the *diagnosis* is broader. The method assumes that the prior CoT is the primary source of the shortcut, which might not hold in all complex reasoning scenarios involving multi-step visual grounding.
This work has significant implications for the reliability of VLMs in safety-critical applications where self-correction is expected to be robust. By identifying that VLMs often "reuse" stale reasoning rather than "recomputing" from visual evidence, the paper highlights a fundamental flaw in current self-reflection paradigms. The proposed FSAF offers a practical, immediate solution to improve VLM reliability without retraining. This could lead to more trustworthy AI systems in domains like medical imaging or autonomous driving, where visual grounding is paramount. The paper makes a significant contribution by diagnosing the "textual shortcut" problem in VLM self-reflection and proposing a novel, training-free attention masking intervention (FSAF) that effectively mitigates this issue, leading to more reliable visual recomputation across a wide range of models.
We present LiveLight, the first diffusion-based framework for real-time streaming video relighting with interactive 3D lighting control. Achieving this is non-trivial, as it requires overcoming three critical challenges: effectively injecting dynamic 3D lighting into a diffusion model, maintaining high-fidelity generation under an extremely low NFE (Number of Function Evaluations) budget for real-time speed, and facilitating continuous streaming for interactive control. To address these pain points, we propose three key designs. First, for accurate lighting injection, we propose a lightweight adapter that feeds Multi-Plane Light Irradiance (MPLI) conditions-depth-aware irradiance maps encoding 3D lighting geometry-directly into the diffusion backbone. Second, to prevent rendering quality degradation at low NFEs towards real-time distillation, we introduce a geometry-guided feedback branch. This training-time constraint leverages a frozen geometry estimator to enforce depth- and normal-consistent relighting, ensuring geometrically plausible shading without adding inference overhead. Finally, to enable streaming interaction, we develop a progressive rolling-window strategy that maintains a denoising ladder of latent chunks at varying noise levels. By propagating intermediate states, this strategy guarantees temporal coherence and supports arbitrarily long video relighting with per-frame reference refresh. Extensive experiments on real-world and synthetic benchmarks demonstrate that LiveLight achieves state-of-the-art relighting quality while running at real-time speed, significantly outperforming offline baselines in temporal stability, lighting controllability, and user preference. To foster real-time interactive relighting research, we will publicly release our models, training data, and synthetic data generator.
Primary: University of Macau
All Institutions: University of Macau, University of Tuebingen
LiveLight presents a significant technical advancement in real-time video relighting by effectively combining diffusion models with 3D lighting priors and a novel streaming denoising strategy, achieving a rare balance of interactivity, quality, and speed.
The paper proposes "LiveLight," a diffusion-based framework for real-time streaming video relighting. The core technical contributions are threefold: 1) A lightweight adapter injecting Multi-Plane Light Irradiance (MPLI) conditions into the diffusion backbone to handle 3D lighting geometry. 2) A geometry-guided feedback branch using a frozen geometry estimator to enforce depth/normal consistency during training, mitigating quality degradation at low Number of Function Evaluations (NFE). 3) A progressive rolling-window strategy for denoising latent chunks to enable continuous streaming and temporal coherence. The approach addresses the specific bottleneck of real-time interactive control in video diffusion models, which is a significant challenge given the computational cost of diffusion steps. The integration of explicit 3D lighting priors (MPLI) with diffusion is a novel architectural choice compared to standard text/image-conditioned relighting.
The authors evaluate LiveLight on real-world and synthetic benchmarks. They claim state-of-the-art performance in relighting quality, temporal stability, and controllability. The key metric is "real-time speed," implying a high frames-per-second (FPS) rate, likely achieved through the low NFE distillation. The comparison against offline baselines highlights the trade-off between quality and speed, arguing that LiveLight achieves a superior balance. The inclusion of user preference studies adds qualitative validation. However, as this is a TOG paper (a top-tier graphics venue), the evaluation is expected to be rigorous regarding visual fidelity and physical plausibility of lighting.
The authors state they will publicly release models, training data, and a synthetic data generator. This is a strong indicator of reproducibility. The use of a frozen geometry estimator suggests that pre-trained models for depth/normal estimation are used, which are widely available (e.g., MiDaS, NormalNet), aiding reproducibility. The specific "MPLI" format and adapter architecture details would need to be clearly documented in the code release for full reproducibility.
The paper acknowledges the challenge of maintaining high fidelity at extremely low NFEs. While the geometry-guided feedback helps, diffusion models at very low steps can still suffer from artifacts or loss of fine detail compared to higher-step offline methods. The reliance on a "frozen geometry estimator" means that if the input video has poor depth/normal estimation (e.g., due to motion blur or occlusion), the relighting quality may degrade. The "rolling-window" strategy might introduce boundary artifacts at the edges of the window if not carefully blended. The term "real-time" is relative; it likely refers to >30 FPS on high-end GPUs, not necessarily mobile devices.
This work enables new applications in virtual production, real-time VR/AR content creation, and interactive video editing. By making high-quality, physically plausible relighting interactive and real-time, it lowers the barrier for creators to manipulate lighting in video content. The release of the synthetic data generator could also benefit the broader community working on 3D-aware video generation. LiveLight presents a significant technical advancement in real-time video relighting by effectively combining diffusion models with 3D lighting priors and a novel streaming denoising strategy, achieving a rare balance of interactivity, quality, and speed.
Language models are increasingly promoted from examinees to examiners: they write the test suites, answer keys, rubrics, and reward functions that define correctness for other systems. We measure the capability that role assumes and find it lacking under the protocol the role is usually deployed with, one-shot greedy authoring with no test-time reasoning. Across four reference constructions - two with complete finite truth, one with a hardened executable reference (HumanEval+/MBPP+), one with an explicitly incomplete lexical reference (WordNet) - models judge whether a candidate belongs far better than they author the set itself. On the incompleteness-proof algorithmic construction the gap is +0.34 to +0.29 F1 over a 24x parameter range and does not close; on executable code, models judging at F1 0.74-0.90 author suites admitting only 19-42% of oracle-correct solutions. A control locates the deficit: asked to emit the predicate rather than its extension, the same models reach F1 about 0.99. The failure is not missing knowledge or an inability to specify, but an inability to materialise the region a specification induces. The dominant error is omission, which resists audit: an over-inclusion is a token a reviewer can challenge, a missing member an absence whose discovery is the authoring problem itself. Models detect planted over-inclusions 6-7x more often than planted omissions, and a production deployment of 43,227 items fails omission-first at 10:1. Wired into RLVR, an authored key costs 1.9 points of accuracy against an exact oracle and 18.5 WordNet-relative (six paired seeds, p=0.031). Gating authored verifiers on a known-correct probe cuts false rejection from 58-92% to at most 5%, but keeps only 5-39% of suites. Repairing them instead, by rewriting each wrong expected value to what a reference execution returns, raises yield 3.3-10.6x across four author families.
Primary: University of Macau
All Institutions: University of Macau
The paper makes a compelling and empirically rigorous case for a fundamental asymmetry in LLM capabilities: while models are proficient at judging membership in a set, they are significantly less capable of authoring the set itself, a deficit driven by silent omissions that resists standard audit and has tangible costs in RLVR pipelines.
The paper employs a rigorous comparative methodology to isolate the "judging-authoring asymmetry" in Large Language Models (LLMs). By constructing ground-truth benchmarks where the acceptable set is mechanically decidable (algorithmic) or executable (code), the authors create a controlled environment to measure the fidelity of authored sets against execution-based judgments. The methodology is strong because it avoids the circularity of using LLMs to grade LLMs, instead relying on oracles and execution. The decomposition of the deficit into "emission," "stopping," and "specification" components via format controls (JSON checkboxes vs. free text) and intensional controls (writing predicates vs. enumerating sets) provides a deep mechanistic understanding of the failure mode. The use of multiple model families and scales ensures the findings are not artifacts of a specific architecture.
The experimental evaluation is comprehensive and robust. It covers four distinct construction types: complete finite truth (word lists with simple predicates), executable truth (HumanEval+/MBPP+), lexical truth (WordNet), and arithmetic truth. The results consistently show that models judge membership significantly better (F1 0.74-0.90) than they author the corresponding sets (F1 0.19-0.42 for code). The paper includes ablation studies on prompt sensitivity, emission format, and model scale. Crucially, it tests frontier models (GPT-5.1, Claude Opus 4.8) and finds the gap persists, although test-time reasoning can close it for simple rules. The inclusion of a "production-scale" field evidence section (43,227 items) adds significant weight to the practical relevance of the findings. The statistical significance is addressed via paired seeds and confidence intervals.
The paper provides detailed descriptions of the construction protocols, including the specific predicates, word lists, and code benchmarks used. The use of deterministic seeds for sampling and the clear definition of the authoring vs. execution interfaces enhance reproducibility. The authors explicitly state that no model output is graded by a model, relying instead on mechanical execution or oracle labels, which facilitates independent verification. The code and data are likely available given the standard practices of such venues, though specific URLs are not in the text. The clear distinction between the "gate" and "repair" mitigations allows other researchers to replicate the baseline failures and improvements.
The primary limitation is the scope of the "authoring" task. The paper focuses on one-shot greedy decoding without test-time reasoning for the main results, acknowledging that reasoning can close the gap for simple rules. This means the findings may not apply to systems that utilize extensive chain-of-thought or self-correction. Additionally, the "lexical" construction suffers from the inherent incompleteness of WordNet, which the authors correctly identify as a confound for precision, though they mitigate this by relying on recall and complete-truth constructions for their main claims. The "repair" mitigation requires a known-correct reference solution, which is not always available in real-world scenarios, limiting the immediate applicability of the proposed fix.
This paper has significant implications for the development of LLM-based agents, automated testing, and reinforcement learning from verifiable rewards (RLVR). By demonstrating that LLM-authored verifiers are prone to silent omissions and over-specification, it warns against blindly trusting model-generated test suites and reward functions. The finding that omission errors resist audit suggests that current evaluation pipelines may be systematically biased towards under-acceptance. The proposed mitigation (gating on known-correct probes) offers a practical path forward for deploying LLM-authored components safely. This work shifts the community's focus from "can LLMs judge?" to "can LLMs define the space of correctness?", a crucial distinction for building reliable AI systems. The paper makes a compelling and empirically rigorous case for a fundamental asymmetry in LLM capabilities: while models are proficient at judging membership in a set, they are significantly less capable of authoring the set itself, a deficit driven by silent omissions that resists standard audit and has tangible costs in RLVR pipelines.
Flow Matching trains continuous-time generative models by regressing the velocity field of a probability path between a simple source distribution and a target data distribution. The coupling that pairs source and target samples strongly affects optimization and sample quality, but structured couplings typically rely on mini-batch transport or assignment procedures whose cost grows at least quadratically in batch size. We propose Quantile Coupling Flow Matching (QC-FM), a lightweight one-sided coupling: rather than matching two pre-sampled batches, it samples only the data batch and constructs each paired source directly. Data ranks projected along a small number of random orthogonal directions are mapped to Gaussian quantiles, and the latent code is completed in the orthogonal complement by conditional Gaussian sampling. The construction is one-dimensional per slice, so the coupling requires no pairwise cost matrix and no assignment to solve. We show that, for each drawn frame, this coupling eliminates the irreducible regression variance along every selected slice and makes the ideal flow exactly straight there, while leaving the sampling prior unchanged: generation still starts from the standard Gaussian, and the training source deviates from it only through the copula of the slice codes, whose transport cost we bound. For training, we apply QC to an anchor subset and complete the remaining source slots with exact Gaussian samples, retaining the QC bias while preserving an explicit signal from the Baseline coupling. Across CIFAR-10, CelebA, FFHQ, and ImageNet-64, QC-FM improves over the Baseline under matched training budgets, reducing FID by up to 12.9%, and outperforms OT-CFM on all four datasets. These results suggest that preserving projected rank structure is a simple and scalable way to inject useful geometric bias into FM couplings without solving a mini-batch transport problem.
Primary: Unknown
All Institutions: Unknown
This paper introduces Quantile Coupling Flow Matching, a computationally efficient, one-sided coupling scheme for Flow Matching that reduces regression variance and improves sample quality without the quadratic cost of mini-batch optimal transport.
The paper proposes Quantile Coupling Flow Matching (QC-FM), a method to structure the coupling between source (noise) and target (data) distributions in Flow Matching (FM). Instead of solving an expensive optimal transport (OT) assignment problem within mini-batches, QC-FM projects data onto random orthogonal directions, ranks the projections, and maps these ranks to Gaussian quantiles to construct the source samples. This "one-sided" coupling is computationally efficient ($O(B \log B)$ vs $O(B^2)$ or $O(B^3)$) and theoretically motivated by the reduction of irreducible regression variance along the selected slices. The authors also introduce hybrid schemes (Mixture and Adjacency) to handle the remaining source slots. The methodology is sound, leveraging well-known concepts from sliced Wasserstein distances and comonotone coupling, but applies them in a novel, efficient way for continuous-time generative modeling.
The authors evaluate QC-FM on standard image generation benchmarks: CIFAR-10, CelebA, FFHQ, and ImageNet-64. They compare against the Baseline (independent coupling) and OT-CFM (minibatch optimal transport). Results show that QC-FM-Mixture consistently outperforms the Baseline and OT-CFM in terms of FID scores under matched training budgets. The paper provides detailed ablation studies on hyperparameters (number of slices $k$, anchor ratio $p$) and computational cost analysis, demonstrating significant speedups in coupling construction time compared to Hungarian/Sinkhorn solvers. The experiments are rigorous and support the claims of improved sample quality and efficiency.
The paper provides a detailed description of the algorithm, including the construction of the source samples, the hybrid completion schemes, and the theoretical bounds. The experimental setup is well-described, specifying model architectures (EDM-based U-Net), optimizers, and training schedules. The code is not explicitly linked in the text provided, but the algorithmic details are sufficient for reproduction. The theoretical proofs in the appendix add to the clarity of the method's properties.
The method is batch-local; the quantiles are estimated from the current mini-batch, which may introduce noise or instability compared to global quantile estimates. The performance gain is moderate (up to 12.9% FID reduction), and the method does not achieve global optimality. The choice of hyperparameters ($k$, $p$) appears dataset-dependent, requiring some tuning or heuristic selection. The theoretical analysis focuses on per-slice properties and does not provide a direct bound on final FID or sample quality.
By providing a scalable alternative to expensive optimal transport couplings in Flow Matching, QC-FM enables more efficient training of continuous-time generative models. This could lower the computational barrier for high-quality image generation and facilitate the use of structured couplings in larger-scale settings. The approach may also inspire similar efficient coupling strategies in other diffusion or flow-based modeling frameworks. This paper introduces Quantile Coupling Flow Matching, a computationally efficient, one-sided coupling scheme for Flow Matching that reduces regression variance and improves sample quality without the quadratic cost of mini-batch optimal transport.
Large language model agents increasingly act through stateful tools, yet model generation and environment execution remain serialized at every step. As decoding accelerates, tool execution becomes a growing bottleneck. Existing action- or observation-only speculation leaves much of this latency exposed: value is concentrated in a few slow calls, some outcomes emerge only through execution, and longer lookahead typically requires an increasingly unlikely chain of action predictions. We present AOSpec, a lossless framework that co-speculates actions and observations across the full agent-environment loop. Expected Value Decoding (EVD) directs observation speculation toward outcomes with the greatest expected latency benefit, optimizing expected time hidden rather than hit rate. For outcomes only execution can reveal, AOSpec launches latency-critical target actions in isolated forks that contain their effects, while Joint Action-State Verification (JASV) verifies both the action and its origin state against committed execution before reuse. JASV recasts long-horizon action dependency from full-chain prediction into target action-state verification, breaking the lookahead--accuracy tradeoff and unlocking long-range overlap without sacrificing serial semantics. Across Terminal-Bench serving settings spanning four harnesses, five actor models, and five serving speeds, AOSpec outperforms every practical baseline, reducing mean end-to-end latency by 11.8-32.5% and p99 latency by up to 42.8%. Its gains increase as decoding accelerates, and its observation model transfers from Terminal-Bench to SWE-bench Verified without retraining.
Primary: Unknown
All Institutions: Unknown
The paper presents a practical and timely optimization for LLM agent serving, offering moderate novelty in adapting speculative decoding to the agent loop. While the latency gains are promising, the technical contribution is incremental rather than transformative, and the full rigor of the claims remains unverified without the complete text and code.
The paper proposes "AOSpec," a framework for low-latency serving of LLM agents. The core technical contribution lies in "co-speculation" of actions and observations, moving beyond traditional speculative decoding which typically only predicts tokens. The method introduces Expected Value Decoding (EVD) to prioritize speculation paths with high latency benefits and Joint Action-State Verification (JASV) to verify the consistency of actions and states in isolated forks. The methodology addresses the specific bottleneck of tool execution in agent loops, which is a valid and important problem. However, the approach is largely an engineering adaptation of speculative decoding principles to the agent domain rather than a fundamental theoretical breakthrough. The "isolated forks" mechanism for containing side effects is a standard virtualization concept applied to LLM inference, which is clever but not novel in computer science terms.
The evaluation is conducted on Terminal-Bench and SWE-bench Verified, which are relevant benchmarks for agent performance. The results claim significant latency reductions (11.8-32.5% mean, up to 42.8% p99). The experimental setup spans multiple models and serving speeds, providing a reasonable level of robustness. However, the abstract-only text provided lacks detailed tables, statistical significance tests, or ablation studies that would confirm the magnitude of these gains. The claim of "lossless" performance is critical; if the speculation introduces any semantic drift in the agent's reasoning, the utility is compromised. The paper asserts this is handled by JASV, but without seeing the verification logic's overhead and failure modes, the net benefit is hard to fully assess. The transferability to SWE-bench is a strong point, suggesting generalizability.
The paper text provided is a skeleton with section headers but no detailed methodology or experimental code. Reproducibility cannot be assessed from the text alone. The claims rely on specific implementations of EVD and JASV which are not described in sufficient detail in the abstract. The lack of code or detailed algorithmic pseudocode in the provided text is a significant barrier to immediate reproducibility.
The primary limitation is the dependency on the accuracy of the observation model. If the observation model is incorrect, the speculation fails, potentially requiring rollback or fallback to sequential execution, which could negate latency gains. The "isolated forks" approach likely incurs memory and computational overhead for maintaining state copies, which might become prohibitive for long-horizon agents with large context windows. Furthermore, the assumption that tool execution is the sole bottleneck may not hold for all agent tasks, particularly those with heavy computational steps or network latency that is not parallelizable with decoding.
This work contributes to the efficiency of LLM agents, making them more viable for real-time applications. By reducing latency, it lowers the cost and improves the user experience of agent-based systems. However, faster agents could also accelerate the deployment of automated systems in critical domains, raising safety and alignment concerns if not properly monitored. The focus on efficiency is positive for the field's scalability. The paper presents a practical and timely optimization for LLM agent serving, offering moderate novelty in adapting speculative decoding to the agent loop. While the latency gains are promising, the technical contribution is incremental rather than transformative, and the full rigor of the claims remains unverified without the complete text and code.
Constrained Markov Decision Processes (CMDPs) provide a natural framework for reinforcement learning in safety-critical applications, where agents maximize long-term reward while satisfying long-term constraints. Although primal-dual actor-critic methods with linear critics are well understood, extending order-optimal convergence guarantees to neural critics in average-reward CMDPs has remained open. The main challenge is a fundamental bias-cost trade-off in neural critic estimation: under Neural Tangent Kernel (NTK) analysis, reducing critic bias substantially increases critic optimization cost, preventing order-optimal convergence in the primal-dual framework. We resolve this bottleneck by introducing a hierarchical Multilevel Monte Carlo (MLMC) neural critic that performs debiasing simultaneously across trajectory sampling and critic optimization. The resulting estimator attains the bias of a long critic optimization run with only logarithmic expected sample cost. Building on this estimator, we develop a primal-dual Natural Actor-Critic algorithm that achieves both an optimality gap and a constraint violation of order $\tilde{O}(T^{-1/2})$. This establishes the first order-optimal convergence guarantees for infinite-horizon average-reward CMDPs with general policy parameterization and neural critics, while eliminating the need to know the underlying mixing time. Our results are novel even in the unconstrained setting.
Primary: unknown
All Institutions: unknown
This paper establishes the first order-optimal convergence guarantees for infinite-horizon average-reward CMDPs with neural critics by introducing a hierarchical MLMC critic that decouples critic bias from optimization cost.
The paper proposes a Hierarchical Multilevel Monte Carlo (MLMC) neural critic to address the bias-cost trade-off in primal-dual actor-critic methods for average-reward Constrained MDPs (CMDPs). The core technical contribution is the application of a two-layer MLMC scheme: an outer layer randomizes the critic optimization horizon to debias the critic parameter, and an inner layer randomizes trajectory lengths to debias the gradient estimation. This allows the algorithm to achieve the bias of a long critic optimization run with logarithmic expected sample cost. The methodology is theoretically rigorous, leveraging Neural Tangent Kernel (NTK) analysis to bound the errors introduced by neural function approximation within the MLMC framework. The approach is novel in combining hierarchical variance/bias reduction specifically for the critic optimization loop in a constrained, average-reward setting.
The paper is purely theoretical. It provides detailed proofs of convergence rates ($\tilde{O}(T^{-1/2})$ for optimality gap and constraint violation) and sample complexity bounds. There are no empirical experiments, simulations, or benchmarks provided in the text. While the theoretical results are significant, the lack of empirical validation on standard RL environments (e.g., MuJoCo, Gym) limits the immediate assessment of practical performance, constant factors, and robustness compared to the theoretical asymptotic guarantees.
The paper provides pseudocode for the HiMLMC-PD-NAC algorithm and detailed mathematical formulations. However, without code or empirical results, reproducibility of the practical performance is not assessable. The theoretical claims are self-contained with assumptions clearly stated (ergodicity, NTK regime, smoothness), which aids in theoretical reproducibility.
The primary limitation is the absence of empirical evaluation. Theoretical guarantees in the NTK regime often assume infinite network width or specific initialization conditions that may not hold in practice. The "logarithmic" sample complexity improvement is asymptotic; the constants involved in the MLMC construction might be large, potentially making the method less efficient than simpler baselines in finite-time regimes. Additionally, the method relies on the NTK regime, which restricts the critic to stay close to initialization, potentially limiting the expressiveness of the critic in complex environments.
This work advances the theoretical understanding of safe reinforcement learning with function approximation. By removing the need for mixing time knowledge and achieving order-optimal rates with neural critics, it paves the way for more robust and theoretically sound safe RL algorithms. However, the gap between theory and practice remains a concern for immediate real-world deployment in safety-critical systems without further empirical validation. This paper establishes the first order-optimal convergence guarantees for infinite-horizon average-reward CMDPs with neural critics by introducing a hierarchical MLMC critic that decouples critic bias from optimization cost.
GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.
Primary: Alibaba Group
All Institutions: Alibaba Group
This paper presents a significant engineering achievement in the field of GUI agents, demonstrating strong performance on key benchmarks through a robust, integrated system leveraging large-scale online RL and synthetic data flywheels.
The paper proposes Qwen-UI-Agent, a foundation GUI agent designed for real-world deployment. The core technical contribution lies in the system architecture rather than a novel algorithmic breakthrough. Key components include a unified action space that interleaves GUI operations with CLI execution, a batched action generation mechanism, and a "data flywheel" inspired by AutoResearch for synthetic data generation and failure diagnosis. The training methodology leverages online Reinforcement Learning (RL) on long-horizon trajectories (100+ turns) using a large-scale infrastructure of 10,000+ concurrent environments. While the integration of these components is sophisticated and the engineering scale is impressive, the underlying approach (RLHF/RL for GUI agents, visual-language models as backbones) is an incremental application of existing techniques to a specific, high-value domain. The novelty is moderate, primarily residing in the system-level integration and the specific data flywheel implementation rather than new theoretical insights.
The experimental section is extensive, evaluating the agent across mobile, computer, and web environments. It reports state-of-the-art results on several benchmarks, including MobileWorld (82.1%), MobileWorld-Real (92.2%), AndroidDaily (97.5%), OSWorld-Verified (79.5%), and WebArena (73.6%). The paper compares against "frontier models" such as Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. The breadth of evaluation is a strong point, demonstrating versatility across different UI paradigms. However, the reliance on synthetic data generation via the agent itself introduces potential circularity risks, which are acknowledged but require careful scrutiny. The results are competitive, particularly on mobile tasks, suggesting strong practical utility.
The paper provides details on the training infrastructure (10,000 concurrent environments) and the use of online RL. However, full reproducibility is hindered by the proprietary nature of the underlying Qwen model weights (unless open-sourced separately), the specific proprietary sandbox environments, and the exact hyperparameters of the RL training loop. The "data flywheel" process is described conceptually but lacks the granular implementation details needed to exactly replicate the data construction pipeline.
The paper acknowledges limitations in its final section, likely including issues with generalization to unseen app layouts, latency in real-time interaction, and safety concerns with autonomous CLI execution. A significant limitation is the dependency on the quality of the underlying VLM and the potential for error propagation in long-horizon tasks. The comparison with "Opus 4.8" and "GPT-5.6 Sol" refers to models that may not yet exist or are hypothetical/future versions in the context of current public knowledge (as of early 2024/2025), which raises questions about the timeliness or accuracy of the benchmarking claims if these are not standard public baselines. If these are internal Alibaba models or specific versions, the comparison should be clearer.
This work has significant potential impact by advancing the state of autonomous agents for digital interaction. It moves beyond simple benchmark solving to "real-world centric" agents capable of long-horizon tasks. This could accelerate the adoption of AI assistants in mobile and desktop computing. However, it also raises ethical and safety concerns regarding autonomous control over devices, potential for misuse, and the reliability of AI in critical workflows. This paper presents a significant engineering achievement in the field of GUI agents, demonstrating strong performance on key benchmarks through a robust, integrated system leveraging large-scale online RL and synthetic data flywheels.
Text-to-image and personalized editing models now synthesize high-fidelity single-subject images with ease. Yet placing multiple named people into shared contact actions such as embrace, carry, or grapple still exposes major failures: fused limbs, invented extremities, and interpenetrating bodies. Existing evaluations largely overlook these anatomical and geometric issues, and VLM-as-a-judge checklists often saturate on Interaction while the errors remain obvious to humans. We introduce MPIE-Bench, a 2,500-sample benchmark of video-mined editing triplets spanning 405 scenes, 14 interaction categories, and four contact densities (C0-C3). We also propose MPIE-Eval, whose two new axes score contact-time geometry from a frozen public multi-person mesh reconstruction. Anatomy asks whether every human-like mass is explained by a complete set of reconstructed bodies, and Interaction asks whether the penetration and surface distance between those bodies match the contact the instruction asked for. Across ten editors, mesh Anatomy tops out at 0.65 and mesh Interaction at 0.72 on two different models, so no single editor is strong on both, while VLM checklists rate the same images above 0.95. A five-rater study confirms that both axes track human judgement more closely than a zero-shot VLM judge, and the rankings hold under ablation of every weight and threshold.
Primary: Zhejiang University
All Institutions: Zhejiang University
MPIE-Bench introduces a rigorous, geometry-aware evaluation framework for multi-person interaction editing, effectively exposing the inadequacy of current VLM-based metrics and providing a new standard for assessing anatomical plausibility in generative models.
The paper proposes MPIE-Bench, a novel benchmark designed to address the specific failure modes of multi-person interaction editing in text-to-image models. The methodology involves constructing a dataset of 2,500 video-mined editing triplets across 405 scenes and 14 interaction categories. Crucially, the authors introduce MPIE-Eval, a geometric evaluation framework that utilizes frozen public multi-person mesh reconstruction to assess "Anatomy" (completeness of human-like masses) and "Interaction" (surface distance and penetration). This shifts evaluation from subjective VLM-based judgments to objective, geometry-aware metrics. The approach is technically sound, leveraging existing mesh reconstruction tools to create a rigorous, automated evaluation protocol that addresses the "fused limbs" and "interpenetrating bodies" issues prevalent in current generative models.
The authors evaluate ten existing editing models on MPIE-Bench. The results highlight a significant discrepancy between VLM-based scores (>0.95) and the proposed geometric metrics (Anatomy max 0.65, Interaction max 0.72). This empirical finding is critical, demonstrating that current VLM-as-a-judge paradigms are saturated and unreliable for this specific task. The ablation studies and five-rater human study confirm that the geometric axes correlate better with human judgment than zero-shot VLMs. The experiments are well-designed to support the claim that existing evaluations are flawed and that the new benchmark provides a more accurate assessment of model capabilities.
The paper presents a frozen evaluation protocol and full-set tables. The use of "frozen public multi-person mesh reconstruction" implies that the core components of the evaluation metric are based on existing, reproducible tools. The dataset construction method (video-mined) is described, though the specific filtering and annotation pipeline details are likely in the appendix. The single-column format without page cap suggests thorough documentation. However, the exact code for the MPIE-Eval metric implementation is not explicitly linked in the provided text, which is a minor barrier to immediate reproducibility, though the methodology is clear.
The benchmark relies on video-mined data, which may introduce biases present in the source videos (e.g., specific poses, lighting, or demographics). The mesh reconstruction models, while public, may have their own failure modes or inaccuracies, particularly with complex occlusions or non-standard body types, which could affect the evaluation scores. The scope is limited to static image editing, not video editing, despite the video source of data. The "contact densities" (C0-C3) provide granularity, but the interaction categories (14) might not cover all nuanced human interactions.
This work has significant implications for the development of more reliable and anatomically correct generative models. By exposing the limitations of VLM-based evaluation, it encourages the community to adopt more rigorous, geometry-aware metrics. This could lead to safer and more usable AI tools for content creation, gaming, and simulation, where physical plausibility is essential. It also highlights the ethical need for better evaluation standards to prevent the spread of misleading or physically impossible synthetic media. MPIE-Bench introduces a rigorous, geometry-aware evaluation framework for multi-person interaction editing, effectively exposing the inadequacy of current VLM-based metrics and providing a new standard for assessing anatomical plausibility in generative models.
Computer-use agents learn from what their actions change, so training one needs applications it can act on, break and reset. The applications that matter most are login-gated and stateful, so synthetic environments stand in for them. Recent pipelines generate such environments in bulk, which moves the bottleneck from how many exist to what is inside each one. The returns, we find, come from three properties: how much behavioural depth an environment carries, whether it targets the interaction an agent actually fails, and whether it improves alongside the model. We present Echoverse, which compiles specifications into stateful applications whose tasks are graded against the application's own database, and a co-evolution loop that reads every graded rollout twice: as repairs to the environment, its tasks and its verifier, and as training signal for the model. Trained on twelve such environments, a 9B model improves from 36.5% to 67.1% across fourteen evaluation splits, within fourteen points of the much larger frontier model that taught it. We examine each property in turn. On the same domains, shallow environments push live-site accuracy below the base model (80.0 to 75.0) while deep ones raise it (80.0 to 85.0 and 48.0 to 65.0); drilling one interface control across many renderings transfers to held-out widget families and to the open web; and repairing a single environment lifts the model trained on it from 16.2% to 38.5%. The same worlds serve as reinforcement-learning environments, where a reward combining the grounded verifier with a dense per-step judge raises held-out score from 58.8% to 68.0%. We release four environments as a benchmark, with their applications, seed data and grounded graders. Code: https://aka.ms/echoverse
Primary: Microsoft Research
All Institutions: Microsoft Research
Echoverse presents a significant methodological advance in agent training by introducing a co-evolution loop for synthetic environments, demonstrating that dynamic environment repair and deep stateful simulations are crucial for scaling computer-use agents, achieving competitive performance with significantly smaller models.
The paper introduces "Echoverse," a framework for generating and evolving synthetic environments specifically for training computer-use agents. The core methodological contribution is a "co-evolution loop" where the agent's failures are used to repair the environment (tasks, verifiers, and state) rather than just updating the model weights. This shifts the training paradigm from static dataset generation to dynamic, self-improving simulation. The use of "grounded verifiers" that grade tasks against the application's own database is a novel approach to reward modeling in non-standardized environments. The approach addresses the critical bottleneck in agent training: the lack of diverse, stateful, and interactive environments that mimic real-world web applications.
The authors evaluate a 9B parameter model trained on twelve Echoverse environments. The results show significant improvement from 36.5% to 67.1% across fourteen evaluation splits. The paper provides ablation studies demonstrating the value of "deep" environments over "shallow" ones, showing that deep environments improve live-site accuracy (80.0 to 85.0) while shallow ones degrade it. The co-evolution loop is shown to lift performance on single environments from 16.2% to 38.5%. Additionally, the environments are used as RL environments with a dense per-step judge, raising held-out scores from 58.8% to 68.0%. The model achieves performance within 14 points of a much larger frontier model, suggesting high data efficiency.
The paper states that four environments, including applications, seed data, and grounded graders, are released as a benchmark. The code is available via a Microsoft shortlink. The description of the co-evolution loop and the grounded verifier suggests a clear methodology, though the specific implementation details of the "repair" mechanism would need to be examined in the full text for full reproducibility. The release of seed data and verifiers significantly aids reproducibility compared to black-box web scraping methods.
The paper relies on synthetic environments. While the "deep" environments are designed to mimic reality, there is always a risk of sim-to-real gap, although the live-site accuracy metrics help mitigate this concern. The evaluation is limited to fourteen splits and twelve environments; generalization to the broader, unstructured web is implied but not fully proven at scale. The "repair" mechanism's scalability to thousands of diverse applications is not explicitly detailed in the abstract, though the framework claims to handle it.
This work addresses a major bottleneck in the development of autonomous agents: the lack of high-quality, stateful training data. By providing a method to generate and evolve these environments, Echoverse could accelerate the development of robust computer-use agents. The release of the benchmark and environments contributes to the open science community. However, the potential for misuse (e.g., automated account creation, scraping) exists, though the focus on training robust agents implies a need for safety and alignment, which is a positive aspect of the research. Echoverse presents a significant methodological advance in agent training by introducing a co-evolution loop for synthetic environments, demonstrating that dynamic environment repair and deep stateful simulations are crucial for scaling computer-use agents, achieving competitive performance with significantly smaller models.
Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.
Primary: Shanghai AI Laboratory
All Institutions: Shanghai AI Laboratory, Zhejiang University, University of Oxford
[One sentence main contribution]. This paper introduces Chimera, a hybrid visual diffusion transformer with a novel scaling recipe and attention mechanism that significantly improves compute efficiency and enables long-context video generation. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The Chimera architecture represents a significant step forward in efficient visual generation by effectively combining multiple attention mechanisms and MoE layers to handle long-context dependencies without the quadratic cost of full attention. The HeteroP scaling scheme provides a practical and principled way to scale such heterogeneous architectures, addressing a key challenge in modern deep learning. The empirical results, including the efficiency gains and zero-shot extrapolation capabilities, demonstrate the practical value of this approach. The derived scaling laws offer actionable insights for future research in model scaling. Overall, this is a high-quality paper with strong technical contributions and significant potential impact on the field of computer vision and generative AI.
The paper proposes "Chimera," a hybrid visual diffusion transformer backbone designed to address the quadratic complexity of full attention in high-resolution generation tasks. The architecture is novel in its combination of Kimi Delta Attention (KDA) for long-context state tracking, Multi-head Latent Attention (MLA) for global interaction, and modality-aware short convolutions for local context. A key methodological contribution is the "HeteroP" scaling scheme, which transfers hyperparameters across width and depth based on functional fan-in and model depth, enabling a principled scaling recipe. The approach of processing text, image, and video tokens in a single raster-ordered stream without positional embeddings is also a distinct architectural choice. The integration of Sparse MoE layers to expand capacity while controlling activated compute adds another layer of engineering sophistication.
The experimental section provides strong empirical evidence for the proposed method. The authors report that the dense backbone is 1.7x more compute-efficient than a matched full-attention baseline (Wan-2.1 2B), and the complete system is 7.3x more efficient. They demonstrate zero-shot extrapolation from 5-second to 30-second video generation with minimal FID degradation (6.5%). The paper also presents "Chinchilla-style" scaling laws for activated model size, training tokens, and data ratios, offering valuable insights into the compute-optimal allocation for image vs. video pretraining. The results are robust and directly support the claims of efficiency and scalability.
The paper provides a detailed description of the architecture, including the specific attention mechanisms and the HeteroP scheme. The mention of specific components like KDA and MLA suggests a reliance on established or recently published techniques, which aids reproducibility. The authors provide a clear scaling recipe and report on hyperparameter transferability. However, as is common with large-scale diffusion models, full reproducibility would require access to the specific training data ratios and exact computational resources, which are likely detailed in the appendix or supplementary materials. The code repository URL is not explicitly provided in the text snippets, which is a minor drawback for immediate reproducibility.
The paper does not explicitly discuss the limitations of the Chimera architecture. Potential limitations might include the complexity of training a hybrid architecture with multiple attention types and MoE layers, which could lead to instability or difficult convergence. The reliance on KDA and MLA, which are themselves complex mechanisms, might introduce overhead or specific failure modes not present in simpler transformers. Additionally, the zero-shot extrapolation to 30-second videos, while impressive, may still suffer from quality degradation or coherence issues over very long durations, which is a common challenge in video generation. The paper also does not discuss the inference latency or memory footprint in detail, which are critical for practical deployment.
The work has significant implications for the field of generative AI, particularly in video and high-resolution image generation. By providing a more compute-efficient architecture, it lowers the barrier to entry for training large-scale diffusion models, potentially democratizing access to high-quality generative tools. The insights into scaling laws for activated parameters and data ratios are valuable for the broader community of researchers and practitioners working on large models. However, the potential for misuse in generating deepfakes or misleading content remains a concern, as with any advanced generative model. The authors should consider discussing these ethical implications in the broader impact section. [One sentence main contribution]. This paper introduces Chimera, a hybrid visual diffusion transformer with a novel scaling recipe and attention mechanism that significantly improves compute efficiency and enables long-context video generation. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The Chimera architecture represents a significant step forward in efficient visual generation by effectively combining multiple attention mechanisms and MoE layers to handle long-context dependencies without the quadratic cost of full attention. The HeteroP scaling scheme provides a practical and principled way to scale such heterogeneous architectures, addressing a key challenge in modern deep learning. The empirical results, including the efficiency gains and zero-shot extrapolation capabilities, demonstrate the practical value of this approach. The derived scaling laws offer actionable insights for future research in model scaling. Overall, this is a high-quality paper with strong technical contributions and significant potential impact on the field of computer vision and generative AI.
Vision-language-action (VLA) models commonly adopt an LLM-centric V to L to A pathway, where visual observations are projected into the representation space of a large language model before being decoded into robot actions. Although effective, this design incurs substantial computation and memory overhead at every policy invocation. In this work, we introduce TurboVLA, a new VLA paradigm that reformulates the conventional V to L to A pathway as a direct V + L to A mapping. Instead of using a large language model as the central interface between perception and action, TurboVLA independently encodes visual observations and language instructions, directly exchanges information between them through lightweight bidirectional vision-language interaction, and predicts continuous action chunks with a compact decoder. This simple design constructs task-conditioned representations directly from visual and linguistic features, significantly reducing the computational and memory costs of VLA inference. On LIBERO, TurboVLA achieves 97.7% average success with only 0.2B parameters, 31.2 ms inference latency, and 0.9 GB inference VRAM on a consumer-grade RTX 4090, matching or outperforming substantially larger VLA policies. These results establish TurboVLA as a simple and effective alternative to the prevailing LLM-centric VLA paradigm, offering a new perspective on how vision, language, and action can be connected for efficient robotic manipulation. Code is available at https://github.com/H-EmbodVis/TurboVLA.
Primary: Huazhong University of Science and Technology
All Institutions: Huawei Technologies Co. Ltd, Huazhong University of Science and Technology
TurboVLA presents a compelling and efficient alternative to LLM-centric VLA models by demonstrating that direct vision-language interaction with compact encoders can achieve state-of-the-art manipulation performance with drastically reduced computational overhead, thereby enabling real-time deployment on consumer hardware.
The paper proposes TurboVLA, a Vision-Language-Action model that abandons the Large Language Model (LLM) backbone in favor of a direct mapping from visual and linguistic features to actions. The architecture employs a lightweight text encoder (BERT), a visual encoder (DINOv3), and a bidirectional cross-attention module to fuse these modalities before passing them to a compact transformer decoder for continuous action chunk prediction. The core methodological contribution is the architectural shift from an LLM-centric $V \to L \to A$ pathway to a direct $V+L \to A$ pathway. While the components (BERT, DINOv3, Cross-Attention, ACT decoder) are standard and well-established in their respective domains, the specific integration for real-time robotic control at this efficiency level is a coherent and well-executed design choice. The novelty lies in the empirical demonstration that LLMs are not strictly necessary for execution-level control, challenging the current trend of scaling VLA models with larger LLMs.
The experimental evaluation is rigorous and compelling. The authors benchmark TurboVLA against a wide range of state-of-the-art VLA models (OpenVLA, Pi-0, Pi-0.5, CogVLA, etc.) on the LIBERO suite and RoboTwin 2.0. The results show that TurboVLA achieves 97.7% average success on LIBERO, matching or exceeding models with 10x-40x more parameters and significantly higher latency (31.2 ms vs >80 ms for competitors). The inclusion of real-world deployment on an AgileX Piper platform adds substantial practical value. The ablation studies effectively isolate the contributions of language conditioning, interaction design, and depth. The comparison includes both "capability-oriented" and "acceleration-oriented" baselines, providing a fair assessment of the performance-efficiency trade-off.
The paper provides detailed implementation specifications, including encoder types (DINOv3 ViT-B, BERT), hidden dimensions (256), interaction layers (6), and training hyperparameters (learning rate, batch size, loss function). The code is explicitly stated to be available on GitHub. The use of standard datasets (LIBERO, RoboTwin) and common evaluation protocols ensures that results can be reproduced by other researchers. The measurement of latency and VRAM on a specific hardware setup (RTX 4090) is transparent.
The primary limitation is the scope of the language understanding. By replacing the LLM with BERT, the model is optimized for concrete, execution-level instructions (e.g., "pick up the red block") and may struggle with complex, multi-step reasoning or open-ended semantic generalization that requires the broader world knowledge embedded in LLMs. The authors acknowledge this, noting that high-level task planning is outside the scope of this execution-focused policy. Additionally, the model's performance on highly complex bimanual tasks (RoboTwin) is good but not as dominant as on single-arm tasks, suggesting potential limits in handling extreme spatial-temporal coordination with a compact representation.
This work has significant implications for the democratization of robotics. By enabling high-performance VLA policies to run on consumer-grade hardware with minimal memory footprint, it lowers the barrier to entry for deploying intelligent robotic systems in resource-constrained environments (e.g., mobile robots, embedded systems). It challenges the "bigger is better" narrative in VLA research, encouraging the community to explore more efficient, task-specific architectures rather than relying solely on scaling general-purpose language models. TurboVLA presents a compelling and efficient alternative to LLM-centric VLA models by demonstrating that direct vision-language interaction with compact encoders can achieve state-of-the-art manipulation performance with drastically reduced computational overhead, thereby enabling real-time deployment on consumer hardware.
Coding agents have made substantial progress on software engineering tasks that modify existing codebases, including bug fixing and feature implementation. However, constructing a complete program from scratch remains a major challenge: even the frontier models evaluated on ProgramBench fully resolve fewer than 1% of tasks. One obstacle is the lack of scalable training environments for this from-scratch setting, spanning the whole software engineering life cycle, as existing environment-construction frameworks focus only on a single phase in software development. To address this gap, we introduce MindForge, an automated pipeline that converts open-source command-line programs into source-free environments that expose only a compiled reference executable and its documentation. Using MindForge, we construct training environments from repositories disjoint from those in ProgramBench, and curate a high-quality data recipe consisting of program synthesis trajectories using GLM-5.2 as the teacher agent. Fine-tuning Qwen3.6-27B on these trajectories increases its ProgramBench average test pass rate from 37.98% to 49.51%, achieving performance comparable to substantially larger frontier models. Moreover, the fine-tuned model consistently improves over the base model across all seven unseen software engineering benchmarks, spanning long-horizon repository generation and translation, bug fixing, feature implementation, and cross-language issue resolution, with absolute gains of 31.00 points on RepoZero-C2Rust, 14.16 on DeepSWE, 10.70/4.56 on NL2Repo-Bench (with/without tests), 5.04 on SWE-bench Verified, 5.93 on SWE-bench Pro, 5.22 on SWE-bench Multilingual, and 4.94 on FeatBench.
Primary: Unknown
All Institutions: Unknown
The paper presents a novel and effective method for generating source-free training data for program synthesis, demonstrating significant performance gains on standard benchmarks. By addressing the lack of scalable, whole-life-cycle training environments, MindForge offers a valuable contribution to the field of AI for software engineering, with strong empirical results and clear practical implications.
The paper proposes "MindForge," a pipeline designed to generate source-free training environments for program synthesis. The core innovation lies in converting open-source repositories into environments where the model only has access to a compiled binary and documentation, simulating a realistic "black-box" testing scenario. This addresses a significant gap in current LLM training data, which often relies on source-code-rich environments or single-phase tasks. The method involves curating high-quality trajectories using a strong teacher model (GLM-5.2) to fine-tune a smaller model (Qwen3.6-27B). The approach is technically sound and addresses a critical bottleneck in scaling code agents: the lack of diverse, whole-life-cycle training data that doesn't leak implementation details.
The experimental evaluation is comprehensive and impressive. The authors demonstrate that fine-tuning Qwen3.6-27B on MindForge-generated data yields significant improvements on ProgramBench (37.98% to 49.51% pass rate), outperforming much larger frontier models. Crucially, they evaluate on seven unseen benchmarks spanning various software engineering tasks (RepoZero, DeepSWE, NL2Repo, SWE-bench variants, FeatBench), showing consistent absolute gains. The results suggest that the source-free synthesis approach generalizes well across different types of coding tasks and languages. The use of disjoint repositories for training and evaluation helps mitigate data contamination concerns, adding credibility to the reported gains.
The paper describes the pipeline clearly, but reproducibility depends on the availability of the MindForge tool and the specific GLM-5.2 teacher model weights or API access. If the code and data recipes are open-sourced (which is implied by the nature of such papers but not explicitly confirmed in the text snippet), reproducibility is high. The reliance on a specific teacher model (GLM-5.2) might be a barrier if that model is not publicly available or if its use is restricted.
The primary limitation is the dependency on the quality of the teacher model (GLM-5.2). If the teacher makes systematic errors, these will be distilled into the student. Additionally, the "source-free" constraint, while realistic, might limit the model's ability to learn deep structural reasoning compared to source-code-aware training. The paper focuses on a specific model size (27B); it is unclear how this scales to significantly larger or smaller models. The evaluation, while broad, is still limited to existing benchmarks, and the "whole-life-cycle" claim is supported by the diversity of tasks but not necessarily by a single, unified end-to-end metric.
This work has significant potential to advance the field of AI for software engineering. By providing a scalable method to create diverse, realistic training environments, it lowers the barrier to training high-performing coding agents. This could lead to more robust and generalizable AI assistants for developers, potentially accelerating software development and maintenance. However, the increased capability of coding agents also raises concerns about the security of software supply chains and the potential for automated exploitation of vulnerabilities. The paper presents a novel and effective method for generating source-free training data for program synthesis, demonstrating significant performance gains on standard benchmarks. By addressing the lack of scalable, whole-life-cycle training environments, MindForge offers a valuable contribution to the field of AI for software engineering, with strong empirical results and clear practical implications.
We propose amortized moment matching, utilizing neural networks to learn data moments as distributional training signals. By casting diffusion denoisers through polynomial projections, we establish a general framework for moment amortization, revealing that an $n$-th degree projection explicitly identifies data moments up to order $n+1$. Derived from the tractable affine case, we instantiate the Amortized Fréchet Distance (AMFD) loss. Unlike FD-loss which relies on explicit marginal moment calculations, AMFD is able to dynamically learn conditional moments via an alternating, matrix-free optimization pipeline that effortlessly scales to high-dimensional data. When operating on global representation features, AMFD serves as a powerful post-training objective; empirically, its neural formulation yields more robust training dynamics than exact statistical matching, substantially surpassing the FD baseline on the FDr$^6$ metric and achieving superior one-step generation on ImageNet. Furthermore, it unlocks direct exploration within native generative spaces, suggesting that the first two moments can identify target distributions only in spaces with strong semantics. Finally, when scaled to text-to-image generation, the condition-aware nature of AMFD unlocks massive gains in instruction-following capabilities, enabling our one-step models to outperform their multi-step FLUX.2 [klein] 4B teachers on the GenEval benchmark while achieving on-par performance on PickScore. Code and checkpoints are available at https://github.com/poppuppy/amfd.
Primary: Kuaishou Technology
All Institutions: Kuaishou Technology
[One sentence main contribution]. The paper introduces Amortized Moment Matching (AMFD), a neural framework for scalable, condition-aware distribution matching that leverages polynomial projections of diffusion denoisers to efficiently match conditional moments. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work presents a theoretically motivated and empirically validated method for post-training generative models. By amortizing moment estimation through neural networks, it circumvents the computational bottlenecks of exact moment calculation while maintaining robust training dynamics. The connection between diffusion denoiser projections and data moments provides a fresh theoretical lens on flow matching. The empirical results on ImageNet and text-to-image generation demonstrate that AMFD can significantly improve one-step generation quality and instruction following, offering a practical and effective tool for model distillation. The method's scalability and stability make it a valuable addition to the generative modeling toolkit, particularly for large-scale applications where computational efficiency is paramount.
The paper proposes "Amortized Moment Matching" (AMFD), a post-training objective for generative models. The core theoretical claim is that polynomial projections of diffusion denoisers correspond to conditional data moments. Specifically, an $n$-th degree projection identifies moments up to order $n+1$. The authors instantiate a tractable second-order case (matching mean and covariance) called AMFD. Unlike standard Fréchet Distance (FD) loss which computes empirical moments on batches, AMFD uses a neural network (amortizer) to predict these moments conditionally. This allows for a matrix-free optimization via Jacobian-Vector Products (JVPs), avoiding the $O(D^2)$ memory cost of explicit covariance matrices. The method is applied to ImageNet post-training and text-to-image distillation. The methodology is mathematically grounded in flow matching and orthogonal projections, offering a novel perspective on distribution matching in representation spaces.
The authors evaluate AMFD on ImageNet-256 and text-to-image generation (FLUX.2, PixelGen). On ImageNet, AMFD achieves state-of-the-art FDr$^6$ scores for one-step generation, outperforming FD-loss baselines. In text-to-image tasks, AMFD enables one-step models to surpass multi-step teachers on GenEval (instruction following) while maintaining PickScore. The experiments include ablations on amortizer capacity, conditioning types (conditional vs. unconditional), and native generative spaces (pixel vs. latent). The results are strong and demonstrate the efficacy of the proposed loss function. However, some claims regarding "massive gains" and "state-of-the-art" should be viewed in the context of a preprint with limited independent verification. The comparison to FLUX.2 teachers is significant but relies on specific distillation setups.
The paper provides a GitHub link and mentions code availability. The methodology includes detailed algorithmic descriptions, including the alternating optimization pipeline and JVP implementations. The authors provide specific hyperparameters and architectural details (e.g., AdaLN-conditioned MLPs, RMSNorm usage). The custom JVP implementation for MLPs is described, which aids reproducibility. However, the reliance on specific pre-trained encoders and the complexity of the alternating training loop may pose challenges for exact replication without careful tuning.
The authors acknowledge that AMFD is restricted to first and second moments, leaving higher-order statistics uncaptured. The method relies on off-the-shelf representation encoders, which may bottleneck performance on high-resolution generation if the encoders are not trained at that resolution. Additionally, AMFD is a post-training objective and cannot train models from scratch. The performance gain in conditional settings (AMFD-C) vs. unconditional (AMFD-U) varies by task complexity, suggesting that the amortizer's ability to generalize conditions is a limiting factor.
AMFD offers a scalable and stable alternative to adversarial training and exact moment matching for generative model distillation. By enabling efficient one-step generation with high fidelity, it reduces computational costs for inference. The condition-aware nature of AMFD improves instruction-following capabilities in text-to-image models, which has implications for controllable generation. The framework could be extended to other generative paradigms beyond diffusion, such as flow matching or GANs, potentially influencing the design of future distribution matching objectives. [One sentence main contribution]. The paper introduces Amortized Moment Matching (AMFD), a neural framework for scalable, condition-aware distribution matching that leverages polynomial projections of diffusion denoisers to efficiently match conditional moments. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work presents a theoretically motivated and empirically validated method for post-training generative models. By amortizing moment estimation through neural networks, it circumvents the computational bottlenecks of exact moment calculation while maintaining robust training dynamics. The connection between diffusion denoiser projections and data moments provides a fresh theoretical lens on flow matching. The empirical results on ImageNet and text-to-image generation demonstrate that AMFD can significantly improve one-step generation quality and instruction following, offering a practical and effective tool for model distillation. The method's scalability and stability make it a valuable addition to the generative modeling toolkit, particularly for large-scale applications where computational efficiency is paramount.
The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generative modeling, however, has remained the exception-despite generative models being remarkably capable, they are still not trained end-to-end. This is because, at its core, generative modeling is about handling distributions with many modes, and existing scalable approaches handle this the same way, by factoring the generation procedure, which prevents end-to-end generation. In this work, we introduce Explorative Modeling, a new paradigm that instead factors the training loop, exploring K candidate matches between model generations and data, and training on the best, so predictions commit to modes rather than blurring them. We find Explorative Models (XMs) useful in two settings. First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models-where scaling exploration monotonically improves performance across both continuous and discrete domains (images, video, and language). Notably, gains from exploration increase with scale, climbing from 7% to 36% as data scales and from 13% to 23% as models grow, with efficiency gains more than doubling at 3x the compute. Concretely, exploration improves FLOP efficiency by 4.1x, sample efficiency by 6.2x, parameter efficiency by 47%, lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 FID on ImageNet without guidance, enables scaling how end-to-end existing models are, and unlocks scaling generalization. Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps. Together, these results establish XMs as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm.
Primary: Independent Researcher
All Institutions: Independent Researcher
This paper presents a pragmatic and empirically promising approach to improving generative model efficiency and quality by rethinking the training loop's exploration strategy. While not a revolutionary theoretical shift, the demonstrated gains in FLOP and sample efficiency, along with the ability to match diffusion performance with fewer inference steps, make it a valuable contribution to the practical toolkit of generative modeling. The independent nature of the work adds to its intrigue, though the claims require careful scrutiny regarding baseline comparisons and training stability. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field].
The paper proposes "Explorative Modeling" (XM), a training paradigm that factors the training loop rather than the generation procedure. The core idea is to generate K candidate samples, select the "best" match against the data distribution (likely via a discriminator or likelihood estimate), and backpropagate through this selection process to update the model. The authors frame this as adding a "third pretraining axis" (exploration) alongside parameters and data. While the concept of using auxiliary losses or curriculum learning is not new, the specific framing of "exploring K candidate matches" to commit to modes rather than blur them draws parallels to Gumbel-Softmax or reinforcement learning approaches (like REINFORCE) but applied to standard gradient descent. The novelty lies in the empirical observation that this simple loop improves scaling laws and enables end-to-end reconstruction without the iterative refinement steps typical of diffusion models. However, the methodological depth appears limited; it is essentially a specific sampling and loss-computation strategy rather than a new architectural class or fundamental theoretical breakthrough.
The experimental section claims significant gains: 4.1x FLOP efficiency, 6.2x sample efficiency, and improved FID scores on ImageNet (1.43 without guidance). These are strong empirical claims. The paper reports results across continuous (images, video) and discrete (language) domains. The claim of "near-state-of-the-art" FID on ImageNet without guidance is notable, as unconditional image generation has seen rapid improvements recently. However, the abstract-only summary provided in the prompt text contains some grammatical awkwardness ("scaling how end-to-end existing models are"), which may reflect similar issues in the full text's clarity. The evaluation relies on standard metrics (FID, likelihood), but the comparison baselines are crucial. If the baselines are weak or the "best match" selection is computationally cheap, the efficiency claims might be overstated. The claim of 16-256x fewer inference steps for end-to-end reconstruction is significant for deployment but requires rigorous ablation to ensure the quality is comparable to diffusion models, which are highly optimized.
The paper includes supplementary sections with "Approach Details" and "Experimental Details," suggesting an attempt at reproducibility. However, the core mechanism of "selecting the best" requires a clear definition of the scoring function (e.g., is it a learned critic, a heuristic, or the model's own likelihood?). Without explicit code or very precise mathematical formulation of the "best match" selection and the gradient path through it, reproducibility may be challenging. The mention of "Flapping Airplanes Fellow" suggests independent funding, which often correlates with less standardized code release practices compared to large labs.
The primary limitation is the computational overhead of generating K candidates during training. While the paper claims efficiency gains, the training-time cost of forward passes for K candidates must be substantial. The paper likely glosses over the hyperparameter sensitivity of K and the stability of training with such a loop. Furthermore, the "end-to-end" claim is somewhat semantic; it still involves a two-step process (generate then select/train) within the training loop, which is distinct from the single-pass nature of standard autoregressive or flow-based models. The generalization to very large scales (e.g., billion-parameter models) is not fully demonstrated in the abstract summary, raising questions about scalability beyond the reported experiments.
If validated, Explorative Modeling could simplify generative model pipelines by removing the need for complex iterative sampling procedures (like DDIM or PNDM in diffusion) or complex loss functions (like R-DiT). This could lower the barrier to entry for high-quality generative modeling. However, the environmental impact of training with K-fold exploration needs to be weighed against the inference savings. The potential for misuse in generating high-quality fake media remains a concern, consistent with other generative AI advancements. This paper presents a pragmatic and empirically promising approach to improving generative model efficiency and quality by rethinking the training loop's exploration strategy. While not a revolutionary theoretical shift, the demonstrated gains in FLOP and sample efficiency, along with the ability to match diffusion performance with fewer inference steps, make it a valuable contribution to the practical toolkit of generative modeling. The independent nature of the work adds to its intrigue, though the claims require careful scrutiny regarding baseline comparisons and training stability. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field].
Modern vision language models (VLMs) turn high-resolution images into long sequences of visual tokens. Every token traverses the language decoder and persists in its prompt KV cache, inflating inference cost and motivating aggressive visual compression. Existing score-based methods assign each token an independent importance score and retain the Top-K. However, text queries consume collective, signed attention messages from the visual population, not isolated patches. Consequently, equally sized Top-K sets can repeatedly cover one salient region, omit sparse but complementary evidence and discard information carried by the removed population. We therefore formulate faithful visual compression as constructing a compact coreset for decoder messages, and introduce our training-free Grounded Message Coreset Pruning (GMC) which jointly allocates support across query-grounded, appearance, and coordinate-aware evidence, then transports discarded states into selected representatives at their original multimodal positions before physical compaction and native attention resume. This decomposes faithful compression into two coupled components, including selecting carriers that cover the required message modes and realizing the signed population message on those carriers. We further derive bounds connecting their errors to signed-message distortion, visual innovation, and candidate-margin stability. Experiments across multiple VLM families and diverse benchmarks demonstrate strong performance, with GMC-H2 retaining 97.78% Full-relative mean capability on Qwen2.5-VL-7B using 80.2% fewer visual tokens, while GMC-L16 reaches 100.36%. Controlled interventions verify that collective support and population realization jointly drive these gains.
Primary: Cardiff University
All Institutions: Cardiff University, Chinese Academy of Sciences, Foundation Model Research Center, Institute of Automation, School of Engineering, School of Future Technology, University of Chinese Academy of Sciences, Wuhan AI Research
[One sentence main contribution]. [The paper introduces GMC, a training-free visual token compression method that formulates faithful compression as constructing a coreset for decoder messages, jointly optimizing support allocation and population realization to preserve signed, complementary evidence, achieving state-of-the-art fidelity with significantly reduced token counts across multiple VLM families.]
The paper proposes "Grounded Message Coreset Pruning" (GMC), a training-free method for compressing visual tokens in Vision-Language Models (VLMs). The core theoretical contribution is reframing token selection not as independent importance scoring, but as constructing a "coreset" for the collective, signed attention messages received by the language decoder. The method involves two coupled steps: (1) Support Allocation: Using a facility-location objective with query-grounded, appearance, and spatial clients to select a subset of tokens that cover complementary evidence modes; (2) Population Realization: Transporting the hidden states of discarded tokens to the selected representatives, preserving their original multimodal coordinates and signed contributions before native attention resumes. The authors provide theoretical bounds connecting the compression error to signed-message distortion and visual innovation. The approach is technically sophisticated, moving beyond simple redundancy reduction to address the specific mechanics of cross-attention in VLMs.
The evaluation is extensive, covering multiple VLM families (Qwen2.5-VL-7B, LLaVA-1.5-7B) and diverse benchmarks (POPE, AMBER, HallusionBench, CHAIR, TextVQA, ChartQA, MME, MMBench, GQA). The results demonstrate that GMC retains significantly higher fidelity than state-of-the-art baselines (VisionZip, MMTok) at aggressive compression ratios (e.g., retaining ~98% capability with 80% fewer tokens). The paper includes rigorous ablation studies isolating the effects of support allocation vs. population realization, and controlled interventions verifying the theoretical claims. The performance gains are consistent across discriminative and generative tasks, and the method shows strong transferability across different model architectures without retraining.
The paper provides detailed descriptions of the algorithm, including the facility-location solver, the transport mechanism, and the coordinate preservation strategy. It mentions frozen probes and specific settings in the appendix (referenced but not fully visible in the provided text snippet, though standard for such submissions). The training-free nature of the method enhances reproducibility as it requires no additional training data or parameter updates. The use of standard benchmarks and open-source base models facilitates independent verification.
The method introduces computational overhead during the pruning phase due to the facility-location optimization and state transport, although this is a one-time prefill cost. The theoretical bounds, while insightful, rely on assumptions about Lipschitz continuity and submodularity that may not hold perfectly in deep non-linear transformers. The paper focuses on static image inputs; the behavior with video or highly dynamic sequences is less explored. Additionally, the "appearance" and "spatial" clients rely on fixed heuristics or banks which might require tuning for specific domains (e.g., dense text vs. open scenes).
This work addresses a critical bottleneck in deploying large VLMs: inference cost and memory usage. By enabling faithful compression without retraining, it lowers the barrier for running powerful multimodal models on edge devices or in high-throughput applications. The theoretical framework of "message coresets" could inspire similar approaches for other sequence-to-sequence or multimodal architectures where collective context matters. It also highlights the importance of preserving signed, complementary information in attention mechanisms, which may influence future model design and pruning strategies. [One sentence main contribution]. [The paper introduces GMC, a training-free visual token compression method that formulates faithful compression as constructing a coreset for decoder messages, jointly optimizing support allocation and population realization to preserve signed, complementary evidence, achieving state-of-the-art fidelity with significantly reduced token counts across multiple VLM families.]
Vision-language models (VLMs) are expected to revise their reasoning when visual evidence changes. Failures to do so are often attributed to insufficient visual attention or contextual inertia, leaving unclear what models reuse instead of recomputing from the current image. We show that evidence-bearing reasoning in a prior chain of thought (CoT) can form a textual shortcut that competes behaviorally with visual recomputation. Across 16 VLMs, a matched counterfactual analysis identifies evidence-bearing content as the most robust carrier of prior-CoT influence. Removing this evidence-bearing content shifts answer preference more than removing length-matched non-evidence context or the final-answer span, with prior control weakening progressively as more stale evidence is removed. Reordering this evidence also weakens prior control, showing that its organization modulates shortcut strength. Beyond the immediate answer, the shortcut can retain residual influence after answer correction: weakening current-image support shifts preference back toward the prior answer, while repeated prior answers and reused premises arise mainly when the shortcut remains active. To limit this influence, we introduce Fresh-State Attention Firewall (FSAF), a training-free intervention that isolates fresh computation from the prior CoT. Across five VLMs, FSAF raises visual update rate from 35.28% to 53.61% and reduces prior-answer rate from 39.22% to 3.67%. Reliable VLM self-reflection therefore requires more than looking again: fresh visual recomputation must be protected from stale textual reuse.
Primary: JD.com
All Institutions: JD.com
The paper makes a significant contribution by diagnosing the "textual shortcut" problem in VLM self-reflection and proposing a novel, training-free attention masking intervention (FSAF) that effectively mitigates this issue, leading to more reliable visual recomputation across a wide range of models.
The paper proposes a rigorous diagnostic framework to isolate "textual shortcuts" in Vision-Language Models (VLMs) during self-reflection. The core methodological contribution is the "Fresh-State Attention Firewall" (FSAF), a training-free intervention that uses attention masking to prevent the model's fresh reasoning trace from attending to the prior, potentially stale, chain-of-thought. The diagnostic methodology involves a matched counterfactual analysis where evidence-bearing content is systematically removed or reordered to measure its causal influence on answer preference. This approach is technically sound, leveraging standard transformer attention mechanisms but applying them in a novel, interventionist manner to diagnose internal model behavior. The distinction between "evidence-bearing" and "non-evidence" context is well-defined and operationally implemented.
The evaluation is extensive, covering 16 different VLMs across multiple families (Qwen, Gemma, InternVL, Kimi). The experiments are carefully controlled, using paired comparisons to isolate the effect of the prior CoT. The results are robust: removing evidence-bearing content consistently shifts answer preference toward the current image, and FSAF significantly improves the "visual update rate" while reducing "prior-answer rate." The inclusion of "support withdrawal" tests to show residual dependence even after correct answers is a strong empirical addition. The use of a large language model as a semantic judge is noted, which is a standard but imperfect practice; however, the paired design mitigates some of this noise. The scale of evaluation (16 models) provides high confidence in the generalizability of the findings.
The paper provides detailed implementation specifics, including the exact attention masking logic, the conversation templates, and the evaluation protocols. The use of vLLM and Transformers backends is standard. The code for FSAF is described in sufficient detail for replication. The dataset (VS-Bench) is referenced from prior work, ensuring consistency. The paper includes an appendix with extensive ablation studies and protocol maps, enhancing reproducibility.
The primary limitation is that FSAF is a training-free intervention that modifies the inference-time attention pattern. While effective, it may not be compatible with all model architectures or inference engines that do not expose low-level attention hooks. Furthermore, the "semantic judge" introduces a potential bias, although the paired design helps. The paper focuses on Qwen models for the FSAF evaluation, which limits the generalizability of the *intervention's* effectiveness to other architectures, although the *diagnosis* is broader. The method assumes that the prior CoT is the primary source of the shortcut, which might not hold in all complex reasoning scenarios involving multi-step visual grounding.
This work has significant implications for the reliability of VLMs in safety-critical applications where self-correction is expected to be robust. By identifying that VLMs often "reuse" stale reasoning rather than "recomputing" from visual evidence, the paper highlights a fundamental flaw in current self-reflection paradigms. The proposed FSAF offers a practical, immediate solution to improve VLM reliability without retraining. This could lead to more trustworthy AI systems in domains like medical imaging or autonomous driving, where visual grounding is paramount. The paper makes a significant contribution by diagnosing the "textual shortcut" problem in VLM self-reflection and proposing a novel, training-free attention masking intervention (FSAF) that effectively mitigates this issue, leading to more reliable visual recomputation across a wide range of models.
We present LiveLight, the first diffusion-based framework for real-time streaming video relighting with interactive 3D lighting control. Achieving this is non-trivial, as it requires overcoming three critical challenges: effectively injecting dynamic 3D lighting into a diffusion model, maintaining high-fidelity generation under an extremely low NFE (Number of Function Evaluations) budget for real-time speed, and facilitating continuous streaming for interactive control. To address these pain points, we propose three key designs. First, for accurate lighting injection, we propose a lightweight adapter that feeds Multi-Plane Light Irradiance (MPLI) conditions-depth-aware irradiance maps encoding 3D lighting geometry-directly into the diffusion backbone. Second, to prevent rendering quality degradation at low NFEs towards real-time distillation, we introduce a geometry-guided feedback branch. This training-time constraint leverages a frozen geometry estimator to enforce depth- and normal-consistent relighting, ensuring geometrically plausible shading without adding inference overhead. Finally, to enable streaming interaction, we develop a progressive rolling-window strategy that maintains a denoising ladder of latent chunks at varying noise levels. By propagating intermediate states, this strategy guarantees temporal coherence and supports arbitrarily long video relighting with per-frame reference refresh. Extensive experiments on real-world and synthetic benchmarks demonstrate that LiveLight achieves state-of-the-art relighting quality while running at real-time speed, significantly outperforming offline baselines in temporal stability, lighting controllability, and user preference. To foster real-time interactive relighting research, we will publicly release our models, training data, and synthetic data generator.
Primary: University of Macau
All Institutions: University of Macau, University of Tuebingen
LiveLight presents a significant technical advancement in real-time video relighting by effectively combining diffusion models with 3D lighting priors and a novel streaming denoising strategy, achieving a rare balance of interactivity, quality, and speed.
The paper proposes "LiveLight," a diffusion-based framework for real-time streaming video relighting. The core technical contributions are threefold: 1) A lightweight adapter injecting Multi-Plane Light Irradiance (MPLI) conditions into the diffusion backbone to handle 3D lighting geometry. 2) A geometry-guided feedback branch using a frozen geometry estimator to enforce depth/normal consistency during training, mitigating quality degradation at low Number of Function Evaluations (NFE). 3) A progressive rolling-window strategy for denoising latent chunks to enable continuous streaming and temporal coherence. The approach addresses the specific bottleneck of real-time interactive control in video diffusion models, which is a significant challenge given the computational cost of diffusion steps. The integration of explicit 3D lighting priors (MPLI) with diffusion is a novel architectural choice compared to standard text/image-conditioned relighting.
The authors evaluate LiveLight on real-world and synthetic benchmarks. They claim state-of-the-art performance in relighting quality, temporal stability, and controllability. The key metric is "real-time speed," implying a high frames-per-second (FPS) rate, likely achieved through the low NFE distillation. The comparison against offline baselines highlights the trade-off between quality and speed, arguing that LiveLight achieves a superior balance. The inclusion of user preference studies adds qualitative validation. However, as this is a TOG paper (a top-tier graphics venue), the evaluation is expected to be rigorous regarding visual fidelity and physical plausibility of lighting.
The authors state they will publicly release models, training data, and a synthetic data generator. This is a strong indicator of reproducibility. The use of a frozen geometry estimator suggests that pre-trained models for depth/normal estimation are used, which are widely available (e.g., MiDaS, NormalNet), aiding reproducibility. The specific "MPLI" format and adapter architecture details would need to be clearly documented in the code release for full reproducibility.
The paper acknowledges the challenge of maintaining high fidelity at extremely low NFEs. While the geometry-guided feedback helps, diffusion models at very low steps can still suffer from artifacts or loss of fine detail compared to higher-step offline methods. The reliance on a "frozen geometry estimator" means that if the input video has poor depth/normal estimation (e.g., due to motion blur or occlusion), the relighting quality may degrade. The "rolling-window" strategy might introduce boundary artifacts at the edges of the window if not carefully blended. The term "real-time" is relative; it likely refers to >30 FPS on high-end GPUs, not necessarily mobile devices.
This work enables new applications in virtual production, real-time VR/AR content creation, and interactive video editing. By making high-quality, physically plausible relighting interactive and real-time, it lowers the barrier for creators to manipulate lighting in video content. The release of the synthetic data generator could also benefit the broader community working on 3D-aware video generation. LiveLight presents a significant technical advancement in real-time video relighting by effectively combining diffusion models with 3D lighting priors and a novel streaming denoising strategy, achieving a rare balance of interactivity, quality, and speed.
In video understanding, vision-language models (VLMs) must ingest massive numbers of visual tokens, causing the computational and memory cost of the prefill stage to rise sharply. Such visual sequences are highly redundant along the spatio-temporal dimension, yet a high compression ratio is often accompanied by the loss of critical details. Existing token-compression methods either employ heuristic, training-free compression with limited content adaptivity or introduce additional modules that require expensive alignment training, leaving the trade-off between efficiency and adaptivity unresolved. To alleviate this limitation, we propose CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens. CRAFT recursively merges tokens by decoupling parameter-free token selection from learnable token fusion: global similarity determines which tokens to merge, while a position-aware weighting module and a content-adaptive channel-wise gate learn how to fuse them. The whole compression pipeline is query-agnostic. Because every retained token is a linear combination of the original tokens, CRAFT preserves their true spatio-temporal coordinates and stays aligned with the pre-trained language model's input distribution. Experiments on multiple representative video benchmarks show that CRAFT consistently outperforms prior state-of-the-art token-compression methods. At about $8\times$ compression, it retains roughly $97\%$ of the backbone's average accuracy and shows significant efficiency improvement.
Primary: Unknown (Paper text does not list author affiliations)
All Institutions: Unknown
CRAFT introduces a decoupled select-then-fuse mechanism for video token compression that balances content adaptivity with distribution preservation, achieving state-of-the-art efficiency-accuracy trade-offs for video VLMs.
The paper proposes CRAFT, a token compression method for Video Vision-Language Models (VLMs). The core methodology decouples token selection (training-free, based on global similarity via random projections) from token fusion (learnable, using a position-aware weighting module and a content-adaptive channel-wise gate). The approach is "query-agnostic," allowing the compressed visual tokens to be cached and reused for multiple questions, addressing a key bottleneck in VLM inference (KV-cache memory and prefill compute). The technical design is sound, leveraging standard components (AdaLN, RoPE) in a novel combination for the specific task of iterative token merging. The "select-then-fuse" iterative loop is a logical extension of static merging methods like ToMe, adding learnability to the fusion step to preserve semantic fidelity.
The authors evaluate CRAFT on six representative video benchmarks (Video-MME, LongVideoBench, MVBench, EgoSchema, MLVU, TempCompass) using Qwen3.5-4B as the primary backbone, with transfer experiments to Qwen2.5-VL-7B and LLaVA-OV-7B. The results show that CRAFT consistently outperforms state-of-the-art training-free methods (VisionZip, FastVID, DivPrune, etc.) and competitive learnable methods at ~8x compression, retaining ~97% of the backbone's accuracy. The efficiency analysis demonstrates significant reductions in prefill latency and FLOPs. The ablation studies effectively isolate the contributions of the selection scaffold and the learnable merging module. The cross-backbone transfer experiments strengthen the claim of generalizability. However, the comparison against other *learnable* compressors is limited to a few baselines on specific backbones due to porting difficulties, which is a fair but potentially incomplete comparison of the learnable landscape.
The paper provides detailed algorithmic pseudocode, hyperparameter settings (learning rates, batch sizes, data sources), and architectural details (dimensions, initialization schemes like AdaLN-Zero). The training data composition is described, and de-duplication against evaluation sets is mentioned. The method is implemented as a plug-and-play module, which aids reproducibility. The reliance on random projections for selection is deterministic given a seed. The code is not explicitly linked in the text provided, but the description is sufficient for implementation.
The paper acknowledges that the method is query-agnostic, which is a feature for caching but might miss query-specific redundancies that query-conditioned methods (like PruneVID) could exploit for single-turn accuracy. However, the authors argue that the loss in single-turn accuracy is outweighed by the gain in multi-turn reusability and overall efficiency. The method adds a small number of parameters (5M) and requires a two-stage training curriculum, which is a non-trivial overhead compared to purely training-free methods, though much lighter than full re-alignment. The performance gain, while consistent, is modest in absolute terms (e.g., 64.6 vs 63.7 on Video-MME), suggesting that the field is approaching diminishing returns on compression without significant architectural changes.
This work contributes to the democratization of large multimodal models by making video understanding more computationally efficient. By enabling the caching of compressed visual tokens, it facilitates multi-turn video dialogue systems that were previously too memory-intensive. The method is broadly applicable to any VLM architecture that accepts visual tokens, promoting efficiency in edge deployment and large-scale inference. CRAFT introduces a decoupled select-then-fuse mechanism for video token compression that balances content adaptivity with distribution preservation, achieving state-of-the-art efficiency-accuracy trade-offs for video VLMs.
Diffusion Transformers (DiTs) have emerged as a core architecture in generative modeling due to their scalability and adaptability to multimodal tasks. DiTs comprise isotropic transformer blocks, and learn representations progressively across depth, where the denoising objective drives later layers to focus on fine-detail reconstruction. This results in degraded representation quality and an imbalanced encoder-decoder behavior. Prior approaches such as representation alignment (REPA) mitigate this by encouraging stronger early representations via training regularization. Alternatively, U-Net-style DiT architectures introduce explicit multi-scale encoder-decoder structures for improved convergence. But they build on standard U-Net wisdom via learnable operators for spatial downsampling, which are not well-suited to transformer architectures, introducing inefficiencies and compatibility issues with components such as cross-attention and representation regularization. In this work, we propose UDT, a U-Net diffusion transformer that combines the representation power of DiTs with the encoding-decoding benefits of U-Nets, through data-adaptive token merging for downsampling and upsampling, while preserving the DiT token dimension. Our baseline UDT architecture outperforms existing U-Net DiTs and achieves performance comparable to REPA across all model sizes. Furthermore, using architectural optimization and REPA, UDT outperforms SiT's 7.9 FID at 1400 epochs (w/o CFG) within 40 epochs (~ 40x faster convergence) for XL model size on 256x256 ImageNet. Finally, it achieves strong image generation performance with CFG, reaching FID of 1.38 (320 epochs) with SD-VAE and 1.35 (500 epochs) with VA-VAE, providing a new backbone for DiTs with strong empirical benefits.
Primary: University of Minnesota
All Institutions: University of Minnesota
The paper presents UDT, a novel U-Net style Diffusion Transformer architecture that leverages data-adaptive token merging to achieve faster convergence and improved generative performance, offering a significant efficiency boost for training large-scale diffusion models.
The paper proposes UDT, a U-Net style Diffusion Transformer that integrates data-adaptive token merging (ToMe) for downsampling and upsampling. The core innovation lies in using token similarity to merge redundant tokens (e.g., background regions) rather than fixed spatial pooling, which preserves semantic information better than standard U-Net downsampling in transformers. The authors argue that this approach mitigates the "imbalanced encoder-decoder" problem in isotropic DiTs by explicitly creating a bottleneck while maintaining token dimensionality. The methodology is sound, leveraging existing efficient transformer techniques (ToMe) in a novel architectural context (Diffusion Transformers). The integration with REPA (Representation Alignment) is also logically derived from the preserved token resolution at the bottleneck.
The experimental evaluation is extensive and rigorous. The authors compare UDT against strong baselines including SiT, U-DiT, and REPA variants on ImageNet 256x256 and 512x512. The results demonstrate significant improvements in training efficiency (convergence speed) and final FID scores. Notably, the claim of achieving SOTA-like FID scores in a fraction of the training epochs (e.g., 40x faster convergence for XL models) is compelling and supported by detailed FID-vs-Epoch curves. The ablation studies on token merge strategies, advanced techniques (RoPE, SwiGLU), and drop-in replacement capabilities for various DiT variants (JiT, MMDiT) add robustness to the claims. The comparison with U-DiT is particularly important as it addresses the specific niche of U-Net style DiTs.
The paper provides a GitHub link to the code, which is a strong positive for reproducibility. The experimental setup closely follows established protocols (SiT, REPA), using standard optimizers, learning rates, and evaluation metrics (FID, IS). The implementation details in the appendix are sufficient for replication. The use of standard datasets (ImageNet) and pre-trained encoders (DINOv2 for REPA) further ensures that the results can be verified by the community.
The paper acknowledges limitations regarding video generation and very high-resolution (2K) images, which is standard for current DiT research. However, a potential limitation is the reliance on the ToMe algorithm's heuristic for token merging; while effective, it introduces a hyperparameter (merge rate) that needs tuning. Additionally, the performance gain in FID is partly attributed to faster convergence, meaning the "final" performance might eventually be matched by longer-trained isotropic DiTs, though the efficiency gain is the primary contribution. The paper does not extensively analyze the impact of token merging on the quality of generated samples beyond FID/IS (e.g., diversity, mode collapse), although qualitative samples are provided.
This work has significant implications for the efficiency and accessibility of training large-scale diffusion models. By reducing the computational cost and training time, UDT lowers the barrier to entry for researchers and practitioners. The architectural improvements also enhance the representational quality of DiTs, potentially benefiting downstream tasks like image editing or inpainting that rely on intermediate features. The broader impact is positive, promoting more efficient and scalable generative AI. The paper presents UDT, a novel U-Net style Diffusion Transformer architecture that leverages data-adaptive token merging to achieve faster convergence and improved generative performance, offering a significant efficiency boost for training large-scale diffusion models.
While feed-forward 3D Gaussian Splatting (3DGS) enables efficient 3D reconstruction, achieving high-fidelity rendering remains challenging. Existing pixel-aligned approaches suffer from spatial inflexibility and massive structural redundancy, whereas query-based methods lack 3D priors and entangle geometry with appearance, yielding blurry, pose-dependent results. To overcome these deficiencies, we propose \textbf{QuerySplat}, a feed-forward 3DGS framework driven by geometric priors and explicit appearance decoupling. Specifically, we design a dual-branch query-based decoder: the geometry branch leverages a pretrained Vision Geometric Model for spatial understanding, which intrinsically endows QuerySplat with pose-free modeling capabilities, while the appearance branch recovers high-frequency details through a dedicated pathway separated from geometric attribute regression. Extensive experiments demonstrate that QuerySplat mitigates the blurry rendering issues of early query-based models and consistently outperforms pixel-aligned approaches in rendering fidelity. On the challenging DL3DV benchmark, it achieves state-of-the-art novel view synthesis performance, with average PSNR gains of 2.30 dB and 1.04 dB over the best pose-free and pose-required baselines, respectively. Project Page: https://inspatio.github.io/querysplat.
Primary: InSpatio Research
All Institutions: InSpatio Research
QuerySplat presents a robust and effective solution to the challenges of feed-forward 3DGS by leveraging geometric priors from VGMs and decoupling attribute prediction, achieving state-of-the-art results on challenging benchmarks and advancing the field of pose-free novel view synthesis.
The paper proposes QuerySplat, a feed-forward 3D Gaussian Splatting (3DGS) framework that decouples geometry and appearance prediction using a dual-branch query-based decoder. The core innovation lies in leveraging a frozen pretrained Vision Geometric Model (VGM, specifically VGGT) to provide geometric priors and a self-calibrated coordinate system, enabling pose-free reconstruction. The geometry branch predicts spatial attributes (center, scale, rotation) while a separate appearance branch predicts opacities and colors. This design addresses the entanglement issues in previous query-based methods (like TokenGS) and the spatial inflexibility of pixel-aligned methods. The methodology is technically sound, combining established components (VGMs, 3DGS, query decoders) in a novel architectural arrangement. The use of transient early-stage regularization (Chamfer distance to VGM depth, opacity floor) is a practical engineering contribution to stabilize training.
The evaluation is conducted on the DL3DV benchmark, a large-scale and challenging dataset for novel view synthesis. The paper reports State-of-the-Art (SOTA) performance in terms of PSNR, SSIM, and LPIPS across 2, 4, and 12-view settings compared to recent posed and pose-free baselines (DepthSplat, TokenGS, YoNoSplat, etc.). The gains are significant (e.g., +2.30 dB PSNR over the best pose-free baseline). Qualitative results demonstrate sharper textures and better geometric coherence. The inclusion of an optional Test-Time Optimization (TTO) module further boosts performance, showing flexibility. The ablation studies effectively validate the contributions of the dual-branch design, regularization, and VGM backbone.
The paper provides detailed implementation details, including the VGM backbone (VGGT), query counts, training schedules, and loss functions. The use of a standard benchmark (DL3DV) and standard metrics facilitates comparison. However, as an arXiv preprint without an accompanying public code repository link (only a project page is listed), immediate reproducibility is slightly hindered, though the description is sufficiently detailed for a competent researcher to implement. The reliance on a specific VGM (VGGT) means results are tied to that model's availability.
The paper acknowledges that the method relies on the quality of the underlying VGM. If the VGM fails to estimate accurate geometry or poses (e.g., in textureless regions or extreme occlusions), the reconstruction may suffer. The computational cost of using a large VGM encoder (VGGT) is not explicitly quantified in terms of inference time compared to lighter baselines, though it is described as "feed-forward." The method is currently evaluated primarily on the DL3DV dataset; generalization to other domains (e.g., indoor vs. outdoor, synthetic vs. real) is implied but not exhaustively proven. The "pose-free" capability is dependent on the VGM's pose estimation accuracy.
This work contributes to the democratization of 3D content creation by enabling high-fidelity 3D reconstruction from unposed, sparse image collections without per-scene optimization. This has significant implications for robotics, augmented reality, and digital twins where rapid, accurate 3D understanding from casual captures is crucial. By decoupling geometry and appearance, it also offers a more modular approach to 3D scene representation that could inspire future research in neural rendering. QuerySplat presents a robust and effective solution to the challenges of feed-forward 3DGS by leveraging geometric priors from VGMs and decoupling attribute prediction, achieving state-of-the-art results on challenging benchmarks and advancing the field of pose-free novel view synthesis.
Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.
Primary: StepFun
All Institutions: StepFun, NTU, PKU, UNSW, SJTU, USTC
ParaASR presents a practical and effective method for accelerating LLM-based ASR through multi-token prediction, achieving significant latency reductions while maintaining high accuracy, thereby addressing a critical bottleneck in deploying large-scale speech recognition systems. The work demonstrates that leveraging the deterministic structure of speech allows for efficient parallel decoding, offering a viable path to low-latency, high-quality ASR without the prohibitive costs of autoregressive decoding.
The paper proposes ParaASR, a method to accelerate LLM-based Automatic Speech Recognition (ASR) by leveraging Multi-Token Prediction (MTP). The core insight is that ASR outputs are strongly constrained by the input audio, allowing the model to predict multiple future tokens in a single forward pass. The methodology involves a two-stage process: first, training a standard autoregressive recognizer on a 4B parameter audio-language foundation model; second, aligning five future-token branches through staged optimization. At inference, the model proposes six tokens per step, and an autoregressive verification step (likely a small verifier or self-consistency check) admits the verified prefix. This approach aims to decouple latency from decoder size while maintaining accuracy. The approach is technically sound and builds on recent trends in speculative decoding and multi-token prediction, adapting them specifically for the acoustic-linguistic alignment problem in ASR.
The evaluation covers Chinese, English, and long-form benchmarks. The reported results show low error rates (2.97%, 3.68%, 3.70%) and a significant speedup with a Real-Time Factor (RTF) as low as 0.0053. The average accepted length of 5.0 out of 6 proposed tokens is a strong indicator of the effectiveness of the multi-token prediction in this domain. The experiments demonstrate that the method retains a 32K context window and can handle long audio (30 minutes) in a single pass. The results are compelling and suggest a practical improvement in serving efficiency for LLM-based ASR systems. However, the lack of comparison against other recent speculative decoding or parallel decoding methods in ASR (e.g., parallel decoding with conformers, or other MTP variants) limits the ability to fully gauge the relative novelty and superiority of the specific implementation details.
The paper mentions starting from a "publicly available audio-language foundation," which aids reproducibility. However, specific details on the staged optimization recipe, the architecture of the verifier, and the exact hyperparameters for the multi-token training are often less detailed in such applied papers. Without open-source code or a very detailed appendix, full reproducibility might be challenging. The authors are from StepFun, a major AI lab, which suggests high-quality engineering, but the academic rigor of the reporting needs to be verified against the full text's technical depth.
The primary limitation is the reliance on a specific foundation model architecture. The performance gain is tied to the quality of the pre-trained model and the effectiveness of the MTP alignment. Additionally, the "verification" step, while fast, adds some overhead and potential error propagation if the verifier is not perfect. The paper does not extensively discuss failure modes or cases where the multi-token prediction fails significantly. The focus on latency reduction might come at the cost of slightly higher training complexity or data requirements for the multi-token branches.
This work has significant implications for the deployment of LLM-based ASR systems, making them more cost-effective and scalable for real-time applications. By reducing latency without sacrificing accuracy, it enables new use cases in voice assistants, live transcription, and accessibility tools. The insight that ASR is a natural setting for multi-token prediction due to acoustic anchoring is a valuable theoretical contribution that could influence future research in other structured prediction tasks. ParaASR presents a practical and effective method for accelerating LLM-based ASR through multi-token prediction, achieving significant latency reductions while maintaining high accuracy, thereby addressing a critical bottleneck in deploying large-scale speech recognition systems. The work demonstrates that leveraging the deterministic structure of speech allows for efficient parallel decoding, offering a viable path to low-latency, high-quality ASR without the prohibitive costs of autoregressive decoding.
World Action Models (WAMs) couple action generation with prediction of future states. Their effectiveness depends on whether future dynamics are modeled in a space that is both aligned with action generation and sufficiently geometry-aware to capture where and how actions change the scene. Existing WAMs typically satisfy only part of this requirement, relying on either perceptually heavy observation-space targets or auxiliary latent spaces that are not jointly structured for action relevance and geometry. We propose SG-WAM, a self-guided framework that learns geometry-aware action-conditioned dynamics directly in the policy-derived representation space. SG-WAM introduces learnable dynamics tokens and a Self-Guided World Predictor that forecasts their future latent states conditioned on intervening robot actions. Prediction targets are generated by an exponential moving average copy of the same policy backbone, providing stable supervision within the representation family used by the action expert. Geometric supervision further structures the policy image-token representations, providing spatially grounded context for the dynamics tokens and yielding a future-alignment space that is both action-relevant and geometry-aware. Latent future prediction, geometric grounding, and flow-matching action generation are jointly optimized end-to-end in a unified framework. Built on a 0.9B model without large-scale embodied pretraining, SG-WAM achieves 98.5% average success on LIBERO and 73% on LIBERO-Plus, while outperforming strong baselines in both in-distribution and out-of-distribution real-world evaluations.
Primary: Nanyang Technological University
All Institutions: Nanyang Technological University, National University of Singapore, The University of Hong Kong
SG-WAM presents a novel self-guided world modeling framework that effectively integrates geometric supervision into policy-derived latent spaces, achieving state-of-the-art results on standard robotic manipulation benchmarks with a moderately sized model.
The paper proposes SG-WAM, a framework for world modeling in robotics that integrates action generation with future state prediction. The core innovation lies in learning dynamics directly within the policy-derived representation space using "dynamics tokens." It employs a Self-Guided World Predictor that forecasts future latent states conditioned on robot actions, using an exponential moving average (EMA) copy of the policy backbone for stable supervision. Crucially, it introduces geometric supervision to structure these representations, aiming to create a space that is both action-relevant and geometry-aware. The method combines latent future prediction, geometric grounding, and flow-matching action generation in a unified end-to-end optimization. While the integration of geometric priors into latent world models is a known direction, the specific mechanism of self-guided supervision within the policy space to align action and geometry is a distinct methodological contribution.
The evaluation focuses on the LIBERO and LIBERO-Plus benchmarks, which are standard for long-horizon robotic manipulation tasks. The reported results are 98.5% average success on LIBERO and 73% on LIBERO-Plus. These results are competitive, particularly on LIBERO, suggesting the method is effective for in-distribution tasks. The paper claims outperformance of strong baselines in both in-distribution and out-of-distribution real-world evaluations. However, the abstract-only score of 60 suggests the initial impression was moderate, and the full text analysis confirms that while the results are strong, they do not represent a paradigm shift in performance (e.g., solving previously unsolvable tasks) but rather an incremental improvement in efficiency and generalization via better representation learning. The use of a 0.9B model without large-scale embodied pretraining is a notable efficiency claim, appealing to resource-constrained settings.
The paper describes the architecture (0.9B model), training objectives (joint optimization of prediction, geometric grounding, and action generation), and benchmarks (LIBERO). However, as an arXiv preprint, the availability of code is not guaranteed, and the URL extraction found none. The description of the "geometric supervision" and "dynamics tokens" provides sufficient detail for a competent researcher to attempt reproduction, but the lack of explicit hyperparameters or code links reduces immediate reproducibility confidence.
The primary limitation is the reliance on the LIBERO suite, which, while standard, may not fully capture the complexity of real-world unstructured environments. The claim of "out-of-distribution real-world evaluations" is significant but requires scrutiny of the specific distribution shifts tested. Furthermore, the complexity of jointly optimizing three distinct objectives (prediction, geometry, action) may introduce training instability or require careful tuning, which is not fully detailed in the abstract. The performance drop on LIBERO-Plus (73%) compared to LIBERO (98.5%) suggests limitations in handling more complex or varied tasks.
This work contributes to the field of embodied AI by providing a more efficient and geometry-aware approach to world modeling, which is critical for sample-efficient learning in robotics. By reducing reliance on large-scale pretraining, it makes advanced world modeling more accessible. The alignment of action and geometry in latent space could lead to more robust and interpretable robotic policies. SG-WAM presents a novel self-guided world modeling framework that effectively integrates geometric supervision into policy-derived latent spaces, achieving state-of-the-art results on standard robotic manipulation benchmarks with a moderately sized model.
3D Gaussian Splatting (3DGS) has emerged as a leading technique for real-time novel view synthesis, yet existing 3DGS accelerators suffer from poor architectural scalability: increasing the number of PEs leads to marginal performance improvement during rendering. We identify that the root cause is the tightly coupled ``checking-while-blending'' dataflow, which exacerbates PE underutilization caused by spatial redundancy from irregular Gaussian coverage and temporal redundancy from asynchronous pixel-wise termination under parallel execution. To address this issue, we propose DeGS, a scalable architecture for efficient 3DGS inference. To systematically eliminate the redundancies inherent in rendering, DeGS exploits a decoupled dataflow, restructuring the coupled $α$-checking, transmittance checking, and $α$-blending of the standard rendering process into consecutive workload parsing, reorganization, and blending stages. This allows the fragmented, length-variable, and temporal-dependent workloads to be reorganized into compact, conflict-free, and dense workloads prior to blending, thereby significantly improving PE utilization during parallel blending. Implemented in 28 nm technology, DeGS achieves 2.36$\times$--7.25$\times$ throughput, 1.82$\times$--6.02$\times$ end-to-end speedup, and 1.59$\times$--4.42$\times$ energy efficiency over state-of-the-art 3DGS accelerators (GSCore, GBU, GCC) across diverse scenes and resolutions (720p to 8K). Moreover, scaling from 16 to 1024 PEs, DeGS maintains over 80\% PE utilization at high resolutions, significantly outperforming existing accelerators.
Primary: Institute of Automation, Chinese Academy of Sciences
All Institutions: Institute of Automation, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Shanghai Jiao Tong University
[One sentence main contribution]. [This paper presents DeGS, a scalable 3DGS hardware accelerator that decouples workload parsing and reorganization from blending, achieving significant throughput and energy efficiency gains by eliminating spatial and temporal redundancies inherent in standard coupled dataflows].
The paper proposes DeGS, a domain-specific hardware accelerator for 3D Gaussian Splatting (3DGS). The core methodological innovation is the decoupling of the rendering dataflow into three stages: Span Parsing, Task Reorganization, and Blending Execution. The authors identify that existing accelerators suffer from poor scalability due to a "checking-while-blending" dataflow that exposes irregular spatial and temporal workloads to the processing elements (PEs). DeGS addresses this by using analytic geometry to extract valid row spans (eliminating spatial redundancy) and a packing scheduler to group these spans into conflict-free, fixed-width packets (eliminating temporal redundancy). This allows the backend Blending Array to operate on dense, regular workloads, significantly improving PE utilization as the array scales. The approach is mathematically grounded, utilizing closed-form solutions for ellipse intersections and digital differential analyzer (DDA) variants for efficient span reconstruction.
The evaluation is comprehensive and rigorous. The authors implement DeGS in RTL (SystemVerilog) and synthesize it in 28nm technology. They compare against three state-of-the-art 3DGS accelerators (GSCore, GBU, GCC) and a contemporary NPU baseline (ORANGE). The evaluation covers multiple datasets (Tanks & Temples, Deep Blending, Mip-NeRF 360) and resolutions (720p to 8K). Key results show significant improvements in throughput (2.36x-7.25x), end-to-end speedup (1.82x-6.02x), and energy efficiency (1.59x-4.42x) over baselines. Crucially, the paper demonstrates scalability: DeGS maintains >80% PE utilization when scaling from 16 to 1024 PEs, whereas baselines degrade significantly. The paper also includes ablation studies and numerical error analysis to prove that the approximations do not degrade rendering quality (PSNR/SSIM/LPIPS are comparable to GPU references).
The paper provides detailed architectural descriptions, including micro-architecture diagrams, dataflow explanations, and mathematical formulations for the span parsing logic. The simulation methodology is clearly defined, using cycle-accurate simulators with trace-driven execution. The baselines are well-defined, and the comparison metrics are standard. The use of a common technology node (28nm) and aligned frequency/storage budgets ensures fair comparison. While the full RTL and simulator code are not explicitly linked in the text provided, the level of detail is sufficient for replication by hardware researchers.
The paper focuses exclusively on the rendering stage of 3DGS. The preprocessing stage (culling, projection, sorting) is acknowledged as memory-bound and relies on existing bandwidth improvements. The scalability analysis suggests that the Span Engine and Packing Scheduler might become bottlenecks at very low resolutions with very large arrays, although the proposed asymmetric scaling rule mitigates this. The numerical approximations, while shown to be negligible for visual quality, introduce minor errors that might be relevant for scientific visualization applications requiring strict precision.
This work significantly advances the field of neural rendering systems by solving a critical scalability bottleneck in 3DGS hardware acceleration. By enabling efficient, high-throughput 3DGS rendering on scalable hardware, it facilitates the deployment of photorealistic novel view synthesis in resource-constrained or high-performance applications such as AR/VR, autonomous driving simulation, and real-time digital twins. The decoupled dataflow paradigm could also inspire optimizations for other irregular rendering or rasterization workloads. [One sentence main contribution]. [This paper presents DeGS, a scalable 3DGS hardware accelerator that decouples workload parsing and reorganization from blending, achieving significant throughput and energy efficiency gains by eliminating spatial and temporal redundancies inherent in standard coupled dataflows].