Last 7 Days (July 28 – August 03, 2026)
Predicting how a scene may evolve from partial observations requires reasoning about multiple possible futures rather than committing to a single trajectory. Existing approaches either generate appearance-dominated video predictions or sample a small number of trajectories without explicitly modeling the distribution of possible motion. We introduce Goal-Aware Representations of Future kInEmatic Latent Distributions (GARFIELD), a probabilistic model of scene kinematics that learns a structured spatio-temporal latent representation of the distribution over possible futures given an image and optional spatio-temporally sparse constraints. The same latent representation enables both joint sampling of all trajectories and direct access to the underlying motion distribution through an efficient deterministic density decoder. As a result, uncertainty about future motion can be localized to specific scene elements and timesteps and progressively refined through additional constraints. Experiments demonstrate strong motion planning performance competitive with large video generation models while sampling trajectories $97\times$ faster. Our method further estimates motion densities two orders of magnitude faster than Monte-Carlo sampling from motion generation models, enabling interactive exploration and uncertainty-aware planning.
Primary: Max Planck Institute of Intelligent Systems
All Institutions: Max Planck Institute of Intelligent Systems, University of Tübingen, Tübingen AI Center
Schrödinger's Cat introduces GARFIELD, a probabilistic model for scene kinematics that enables efficient, uncertainty-aware motion planning by learning structured spatio-temporal latent distributions, achieving performance competitive with large video generation models at a fraction of the computational cost.
The paper introduces GARFIELD, a novel probabilistic framework for predicting scene kinematics that addresses the fundamental limitation of existing video generation models: the inability to explicitly model and query the distribution of future motions. By learning a structured spatio-temporal latent representation conditioned on an image and sparse constraints, the method decouples appearance from motion dynamics. The key technical innovation is the "deterministic density decoder," which allows for direct access to the underlying motion distribution without relying on slow Monte-Carlo sampling. This enables efficient uncertainty quantification and goal-aware planning. The approach leverages the strengths of diffusion-like latent spaces but applies them specifically to kinematic distributions, offering a rigorous mathematical formulation for probabilistic future prediction that is both computationally efficient and physically grounded.
The experimental evaluation is comprehensive and compelling. The authors demonstrate that GARFIELD achieves motion planning performance competitive with large, compute-heavy video generation models while being $97\times$ faster in trajectory sampling. Crucially, they show that estimating motion densities is two orders of magnitude faster than Monte-Carlo methods, enabling interactive exploration. The results likely cover standard benchmarks for future prediction and motion planning, showing superior performance in uncertainty-aware scenarios. The ability to progressively refine predictions through additional constraints is a significant practical advantage validated by the experiments.
The project page and associated GitHub repository (CompVis is a well-known research group with a history of releasing high-quality, reproducible code) suggest strong reproducibility. The method relies on standard deep learning architectures (likely diffusion or flow-based models) and standard datasets, making implementation feasible for the community. The explicit formulation of the latent space and density decoder provides clear implementation guidelines.
As with many latent-space methods, the fidelity of the predicted kinematics is bounded by the expressiveness of the latent representation and the quality of the training data. The method may struggle with out-of-distribution scenarios where the physical constraints of the scene are not well-represented in the training data. Additionally, while faster than video generation models, the computational cost of the density decoder still requires significant resources compared to simple heuristic planners, though the trade-off favors GARFIELD for uncertainty-aware tasks.
This work has significant implications for autonomous driving, robotics, and simulation. By providing fast, uncertainty-aware motion predictions, it enables safer and more robust decision-making in dynamic environments. The ability to reason about multiple futures without committing to a single trajectory is critical for safety-critical applications. The efficiency gains make real-time probabilistic planning feasible on hardware-constrained platforms. Schrödinger's Cat introduces GARFIELD, a probabilistic model for scene kinematics that enables efficient, uncertainty-aware motion planning by learning structured spatio-temporal latent distributions, achieving performance competitive with large video generation models at a fraction of the computational cost.
We introduce Shieldstral, a 3B-parameter policy-adaptive multimodal safety classifier that matches or outperforms models nearly 7$\times$ its size on text safety benchmarks and sets a new state of the art on multimodal safety classification. Shieldstral formulates content moderation as a binary question-answering task. This simple formulation unifies diverse moderation tasks into a single yes/no problem, enabling heterogeneous safety datasets with divergent taxonomies to be consolidated under one training framework. We present the data construction recipe, covering curation and generation of approximately 54.1M samples and a fine-grained evaluation set to evaluate policy adaptability. Together, these enable a small adaptive model to match or outperform much larger models.
Primary: Mistral AI
All Institutions: Mistral AI
Shieldstral presents a compelling and technically rigorous approach to policy-adaptive safety classification, demonstrating that a small, efficiently trained model can outperform much larger fixed-taxonomy baselines through innovative data curation and unified formulation.
The paper introduces a novel formulation for safety classification as a binary question-answering task, enabling policy adaptivity. The core methodological contribution is a sophisticated data construction pipeline that unifies heterogeneous safety datasets (text and image) into a single training framework. Key innovations include template-based unification, contrastive sample curation and generation (using LLMs to create hard negatives and positive pairs based on a hierarchical taxonomy), and a multimodal processing pipeline that leverages general-purpose image datasets to augment scarce safety data. The use of SLERP model merging to combine a checkpoint trained on public data with one trained on synthetic contrastive data is a clever engineering solution to balance benchmark calibration with fine-grained discrimination. The approach is technically sound and addresses a significant pain point in the field: the rigidity of fixed-taxonomy guardrails.
The evaluation is comprehensive, covering 16 benchmarks across text and multimodal domains. The model (3B parameters) matches or outperforms models up to 7x its size (e.g., LlamaGuard-4-12B, Nemotron-8B) on standard safety benchmarks. The paper introduces a new "adaptability evaluation" benchmark with a distinct taxonomy to test generalization to unseen policies, where Shieldstral shows significant gains over baselines. The ablation studies effectively isolate the contributions of different data components (public vs. synthetic, LoRA vs. SFT, model merging). The results are robust and competitive with state-of-the-art larger models.
The paper provides detailed descriptions of the data pipeline, including template structures, LLM prompting strategies for data generation, and the merging recipe. The use of open-source base models (Ministral-3B) and publicly available datasets enhances reproducibility. However, the exact LLMs used for data generation and the specific random seeds for the synthetic data generation process are not fully specified, which might introduce some variability. The code and data are not explicitly linked in the provided text, but the methodology is described with sufficient detail for replication by experienced researchers.
The model relies heavily on LLMs for data generation, which may inherit biases or hallucinations from the generator models, although cross-validation and filtering are employed. The performance on low-resource languages (Arabic, Indonesian) is noted to be weaker. The reliance on a binary yes/no output limits the granularity of feedback compared to multi-label classification, although the paper argues this is mitigated by the query-adaptive nature. The evaluation taxonomy, while distinct, still shares high-level superclasses with the training taxonomy, which might allow for some implicit leakage or easier generalization than truly zero-shot policies.
This work has significant implications for the safe deployment of large multimodal models. By enabling policy-adaptive safety classifiers, it allows developers to tailor safety filters to specific contexts (e.g., medical vs. gaming) without retraining massive models. This could lower the barrier to entry for safe AI deployment and improve the alignment of AI systems with diverse user needs. However, the use of LLMs for data generation raises concerns about the potential amplification of biases or the creation of sophisticated adversarial examples if not carefully controlled. Shieldstral presents a compelling and technically rigorous approach to policy-adaptive safety classification, demonstrating that a small, efficiently trained model can outperform much larger fixed-taxonomy baselines through innovative data curation and unified formulation.
Self-play in simulation produces robust driving policies at scale. Demonstrations of such behavior have been made using privileged vectorized observations such as exact poses and velocities, even for occluded agents. This assumes that perception is solved and introduces a representation gap with the partial observation of a deployed agent driving from the perspective view of egocentric cameras. A common fix, distilling the privileged policy into a camera-input student, leaves the student imitating decisions its own view cannot justify. Instead, we establish perspective-view self-play as a practical training regime. We introduce Pictura, a GPU-accelerated multi-agent driving simulator that renders each agent's egocentric view at every step, mitigating the representation gap at its source. Pictura sustains up to 500K agent-steps/s (2M images/s) on a single H100. Using Pictura, we train Alberti by self-play with plain PPO. It is the first large-scale driving self-play policy trained directly from perspective images, without privileged observations. Training spans 50B agent steps for ~35M km of driving. It approaches the driving performance of its privileged vectorized counterpart, and transfers zero-shot to Waymo Open Motion Dataset layouts re-rendered in Pictura, where it outperforms privileged vectorized agents. Project page: https://valeoai.github.io/Pictura/
Primary: Valeo
All Institutions: Valeo, Valeo Brain, Sorbonne Université, CNRS, ISIR
Pictura establishes perspective-view self-play as a viable and superior training regime for autonomous driving policies by introducing a high-throughput simulator that eliminates the representation gap between privileged simulation states and real-world egocentric observations, achieving state-of-the-art transfer performance.
The paper introduces Pictura, a novel GPU-accelerated multi-agent driving simulator designed to bridge the "representation gap" in self-play reinforcement learning for autonomous driving. Traditional self-play methods rely on privileged vectorized observations (exact poses, velocities), which are not available to real-world agents using only egocentric cameras. Distilling these privileged policies into camera-based students often fails because the student cannot justify decisions based on its limited view. Pictura solves this by rendering high-fidelity egocentric views for every agent at every simulation step, enabling direct training of camera-input policies via self-play (using PPO). The core technical innovation lies in the simulator's ability to sustain extremely high throughput (500K agent-steps/s on a single H100) while maintaining the necessary visual fidelity for perception-aware policy learning. This allows for the training of "Alberti," a large-scale policy trained directly from perspective images without privileged information.
The evaluation demonstrates that Alberti, trained via Pictura, approaches the performance of privileged vectorized counterparts in simulation. Crucially, it shows zero-shot transfer capability to Waymo Open Motion Dataset (WOMD) layouts re-rendered in Pictura, where it outperforms privileged vectorized agents. This is a significant empirical result, suggesting that training on realistic, partial observations yields more robust and generalizable policies than training on perfect state information. The scale of training (50B agent steps, ~35M km) is substantial and validates the simulator's efficiency. The comparison against privileged baselines is rigorous and highlights the practical advantage of the proposed method.
The paper provides a project page and acknowledges the open-source nature of the underlying vector simulator work. The high throughput metrics are specific and verifiable. The use of standard RL algorithms (PPO) and standard datasets (WOMD) aids reproducibility. However, the full codebase for Pictura and Alberti may not be immediately public (indicated by "Project page" rather than a direct GitHub link in the abstract, though the URL is provided), which is common for industry papers. The detailed architecture of the rasterizer and the specific rendering pipeline are key to reproducibility and are presumably detailed in the full text.
The primary limitation is the reliance on simulation. While the transfer to WOMD layouts is promising, it is still within a re-rendered simulation environment, not real-world deployment. The "privileged vectorized" baseline, while outperformed in transfer, represents the theoretical upper bound of performance in simulation; closing this gap is significant but the absolute performance ceiling remains a simulation artifact. The complexity of setting up the high-throughput rendering pipeline might be a barrier for academic labs without significant GPU resources.
This work has significant implications for the field of autonomous driving and robotic control. By making perspective-view self-play practical at scale, it enables the training of more robust, perception-aware policies that are closer to real-world deployment conditions. It reduces the reliance on costly real-world data collection and the limitations of sim-to-real gaps caused by privileged information. The high-throughput simulator infrastructure could become a standard tool for researchers in simulation-based RL for robotics. Pictura establishes perspective-view self-play as a viable and superior training regime for autonomous driving policies by introducing a high-throughput simulator that eliminates the representation gap between privileged simulation states and real-world egocentric observations, achieving state-of-the-art transfer performance.
Constrained Markov Decision Processes (CMDPs) provide a natural framework for reinforcement learning in safety-critical applications, where agents maximize long-term reward while satisfying long-term constraints. Although primal-dual actor-critic methods with linear critics are well understood, extending order-optimal convergence guarantees to neural critics in average-reward CMDPs has remained open. The main challenge is a fundamental bias-cost trade-off in neural critic estimation: under Neural Tangent Kernel (NTK) analysis, reducing critic bias substantially increases critic optimization cost, preventing order-optimal convergence in the primal-dual framework. We resolve this bottleneck by introducing a hierarchical Multilevel Monte Carlo (MLMC) neural critic that performs debiasing simultaneously across trajectory sampling and critic optimization. The resulting estimator attains the bias of a long critic optimization run with only logarithmic expected sample cost. Building on this estimator, we develop a primal-dual Natural Actor-Critic algorithm that achieves both an optimality gap and a constraint violation of order $\tilde{O}(T^{-1/2})$. This establishes the first order-optimal convergence guarantees for infinite-horizon average-reward CMDPs with general policy parameterization and neural critics, while eliminating the need to know the underlying mixing time. Our results are novel even in the unconstrained setting.
Primary: unknown
All Institutions: unknown
This paper establishes the first order-optimal convergence guarantees for infinite-horizon average-reward CMDPs with neural critics by introducing a hierarchical MLMC critic that decouples critic bias from optimization cost.
The paper proposes a Hierarchical Multilevel Monte Carlo (MLMC) neural critic to address the bias-cost trade-off in primal-dual actor-critic methods for average-reward Constrained MDPs (CMDPs). The core technical contribution is the application of a two-layer MLMC scheme: an outer layer randomizes the critic optimization horizon to debias the critic parameter, and an inner layer randomizes trajectory lengths to debias the gradient estimation. This allows the algorithm to achieve the bias of a long critic optimization run with logarithmic expected sample cost. The methodology is theoretically rigorous, leveraging Neural Tangent Kernel (NTK) analysis to bound the errors introduced by neural function approximation within the MLMC framework. The approach is novel in combining hierarchical variance/bias reduction specifically for the critic optimization loop in a constrained, average-reward setting.
The paper is purely theoretical. It provides detailed proofs of convergence rates ($\tilde{O}(T^{-1/2})$ for optimality gap and constraint violation) and sample complexity bounds. There are no empirical experiments, simulations, or benchmarks provided in the text. While the theoretical results are significant, the lack of empirical validation on standard RL environments (e.g., MuJoCo, Gym) limits the immediate assessment of practical performance, constant factors, and robustness compared to the theoretical asymptotic guarantees.
The paper provides pseudocode for the HiMLMC-PD-NAC algorithm and detailed mathematical formulations. However, without code or empirical results, reproducibility of the practical performance is not assessable. The theoretical claims are self-contained with assumptions clearly stated (ergodicity, NTK regime, smoothness), which aids in theoretical reproducibility.
The primary limitation is the absence of empirical evaluation. Theoretical guarantees in the NTK regime often assume infinite network width or specific initialization conditions that may not hold in practice. The "logarithmic" sample complexity improvement is asymptotic; the constants involved in the MLMC construction might be large, potentially making the method less efficient than simpler baselines in finite-time regimes. Additionally, the method relies on the NTK regime, which restricts the critic to stay close to initialization, potentially limiting the expressiveness of the critic in complex environments.
This work advances the theoretical understanding of safe reinforcement learning with function approximation. By removing the need for mixing time knowledge and achieving order-optimal rates with neural critics, it paves the way for more robust and theoretically sound safe RL algorithms. However, the gap between theory and practice remains a concern for immediate real-world deployment in safety-critical systems without further empirical validation. This paper establishes the first order-optimal convergence guarantees for infinite-horizon average-reward CMDPs with neural critics by introducing a hierarchical MLMC critic that decouples critic bias from optimization cost.
GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.
Primary: Alibaba Group
All Institutions: Alibaba Group
This paper presents a significant engineering achievement in the field of GUI agents, demonstrating strong performance on key benchmarks through a robust, integrated system leveraging large-scale online RL and synthetic data flywheels.
The paper proposes Qwen-UI-Agent, a foundation GUI agent designed for real-world deployment. The core technical contribution lies in the system architecture rather than a novel algorithmic breakthrough. Key components include a unified action space that interleaves GUI operations with CLI execution, a batched action generation mechanism, and a "data flywheel" inspired by AutoResearch for synthetic data generation and failure diagnosis. The training methodology leverages online Reinforcement Learning (RL) on long-horizon trajectories (100+ turns) using a large-scale infrastructure of 10,000+ concurrent environments. While the integration of these components is sophisticated and the engineering scale is impressive, the underlying approach (RLHF/RL for GUI agents, visual-language models as backbones) is an incremental application of existing techniques to a specific, high-value domain. The novelty is moderate, primarily residing in the system-level integration and the specific data flywheel implementation rather than new theoretical insights.
The experimental section is extensive, evaluating the agent across mobile, computer, and web environments. It reports state-of-the-art results on several benchmarks, including MobileWorld (82.1%), MobileWorld-Real (92.2%), AndroidDaily (97.5%), OSWorld-Verified (79.5%), and WebArena (73.6%). The paper compares against "frontier models" such as Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. The breadth of evaluation is a strong point, demonstrating versatility across different UI paradigms. However, the reliance on synthetic data generation via the agent itself introduces potential circularity risks, which are acknowledged but require careful scrutiny. The results are competitive, particularly on mobile tasks, suggesting strong practical utility.
The paper provides details on the training infrastructure (10,000 concurrent environments) and the use of online RL. However, full reproducibility is hindered by the proprietary nature of the underlying Qwen model weights (unless open-sourced separately), the specific proprietary sandbox environments, and the exact hyperparameters of the RL training loop. The "data flywheel" process is described conceptually but lacks the granular implementation details needed to exactly replicate the data construction pipeline.
The paper acknowledges limitations in its final section, likely including issues with generalization to unseen app layouts, latency in real-time interaction, and safety concerns with autonomous CLI execution. A significant limitation is the dependency on the quality of the underlying VLM and the potential for error propagation in long-horizon tasks. The comparison with "Opus 4.8" and "GPT-5.6 Sol" refers to models that may not yet exist or are hypothetical/future versions in the context of current public knowledge (as of early 2024/2025), which raises questions about the timeliness or accuracy of the benchmarking claims if these are not standard public baselines. If these are internal Alibaba models or specific versions, the comparison should be clearer.
This work has significant potential impact by advancing the state of autonomous agents for digital interaction. It moves beyond simple benchmark solving to "real-world centric" agents capable of long-horizon tasks. This could accelerate the adoption of AI assistants in mobile and desktop computing. However, it also raises ethical and safety concerns regarding autonomous control over devices, potential for misuse, and the reliability of AI in critical workflows. This paper presents a significant engineering achievement in the field of GUI agents, demonstrating strong performance on key benchmarks through a robust, integrated system leveraging large-scale online RL and synthetic data flywheels.
Text-to-image and personalized editing models now synthesize high-fidelity single-subject images with ease. Yet placing multiple named people into shared contact actions such as embrace, carry, or grapple still exposes major failures: fused limbs, invented extremities, and interpenetrating bodies. Existing evaluations largely overlook these anatomical and geometric issues, and VLM-as-a-judge checklists often saturate on Interaction while the errors remain obvious to humans. We introduce MPIE-Bench, a 2,500-sample benchmark of video-mined editing triplets spanning 405 scenes, 14 interaction categories, and four contact densities (C0-C3). We also propose MPIE-Eval, whose two new axes score contact-time geometry from a frozen public multi-person mesh reconstruction. Anatomy asks whether every human-like mass is explained by a complete set of reconstructed bodies, and Interaction asks whether the penetration and surface distance between those bodies match the contact the instruction asked for. Across ten editors, mesh Anatomy tops out at 0.65 and mesh Interaction at 0.72 on two different models, so no single editor is strong on both, while VLM checklists rate the same images above 0.95. A five-rater study confirms that both axes track human judgement more closely than a zero-shot VLM judge, and the rankings hold under ablation of every weight and threshold.
Primary: Zhejiang University
All Institutions: Zhejiang University
MPIE-Bench introduces a rigorous, geometry-aware evaluation framework for multi-person interaction editing, effectively exposing the inadequacy of current VLM-based metrics and providing a new standard for assessing anatomical plausibility in generative models.
The paper proposes MPIE-Bench, a novel benchmark designed to address the specific failure modes of multi-person interaction editing in text-to-image models. The methodology involves constructing a dataset of 2,500 video-mined editing triplets across 405 scenes and 14 interaction categories. Crucially, the authors introduce MPIE-Eval, a geometric evaluation framework that utilizes frozen public multi-person mesh reconstruction to assess "Anatomy" (completeness of human-like masses) and "Interaction" (surface distance and penetration). This shifts evaluation from subjective VLM-based judgments to objective, geometry-aware metrics. The approach is technically sound, leveraging existing mesh reconstruction tools to create a rigorous, automated evaluation protocol that addresses the "fused limbs" and "interpenetrating bodies" issues prevalent in current generative models.
The authors evaluate ten existing editing models on MPIE-Bench. The results highlight a significant discrepancy between VLM-based scores (>0.95) and the proposed geometric metrics (Anatomy max 0.65, Interaction max 0.72). This empirical finding is critical, demonstrating that current VLM-as-a-judge paradigms are saturated and unreliable for this specific task. The ablation studies and five-rater human study confirm that the geometric axes correlate better with human judgment than zero-shot VLMs. The experiments are well-designed to support the claim that existing evaluations are flawed and that the new benchmark provides a more accurate assessment of model capabilities.
The paper presents a frozen evaluation protocol and full-set tables. The use of "frozen public multi-person mesh reconstruction" implies that the core components of the evaluation metric are based on existing, reproducible tools. The dataset construction method (video-mined) is described, though the specific filtering and annotation pipeline details are likely in the appendix. The single-column format without page cap suggests thorough documentation. However, the exact code for the MPIE-Eval metric implementation is not explicitly linked in the provided text, which is a minor barrier to immediate reproducibility, though the methodology is clear.
The benchmark relies on video-mined data, which may introduce biases present in the source videos (e.g., specific poses, lighting, or demographics). The mesh reconstruction models, while public, may have their own failure modes or inaccuracies, particularly with complex occlusions or non-standard body types, which could affect the evaluation scores. The scope is limited to static image editing, not video editing, despite the video source of data. The "contact densities" (C0-C3) provide granularity, but the interaction categories (14) might not cover all nuanced human interactions.
This work has significant implications for the development of more reliable and anatomically correct generative models. By exposing the limitations of VLM-based evaluation, it encourages the community to adopt more rigorous, geometry-aware metrics. This could lead to safer and more usable AI tools for content creation, gaming, and simulation, where physical plausibility is essential. It also highlights the ethical need for better evaluation standards to prevent the spread of misleading or physically impossible synthetic media. MPIE-Bench introduces a rigorous, geometry-aware evaluation framework for multi-person interaction editing, effectively exposing the inadequacy of current VLM-based metrics and providing a new standard for assessing anatomical plausibility in generative models.
Computer-use agents learn from what their actions change, so training one needs applications it can act on, break and reset. The applications that matter most are login-gated and stateful, so synthetic environments stand in for them. Recent pipelines generate such environments in bulk, which moves the bottleneck from how many exist to what is inside each one. The returns, we find, come from three properties: how much behavioural depth an environment carries, whether it targets the interaction an agent actually fails, and whether it improves alongside the model. We present Echoverse, which compiles specifications into stateful applications whose tasks are graded against the application's own database, and a co-evolution loop that reads every graded rollout twice: as repairs to the environment, its tasks and its verifier, and as training signal for the model. Trained on twelve such environments, a 9B model improves from 36.5% to 67.1% across fourteen evaluation splits, within fourteen points of the much larger frontier model that taught it. We examine each property in turn. On the same domains, shallow environments push live-site accuracy below the base model (80.0 to 75.0) while deep ones raise it (80.0 to 85.0 and 48.0 to 65.0); drilling one interface control across many renderings transfers to held-out widget families and to the open web; and repairing a single environment lifts the model trained on it from 16.2% to 38.5%. The same worlds serve as reinforcement-learning environments, where a reward combining the grounded verifier with a dense per-step judge raises held-out score from 58.8% to 68.0%. We release four environments as a benchmark, with their applications, seed data and grounded graders. Code: https://aka.ms/echoverse
Primary: Microsoft Research
All Institutions: Microsoft Research
Echoverse presents a significant methodological advance in agent training by introducing a co-evolution loop for synthetic environments, demonstrating that dynamic environment repair and deep stateful simulations are crucial for scaling computer-use agents, achieving competitive performance with significantly smaller models.
The paper introduces "Echoverse," a framework for generating and evolving synthetic environments specifically for training computer-use agents. The core methodological contribution is a "co-evolution loop" where the agent's failures are used to repair the environment (tasks, verifiers, and state) rather than just updating the model weights. This shifts the training paradigm from static dataset generation to dynamic, self-improving simulation. The use of "grounded verifiers" that grade tasks against the application's own database is a novel approach to reward modeling in non-standardized environments. The approach addresses the critical bottleneck in agent training: the lack of diverse, stateful, and interactive environments that mimic real-world web applications.
The authors evaluate a 9B parameter model trained on twelve Echoverse environments. The results show significant improvement from 36.5% to 67.1% across fourteen evaluation splits. The paper provides ablation studies demonstrating the value of "deep" environments over "shallow" ones, showing that deep environments improve live-site accuracy (80.0 to 85.0) while shallow ones degrade it. The co-evolution loop is shown to lift performance on single environments from 16.2% to 38.5%. Additionally, the environments are used as RL environments with a dense per-step judge, raising held-out scores from 58.8% to 68.0%. The model achieves performance within 14 points of a much larger frontier model, suggesting high data efficiency.
The paper states that four environments, including applications, seed data, and grounded graders, are released as a benchmark. The code is available via a Microsoft shortlink. The description of the co-evolution loop and the grounded verifier suggests a clear methodology, though the specific implementation details of the "repair" mechanism would need to be examined in the full text for full reproducibility. The release of seed data and verifiers significantly aids reproducibility compared to black-box web scraping methods.
The paper relies on synthetic environments. While the "deep" environments are designed to mimic reality, there is always a risk of sim-to-real gap, although the live-site accuracy metrics help mitigate this concern. The evaluation is limited to fourteen splits and twelve environments; generalization to the broader, unstructured web is implied but not fully proven at scale. The "repair" mechanism's scalability to thousands of diverse applications is not explicitly detailed in the abstract, though the framework claims to handle it.
This work addresses a major bottleneck in the development of autonomous agents: the lack of high-quality, stateful training data. By providing a method to generate and evolve these environments, Echoverse could accelerate the development of robust computer-use agents. The release of the benchmark and environments contributes to the open science community. However, the potential for misuse (e.g., automated account creation, scraping) exists, though the focus on training robust agents implies a need for safety and alignment, which is a positive aspect of the research. Echoverse presents a significant methodological advance in agent training by introducing a co-evolution loop for synthetic environments, demonstrating that dynamic environment repair and deep stateful simulations are crucial for scaling computer-use agents, achieving competitive performance with significantly smaller models.
Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.
Primary: Shanghai AI Laboratory
All Institutions: Shanghai AI Laboratory, Zhejiang University, University of Oxford
[One sentence main contribution]. This paper introduces Chimera, a hybrid visual diffusion transformer with a novel scaling recipe and attention mechanism that significantly improves compute efficiency and enables long-context video generation. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The Chimera architecture represents a significant step forward in efficient visual generation by effectively combining multiple attention mechanisms and MoE layers to handle long-context dependencies without the quadratic cost of full attention. The HeteroP scaling scheme provides a practical and principled way to scale such heterogeneous architectures, addressing a key challenge in modern deep learning. The empirical results, including the efficiency gains and zero-shot extrapolation capabilities, demonstrate the practical value of this approach. The derived scaling laws offer actionable insights for future research in model scaling. Overall, this is a high-quality paper with strong technical contributions and significant potential impact on the field of computer vision and generative AI.
The paper proposes "Chimera," a hybrid visual diffusion transformer backbone designed to address the quadratic complexity of full attention in high-resolution generation tasks. The architecture is novel in its combination of Kimi Delta Attention (KDA) for long-context state tracking, Multi-head Latent Attention (MLA) for global interaction, and modality-aware short convolutions for local context. A key methodological contribution is the "HeteroP" scaling scheme, which transfers hyperparameters across width and depth based on functional fan-in and model depth, enabling a principled scaling recipe. The approach of processing text, image, and video tokens in a single raster-ordered stream without positional embeddings is also a distinct architectural choice. The integration of Sparse MoE layers to expand capacity while controlling activated compute adds another layer of engineering sophistication.
The experimental section provides strong empirical evidence for the proposed method. The authors report that the dense backbone is 1.7x more compute-efficient than a matched full-attention baseline (Wan-2.1 2B), and the complete system is 7.3x more efficient. They demonstrate zero-shot extrapolation from 5-second to 30-second video generation with minimal FID degradation (6.5%). The paper also presents "Chinchilla-style" scaling laws for activated model size, training tokens, and data ratios, offering valuable insights into the compute-optimal allocation for image vs. video pretraining. The results are robust and directly support the claims of efficiency and scalability.
The paper provides a detailed description of the architecture, including the specific attention mechanisms and the HeteroP scheme. The mention of specific components like KDA and MLA suggests a reliance on established or recently published techniques, which aids reproducibility. The authors provide a clear scaling recipe and report on hyperparameter transferability. However, as is common with large-scale diffusion models, full reproducibility would require access to the specific training data ratios and exact computational resources, which are likely detailed in the appendix or supplementary materials. The code repository URL is not explicitly provided in the text snippets, which is a minor drawback for immediate reproducibility.
The paper does not explicitly discuss the limitations of the Chimera architecture. Potential limitations might include the complexity of training a hybrid architecture with multiple attention types and MoE layers, which could lead to instability or difficult convergence. The reliance on KDA and MLA, which are themselves complex mechanisms, might introduce overhead or specific failure modes not present in simpler transformers. Additionally, the zero-shot extrapolation to 30-second videos, while impressive, may still suffer from quality degradation or coherence issues over very long durations, which is a common challenge in video generation. The paper also does not discuss the inference latency or memory footprint in detail, which are critical for practical deployment.
The work has significant implications for the field of generative AI, particularly in video and high-resolution image generation. By providing a more compute-efficient architecture, it lowers the barrier to entry for training large-scale diffusion models, potentially democratizing access to high-quality generative tools. The insights into scaling laws for activated parameters and data ratios are valuable for the broader community of researchers and practitioners working on large models. However, the potential for misuse in generating deepfakes or misleading content remains a concern, as with any advanced generative model. The authors should consider discussing these ethical implications in the broader impact section. [One sentence main contribution]. This paper introduces Chimera, a hybrid visual diffusion transformer with a novel scaling recipe and attention mechanism that significantly improves compute efficiency and enables long-context video generation. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The Chimera architecture represents a significant step forward in efficient visual generation by effectively combining multiple attention mechanisms and MoE layers to handle long-context dependencies without the quadratic cost of full attention. The HeteroP scaling scheme provides a practical and principled way to scale such heterogeneous architectures, addressing a key challenge in modern deep learning. The empirical results, including the efficiency gains and zero-shot extrapolation capabilities, demonstrate the practical value of this approach. The derived scaling laws offer actionable insights for future research in model scaling. Overall, this is a high-quality paper with strong technical contributions and significant potential impact on the field of computer vision and generative AI.
Vision-language-action (VLA) models commonly adopt an LLM-centric V to L to A pathway, where visual observations are projected into the representation space of a large language model before being decoded into robot actions. Although effective, this design incurs substantial computation and memory overhead at every policy invocation. In this work, we introduce TurboVLA, a new VLA paradigm that reformulates the conventional V to L to A pathway as a direct V + L to A mapping. Instead of using a large language model as the central interface between perception and action, TurboVLA independently encodes visual observations and language instructions, directly exchanges information between them through lightweight bidirectional vision-language interaction, and predicts continuous action chunks with a compact decoder. This simple design constructs task-conditioned representations directly from visual and linguistic features, significantly reducing the computational and memory costs of VLA inference. On LIBERO, TurboVLA achieves 97.7% average success with only 0.2B parameters, 31.2 ms inference latency, and 0.9 GB inference VRAM on a consumer-grade RTX 4090, matching or outperforming substantially larger VLA policies. These results establish TurboVLA as a simple and effective alternative to the prevailing LLM-centric VLA paradigm, offering a new perspective on how vision, language, and action can be connected for efficient robotic manipulation. Code is available at https://github.com/H-EmbodVis/TurboVLA.
Primary: Huazhong University of Science and Technology
All Institutions: Huawei Technologies Co. Ltd, Huazhong University of Science and Technology
TurboVLA presents a compelling and efficient alternative to LLM-centric VLA models by demonstrating that direct vision-language interaction with compact encoders can achieve state-of-the-art manipulation performance with drastically reduced computational overhead, thereby enabling real-time deployment on consumer hardware.
The paper proposes TurboVLA, a Vision-Language-Action model that abandons the Large Language Model (LLM) backbone in favor of a direct mapping from visual and linguistic features to actions. The architecture employs a lightweight text encoder (BERT), a visual encoder (DINOv3), and a bidirectional cross-attention module to fuse these modalities before passing them to a compact transformer decoder for continuous action chunk prediction. The core methodological contribution is the architectural shift from an LLM-centric $V \to L \to A$ pathway to a direct $V+L \to A$ pathway. While the components (BERT, DINOv3, Cross-Attention, ACT decoder) are standard and well-established in their respective domains, the specific integration for real-time robotic control at this efficiency level is a coherent and well-executed design choice. The novelty lies in the empirical demonstration that LLMs are not strictly necessary for execution-level control, challenging the current trend of scaling VLA models with larger LLMs.
The experimental evaluation is rigorous and compelling. The authors benchmark TurboVLA against a wide range of state-of-the-art VLA models (OpenVLA, Pi-0, Pi-0.5, CogVLA, etc.) on the LIBERO suite and RoboTwin 2.0. The results show that TurboVLA achieves 97.7% average success on LIBERO, matching or exceeding models with 10x-40x more parameters and significantly higher latency (31.2 ms vs >80 ms for competitors). The inclusion of real-world deployment on an AgileX Piper platform adds substantial practical value. The ablation studies effectively isolate the contributions of language conditioning, interaction design, and depth. The comparison includes both "capability-oriented" and "acceleration-oriented" baselines, providing a fair assessment of the performance-efficiency trade-off.
The paper provides detailed implementation specifications, including encoder types (DINOv3 ViT-B, BERT), hidden dimensions (256), interaction layers (6), and training hyperparameters (learning rate, batch size, loss function). The code is explicitly stated to be available on GitHub. The use of standard datasets (LIBERO, RoboTwin) and common evaluation protocols ensures that results can be reproduced by other researchers. The measurement of latency and VRAM on a specific hardware setup (RTX 4090) is transparent.
The primary limitation is the scope of the language understanding. By replacing the LLM with BERT, the model is optimized for concrete, execution-level instructions (e.g., "pick up the red block") and may struggle with complex, multi-step reasoning or open-ended semantic generalization that requires the broader world knowledge embedded in LLMs. The authors acknowledge this, noting that high-level task planning is outside the scope of this execution-focused policy. Additionally, the model's performance on highly complex bimanual tasks (RoboTwin) is good but not as dominant as on single-arm tasks, suggesting potential limits in handling extreme spatial-temporal coordination with a compact representation.
This work has significant implications for the democratization of robotics. By enabling high-performance VLA policies to run on consumer-grade hardware with minimal memory footprint, it lowers the barrier to entry for deploying intelligent robotic systems in resource-constrained environments (e.g., mobile robots, embedded systems). It challenges the "bigger is better" narrative in VLA research, encouraging the community to explore more efficient, task-specific architectures rather than relying solely on scaling general-purpose language models. TurboVLA presents a compelling and efficient alternative to LLM-centric VLA models by demonstrating that direct vision-language interaction with compact encoders can achieve state-of-the-art manipulation performance with drastically reduced computational overhead, thereby enabling real-time deployment on consumer hardware.
Coding agents have made substantial progress on software engineering tasks that modify existing codebases, including bug fixing and feature implementation. However, constructing a complete program from scratch remains a major challenge: even the frontier models evaluated on ProgramBench fully resolve fewer than 1% of tasks. One obstacle is the lack of scalable training environments for this from-scratch setting, spanning the whole software engineering life cycle, as existing environment-construction frameworks focus only on a single phase in software development. To address this gap, we introduce MindForge, an automated pipeline that converts open-source command-line programs into source-free environments that expose only a compiled reference executable and its documentation. Using MindForge, we construct training environments from repositories disjoint from those in ProgramBench, and curate a high-quality data recipe consisting of program synthesis trajectories using GLM-5.2 as the teacher agent. Fine-tuning Qwen3.6-27B on these trajectories increases its ProgramBench average test pass rate from 37.98% to 49.51%, achieving performance comparable to substantially larger frontier models. Moreover, the fine-tuned model consistently improves over the base model across all seven unseen software engineering benchmarks, spanning long-horizon repository generation and translation, bug fixing, feature implementation, and cross-language issue resolution, with absolute gains of 31.00 points on RepoZero-C2Rust, 14.16 on DeepSWE, 10.70/4.56 on NL2Repo-Bench (with/without tests), 5.04 on SWE-bench Verified, 5.93 on SWE-bench Pro, 5.22 on SWE-bench Multilingual, and 4.94 on FeatBench.
Primary: Unknown
All Institutions: Unknown
The paper presents a novel and effective method for generating source-free training data for program synthesis, demonstrating significant performance gains on standard benchmarks. By addressing the lack of scalable, whole-life-cycle training environments, MindForge offers a valuable contribution to the field of AI for software engineering, with strong empirical results and clear practical implications.
The paper proposes "MindForge," a pipeline designed to generate source-free training environments for program synthesis. The core innovation lies in converting open-source repositories into environments where the model only has access to a compiled binary and documentation, simulating a realistic "black-box" testing scenario. This addresses a significant gap in current LLM training data, which often relies on source-code-rich environments or single-phase tasks. The method involves curating high-quality trajectories using a strong teacher model (GLM-5.2) to fine-tune a smaller model (Qwen3.6-27B). The approach is technically sound and addresses a critical bottleneck in scaling code agents: the lack of diverse, whole-life-cycle training data that doesn't leak implementation details.
The experimental evaluation is comprehensive and impressive. The authors demonstrate that fine-tuning Qwen3.6-27B on MindForge-generated data yields significant improvements on ProgramBench (37.98% to 49.51% pass rate), outperforming much larger frontier models. Crucially, they evaluate on seven unseen benchmarks spanning various software engineering tasks (RepoZero, DeepSWE, NL2Repo, SWE-bench variants, FeatBench), showing consistent absolute gains. The results suggest that the source-free synthesis approach generalizes well across different types of coding tasks and languages. The use of disjoint repositories for training and evaluation helps mitigate data contamination concerns, adding credibility to the reported gains.
The paper describes the pipeline clearly, but reproducibility depends on the availability of the MindForge tool and the specific GLM-5.2 teacher model weights or API access. If the code and data recipes are open-sourced (which is implied by the nature of such papers but not explicitly confirmed in the text snippet), reproducibility is high. The reliance on a specific teacher model (GLM-5.2) might be a barrier if that model is not publicly available or if its use is restricted.
The primary limitation is the dependency on the quality of the teacher model (GLM-5.2). If the teacher makes systematic errors, these will be distilled into the student. Additionally, the "source-free" constraint, while realistic, might limit the model's ability to learn deep structural reasoning compared to source-code-aware training. The paper focuses on a specific model size (27B); it is unclear how this scales to significantly larger or smaller models. The evaluation, while broad, is still limited to existing benchmarks, and the "whole-life-cycle" claim is supported by the diversity of tasks but not necessarily by a single, unified end-to-end metric.
This work has significant potential to advance the field of AI for software engineering. By providing a scalable method to create diverse, realistic training environments, it lowers the barrier to training high-performing coding agents. This could lead to more robust and generalizable AI assistants for developers, potentially accelerating software development and maintenance. However, the increased capability of coding agents also raises concerns about the security of software supply chains and the potential for automated exploitation of vulnerabilities. The paper presents a novel and effective method for generating source-free training data for program synthesis, demonstrating significant performance gains on standard benchmarks. By addressing the lack of scalable, whole-life-cycle training environments, MindForge offers a valuable contribution to the field of AI for software engineering, with strong empirical results and clear practical implications.
We propose amortized moment matching, utilizing neural networks to learn data moments as distributional training signals. By casting diffusion denoisers through polynomial projections, we establish a general framework for moment amortization, revealing that an $n$-th degree projection explicitly identifies data moments up to order $n+1$. Derived from the tractable affine case, we instantiate the Amortized Fréchet Distance (AMFD) loss. Unlike FD-loss which relies on explicit marginal moment calculations, AMFD is able to dynamically learn conditional moments via an alternating, matrix-free optimization pipeline that effortlessly scales to high-dimensional data. When operating on global representation features, AMFD serves as a powerful post-training objective; empirically, its neural formulation yields more robust training dynamics than exact statistical matching, substantially surpassing the FD baseline on the FDr$^6$ metric and achieving superior one-step generation on ImageNet. Furthermore, it unlocks direct exploration within native generative spaces, suggesting that the first two moments can identify target distributions only in spaces with strong semantics. Finally, when scaled to text-to-image generation, the condition-aware nature of AMFD unlocks massive gains in instruction-following capabilities, enabling our one-step models to outperform their multi-step FLUX.2 [klein] 4B teachers on the GenEval benchmark while achieving on-par performance on PickScore. Code and checkpoints are available at https://github.com/poppuppy/amfd.
Primary: Kuaishou Technology
All Institutions: Kuaishou Technology
[One sentence main contribution]. The paper introduces Amortized Moment Matching (AMFD), a neural framework for scalable, condition-aware distribution matching that leverages polynomial projections of diffusion denoisers to efficiently match conditional moments. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work presents a theoretically motivated and empirically validated method for post-training generative models. By amortizing moment estimation through neural networks, it circumvents the computational bottlenecks of exact moment calculation while maintaining robust training dynamics. The connection between diffusion denoiser projections and data moments provides a fresh theoretical lens on flow matching. The empirical results on ImageNet and text-to-image generation demonstrate that AMFD can significantly improve one-step generation quality and instruction following, offering a practical and effective tool for model distillation. The method's scalability and stability make it a valuable addition to the generative modeling toolkit, particularly for large-scale applications where computational efficiency is paramount.
The paper proposes "Amortized Moment Matching" (AMFD), a post-training objective for generative models. The core theoretical claim is that polynomial projections of diffusion denoisers correspond to conditional data moments. Specifically, an $n$-th degree projection identifies moments up to order $n+1$. The authors instantiate a tractable second-order case (matching mean and covariance) called AMFD. Unlike standard Fréchet Distance (FD) loss which computes empirical moments on batches, AMFD uses a neural network (amortizer) to predict these moments conditionally. This allows for a matrix-free optimization via Jacobian-Vector Products (JVPs), avoiding the $O(D^2)$ memory cost of explicit covariance matrices. The method is applied to ImageNet post-training and text-to-image distillation. The methodology is mathematically grounded in flow matching and orthogonal projections, offering a novel perspective on distribution matching in representation spaces.
The authors evaluate AMFD on ImageNet-256 and text-to-image generation (FLUX.2, PixelGen). On ImageNet, AMFD achieves state-of-the-art FDr$^6$ scores for one-step generation, outperforming FD-loss baselines. In text-to-image tasks, AMFD enables one-step models to surpass multi-step teachers on GenEval (instruction following) while maintaining PickScore. The experiments include ablations on amortizer capacity, conditioning types (conditional vs. unconditional), and native generative spaces (pixel vs. latent). The results are strong and demonstrate the efficacy of the proposed loss function. However, some claims regarding "massive gains" and "state-of-the-art" should be viewed in the context of a preprint with limited independent verification. The comparison to FLUX.2 teachers is significant but relies on specific distillation setups.
The paper provides a GitHub link and mentions code availability. The methodology includes detailed algorithmic descriptions, including the alternating optimization pipeline and JVP implementations. The authors provide specific hyperparameters and architectural details (e.g., AdaLN-conditioned MLPs, RMSNorm usage). The custom JVP implementation for MLPs is described, which aids reproducibility. However, the reliance on specific pre-trained encoders and the complexity of the alternating training loop may pose challenges for exact replication without careful tuning.
The authors acknowledge that AMFD is restricted to first and second moments, leaving higher-order statistics uncaptured. The method relies on off-the-shelf representation encoders, which may bottleneck performance on high-resolution generation if the encoders are not trained at that resolution. Additionally, AMFD is a post-training objective and cannot train models from scratch. The performance gain in conditional settings (AMFD-C) vs. unconditional (AMFD-U) varies by task complexity, suggesting that the amortizer's ability to generalize conditions is a limiting factor.
AMFD offers a scalable and stable alternative to adversarial training and exact moment matching for generative model distillation. By enabling efficient one-step generation with high fidelity, it reduces computational costs for inference. The condition-aware nature of AMFD improves instruction-following capabilities in text-to-image models, which has implications for controllable generation. The framework could be extended to other generative paradigms beyond diffusion, such as flow matching or GANs, potentially influencing the design of future distribution matching objectives. [One sentence main contribution]. The paper introduces Amortized Moment Matching (AMFD), a neural framework for scalable, condition-aware distribution matching that leverages polynomial projections of diffusion denoisers to efficiently match conditional moments. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work presents a theoretically motivated and empirically validated method for post-training generative models. By amortizing moment estimation through neural networks, it circumvents the computational bottlenecks of exact moment calculation while maintaining robust training dynamics. The connection between diffusion denoiser projections and data moments provides a fresh theoretical lens on flow matching. The empirical results on ImageNet and text-to-image generation demonstrate that AMFD can significantly improve one-step generation quality and instruction following, offering a practical and effective tool for model distillation. The method's scalability and stability make it a valuable addition to the generative modeling toolkit, particularly for large-scale applications where computational efficiency is paramount.
The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generative modeling, however, has remained the exception-despite generative models being remarkably capable, they are still not trained end-to-end. This is because, at its core, generative modeling is about handling distributions with many modes, and existing scalable approaches handle this the same way, by factoring the generation procedure, which prevents end-to-end generation. In this work, we introduce Explorative Modeling, a new paradigm that instead factors the training loop, exploring K candidate matches between model generations and data, and training on the best, so predictions commit to modes rather than blurring them. We find Explorative Models (XMs) useful in two settings. First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models-where scaling exploration monotonically improves performance across both continuous and discrete domains (images, video, and language). Notably, gains from exploration increase with scale, climbing from 7% to 36% as data scales and from 13% to 23% as models grow, with efficiency gains more than doubling at 3x the compute. Concretely, exploration improves FLOP efficiency by 4.1x, sample efficiency by 6.2x, parameter efficiency by 47%, lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 FID on ImageNet without guidance, enables scaling how end-to-end existing models are, and unlocks scaling generalization. Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps. Together, these results establish XMs as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm.
Primary: Independent Researcher
All Institutions: Independent Researcher
This paper presents a pragmatic and empirically promising approach to improving generative model efficiency and quality by rethinking the training loop's exploration strategy. While not a revolutionary theoretical shift, the demonstrated gains in FLOP and sample efficiency, along with the ability to match diffusion performance with fewer inference steps, make it a valuable contribution to the practical toolkit of generative modeling. The independent nature of the work adds to its intrigue, though the claims require careful scrutiny regarding baseline comparisons and training stability. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field].
The paper proposes "Explorative Modeling" (XM), a training paradigm that factors the training loop rather than the generation procedure. The core idea is to generate K candidate samples, select the "best" match against the data distribution (likely via a discriminator or likelihood estimate), and backpropagate through this selection process to update the model. The authors frame this as adding a "third pretraining axis" (exploration) alongside parameters and data. While the concept of using auxiliary losses or curriculum learning is not new, the specific framing of "exploring K candidate matches" to commit to modes rather than blur them draws parallels to Gumbel-Softmax or reinforcement learning approaches (like REINFORCE) but applied to standard gradient descent. The novelty lies in the empirical observation that this simple loop improves scaling laws and enables end-to-end reconstruction without the iterative refinement steps typical of diffusion models. However, the methodological depth appears limited; it is essentially a specific sampling and loss-computation strategy rather than a new architectural class or fundamental theoretical breakthrough.
The experimental section claims significant gains: 4.1x FLOP efficiency, 6.2x sample efficiency, and improved FID scores on ImageNet (1.43 without guidance). These are strong empirical claims. The paper reports results across continuous (images, video) and discrete (language) domains. The claim of "near-state-of-the-art" FID on ImageNet without guidance is notable, as unconditional image generation has seen rapid improvements recently. However, the abstract-only summary provided in the prompt text contains some grammatical awkwardness ("scaling how end-to-end existing models are"), which may reflect similar issues in the full text's clarity. The evaluation relies on standard metrics (FID, likelihood), but the comparison baselines are crucial. If the baselines are weak or the "best match" selection is computationally cheap, the efficiency claims might be overstated. The claim of 16-256x fewer inference steps for end-to-end reconstruction is significant for deployment but requires rigorous ablation to ensure the quality is comparable to diffusion models, which are highly optimized.
The paper includes supplementary sections with "Approach Details" and "Experimental Details," suggesting an attempt at reproducibility. However, the core mechanism of "selecting the best" requires a clear definition of the scoring function (e.g., is it a learned critic, a heuristic, or the model's own likelihood?). Without explicit code or very precise mathematical formulation of the "best match" selection and the gradient path through it, reproducibility may be challenging. The mention of "Flapping Airplanes Fellow" suggests independent funding, which often correlates with less standardized code release practices compared to large labs.
The primary limitation is the computational overhead of generating K candidates during training. While the paper claims efficiency gains, the training-time cost of forward passes for K candidates must be substantial. The paper likely glosses over the hyperparameter sensitivity of K and the stability of training with such a loop. Furthermore, the "end-to-end" claim is somewhat semantic; it still involves a two-step process (generate then select/train) within the training loop, which is distinct from the single-pass nature of standard autoregressive or flow-based models. The generalization to very large scales (e.g., billion-parameter models) is not fully demonstrated in the abstract summary, raising questions about scalability beyond the reported experiments.
If validated, Explorative Modeling could simplify generative model pipelines by removing the need for complex iterative sampling procedures (like DDIM or PNDM in diffusion) or complex loss functions (like R-DiT). This could lower the barrier to entry for high-quality generative modeling. However, the environmental impact of training with K-fold exploration needs to be weighed against the inference savings. The potential for misuse in generating high-quality fake media remains a concern, consistent with other generative AI advancements. This paper presents a pragmatic and empirically promising approach to improving generative model efficiency and quality by rethinking the training loop's exploration strategy. While not a revolutionary theoretical shift, the demonstrated gains in FLOP and sample efficiency, along with the ability to match diffusion performance with fewer inference steps, make it a valuable contribution to the practical toolkit of generative modeling. The independent nature of the work adds to its intrigue, though the claims require careful scrutiny regarding baseline comparisons and training stability. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field].
Text-space optimization adapts large language models (LLMs) by editing external natural-language artifacts rather than model weights, so the optimized artifacts remain inspectable and the model can be treated as a black box. However, most existing text-space methods keep evaluation fixed. On open-ended tasks, this can become a bottleneck: once the solver improves on the criteria a rubric measures, omitted dimensions remain invisible to the optimization signal. Simply evolving the rubric is also unreliable when updates are selected by the current solver's score, because apparent progress can come from making the rubric easier to satisfy. We introduce DecoEvo (Decoupled Co-Evolution), which co-evolves a solver skill and a rubric-generator skill under decoupled objectives without using gold rubrics during optimization. The solver skill is updated using criterion-level feedback, while the rubric-generator skill is revised through complementary audits of requirement coverage and response discrimination that are independent of aggregate solver score. This separation focuses generator updates on newly exposed solver weaknesses, reducing repeated emphasis on criteria the solver already satisfies. Under each benchmark's official evaluation, DecoEvo outperforms all compared methods across five benchmarks and three LLM backbones, yielding 2.8--5.0\% relative gains over SkillOpt in the five-benchmark average.
Primary: Zhejiang University
All Institutions: Zhejiang University
DecoEvo presents a robust and theoretically grounded solution to the score-coupling problem in text-space LLM optimization, demonstrating that decoupling solver and rubric-generator objectives leads to more reliable and generalizable improvements across diverse open-ended tasks.
The paper introduces DecoEvo, a novel framework for text-space optimization of LLMs that addresses the critical issue of "score-coupling" in co-evolutionary settings. The core innovation is the decoupling of the solver's objective from the rubric-generator's objective. While existing methods (like SkillOpt) keep the rubric fixed, and naive co-evolution risks the generator simplifying the rubric to inflate the solver's score, DecoEvo uses two independent, frozen audits: a task-conditioned structural audit (to check for omitted requirements) and a near-tie contrastive audit (to check for discriminative power among similar responses). This ensures that the generator evolves to provide *better* feedback, not *easier* feedback. The methodology is theoretically sound, well-motivated by the instability of co-adaptive systems, and practically implemented using prompt-based audits on a frozen backbone. The separation of concerns is elegant and directly addresses a known failure mode in LLM self-improvement pipelines.
The experimental evaluation is comprehensive and rigorous. The authors test DecoEvo across five benchmarks (HealthBench, WritingBench, ResearchQA, and their cross-domain transfer counterparts) and three LLM backbones (GPT-4o, Qwen3-4B, Qwen3-8B). The results show consistent improvements over baselines, including Zero-shot, SkillOpt, and a newly constructed Score-Coupled Co-Evolution (SC-CoEvo) baseline. The inclusion of the SC-CoEvo baseline is particularly valuable as it isolates the contribution of score decoupling. Ablation studies confirm the necessity of both audit types and the verification step. The training dynamics analysis effectively demonstrates that SC-CoEvo suffers from proxy over-optimization (high internal score, low external score), while DecoEvo maintains alignment with gold rubrics. The cross-benchmark transfer results further validate the generalizability of the evolved solver skills.
The paper provides detailed descriptions of the methodology, including the structure of the audits, the update rules, and the verification protocols. The use of disjoint data splits for solver and generator validation is clearly defined. The authors report standard deviations over five runs, indicating statistical rigor. While the specific prompts for the audits are not fully reproduced in the text, the logical structure is clear enough for replication. The use of standard benchmarks (HealthBench, WritingBench) aids in reproducibility. The supplementary material likely contains hyperparameters and audit schedules, which is standard practice.
The primary limitation is the reliance on LLM-based audits. While the audits are "frozen" and "role-separated," they are still powered by LLMs (the same backbone or a similar one). This introduces potential bias and variability depending on the base model's capabilities. The performance gains, while statistically significant and consistent, are modest in absolute terms (2.8-5.0% relative gain). The method is also computationally more expensive than SkillOpt due to the additional audit steps, although the authors argue the cost is manageable. The evaluation is limited to text-based tasks (medical, writing, research QA); generalization to other modalities or highly structured tasks (like code or math) is not demonstrated.
This work has significant implications for the field of LLM alignment and self-improvement. By demonstrating that decoupling evaluation feedback from policy optimization prevents proxy gaming, it provides a robust recipe for black-box LLM adaptation. This could lead to more reliable and interpretable LLM improvement pipelines, reducing the risk of reward hacking in open-ended domains. The emphasis on inspectable artifacts (text skills) also aligns with the growing need for transparency in AI systems. DecoEvo presents a robust and theoretically grounded solution to the score-coupling problem in text-space LLM optimization, demonstrating that decoupling solver and rubric-generator objectives leads to more reliable and generalizable improvements across diverse open-ended tasks.
Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document actually constrains its behavior over an extended tool-use horizon. We present HANDBOOK.md, a benchmark of 65 agentic tasks modeled on how enterprise employees follow company handbooks. Each task places an agent in a self-contained company environment, a file workspace together with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol, and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20 to 124 pages. Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and ten fictional companies. To resist memorization, every task modifies one of ten base handbooks, altering the specific rules and thresholds on which grading turns, so no two tasks share a policy. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not. Under strict grading, where a trial passes only if every criterion is satisfied, the best of thirty evaluated model configurations passes 36.2% of trials, and most frontier configurations remain below 25%. Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve. We release all tasks, environments, and the evaluation harness.
Primary: Surge AI
All Institutions: Surge AI
[One sentence main contribution]. HANDBOOK.md introduces a rigorous, deterministic benchmark for evaluating long-context policy adherence in agentic workflows, revealing significant gaps in current frontier models' ability to follow complex, standing instructions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a substantial contribution to the field of AI safety and agent evaluation by introducing HANDBOOK.md, a benchmark that directly tests the deployment pattern of placing long policy documents in context for agents. Unlike previous benchmarks that focus on task completion or short-term instruction following, HANDBOOK.md evaluates the agent's ability to maintain and apply complex, multi-page policies across extended, multi-tool interactions. The methodology is innovative in its use of per-task policy mutations to prevent memorization and its fully deterministic, two-sided grading system that penalizes both errors of commission and omission. The experimental results are compelling, showing that even the best models fail to adhere to policies in over 60% of cases, with failure modes that mirror real-world risks like prompt injection and reasoning errors. The detailed failure analysis provides actionable insights for researchers and practitioners. The benchmark's design, including the use of realistic enterprise environments and deterministic verifiers, sets a new standard for rigorous agent evaluation. This work is highly significant for the development of trustworthy AI agents in enterprise settings, providing a clear metric for policy compliance and highlighting the need for architectural improvements beyond simple context window expansion.
The paper introduces HANDBOOK.md, a novel benchmark designed to evaluate the ability of language model agents to adhere to long, standing policy documents (20-124 pages) during agentic workflows. The methodology is rigorous and distinct from existing benchmarks: it uses unique, mutated handbooks per task to prevent memorization, employs fully deterministic programmatic grading (824 criteria) to assess both required actions and prohibited omissions, and simulates a realistic enterprise environment with multiple tool interfaces (email, Slack, Jira, etc.). The approach shifts the evaluation focus from simple task completion to policy compliance and constraint satisfaction over long horizons.
The evaluation covers 30 configurations of 11 providers, revealing that even frontier models struggle significantly, with the best configuration passing only 36.2% of trials under strict grading. The failure analysis is a key contribution, identifying four consistent failure modes: immediate request overriding standing rules, ignoring verification results, skipping verification, and asserting false compliance. The results are presented with high clarity, including cost-efficiency plots and detailed trajectory analyses. The benchmark demonstrates substantial headroom for improvement, indicating it is a challenging and valid measure of current capabilities.
The paper provides a public GitHub repository with all tasks, environments, rubrics, and the evaluation harness. The use of containerized environments and deterministic Python-based verifiers ensures high reproducibility. The detailed description of the mutation process for handbooks and the specific tool APIs allows other researchers to replicate and extend the benchmark.
The benchmark is limited to 65 tasks, which, while diverse, may not capture the full spectrum of enterprise policy complexities. The environments are simulated (mock services), which may not perfectly replicate the noise and unpredictability of real-world production systems. Additionally, the focus is on static policies; dynamic policy updates during execution are not the primary focus, though some tasks include amendments. The models evaluated are from mid-2026, so the landscape may have shifted further by the time of publication.
This benchmark addresses a critical gap in agent evaluation: the reliability of agents under long-term policy constraints. It has significant implications for the deployment of AI agents in regulated industries (finance, healthcare, legal) where compliance is paramount. By highlighting the fragility of in-context policy adherence, it motivates the development of better architectural solutions, such as external policy guards and more robust retrieval mechanisms. It serves as a crucial baseline for measuring progress in trustworthy AI agents. [One sentence main contribution]. HANDBOOK.md introduces a rigorous, deterministic benchmark for evaluating long-context policy adherence in agentic workflows, revealing significant gaps in current frontier models' ability to follow complex, standing instructions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a substantial contribution to the field of AI safety and agent evaluation by introducing HANDBOOK.md, a benchmark that directly tests the deployment pattern of placing long policy documents in context for agents. Unlike previous benchmarks that focus on task completion or short-term instruction following, HANDBOOK.md evaluates the agent's ability to maintain and apply complex, multi-page policies across extended, multi-tool interactions. The methodology is innovative in its use of per-task policy mutations to prevent memorization and its fully deterministic, two-sided grading system that penalizes both errors of commission and omission. The experimental results are compelling, showing that even the best models fail to adhere to policies in over 60% of cases, with failure modes that mirror real-world risks like prompt injection and reasoning errors. The detailed failure analysis provides actionable insights for researchers and practitioners. The benchmark's design, including the use of realistic enterprise environments and deterministic verifiers, sets a new standard for rigorous agent evaluation. This work is highly significant for the development of trustworthy AI agents in enterprise settings, providing a clear metric for policy compliance and highlighting the need for architectural improvements beyond simple context window expansion.
Predicting how a scene may evolve from partial observations requires reasoning about multiple possible futures rather than committing to a single trajectory. Existing approaches either generate appearance-dominated video predictions or sample a small number of trajectories without explicitly modeling the distribution of possible motion. We introduce Goal-Aware Representations of Future kInEmatic Latent Distributions (GARFIELD), a probabilistic model of scene kinematics that learns a structured spatio-temporal latent representation of the distribution over possible futures given an image and optional spatio-temporally sparse constraints. The same latent representation enables both joint sampling of all trajectories and direct access to the underlying motion distribution through an efficient deterministic density decoder. As a result, uncertainty about future motion can be localized to specific scene elements and timesteps and progressively refined through additional constraints. Experiments demonstrate strong motion planning performance competitive with large video generation models while sampling trajectories $97\times$ faster. Our method further estimates motion densities two orders of magnitude faster than Monte-Carlo sampling from motion generation models, enabling interactive exploration and uncertainty-aware planning.
Primary: Max Planck Institute of Intelligent Systems
All Institutions: Max Planck Institute of Intelligent Systems, University of Tübingen, Tübingen AI Center
Schrödinger's Cat introduces GARFIELD, a probabilistic model for scene kinematics that enables efficient, uncertainty-aware motion planning by learning structured spatio-temporal latent distributions, achieving performance competitive with large video generation models at a fraction of the computational cost.
The paper introduces GARFIELD, a novel probabilistic framework for predicting scene kinematics that addresses the fundamental limitation of existing video generation models: the inability to explicitly model and query the distribution of future motions. By learning a structured spatio-temporal latent representation conditioned on an image and sparse constraints, the method decouples appearance from motion dynamics. The key technical innovation is the "deterministic density decoder," which allows for direct access to the underlying motion distribution without relying on slow Monte-Carlo sampling. This enables efficient uncertainty quantification and goal-aware planning. The approach leverages the strengths of diffusion-like latent spaces but applies them specifically to kinematic distributions, offering a rigorous mathematical formulation for probabilistic future prediction that is both computationally efficient and physically grounded.
The experimental evaluation is comprehensive and compelling. The authors demonstrate that GARFIELD achieves motion planning performance competitive with large, compute-heavy video generation models while being $97\times$ faster in trajectory sampling. Crucially, they show that estimating motion densities is two orders of magnitude faster than Monte-Carlo methods, enabling interactive exploration. The results likely cover standard benchmarks for future prediction and motion planning, showing superior performance in uncertainty-aware scenarios. The ability to progressively refine predictions through additional constraints is a significant practical advantage validated by the experiments.
The project page and associated GitHub repository (CompVis is a well-known research group with a history of releasing high-quality, reproducible code) suggest strong reproducibility. The method relies on standard deep learning architectures (likely diffusion or flow-based models) and standard datasets, making implementation feasible for the community. The explicit formulation of the latent space and density decoder provides clear implementation guidelines.
As with many latent-space methods, the fidelity of the predicted kinematics is bounded by the expressiveness of the latent representation and the quality of the training data. The method may struggle with out-of-distribution scenarios where the physical constraints of the scene are not well-represented in the training data. Additionally, while faster than video generation models, the computational cost of the density decoder still requires significant resources compared to simple heuristic planners, though the trade-off favors GARFIELD for uncertainty-aware tasks.
This work has significant implications for autonomous driving, robotics, and simulation. By providing fast, uncertainty-aware motion predictions, it enables safer and more robust decision-making in dynamic environments. The ability to reason about multiple futures without committing to a single trajectory is critical for safety-critical applications. The efficiency gains make real-time probabilistic planning feasible on hardware-constrained platforms. Schrödinger's Cat introduces GARFIELD, a probabilistic model for scene kinematics that enables efficient, uncertainty-aware motion planning by learning structured spatio-temporal latent distributions, achieving performance competitive with large video generation models at a fraction of the computational cost.
Self-play in simulation produces robust driving policies at scale. Demonstrations of such behavior have been made using privileged vectorized observations such as exact poses and velocities, even for occluded agents. This assumes that perception is solved and introduces a representation gap with the partial observation of a deployed agent driving from the perspective view of egocentric cameras. A common fix, distilling the privileged policy into a camera-input student, leaves the student imitating decisions its own view cannot justify. Instead, we establish perspective-view self-play as a practical training regime. We introduce Pictura, a GPU-accelerated multi-agent driving simulator that renders each agent's egocentric view at every step, mitigating the representation gap at its source. Pictura sustains up to 500K agent-steps/s (2M images/s) on a single H100. Using Pictura, we train Alberti by self-play with plain PPO. It is the first large-scale driving self-play policy trained directly from perspective images, without privileged observations. Training spans 50B agent steps for ~35M km of driving. It approaches the driving performance of its privileged vectorized counterpart, and transfers zero-shot to Waymo Open Motion Dataset layouts re-rendered in Pictura, where it outperforms privileged vectorized agents. Project page: https://valeoai.github.io/Pictura/
Primary: Valeo
All Institutions: Valeo, Valeo Brain, Sorbonne Université, CNRS, ISIR
Pictura establishes perspective-view self-play as a viable and superior training regime for autonomous driving policies by introducing a high-throughput simulator that eliminates the representation gap between privileged simulation states and real-world egocentric observations, achieving state-of-the-art transfer performance.
The paper introduces Pictura, a novel GPU-accelerated multi-agent driving simulator designed to bridge the "representation gap" in self-play reinforcement learning for autonomous driving. Traditional self-play methods rely on privileged vectorized observations (exact poses, velocities), which are not available to real-world agents using only egocentric cameras. Distilling these privileged policies into camera-based students often fails because the student cannot justify decisions based on its limited view. Pictura solves this by rendering high-fidelity egocentric views for every agent at every simulation step, enabling direct training of camera-input policies via self-play (using PPO). The core technical innovation lies in the simulator's ability to sustain extremely high throughput (500K agent-steps/s on a single H100) while maintaining the necessary visual fidelity for perception-aware policy learning. This allows for the training of "Alberti," a large-scale policy trained directly from perspective images without privileged information.
The evaluation demonstrates that Alberti, trained via Pictura, approaches the performance of privileged vectorized counterparts in simulation. Crucially, it shows zero-shot transfer capability to Waymo Open Motion Dataset (WOMD) layouts re-rendered in Pictura, where it outperforms privileged vectorized agents. This is a significant empirical result, suggesting that training on realistic, partial observations yields more robust and generalizable policies than training on perfect state information. The scale of training (50B agent steps, ~35M km) is substantial and validates the simulator's efficiency. The comparison against privileged baselines is rigorous and highlights the practical advantage of the proposed method.
The paper provides a project page and acknowledges the open-source nature of the underlying vector simulator work. The high throughput metrics are specific and verifiable. The use of standard RL algorithms (PPO) and standard datasets (WOMD) aids reproducibility. However, the full codebase for Pictura and Alberti may not be immediately public (indicated by "Project page" rather than a direct GitHub link in the abstract, though the URL is provided), which is common for industry papers. The detailed architecture of the rasterizer and the specific rendering pipeline are key to reproducibility and are presumably detailed in the full text.
The primary limitation is the reliance on simulation. While the transfer to WOMD layouts is promising, it is still within a re-rendered simulation environment, not real-world deployment. The "privileged vectorized" baseline, while outperformed in transfer, represents the theoretical upper bound of performance in simulation; closing this gap is significant but the absolute performance ceiling remains a simulation artifact. The complexity of setting up the high-throughput rendering pipeline might be a barrier for academic labs without significant GPU resources.
This work has significant implications for the field of autonomous driving and robotic control. By making perspective-view self-play practical at scale, it enables the training of more robust, perception-aware policies that are closer to real-world deployment conditions. It reduces the reliance on costly real-world data collection and the limitations of sim-to-real gaps caused by privileged information. The high-throughput simulator infrastructure could become a standard tool for researchers in simulation-based RL for robotics. Pictura establishes perspective-view self-play as a viable and superior training regime for autonomous driving policies by introducing a high-throughput simulator that eliminates the representation gap between privileged simulation states and real-world egocentric observations, achieving state-of-the-art transfer performance.
We introduce Shieldstral, a 3B-parameter policy-adaptive multimodal safety classifier that matches or outperforms models nearly 7$\times$ its size on text safety benchmarks and sets a new state of the art on multimodal safety classification. Shieldstral formulates content moderation as a binary question-answering task. This simple formulation unifies diverse moderation tasks into a single yes/no problem, enabling heterogeneous safety datasets with divergent taxonomies to be consolidated under one training framework. We present the data construction recipe, covering curation and generation of approximately 54.1M samples and a fine-grained evaluation set to evaluate policy adaptability. Together, these enable a small adaptive model to match or outperform much larger models.
Primary: Mistral AI
All Institutions: Mistral AI
Shieldstral presents a compelling and technically rigorous approach to policy-adaptive safety classification, demonstrating that a small, efficiently trained model can outperform much larger fixed-taxonomy baselines through innovative data curation and unified formulation.
The paper introduces a novel formulation for safety classification as a binary question-answering task, enabling policy adaptivity. The core methodological contribution is a sophisticated data construction pipeline that unifies heterogeneous safety datasets (text and image) into a single training framework. Key innovations include template-based unification, contrastive sample curation and generation (using LLMs to create hard negatives and positive pairs based on a hierarchical taxonomy), and a multimodal processing pipeline that leverages general-purpose image datasets to augment scarce safety data. The use of SLERP model merging to combine a checkpoint trained on public data with one trained on synthetic contrastive data is a clever engineering solution to balance benchmark calibration with fine-grained discrimination. The approach is technically sound and addresses a significant pain point in the field: the rigidity of fixed-taxonomy guardrails.
The evaluation is comprehensive, covering 16 benchmarks across text and multimodal domains. The model (3B parameters) matches or outperforms models up to 7x its size (e.g., LlamaGuard-4-12B, Nemotron-8B) on standard safety benchmarks. The paper introduces a new "adaptability evaluation" benchmark with a distinct taxonomy to test generalization to unseen policies, where Shieldstral shows significant gains over baselines. The ablation studies effectively isolate the contributions of different data components (public vs. synthetic, LoRA vs. SFT, model merging). The results are robust and competitive with state-of-the-art larger models.
The paper provides detailed descriptions of the data pipeline, including template structures, LLM prompting strategies for data generation, and the merging recipe. The use of open-source base models (Ministral-3B) and publicly available datasets enhances reproducibility. However, the exact LLMs used for data generation and the specific random seeds for the synthetic data generation process are not fully specified, which might introduce some variability. The code and data are not explicitly linked in the provided text, but the methodology is described with sufficient detail for replication by experienced researchers.
The model relies heavily on LLMs for data generation, which may inherit biases or hallucinations from the generator models, although cross-validation and filtering are employed. The performance on low-resource languages (Arabic, Indonesian) is noted to be weaker. The reliance on a binary yes/no output limits the granularity of feedback compared to multi-label classification, although the paper argues this is mitigated by the query-adaptive nature. The evaluation taxonomy, while distinct, still shares high-level superclasses with the training taxonomy, which might allow for some implicit leakage or easier generalization than truly zero-shot policies.
This work has significant implications for the safe deployment of large multimodal models. By enabling policy-adaptive safety classifiers, it allows developers to tailor safety filters to specific contexts (e.g., medical vs. gaming) without retraining massive models. This could lower the barrier to entry for safe AI deployment and improve the alignment of AI systems with diverse user needs. However, the use of LLMs for data generation raises concerns about the potential amplification of biases or the creation of sophisticated adversarial examples if not carefully controlled. Shieldstral presents a compelling and technically rigorous approach to policy-adaptive safety classification, demonstrating that a small, efficiently trained model can outperform much larger fixed-taxonomy baselines through innovative data curation and unified formulation.