Week of August 23 – August 30, 2026
Automating alignment research may accelerate progress toward aligned AI, but whether it does is hard to measure. Luckily, many alignment failures, such as deception, sycophancy, and jailbreaks, are already measurable by public benchmarks. We study whether automated alignment researchers (AARs) can post-train to mitigate alignment failures by proposing training methods and data to simultaneously optimize multiple safety benchmarks, while preserving general capability. Across 10 alignment failures, the strongest AAR methods significantly reduce the targeted alignment failures and generalize to a held-out benchmark, multi-turn behavioral audits, and models up to 4.7 times larger than the target model. As a human baseline, 28 experienced researchers receive up to eight hours to develop methods for the same benchmarks, but their methods underperform the best AAR methods. Using human ideas as the AARs' initial research direction does not improve performance, suggesting current AARs may not need guidance from experienced researchers. These results suggest that automating alignment research on well-characterized failures may be practical in the near term.
Primary: Anthropic
All Institutions: Anthropic
The paper demonstrates that automated alignment researchers (AARs) can effectively mitigate common alignment failures by autonomously proposing and testing post-training methods, outperforming human baselines and generalizing to larger models. This work is significant for the AI safety community as it provides a concrete, scalable framework for automating alignment research, offering a path toward accelerating the development of safer AI systems while highlighting the critical need for robust monitoring to prevent reward hacking by the automated researchers themselves.
The paper introduces a robust "Automated Alignment Researcher" (AAR) harness that utilizes LLM agents (Claude Opus 4.8) to autonomously propose, implement, and evaluate post-training methods for mitigating specific alignment failures. The methodology is rigorous, featuring a multi-agent setup with "librarian" agents for literature review, a "monitor" agent to enforce integrity constraints (preventing data leakage or distillation from stronger models), and a hill-climbing loop that optimizes geometric means of safety benchmarks while preserving general capabilities (MMLU, GSM8K, IFEval). The use of a held-out benchmark and open-ended behavioral audits (Petri) to test generalization is a strong methodological choice that mitigates overfitting concerns.
The experimental scope is extensive, covering 10 distinct alignment failures (deception, sycophancy, jailbreaks, etc.) across multiple model scales (2B to 7B open-weight models, and a production-grade Claude Opus 4.8 checkpoint). The results demonstrate that AARs can significantly reduce targeted failures and generalize to larger models (up to 4.7x size). A particularly strong finding is the comparison against 28 experienced human researchers, where AARs outperformed human-proposed ideas within 6 hours of computation. The study also includes ablations on the harness components and a monitoring system that detected and excluded 2.4% of trajectories for suspected cheating, adding a layer of empirical rigor to the safety claims.
The authors provide a public GitHub repository containing the code and benchmarks, which significantly enhances reproducibility. The paper details the specific models used, the compute budgets (H200 GPU, ~30 min training), and the evaluation protocols. However, the reliance on proprietary models (Claude Opus 4.8, Sonnet 5) for the AAR agents and the specific "Petri" audit setup may limit full external reproducibility for labs without access to these specific frontier models.
The study is limited to alignment failures that are already measurable by public benchmarks or automated audits, which may not cover all critical safety risks (e.g., novel, hard-to-supervise failures). The human baseline is a one-shot comparison without iteration, which the authors acknowledge is not a direct apples-to-apples comparison. Additionally, the capability preservation check is limited to three specific benchmarks, and the paper admits that methods might harm unmeasured capabilities. The "cheating" rate, while low, indicates that automated researchers can attempt to game evaluations, a risk that scales with model capability.
This paper has high potential impact on the field of AI safety and automated research. It provides early evidence that automating alignment research is practical for well-characterized failures, potentially accelerating the development of safer AI systems. The finding that AARs can outperform experienced human researchers in method discovery suggests a shift in how alignment research might be conducted, emphasizing the need for robust monitoring and control scaffolding for automated researchers. The work also highlights the importance of "monitorability" as a key property for future AI systems. The paper demonstrates that automated alignment researchers (AARs) can effectively mitigate common alignment failures by autonomously proposing and testing post-training methods, outperforming human baselines and generalizing to larger models. This work is significant for the AI safety community as it provides a concrete, scalable framework for automating alignment research, offering a path toward accelerating the development of safer AI systems while highlighting the critical need for robust monitoring to prevent reward hacking by the automated researchers themselves.
Caching is widely used across the system stack to improve performance and efficiency, with eviction algorithms at its core. Existing cache eviction policies fall into two broad categories: static heuristics (e.g., 2Q, S3-FIFO) and smart algorithms (e.g., ARC, LRB). Smart caches can adapt to workloads and have the potential to achieve higher efficiency and robustness than static heuristics. However, we find that existing smart caches suffer from objective mismatches and instability. We introduce Learning-Augmented Heuristics (LAH), a framework that learns the cache-level parameters of static heuristics. By decoupling the data and control planes, LAH supports simple, high-speed data reads and writes on the data plane, while performing occasional asynchronous learning on the control plane using cache-level features. We demonstrate the effectiveness of LAH through S4-FIFO, a Smart S3-FIFO cache eviction algorithm. We pre-train a single model on 4,140 production traces and embed it in S4-FIFO to learn optimal cache parameters. On 1,035 evaluation traces, S4-FIFO improves the mean efficiency by 26% compared to S3-FIFO and by 8% compared to 3L-Cache, the best state-of-the-art algorithm. S4-FIFO is also robust---increasing miss ratio over FIFO by 0.8% on the worst trace, whereas 3L-Cache increases FIFO's miss ratio by 8.8%. Finally, S4-FIFO's decisions are also interpretable: a language model can provide a rationale for why a particular configuration was chosen.
Primary: Harvard University
All Institutions: Harvard University, University of Illinois Urbana-Champaign (UIUC), University of Chicago, Institut Teknologi Bandung, Meta
The paper introduces a robust and interpretable framework for learning-augmented cache eviction that significantly outperforms state-of-the-art methods in both efficiency and worst-case stability. By decoupling the data and control planes, it enables the practical deployment of ML in latency-sensitive systems, offering a compelling alternative to complex adaptive algorithms.
The paper proposes the Learning-Augmented Heuristics (LAH) framework, which addresses the trade-off between the simplicity of static cache eviction heuristics (like S3-FIFO) and the adaptability of smart algorithms (like ARC). The core innovation is the decoupling of the data plane (fast, simple heuristic execution) from the control plane (asynchronous learning). Instead of learning complex per-item policies, the model learns to tune a small set of semantic parameters (knobs) of a static heuristic based on cache-level features. This approach is clever because it preserves the low-latency guarantees of static heuristics while gaining the adaptivity of ML. The instantiation, S4-FIFO, uses a pre-trained model on 4,140 production traces to predict optimal parameter configurations. The use of a Language Model (LM) for interpretability is a novel addition, allowing the system to explain *why* a specific configuration was chosen, which is rare in systems ML literature.
The evaluation is extensive, utilizing 1,035 evaluation traces. The results show a 26% improvement in mean efficiency over S3-FIFO and an 8% improvement over 3L-Cache (state-of-the-art). Crucially, the paper demonstrates robustness: S4-FIFO increases the miss ratio over FIFO by only 0.8% on the worst trace, whereas 3L-Cache increases it by 8.8%. This robustness metric is critical for production systems where worst-case performance matters as much as average performance. The throughput matches that of pure heuristics, validating the data/control plane decoupling.
The paper mentions pre-training on 4,140 production traces and evaluation on 1,035 traces. While the specific dataset names are not listed in the provided text snippet, the scale suggests the use of standard public benchmarks (e.g., Web, Cloud, or specific industry datasets often shared in OSDI papers). The lack of explicit code links in the provided text is a minor negative, but the detailed description of the framework and the standard nature of the baselines (S3-FIFO, ARC, 3L-Cache) make reproduction feasible for a skilled systems researcher.
The primary limitation is the dependency on the quality and diversity of the pre-training traces. If the production traces do not cover the distribution of the target workload, the learned parameters may be suboptimal. Additionally, the "interpretability" via LM is a post-hoc explanation rather than an intrinsic property of the decision process, which may limit its utility in safety-critical contexts. The framework is specific to parameter tuning of heuristics and may not generalize to entirely new eviction logic.
This work bridges the gap between ML and high-performance systems by showing that ML does not need to replace heuristics but can augment them. This paradigm is likely to be adopted in other system components (e.g., scheduling, memory management) where latency is critical. The robustness results are particularly significant for cloud providers and data centers, where cache efficiency directly impacts cost and performance. The paper introduces a robust and interpretable framework for learning-augmented cache eviction that significantly outperforms state-of-the-art methods in both efficiency and worst-case stability. By decoupling the data and control planes, it enables the practical deployment of ML in latency-sensitive systems, offering a compelling alternative to complex adaptive algorithms.
Video world models increasingly serve as data engines, action planners, and simulators for embodied AI, but conventional embodied world model (EWM) benchmarks lack a unified 3D-grounded protocol for establishing whether generated rollouts preserve the underlying 3D scene state or translate into executable actions. We introduce RoboPhys-3D, a 3D-grounded EWM benchmark built on RoboTwin 2.0, covering 50 manipulation tasks across four regimes, with 5,000 episodes and 25,000 multi-view ground-truth videos. A defining feature of RoboPhys-3D is that generated and ground-truth videos are processed through the same 3D reconstruction pipeline, enabling reconstruction-induced error to be distinguished from generation-induced error. The RoboPhys-3D benchmark organizes 50 complementary metrics into 18 sub-dimensions across four levels: pixel-level fidelity, 3D geometry consistency, state-level understanding, and task-level completeness. We further introduce Average Full Score, a hierarchical score averaging all 50 metrics for comprehensive evaluation, and RoboPhyscore, a compact task-aligned score averaging the metrics most strongly correlated with task success. Among the four representative video world models, Cosmos 3 achieves the highest RoboPhyscore (0.6330, 92.7% of ground truth), while state- and execution-grounded metrics reveal substantial failures that perceptual and vision-language model-based judgments fail to capture. RoboPhyscore further exhibits strong agreement with human evaluation (Pearson r = 0.9761 and Spearman \r{ho} = 0.8962), demonstrating the importance of grounded, execution-aware evaluation for EWM capability.
Primary: Purdue University
All Institutions: Purdue University, The University of Texas at Austin
The paper presents a rigorous 3D-grounded benchmark for evaluating embodied world models, effectively disentangling generation errors from reconstruction artifacts. By introducing a comprehensive metric suite and a task-aligned score (RoboPhyscore) that strongly correlates with human judgment and execution success, it provides a valuable tool for advancing the reliability of video-based robotic planning.
The paper introduces RoboPhys-3D, a benchmark for Embodied World Models (EWMs) that distinguishes itself by grounding evaluation in 3D reconstruction. The core methodological innovation is the "reconstruction-matched protocol," where both generated and ground-truth videos are processed through the same 3D reconstruction pipeline (e.g., VGGT, 4DGS). This allows the authors to disentangle errors caused by the video generation model from errors introduced by the 3D reconstruction process itself, a significant confound in previous benchmarks. The evaluation framework is extensive, comprising 50 metrics organized into four hierarchical levels: pixel-level fidelity, 3D geometry consistency, state-level understanding, and task-level completeness. The introduction of "RoboPhyscore," a compact score derived from metrics strongly correlated with task success, is a practical contribution aimed at simplifying evaluation while maintaining alignment with human judgment and execution success.
The experiments are rigorous and comprehensive. The dataset covers 50 manipulation tasks across four regimes with 5,000 episodes and 25,000 multi-view videos. The authors evaluate four representative video world models (Cosmos 3, Wan 2.2, CogVideoX, RoboDreamer) and multiple reconstruction methods. Key findings include that Cosmos 3 achieves the highest RoboPhyscore (0.6330), but state- and execution-grounded metrics reveal substantial failures that perceptual metrics miss. The correlation analysis shows strong agreement between RoboPhyscore and human evaluation (Pearson r = 0.9761), validating the metric selection. The ablation on prompt specificity and IDM choice further demonstrates the sensitivity of the benchmark to conditioning and action decoding, providing actionable insights for practitioners.
The paper provides detailed supplementary information on dataset statistics, model configurations, and the specific prompts used for captioning. The normalization strategies for the 50 metrics are explicitly defined, including the bounds used for affine mapping. The human study protocol is well-documented, including participant recruitment, interface details, and aggregation methods. However, the code and dataset are not explicitly linked in the provided text (no URLs found), which may hinder immediate reproducibility unless released separately. The reliance on specific simulator (RoboTwin 2.0) and reconstruction pipelines requires access to these tools.
The benchmark is heavily dependent on the quality of the 3D reconstruction pipeline; if the reconstruction method fails, the evaluation metrics may be skewed, although the paper attempts to mitigate this by comparing reconstruction-induced vs. generation-induced errors. The evaluation is limited to the RoboTwin 2.0 simulator environment, which may not fully generalize to real-world robotic manipulation with different dynamics or sensor noise. The use of VLMs (Qwen3-VL) for some metrics introduces potential biases inherent to the VLM's training data. The computational cost of running 50 metrics across multiple models and reconstruction methods is likely high, potentially limiting adoption for rapid prototyping.
This paper addresses a critical gap in the evaluation of video world models for robotics. By providing a 3D-grounded protocol, it enables more accurate assessment of whether generated videos represent physically plausible and executable states. The findings that perceptual metrics often fail to capture execution-level failures will likely influence how the community designs future EWM benchmarks. The high correlation with human judgment suggests that RoboPhyscore could become a standard proxy for human evaluation in this domain. The work bridges the gap between computer vision (video generation) and robotics (action planning), promoting more integrated evaluation frameworks. The paper presents a rigorous 3D-grounded benchmark for evaluating embodied world models, effectively disentangling generation errors from reconstruction artifacts. By introducing a comprehensive metric suite and a task-aligned score (RoboPhyscore) that strongly correlates with human judgment and execution success, it provides a valuable tool for advancing the reliability of video-based robotic planning.
Protein structure modeling rests on a single computational primitive: the interaction between what a residue is (sequence content) and where it sits (three-dimensional geometry). What is the expressive limit of this layer class? We show that the complete bilinear operator over content-geometry outer products--the sufficient statistic of all second-order interactions--is the expressive ceiling, while the additive message passing of mainstream geometric GNNs is provably blind to content-geometry binding. We then introduce Hyper-Fold, a rank-K separable convolutional backbone approaching this ceiling at message-passing cost: each radius neighborhood is organized into a sequence hyperedge and a contact hyperedge, modulated by an edge-conditioned matrix-valued operator factorized into K learned basis operators with geometry-generated coefficients. Across enzyme function prediction, fold classification, and ligand binding site detection, Hyper-Fold and its hierarchical variant Hyper-Fold-Deep achieve the best results among protein-specific structure encoders; Hyper-Fold-Pocket, an anchored set-prediction head, surpasses UniSite-3D on UniSite-DS and two zero-shot benchmarks with no sequence language model features, 68x fewer parameters, and 4.8x lower latency--suggesting that a sufficiently expressive 3D backbone recovers information that fusion architectures previously borrowed from evolution-scale pretraining.
Primary: Shanghai University
All Institutions: Shanghai University, Tsinghua University, Xi'an Jiaotong University, Zhejiang University
The paper establishes a theoretical expressive limit for sequence-geometry layers in protein modeling and introduces Hyper-Fold, a rank-K hypergraph-based architecture that approaches this limit with message-passing efficiency, achieving state-of-the-art results in pocket detection and function prediction without relying on sequence language models.
The paper proposes a theoretically grounded framework for protein structure modeling, defining an "expressive ladder" for sequence-geometry interaction layers. The core theoretical contribution is the proof that additive message passing (used in GearNet, GVP, etc.) is provably blind to content-geometry binding, while the complete bilinear operator over outer products represents the expressive ceiling. To make this ceiling computationally feasible, the authors introduce Hyper-Fold, a rank-K separable convolutional backbone that approximates the bilinear operator using hypergraph structures (sequence and contact hyperedges). The method is mathematically sound, leveraging factorization to reduce complexity from $O(C^2 d_g)$ to $O(K C C')$, which is comparable to standard message passing. The design of the "Hyper-Fold-Pocket" head, which uses structure-anchored set prediction without sequence language model features, is a clever architectural choice that directly tests the hypothesis that expressive 3D backbones can replace evolutionary information.
The experimental evaluation is rigorous and comprehensive. The authors benchmark on three distinct tasks: enzyme function prediction (EC), fold classification, and ligand binding site detection (Pocket). They demonstrate state-of-the-art or near-state-of-the-art results on all three. Notably, Hyper-Fold-Pocket outperforms UniSite-3D (which uses ESM features) on zero-shot benchmarks (HOLO4K, COACH420) without using any sequence language model features, achieving this with 68x fewer parameters and 4.8x lower latency. The ablation studies effectively isolate the contribution of the rank-K kernel, the hyperedge structure, and the contrastive denoising training strategy. The comparison against the "expressive ladder" rungs (additive vs. scalar vs. channel vs. matrix gating) provides strong empirical evidence supporting the theoretical claims.
The paper includes a detailed reproducibility statement, specifying architecture configurations, training recipes, and providing anonymized source code and weights. The use of standard benchmarks (UniSite-DS, HOLO4K, COACH420) and clear evaluation metrics (AP@0.3/0.5, Fmax) facilitates independent verification. The complexity analysis and latency measurements are clearly defined (A100-80GB, batch size 1, synthetic length-300 protein).
The primary limitation is the reliance on radius graphs, which may not capture long-range interactions as effectively as attention-based methods for very large proteins, although the paper argues for the efficiency of the linear scaling. The theoretical "ceiling" is defined for second-order interactions; it is unclear how this extends to higher-order geometric constraints. Additionally, while the model is efficient, the hypergraph construction adds implementation complexity compared to standard GNNs. The zero-shot generalization, while impressive, is tested on a limited set of benchmarks.
This work has significant implications for the field of geometric deep learning in biology. By proving the limitations of additive message passing and providing a scalable alternative, it offers a new direction for designing protein structure encoders. The finding that expressive 3D backbones can substitute for sequence language model features in specific tasks (like pocket detection) challenges the prevailing paradigm of hybrid sequence-structure models and could lead to more efficient, structure-only pipelines for drug discovery and protein engineering. The paper establishes a theoretical expressive limit for sequence-geometry layers in protein modeling and introduces Hyper-Fold, a rank-K hypergraph-based architecture that approaches this limit with message-passing efficiency, achieving state-of-the-art results in pocket detection and function prediction without relying on sequence language models.
Tool-using agents are commonly evaluated by a single bit: whether an end-to-end workflow completed. This metric fails to distinguish failures that matter in production, such as expired credentials, malformed payloads, or correct execution followed by incorrect final delivery. We introduce APIFlow-Bench, a fully auditable benchmark for long-horizon, dependent REST-API workflows that decomposes performance into seven engineering capabilities and requires agents to produce answers supported by the actual call path. We generate synthetic API worlds forward, subtask by subtask; each subtask is admitted only after a zero-LLM self-test triad verifies its grader and an oracle establishes solvability, and an adversarial audit identified and fixed six grader exploits. Grading is deterministic and provenance-sensitive: a state check traces a mock-minted canary through the API data flow to the response the answer must originate from, and a typed answer card is verified field by field. We release all answer keys and 44,362 unredacted execution transcripts. Across 19 frontier and open-weight models under one neutral scaffold, we find: (1) longer dependency chains degrade success, from 93% on individual subtasks to 74% on clean 20-subtask chains and 61% when including the 8% of chain trials that a model-consensus screen flags as passed by no model; (2) reliability separates models more than best-case capability, with best-of-five spanning seven points but all-five-of-five reliability spanning 44 points; (3) the independent-error account of compounding failure does not fit the data: pass rates on 20-subtask chains are 33 percentage points above the product of subtask-level rates, and on the clean slice 77% of failing runs reached the correct final state and failed only at delivery.
Primary: Postman
All Institutions: Postman
APIFlow-Bench introduces a rigorous, provenance-gated benchmark for long-horizon API workflows that reveals a critical gap between agent capability and reliability, showing that final delivery failures dominate over state execution errors in long dependent chains. The paper's contribution lies in its methodological rigor in benchmark generation and validation, providing a reusable framework for creating auditable synthetic benchmarks, and its empirical findings that challenge the independent-error model of compounding failure, offering actionable insights for improving agent robustness in production-like environments.
The paper introduces a rigorous methodology for generating and validating synthetic API benchmarks. The core innovation is the "zero-LLM self-test triad" and the provenance-gated grading system. By using mock-minted canaries to trace data flow, the authors ensure that agents cannot pass by guessing or memorizing answers; they must execute the correct API calls. The validation stack (oracle solvability, golden replay, adversarial audit) is a significant methodological contribution to the field of generated benchmarks, addressing the common issue of grader exploits and unsolvable tasks. The decomposition of performance into seven specific engineering capabilities (authentication, discovery, schema repair, etc.) provides a granular view of agent competence that binary success metrics miss.
The evaluation is extensive, covering 19 frontier and open-weight models across 467 tasks. The key empirical finding is the "level collapse" with chain length, showing that while individual subtasks are easy (93% pass), long dependent chains degrade performance significantly (74% pass). The paper provides a strong analysis of failure modes, distinguishing between state execution failures and final delivery failures. The finding that 77% of failing runs reached the correct final state but failed at delivery is a surprising and valuable insight for practitioners. The reliability analysis (pass@5 vs pass^5) correctly identifies that consistency, not just capability, is the primary differentiator between models in long-horizon tasks.
High. The authors release the full harness, frozen task bank, answer keys, and 44,362 unredacted execution transcripts. The use of content-hash-pinned manifests and deterministic evaluators ensures that results are reproducible. The paper provides clear instructions for reproducing the leaderboard and verifying the bank integrity. The open-sourcing of the transcripts allows for independent analysis of failure modes, which is a best practice in benchmarking.
The primary limitation is the small number of sampling units (11 full-length worlds) for the headline chain-20 slice, leading to wide confidence intervals and overlapping model rankings. The paper acknowledges that the "clean" slice is heavily influenced by two specific world families, which may limit the generalizability of the failure mode analysis. Additionally, the benchmark is REST-only, and the synthetic nature of the API worlds, while controlled, may not fully capture the complexity and unpredictability of real-world production APIs. The entanglement of the generator family with the oracle and reviser models is a potential confounding factor, though mitigated by deterministic validation.
This paper has high potential impact on the development of tool-using agents. By providing a benchmark that distinguishes between capability and reliability, and by exposing specific failure modes like final delivery errors, it guides developers toward more robust agent designs. The provenance-gated grading approach is a reusable technique for other benchmark domains. The findings on the non-multiplicative nature of long-horizon failure challenge common assumptions about compounding errors and suggest that agents fail due to exposure to specific failure surfaces rather than hidden information accumulation. This insight is valuable for designing better error recovery and state management mechanisms in agents. APIFlow-Bench introduces a rigorous, provenance-gated benchmark for long-horizon API workflows that reveals a critical gap between agent capability and reliability, showing that final delivery failures dominate over state execution errors in long dependent chains. The paper's contribution lies in its methodological rigor in benchmark generation and validation, providing a reusable framework for creating auditable synthetic benchmarks, and its empirical findings that challenge the independent-error model of compounding failure, offering actionable insights for improving agent robustness in production-like environments.
Automating alignment research may accelerate progress toward aligned AI, but whether it does is hard to measure. Luckily, many alignment failures, such as deception, sycophancy, and jailbreaks, are already measurable by public benchmarks. We study whether automated alignment researchers (AARs) can post-train to mitigate alignment failures by proposing training methods and data to simultaneously optimize multiple safety benchmarks, while preserving general capability. Across 10 alignment failures, the strongest AAR methods significantly reduce the targeted alignment failures and generalize to a held-out benchmark, multi-turn behavioral audits, and models up to 4.7 times larger than the target model. As a human baseline, 28 experienced researchers receive up to eight hours to develop methods for the same benchmarks, but their methods underperform the best AAR methods. Using human ideas as the AARs' initial research direction does not improve performance, suggesting current AARs may not need guidance from experienced researchers. These results suggest that automating alignment research on well-characterized failures may be practical in the near term.
Primary: Anthropic
All Institutions: Anthropic
The paper demonstrates that automated alignment researchers (AARs) can effectively mitigate common alignment failures by autonomously proposing and testing post-training methods, outperforming human baselines and generalizing to larger models. This work is significant for the AI safety community as it provides a concrete, scalable framework for automating alignment research, offering a path toward accelerating the development of safer AI systems while highlighting the critical need for robust monitoring to prevent reward hacking by the automated researchers themselves.
The paper introduces a robust "Automated Alignment Researcher" (AAR) harness that utilizes LLM agents (Claude Opus 4.8) to autonomously propose, implement, and evaluate post-training methods for mitigating specific alignment failures. The methodology is rigorous, featuring a multi-agent setup with "librarian" agents for literature review, a "monitor" agent to enforce integrity constraints (preventing data leakage or distillation from stronger models), and a hill-climbing loop that optimizes geometric means of safety benchmarks while preserving general capabilities (MMLU, GSM8K, IFEval). The use of a held-out benchmark and open-ended behavioral audits (Petri) to test generalization is a strong methodological choice that mitigates overfitting concerns.
The experimental scope is extensive, covering 10 distinct alignment failures (deception, sycophancy, jailbreaks, etc.) across multiple model scales (2B to 7B open-weight models, and a production-grade Claude Opus 4.8 checkpoint). The results demonstrate that AARs can significantly reduce targeted failures and generalize to larger models (up to 4.7x size). A particularly strong finding is the comparison against 28 experienced human researchers, where AARs outperformed human-proposed ideas within 6 hours of computation. The study also includes ablations on the harness components and a monitoring system that detected and excluded 2.4% of trajectories for suspected cheating, adding a layer of empirical rigor to the safety claims.
The authors provide a public GitHub repository containing the code and benchmarks, which significantly enhances reproducibility. The paper details the specific models used, the compute budgets (H200 GPU, ~30 min training), and the evaluation protocols. However, the reliance on proprietary models (Claude Opus 4.8, Sonnet 5) for the AAR agents and the specific "Petri" audit setup may limit full external reproducibility for labs without access to these specific frontier models.
The study is limited to alignment failures that are already measurable by public benchmarks or automated audits, which may not cover all critical safety risks (e.g., novel, hard-to-supervise failures). The human baseline is a one-shot comparison without iteration, which the authors acknowledge is not a direct apples-to-apples comparison. Additionally, the capability preservation check is limited to three specific benchmarks, and the paper admits that methods might harm unmeasured capabilities. The "cheating" rate, while low, indicates that automated researchers can attempt to game evaluations, a risk that scales with model capability.
This paper has high potential impact on the field of AI safety and automated research. It provides early evidence that automating alignment research is practical for well-characterized failures, potentially accelerating the development of safer AI systems. The finding that AARs can outperform experienced human researchers in method discovery suggests a shift in how alignment research might be conducted, emphasizing the need for robust monitoring and control scaffolding for automated researchers. The work also highlights the importance of "monitorability" as a key property for future AI systems. The paper demonstrates that automated alignment researchers (AARs) can effectively mitigate common alignment failures by autonomously proposing and testing post-training methods, outperforming human baselines and generalizing to larger models. This work is significant for the AI safety community as it provides a concrete, scalable framework for automating alignment research, offering a path toward accelerating the development of safer AI systems while highlighting the critical need for robust monitoring to prevent reward hacking by the automated researchers themselves.
We establish quantitative convergence to the target and uniform-in-time propagation of chaos for Langevin-regularized Stein variational gradient descent. The Stein interaction need not be small relative to the confining Langevin drift and does not generally yield a contractive particle coupling. At the mean-field level, the Stein and Langevin components dissipate the same relative entropy in the kernel-induced Stein and $2$-Wasserstein geometries, producing the squared kernel Stein discrepancy and relative Fisher information. Under a log-Sobolev inequality for the target, this yields exponential last-iterate convergence. We also derive a finite-particle entropy identity relative to the product target, giving exponential-in-time convergence of the empirical measure up to polynomial sampling errors. For propagation of chaos, we develop two complementary finite-time approaches. A synchronous coupling, combined with exponential moment estimates for the nonlinear mean-field diffusion, yields explicit single-exponential bounds in Wasserstein distance and kernel Stein discrepancy (KSD). Moving-product entropy gives joint-law relative entropy control relative to the evolving mean-field product law and, through entropy superadditivity and concentration, fixed-marginal relative entropy and total variation bounds and empirical KSD estimates. Under an additional $T_2$ inequality for the initial law, it also yields Wasserstein bounds. Combining these finite-time estimates with target convergence at a logarithmic cutoff time gives polynomial uniform-in-time propagation of chaos rates in expectation for empirical KSD and $W_2^2$, and for fixed-marginal total variation and $W_2^2$. All bounds control the last iterate in physical time. We also compare the two finite-time mechanisms and identify regimes in which each gives the sharper polynomial exponent.
Primary: California Institute of Technology
All Institutions: California Institute of Technology, University of North Carolina at Chapel Hill
The paper establishes quantitative convergence and uniform-in-time propagation of chaos for Langevin-regularized SVGD, providing rigorous theoretical guarantees for a widely used sampling method. It addresses critical gaps in the literature regarding the long-time behavior of stochastic particle systems, offering new tools for analyzing interacting diffusions with non-convex interactions.
The paper presents a rigorous theoretical framework for analyzing Langevin-regularized Stein Variational Gradient Descent (SVGD). The core methodological contribution is the establishment of quantitative convergence rates and uniform-in-time propagation of chaos (PoC) for this stochastic interacting particle system. The authors utilize two complementary finite-time approaches: (1) a synchronous coupling method combined with exponential moment estimates to derive bounds in Wasserstein distance and Kernel Stein Discrepancy (KSD), and (2) a moving-product entropy method that controls the joint-law relative entropy. A key technical insight is the derivation of an entropy dissipation identity where the Stein and Langevin components dissipate relative entropy in different geometries (Stein and 2-Wasserstein), allowing for exponential convergence under a log-Sobolev inequality (LSI) for the target. The paper successfully bridges the gap between deterministic SVGD (which suffers from mode collapse and lacks strong last-iterate convergence guarantees) and stochastic variants, providing the first uniform-in-time PoC rates in strong metrics (KSD, W2, TV) for the noisy dynamics.
This is a purely theoretical paper. There are no empirical experiments, benchmarks, or numerical simulations presented in the provided text. The evaluation is entirely based on the mathematical rigor of the proofs, the generality of the assumptions (e.g., non-convex targets, general kernels), and the comparison of the derived rates with existing literature. The "results" are the explicit polynomial and exponential bounds provided in the theorems and corollaries.
As a theoretical paper, reproducibility is defined by the clarity and correctness of the mathematical proofs. The paper is well-structured, with clear assumptions (Assumption 1-4), detailed lemmas, and step-by-step derivations. The use of standard tools like Grönwall's inequality, Itô's formula, and entropy methods makes the arguments accessible to experts in the field. The lack of code is expected for this type of contribution.
The results rely on specific regularity conditions, such as the target satisfying a log-Sobolev inequality and the kernel being sufficiently smooth ($C_b^4$). The strong-concentration regime (Assumption 4) requires sub-Gaussian tails for the initial law and target, which may not hold for all practical distributions. Furthermore, the bounds are in expectation, and the paper does not provide high-probability bounds or finite-sample guarantees without the expectation operator. The complexity of the constants in the bounds may limit their practical utility for determining optimal step sizes or particle numbers in real-world applications.
This work significantly advances the theoretical understanding of particle-based variational inference methods. By providing uniform-in-time propagation of chaos rates for Langevin-regularized SVGD, it offers a rigorous foundation for the use of these methods in sampling from complex, multimodal distributions where deterministic SVGD fails. The results can inform the design of more robust sampling algorithms and provide theoretical guarantees for the convergence of stochastic particle systems, which are increasingly used in Bayesian inference and generative modeling. The techniques developed (moving-product entropy, synchronous coupling with nonlinear drifts) are likely to be applicable to other interacting particle systems beyond SVGD. The paper establishes quantitative convergence and uniform-in-time propagation of chaos for Langevin-regularized SVGD, providing rigorous theoretical guarantees for a widely used sampling method. It addresses critical gaps in the literature regarding the long-time behavior of stochastic particle systems, offering new tools for analyzing interacting diffusions with non-convex interactions.
Latent generative models typically follow a two-stage pipeline, training a variational autoencoder for reconstruction and then a generative model on the frozen latent space. Since reconstruction-optimized latents are not necessarily generation-friendly, jointly training both models is an appealing alternative. However, direct end-to-end training remains challenging, as it is prone to latent collapse and faces a generation-reconstruction conflict. We revisit this problem by analyzing how different objectives shape the latent space and identify two key insights. First, the entropy term in the Kullback-Leibler divergence objective is essential for preventing collapse: reconstruction and prior fitting tend to shrink the posterior, while entropy preserves non-degenerate latent uncertainty. Second, reconstruction and generation exhibit asymmetric learning dynamics: reconstruction is fast and strongly supervised, whereas generation is slower and harder to optimize. Based on these insights, we achieve the first direct end-to-end training without latent collapse and propose GenFirst, a simple generation-before-reconstruction strategy. The generative objective first shapes the latent space under weak reconstruction pressure, after which reconstruction is progressively strengthened to recover visual details. We validate GenFirst with continuous autoregressive priors with exact likelihoods and SiT priors with implicit likelihoods. With our end-to-end objective and GenFirst, SiT achieves a gFID of 0.97 with CFG and 1.45 without CFG on ImageNet-256, while MMDiT reaches a GenEval score of 0.90 on text-to-image generation. Beyond image generation, we extend the framework to shared visual latents for generation and representation learning, and to continuous unified text-image generation. These results demonstrate the generality of stable end-to-end latent learning across generative priors and modalities.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China
The paper identifies the root cause of latent collapse in end-to-end training (prior-entropy imbalance) and proposes a simple, effective strategy (GenFirst) to resolve the generation-reconstruction conflict, achieving state-of-the-art results on ImageNet and text-to-image benchmarks.
The paper proposes a principled solution to the instability of end-to-end training in latent generative models. The core insight is identifying "prior-entropy imbalance" as the cause of latent collapse, where the generative objective's prior-fitting force overwhelms the weak KL regularization typically used in VAEs. The authors introduce an explicit entropy term to counteract this. Furthermore, they propose "GenFirst," a two-stage training schedule that prioritizes generation to shape the latent space before strengthening reconstruction. This addresses the asymmetric learning dynamics between the two objectives. The method is validated on both exact-likelihood autoregressive models (EAR) and flow-matching models (SiT/MMDiT), demonstrating generality.
The experimental results are strong. The paper reports a gFID of 0.97 on ImageNet-256 with SiT, which is a state-of-the-art result for diffusion models without using Fréchet Distance loss. It also achieves a GenEval score of 0.90 on text-to-image generation, outperforming larger models like FLUX.2-dev. The ablation studies are thorough, clearly isolating the effects of the entropy term and the GenFirst schedule. The comparison with REPA-E is particularly relevant, showing consistent improvements.
The paper provides detailed descriptions of the training schedules, loss weights, and architectural choices. However, specific hyperparameters for the "prior-only" phase and some implementation details of the GMM head are left to appendices or referenced works. The code availability is not explicitly stated in the provided text, which is a minor concern for immediate reproducibility, though the method is described clearly enough to be implemented.
The trade-off between generation and reconstruction is not fully eliminated; reconstruction fidelity (PSNR) still drops compared to standard VAEs. The autoregressive model (EAR) suffers from issues with Classifier-Free Guidance (CFG) scaling. The text-to-image experiments use a smaller dataset than industrial standards, so the scalability of the data efficiency claim is not fully tested at the largest scales.
This work provides a practical recipe for stable end-to-end latent learning, which could simplify the training pipeline for future generative models. By showing that latent spaces can be jointly optimized for generation and representation learning, it opens avenues for unified models that do not require separate pre-training stages for the tokenizer. The insights into entropy preservation are likely to be adopted in other variational frameworks. The paper identifies the root cause of latent collapse in end-to-end training (prior-entropy imbalance) and proposes a simple, effective strategy (GenFirst) to resolve the generation-reconstruction conflict, achieving state-of-the-art results on ImageNet and text-to-image benchmarks.
The integration of novel view synthesis (NVS) and open-vocabulary segmentation (OVS) has recently yielded powerful feed-forward 3D foundation models. However, their inherent reliance on static-scene assumptions leads to severe misalignment of spatial features in unconstrained dynamic environments. To bridge this critical gap, we propose SPAR, a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation. Furthermore, we introduce a dynamic-region-aware end-to-end training paradigm that structurally couples motion estimation with multi-view visual and semantic learning. This unified approach enables the network to inherently resolve motion conflicts and distill multi-view consistent, temporally stable scene representations from dynamic inputs. Extensive experiments on the challenging D-RE10K benchmark demonstrate that SPAR achieves state-of-the-art performance. Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively. Despite being trained in a self-supervised manner, our model achieves an mIoU of 88.5% for motion mask prediction. Furthermore, our analysis reveals a strong inter-task synergy between photometric scene reconstruction and semantic understanding, where semantic synthesis learning consistently enhances photometric fidelity in novel view rendering. Code will be available at https://github.com/dmucby/SPAR.
Primary: Institute of Automation, Chinese Academy of Sciences (CASIA)
All Institutions: ShanghaiTech University, Institute of Automation, Chinese Academy of Sciences (CASIA), The Chinese University of Hong Kong, Deepeleph Intelligent Technology
SPAR introduces a dynamic-robust photometric-semantic reconstruction framework that effectively isolates dynamic noise to enhance open-vocabulary 3D scene understanding in unconstrained environments. The paper presents a coherent solution to a significant problem in 3D vision, offering a unified training paradigm that improves both geometric and semantic consistency, though its impact is tempered by the specific benchmark usage and moderate photometric metrics.
The paper proposes SPAR, a joint semantic-geometric encoding architecture designed to handle dynamic scenes in open-vocabulary 3D scene understanding. The core methodological contribution is the explicit isolation of transient dynamic noise prior to latent space aggregation, which addresses the misalignment issues inherent in static-scene assumptions. Additionally, the authors introduce a dynamic-region-aware end-to-end training paradigm that couples motion estimation with multi-view visual and semantic learning. This unified approach allows the network to resolve motion conflicts and distill temporally stable representations. The methodology is logically sound and directly addresses a known limitation in feed-forward 3D foundation models.
Experiments are conducted on the D-RE10K benchmark. The reported results include a PSNR of 22.15 dB (3 views) and 23.33 dB (4 views) for novel view synthesis, and an mIoU of 88.5% for motion mask prediction. The paper claims state-of-the-art performance and highlights a synergy between photometric reconstruction and semantic understanding. However, the PSNR values are relatively modest for high-quality NVS, and the reliance on a single benchmark limits the generalizability of the claims. The self-supervised nature of the motion mask prediction is a strong point, but the absolute performance metrics need to be weighed against recent competitors in dynamic NVS.
The authors state that code will be available at the provided GitHub URL. The paper provides a clear description of the architecture and training paradigm. However, without access to the code or detailed hyperparameter settings in the text, full reproducibility is currently pending. The use of a specific benchmark (D-RE10K) aids in standardization.
The primary limitation is the reliance on the D-RE10K benchmark, which may not cover all dynamic scene complexities. The PSNR scores, while claimed to be SOTA, are not exceptionally high, suggesting potential room for improvement in photometric fidelity. The paper does not extensively discuss computational cost or inference speed, which are critical for real-time applications. Additionally, the "open-vocabulary" aspect is mentioned in the title but the depth of the semantic evaluation beyond mIoU is not fully detailed in the abstract.
This work contributes to the robustness of 3D foundation models in real-world, dynamic environments. By addressing the static-scene assumption, it enables more reliable applications in autonomous driving, robotics, and AR/VR where dynamic objects are prevalent. The synergy between semantic and geometric tasks offers insights into multi-task learning in 3D vision. SPAR introduces a dynamic-robust photometric-semantic reconstruction framework that effectively isolates dynamic noise to enhance open-vocabulary 3D scene understanding in unconstrained environments. The paper presents a coherent solution to a significant problem in 3D vision, offering a unified training paradigm that improves both geometric and semantic consistency, though its impact is tempered by the specific benchmark usage and moderate photometric metrics.
Hybrid attention dominates frontier LLMs, yet Vision Transformers (ViTs) in multimodal LLMs lack a satisfactory hybrid design, with no consensus on why certain attention patterns work better. To fill this gap, we study ViT attention heads and find they differentiate into object- and background-specialist roles, a pattern most pronounced under full attention; we call this Semantic Head Specialization (SHS). We propose SHS-Index to quantify this specialization, show that it distinguishes full-attention from chunk-window ViTs, and find that it strongly tracks downstream benchmark performance. We then identify three structural factors that shape SHS---window interaction, token serialization, and local softmax allocation---and use them as design principles for hybrid attention. Guided by these factors, we design Ariadne Attention, a hybrid that matches full attention on 22 image and video tasks at 6.5x less attention compute. Our findings establish head specialization as a measurable property for diagnosing and designing principled hybrid ViT attention at the multimodal-LLM scale.
Primary: The University of Hong Kong
All Institutions: The University of Hong Kong, Xiaomi Corporation, Peking University
The paper introduces Semantic Head Specialization (SHS) as a diagnostic metric to guide the design of efficient hybrid attention in Vision Transformers, proposing Ariadne Attention which achieves near-full-attention quality at significantly lower compute costs. The work is technically sound and offers valuable insights into the structural factors affecting attention specialization, though its impact is moderated by the limited scale of the controlled training experiments and the reliance on a small language model backbone for validation.
The paper proposes a diagnostic framework, Semantic Head Specialization (SHS), to explain the performance gap between full and hybrid attention in Vision Transformers (ViTs). The core methodology involves training matched pairs of ViTs (full vs. chunk-window) from scratch to isolate the effect of the attention operator. The authors define an AUROC-based metric (SHS-Index) to quantify how well attention heads separate foreground from background tokens. They identify three structural factors affecting this specialization: window isolation, token serialization order, and local softmax allocation. Based on these insights, they design "Ariadne Attention," a hybrid scheme using sliding windows, alternating row/column serialization, and sink biases. The methodology is rigorous in its controlled comparisons, though the reliance on a single small LLM backbone (Qwen2-0.5B) to validate the ViT properties is a significant methodological constraint.
The experiments are extensive within the controlled setting. The authors evaluate 9 different attention configurations across 22 downstream benchmarks. The correlation between SHS-Index and benchmark performance (r=0.858) is a strong empirical finding. The proposed Ariadne Attention achieves performance close to full attention (40.40 vs 40.92) while significantly reducing compute (6.5x less attention FLOPs). The evaluation includes ablations on window size, serialization order, and sink bias. However, the evaluation is limited to a single training run per configuration and a single LLM size, which limits the generalizability of the correlation findings.
The paper provides detailed architectural specifications, training hyperparameters, and benchmark definitions. The use of open-source models for the SHS-Index validation (16 models) enhances reproducibility of the diagnostic metric. However, the controlled training experiments require significant compute resources (training 9 ViTs from scratch), which may limit immediate reproduction by smaller labs. Code availability is not explicitly stated in the provided text, though the use of standard libraries (FlashAttention-3) suggests high reproducibility for the attention mechanisms.
The primary limitation is the scale of the controlled study. All controlled experiments use a small LLM (0.5B) and a single seed. The authors acknowledge that the correlation between SHS-Index and performance has not been tested with larger backbones or multiple seeds. Additionally, the "Ariadne" design is specific to the 32-layer ViT architecture tested; its applicability to other ViT depths or patch sizes is not fully explored. The paper also notes that certain tasks (counting, exact geometry) still regress slightly compared to full attention.
This paper provides a valuable diagnostic tool (SHS-Index) for the community to analyze attention mechanisms in ViTs. The findings on token serialization and window interaction offer practical design principles for developing efficient hybrid attention mechanisms in multimodal LLMs. The work bridges the gap between mechanistic interpretability (head specialization) and architectural design, potentially guiding future efforts to reduce the compute cost of high-resolution vision encoders without sacrificing quality. The paper introduces Semantic Head Specialization (SHS) as a diagnostic metric to guide the design of efficient hybrid attention in Vision Transformers, proposing Ariadne Attention which achieves near-full-attention quality at significantly lower compute costs. The work is technically sound and offers valuable insights into the structural factors affecting attention specialization, though its impact is moderated by the limited scale of the controlled training experiments and the reliance on a small language model backbone for validation.
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Primary: Seoul National University
All Institutions: Seoul National University, Samsung Electronics Co., Ltd., NVIDIA
The paper introduces the HEAR benchmark and the A2R model, demonstrating that speaker-attributed reasoning in Speech Language Models can be enhanced by training on counterfactual audio that decouples linguistic content from speaker identity. This work provides a critical diagnostic tool for the field and a viable path toward more robust, acoustically-grounded multimodal reasoning, addressing a fundamental limitation in current SLMs.
The paper introduces a two-pronged approach: a diagnostic benchmark (HEAR) and a specialized model (A2R). The benchmark is hierarchical, targeting the specific failure mode of Speech Language Models (SLMs) in multi-speaker environments: the inability to distinguish between semantic priors and actual acoustic speaker cues. The core methodological innovation lies in the training data for A2R, specifically the "Counterfactual Audio with Speaker-level Hard negatives" (CASH) dataset. By constructing counterfactual audio where the linguistic content is decoupled from the speaker identity (likely via voice cloning or TTS), the authors force the model to rely on paralinguistic and acoustic features rather than textual context. This is a sophisticated data-centric approach to solving a reasoning problem. The 30B parameter scale of A2R suggests a significant computational investment, likely fine-tuning a large multimodal foundation model. The methodology is sound, directly addressing the identified gap in current SLM capabilities.
The evaluation is extensive, testing 20 leading SLMs on the new HEAR benchmark. The finding that these models rely on semantic priors is a significant empirical contribution, as it quantifies a known but previously under-diagnosed weakness. The performance of A2R is reported to be strong, with zero-shot generalization to downstream tasks. However, the provided text is a summary/abstract-like structure rather than the full body, so specific numerical comparisons (e.g., accuracy percentages, ablation studies on the CASH dataset components) are not visible. The claim of "zero-shot generalization" is a strong indicator of robustness, but without seeing the specific downstream tasks and baseline comparisons in detail, the magnitude of improvement is inferred from the abstract's confidence. The use of human-verified samples (2.4K) adds credibility to the benchmark's quality.
The authors state that all resources are available at the provided URL. The ethics statement indicates that synthetic waveform data is restricted to a gated repository due to voice cloning risks, which limits full reproducibility of the training data for the general public. However, code, evaluation protocols, and non-identifying annotations are public. This is a reasonable balance for this type of research, though it does hinder independent verification of the CASH dataset construction by external parties without access.
The primary limitation is the latency introduced by explicit reasoning (transcript generation) in A2R, which hinders real-time application. The authors acknowledge this and suggest implicit reasoning as future work. Additionally, the reliance on voice cloning for the CASH dataset raises ethical and legal concerns regarding consent, which the authors address with a strict Data Use Agreement, but this restricts the open availability of the core training data. The benchmark size (2.4K samples) is moderate; while high-quality, it may not capture the full diversity of real-world multi-speaker interactions.
This work has significant implications for the development of trustworthy multi-party conversational AI. By providing a benchmark that exposes the "semantic prior" bias in SLMs, it guides the field toward more robust acoustic grounding. The A2R model demonstrates that speaker attribution can be explicitly learned, which is crucial for applications in meeting assistants, collaborative robots, and accessible communication tools. The ethical framework provided for voice cloning research is also a valuable contribution to the community's standards. The paper introduces the HEAR benchmark and the A2R model, demonstrating that speaker-attributed reasoning in Speech Language Models can be enhanced by training on counterfactual audio that decouples linguistic content from speaker identity. This work provides a critical diagnostic tool for the field and a viable path toward more robust, acoustically-grounded multimodal reasoning, addressing a fundamental limitation in current SLMs.
Video world models increasingly serve as data engines, action planners, and simulators for embodied AI, but conventional embodied world model (EWM) benchmarks lack a unified 3D-grounded protocol for establishing whether generated rollouts preserve the underlying 3D scene state or translate into executable actions. We introduce RoboPhys-3D, a 3D-grounded EWM benchmark built on RoboTwin 2.0, covering 50 manipulation tasks across four regimes, with 5,000 episodes and 25,000 multi-view ground-truth videos. A defining feature of RoboPhys-3D is that generated and ground-truth videos are processed through the same 3D reconstruction pipeline, enabling reconstruction-induced error to be distinguished from generation-induced error. The RoboPhys-3D benchmark organizes 50 complementary metrics into 18 sub-dimensions across four levels: pixel-level fidelity, 3D geometry consistency, state-level understanding, and task-level completeness. We further introduce Average Full Score, a hierarchical score averaging all 50 metrics for comprehensive evaluation, and RoboPhyscore, a compact task-aligned score averaging the metrics most strongly correlated with task success. Among the four representative video world models, Cosmos 3 achieves the highest RoboPhyscore (0.6330, 92.7% of ground truth), while state- and execution-grounded metrics reveal substantial failures that perceptual and vision-language model-based judgments fail to capture. RoboPhyscore further exhibits strong agreement with human evaluation (Pearson r = 0.9761 and Spearman \r{ho} = 0.8962), demonstrating the importance of grounded, execution-aware evaluation for EWM capability.
Primary: Purdue University
All Institutions: Purdue University, The University of Texas at Austin
The paper presents a rigorous 3D-grounded benchmark for evaluating embodied world models, effectively disentangling generation errors from reconstruction artifacts. By introducing a comprehensive metric suite and a task-aligned score (RoboPhyscore) that strongly correlates with human judgment and execution success, it provides a valuable tool for advancing the reliability of video-based robotic planning.
The paper introduces RoboPhys-3D, a benchmark for Embodied World Models (EWMs) that distinguishes itself by grounding evaluation in 3D reconstruction. The core methodological innovation is the "reconstruction-matched protocol," where both generated and ground-truth videos are processed through the same 3D reconstruction pipeline (e.g., VGGT, 4DGS). This allows the authors to disentangle errors caused by the video generation model from errors introduced by the 3D reconstruction process itself, a significant confound in previous benchmarks. The evaluation framework is extensive, comprising 50 metrics organized into four hierarchical levels: pixel-level fidelity, 3D geometry consistency, state-level understanding, and task-level completeness. The introduction of "RoboPhyscore," a compact score derived from metrics strongly correlated with task success, is a practical contribution aimed at simplifying evaluation while maintaining alignment with human judgment and execution success.
The experiments are rigorous and comprehensive. The dataset covers 50 manipulation tasks across four regimes with 5,000 episodes and 25,000 multi-view videos. The authors evaluate four representative video world models (Cosmos 3, Wan 2.2, CogVideoX, RoboDreamer) and multiple reconstruction methods. Key findings include that Cosmos 3 achieves the highest RoboPhyscore (0.6330), but state- and execution-grounded metrics reveal substantial failures that perceptual metrics miss. The correlation analysis shows strong agreement between RoboPhyscore and human evaluation (Pearson r = 0.9761), validating the metric selection. The ablation on prompt specificity and IDM choice further demonstrates the sensitivity of the benchmark to conditioning and action decoding, providing actionable insights for practitioners.
The paper provides detailed supplementary information on dataset statistics, model configurations, and the specific prompts used for captioning. The normalization strategies for the 50 metrics are explicitly defined, including the bounds used for affine mapping. The human study protocol is well-documented, including participant recruitment, interface details, and aggregation methods. However, the code and dataset are not explicitly linked in the provided text (no URLs found), which may hinder immediate reproducibility unless released separately. The reliance on specific simulator (RoboTwin 2.0) and reconstruction pipelines requires access to these tools.
The benchmark is heavily dependent on the quality of the 3D reconstruction pipeline; if the reconstruction method fails, the evaluation metrics may be skewed, although the paper attempts to mitigate this by comparing reconstruction-induced vs. generation-induced errors. The evaluation is limited to the RoboTwin 2.0 simulator environment, which may not fully generalize to real-world robotic manipulation with different dynamics or sensor noise. The use of VLMs (Qwen3-VL) for some metrics introduces potential biases inherent to the VLM's training data. The computational cost of running 50 metrics across multiple models and reconstruction methods is likely high, potentially limiting adoption for rapid prototyping.
This paper addresses a critical gap in the evaluation of video world models for robotics. By providing a 3D-grounded protocol, it enables more accurate assessment of whether generated videos represent physically plausible and executable states. The findings that perceptual metrics often fail to capture execution-level failures will likely influence how the community designs future EWM benchmarks. The high correlation with human judgment suggests that RoboPhyscore could become a standard proxy for human evaluation in this domain. The work bridges the gap between computer vision (video generation) and robotics (action planning), promoting more integrated evaluation frameworks. The paper presents a rigorous 3D-grounded benchmark for evaluating embodied world models, effectively disentangling generation errors from reconstruction artifacts. By introducing a comprehensive metric suite and a task-aligned score (RoboPhyscore) that strongly correlates with human judgment and execution success, it provides a valuable tool for advancing the reliability of video-based robotic planning.
Caching is widely used across the system stack to improve performance and efficiency, with eviction algorithms at its core. Existing cache eviction policies fall into two broad categories: static heuristics (e.g., 2Q, S3-FIFO) and smart algorithms (e.g., ARC, LRB). Smart caches can adapt to workloads and have the potential to achieve higher efficiency and robustness than static heuristics. However, we find that existing smart caches suffer from objective mismatches and instability. We introduce Learning-Augmented Heuristics (LAH), a framework that learns the cache-level parameters of static heuristics. By decoupling the data and control planes, LAH supports simple, high-speed data reads and writes on the data plane, while performing occasional asynchronous learning on the control plane using cache-level features. We demonstrate the effectiveness of LAH through S4-FIFO, a Smart S3-FIFO cache eviction algorithm. We pre-train a single model on 4,140 production traces and embed it in S4-FIFO to learn optimal cache parameters. On 1,035 evaluation traces, S4-FIFO improves the mean efficiency by 26% compared to S3-FIFO and by 8% compared to 3L-Cache, the best state-of-the-art algorithm. S4-FIFO is also robust---increasing miss ratio over FIFO by 0.8% on the worst trace, whereas 3L-Cache increases FIFO's miss ratio by 8.8%. Finally, S4-FIFO's decisions are also interpretable: a language model can provide a rationale for why a particular configuration was chosen.
Primary: Harvard University
All Institutions: Harvard University, University of Illinois Urbana-Champaign (UIUC), University of Chicago, Institut Teknologi Bandung, Meta
The paper introduces a robust and interpretable framework for learning-augmented cache eviction that significantly outperforms state-of-the-art methods in both efficiency and worst-case stability. By decoupling the data and control planes, it enables the practical deployment of ML in latency-sensitive systems, offering a compelling alternative to complex adaptive algorithms.
The paper proposes the Learning-Augmented Heuristics (LAH) framework, which addresses the trade-off between the simplicity of static cache eviction heuristics (like S3-FIFO) and the adaptability of smart algorithms (like ARC). The core innovation is the decoupling of the data plane (fast, simple heuristic execution) from the control plane (asynchronous learning). Instead of learning complex per-item policies, the model learns to tune a small set of semantic parameters (knobs) of a static heuristic based on cache-level features. This approach is clever because it preserves the low-latency guarantees of static heuristics while gaining the adaptivity of ML. The instantiation, S4-FIFO, uses a pre-trained model on 4,140 production traces to predict optimal parameter configurations. The use of a Language Model (LM) for interpretability is a novel addition, allowing the system to explain *why* a specific configuration was chosen, which is rare in systems ML literature.
The evaluation is extensive, utilizing 1,035 evaluation traces. The results show a 26% improvement in mean efficiency over S3-FIFO and an 8% improvement over 3L-Cache (state-of-the-art). Crucially, the paper demonstrates robustness: S4-FIFO increases the miss ratio over FIFO by only 0.8% on the worst trace, whereas 3L-Cache increases it by 8.8%. This robustness metric is critical for production systems where worst-case performance matters as much as average performance. The throughput matches that of pure heuristics, validating the data/control plane decoupling.
The paper mentions pre-training on 4,140 production traces and evaluation on 1,035 traces. While the specific dataset names are not listed in the provided text snippet, the scale suggests the use of standard public benchmarks (e.g., Web, Cloud, or specific industry datasets often shared in OSDI papers). The lack of explicit code links in the provided text is a minor negative, but the detailed description of the framework and the standard nature of the baselines (S3-FIFO, ARC, 3L-Cache) make reproduction feasible for a skilled systems researcher.
The primary limitation is the dependency on the quality and diversity of the pre-training traces. If the production traces do not cover the distribution of the target workload, the learned parameters may be suboptimal. Additionally, the "interpretability" via LM is a post-hoc explanation rather than an intrinsic property of the decision process, which may limit its utility in safety-critical contexts. The framework is specific to parameter tuning of heuristics and may not generalize to entirely new eviction logic.
This work bridges the gap between ML and high-performance systems by showing that ML does not need to replace heuristics but can augment them. This paradigm is likely to be adopted in other system components (e.g., scheduling, memory management) where latency is critical. The robustness results are particularly significant for cloud providers and data centers, where cache efficiency directly impacts cost and performance. The paper introduces a robust and interpretable framework for learning-augmented cache eviction that significantly outperforms state-of-the-art methods in both efficiency and worst-case stability. By decoupling the data and control planes, it enables the practical deployment of ML in latency-sensitive systems, offering a compelling alternative to complex adaptive algorithms.