Why we think this paper is great for you:
This paper directly addresses Reinforcement Learning, focusing on how reward functions guide agent learning. It's highly relevant to your core interest in RL and agent development.

Abstract
Reinforcement learning (RL) has been recognized as a powerful tool for robot control tasks. RL typically employs reward functions to define task objectives and guide agent learning. However, since the reward function serves the dual purpose of defining the optimal goal and guiding learning, it is challenging to design the reward function manually, which often results in a suboptimal task representation. To tackle the reward design challenge in RL, inspired by the satisficing theory, we propose a Test-driven Reinforcement Learning (TdRL) framework. In the TdRL framework, multiple test functions are used to represent the task objective rather than a single reward function. Test functions can be categorized as pass-fail tests and indicative tests, each dedicated to defining the optimal objective and guiding the learning process, respectively, thereby making defining tasks easier. Building upon such a task definition, we first prove that if a trajectory return function assigns higher returns to trajectories closer to the optimal trajectory set, maximum entropy policy optimization based on this return function will yield a policy that is closer to the optimal policy set. Then, we introduce a lexicographic heuristic approach to compare the relative distance relationship between trajectories and the optimal trajectory set for learning the trajectory return function. Furthermore, we develop an algorithm implementation of TdRL. Experimental results on the DeepMind Control Suite benchmark demonstrate that TdRL matches or outperforms handcrafted reward methods in policy training, with greater design simplicity and inherent support for multi-objective optimization. We argue that TdRL offers a novel perspective for representing task objectives, which could be helpful in addressing the reward design challenges in RL applications.

AI Summary

• TdRL replaces a single reward function with multiple test functions (pass-fail and indicative) to define task objectives, simplifying multi-objective RL design by eliminating manual weight tuning. [3]
• The framework provides theoretical guarantees: if a trajectory return function assigns higher returns to trajectories closer to the optimal set, maximum entropy policy optimization will yield a policy closer to the optimal policy set. [3]
• A lexicographic heuristic is introduced to compare trajectory distances to the optimal set, enabling the learning of the trajectory return function without direct knowledge of the optimal trajectory set. [3]
• TdRL's reward learning process decomposes a learned trajectory return function into a state-action reward function, mitigating designer-induced bias by operating at the trajectory level rather than state-action pairs. [3]
• Experimental results demonstrate TdRL matches or outperforms handcrafted reward methods on DeepMind Control Suite tasks, showcasing comparable performance with greater design simplicity and inherent multi-objective support. [3]
• Robustness in return function learning is achieved by balancing a distance-based cross-entropy loss with a penalty term (MSE loss), using techniques like gradient norm rescaling or early stopping to prevent numerical instability. [3]
• Test-driven Reinforcement Learning (TdRL): A framework that uses multiple test functions (pass-fail and indicative) instead of a single reward function to represent task objectives, aiming for 'satisficing' solutions. [3]
• Pass-fail tests (z_pf): Binary functions that evaluate whether a trajectory meets required criteria, dedicated to defining the optimal objective. [3]
• Indicative tests (z_ind): Real-valued functions that quantify a trajectory's performance in a specific metric, providing informative guiding signals for policy learning. [3]
• The method effectively achieves 'satisficing' solutions across multiple objectives, ensuring all predefined criteria are met, rather than optimizing a single metric to its maximum at the expense of others. [2]

Why we think this paper is great for you:
This paper explores offline reinforcement learning, leveraging diffusion models which are a form of advanced machine learning. It aligns well with your interest in combining advanced models with RL.

Abstract
In offline reinforcement learning, value overestimation caused by out-of-distribution (OOD) actions significantly limits policy performance. Recently, diffusion models have been leveraged for their strong distribution-matching capabilities, enforcing conservatism through behavior policy constraints. However, existing methods often apply indiscriminate regularization to redundant actions in low-quality datasets, resulting in excessive conservatism and an imbalance between the expressiveness and efficiency of diffusion modeling. To address these issues, we propose DIffusion policies with Value-conditional Optimization (DIVO), a novel approach that leverages diffusion models to generate high-quality, broadly covered in-distribution state-action samples while facilitating efficient policy improvement. Specifically, DIVO introduces a binary-weighted mechanism that utilizes the advantage values of actions in the offline dataset to guide diffusion model training. This enables a more precise alignment with the dataset's distribution while selectively expanding the boundaries of high-advantage actions. During policy improvement, DIVO dynamically filters high-return-potential actions from the diffusion model, effectively guiding the learned policy toward better performance. This approach achieves a critical balance between conservatism and explorability in offline RL. We evaluate DIVO on the D4RL benchmark and compare it against state-of-the-art baselines. Empirical results demonstrate that DIVO achieves superior performance, delivering significant improvements in average returns across locomotion tasks and outperforming existing methods in the challenging AntMaze domain, where sparse rewards pose a major difficulty.

Why we think this paper is great for you:
This paper discusses Reinforcement Learning with Verifiable Rewards (RLVR) in the context of large language models. It connects your interest in RL with advanced model applications and reasoning.

Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) reliably improves the reasoning performance of large language models, yet it appears to modify only a small fraction of parameters. We revisit this paradox and show that sparsity is a surface artifact of a model-conditioned optimization bias: for a fixed pretrained model, updates consistently localize to preferred parameter regions, highly consistent across runs and largely invariant to datasets and RL recipes. We mechanistically explain these dynamics with a Three-Gate Theory: Gate I (KL Anchor) imposes a KL-constrained update; Gate II (Model Geometry) steers the step off principal directions into low-curvature, spectrum-preserving subspaces; and Gate III (Precision) hides micro-updates in non-preferred regions, making the off-principal bias appear as sparsity. We then validate this theory and, for the first time, provide a parameter-level characterization of RLVR's learning dynamics: RLVR learns off principal directions in weight space, achieving gains via minimal spectral drift, reduced principal-subspace rotation, and off-principal update alignment. In contrast, SFT targets principal weights, distorts the spectrum, and even lags RLVR. Together, these results provide the first parameter-space account of RLVR's training dynamics, revealing clear regularities in how parameters evolve. Crucially, we show that RL operates in a distinct optimization regime from SFT, so directly adapting SFT-era parameter-efficient fine-tuning (PEFT) methods can be flawed, as evidenced by our case studies on advanced sparse fine-tuning and LoRA variants. We hope this work charts a path toward a white-box understanding of RLVR and the design of geometry-aware, RLVR-native learning algorithms, rather than repurposed SFT-era heuristics.

Why we think this paper is great for you:
This paper focuses on autonomous agents and their evolution, which directly aligns with your interest in agent systems. It explores how agents can reason and execute complex tasks.

Abstract
Autonomous agents powered by large language models (LLMs) have the potential to significantly enhance human productivity by reasoning, using tools, and executing complex tasks in diverse environments. However, current approaches to developing such agents remain costly and inefficient, as they typically require manually constructed task datasets and reinforcement learning (RL) pipelines with extensive random exploration. These limitations lead to prohibitively high data-construction costs, low exploration efficiency, and poor sample utilization. To address these challenges, we present AgentEvolver, a self-evolving agent system that leverages the semantic understanding and reasoning capabilities of LLMs to drive autonomous agent learning. AgentEvolver introduces three synergistic mechanisms: (i) self-questioning, which enables curiosity-driven task generation in novel environments, reducing dependence on handcrafted datasets; (ii) self-navigating, which improves exploration efficiency through experience reuse and hybrid policy guidance; and (iii) self-attributing, which enhances sample efficiency by assigning differentiated rewards to trajectory states and actions based on their contribution. By integrating these mechanisms into a unified framework, AgentEvolver enables scalable, cost-effective, and continual improvement of agent capabilities. Preliminary experiments indicate that AgentEvolver achieves more efficient exploration, better sample utilization, and faster adaptation compared to traditional RL-based baselines.

Why we think this paper is great for you:
This paper introduces a deep neural-operator framework for probabilistic models. While it involves deep learning, it's less directly connected to reinforcement learning or agentic systems compared to the others.

Abstract
We propose a deep neural-operator framework for a general class of probability models. Under global Lipschitz conditions on the operator over the entire Euclidean space-and for a broad class of probabilistic models-we establish a universal approximation theorem with explicit network-size bounds for the proposed architecture. The underlying stochastic processes are required only to satisfy integrability and general tail-probability conditions. We verify these assumptions for both European and American option-pricing problems within the forward-backward SDE (FBSDE) framework, which in turn covers a broad class of operators arising from parabolic PDEs, with or without free boundaries. Finally, we present a numerical example for a basket of American options, demonstrating that the learned model produces optimal stopping boundaries for new strike prices without retraining.

Why we think this paper is great for you:
This paper directly addresses Reinforcement Learning, focusing on how reward functions guide agent learning. It's highly relevant to your core interest in RL and agent development.

Abstract
Reinforcement learning (RL) has been recognized as a powerful tool for robot control tasks. RL typically employs reward functions to define task objectives and guide agent learning. However, since the reward function serves the dual purpose of defining the optimal goal and guiding learning, it is challenging to design the reward function manually, which often results in a suboptimal task representation. To tackle the reward design challenge in RL, inspired by the satisficing theory, we propose a Test-driven Reinforcement Learning (TdRL) framework. In the TdRL framework, multiple test functions are used to represent the task objective rather than a single reward function. Test functions can be categorized as pass-fail tests and indicative tests, each dedicated to defining the optimal objective and guiding the learning process, respectively, thereby making defining tasks easier. Building upon such a task definition, we first prove that if a trajectory return function assigns higher returns to trajectories closer to the optimal trajectory set, maximum entropy policy optimization based on this return function will yield a policy that is closer to the optimal policy set. Then, we introduce a lexicographic heuristic approach to compare the relative distance relationship between trajectories and the optimal trajectory set for learning the trajectory return function. Furthermore, we develop an algorithm implementation of TdRL. Experimental results on the DeepMind Control Suite benchmark demonstrate that TdRL matches or outperforms handcrafted reward methods in policy training, with greater design simplicity and inherent support for multi-objective optimization. We argue that TdRL offers a novel perspective for representing task objectives, which could be helpful in addressing the reward design challenges in RL applications.

AI Summary

• TdRL replaces a single reward function with multiple test functions (pass-fail and indicative) to define task objectives, simplifying multi-objective RL design by eliminating manual weight tuning. [3]
• The framework provides theoretical guarantees: if a trajectory return function assigns higher returns to trajectories closer to the optimal set, maximum entropy policy optimization will yield a policy closer to the optimal policy set. [3]
• A lexicographic heuristic is introduced to compare trajectory distances to the optimal set, enabling the learning of the trajectory return function without direct knowledge of the optimal trajectory set. [3]
• TdRL's reward learning process decomposes a learned trajectory return function into a state-action reward function, mitigating designer-induced bias by operating at the trajectory level rather than state-action pairs. [3]
• Experimental results demonstrate TdRL matches or outperforms handcrafted reward methods on DeepMind Control Suite tasks, showcasing comparable performance with greater design simplicity and inherent multi-objective support. [3]
• Robustness in return function learning is achieved by balancing a distance-based cross-entropy loss with a penalty term (MSE loss), using techniques like gradient norm rescaling or early stopping to prevent numerical instability. [3]
• Test-driven Reinforcement Learning (TdRL): A framework that uses multiple test functions (pass-fail and indicative) instead of a single reward function to represent task objectives, aiming for 'satisficing' solutions. [3]
• Pass-fail tests (z_pf): Binary functions that evaluate whether a trajectory meets required criteria, dedicated to defining the optimal objective. [3]
• Indicative tests (z_ind): Real-valued functions that quantify a trajectory's performance in a specific metric, providing informative guiding signals for policy learning. [3]
• The method effectively achieves 'satisficing' solutions across multiple objectives, ensuring all predefined criteria are met, rather than optimizing a single metric to its maximum at the expense of others. [2]

Why we think this paper is great for you:
This paper explores offline reinforcement learning, leveraging diffusion models which are a form of advanced machine learning. It aligns well with your interest in combining advanced models with RL.

Abstract
In offline reinforcement learning, value overestimation caused by out-of-distribution (OOD) actions significantly limits policy performance. Recently, diffusion models have been leveraged for their strong distribution-matching capabilities, enforcing conservatism through behavior policy constraints. However, existing methods often apply indiscriminate regularization to redundant actions in low-quality datasets, resulting in excessive conservatism and an imbalance between the expressiveness and efficiency of diffusion modeling. To address these issues, we propose DIffusion policies with Value-conditional Optimization (DIVO), a novel approach that leverages diffusion models to generate high-quality, broadly covered in-distribution state-action samples while facilitating efficient policy improvement. Specifically, DIVO introduces a binary-weighted mechanism that utilizes the advantage values of actions in the offline dataset to guide diffusion model training. This enables a more precise alignment with the dataset's distribution while selectively expanding the boundaries of high-advantage actions. During policy improvement, DIVO dynamically filters high-return-potential actions from the diffusion model, effectively guiding the learned policy toward better performance. This approach achieves a critical balance between conservatism and explorability in offline RL. We evaluate DIVO on the D4RL benchmark and compare it against state-of-the-art baselines. Empirical results demonstrate that DIVO achieves superior performance, delivering significant improvements in average returns across locomotion tasks and outperforming existing methods in the challenging AntMaze domain, where sparse rewards pose a major difficulty.