🎯 Top Personalized Recommendations
Universit Roma Tre, Ban
Why we think this paper is great for you:
This paper offers a compelling application of advanced learning techniques to real-world resource management, directly aligning with your interest in practical deep learning solutions. It demonstrates how complex systems can be optimized using these methods.
This paper offers a compelling application of advanced learning techniques to real-world resource management, directly aligning with your interest in practical deep learning solutions. It demonstrates how complex systems can be optimized using these methods.
Abstract
We introduce GasRL, a simulator that couples a calibrated representation of the natural gas market with a model of storage-operator policies trained with deep reinforcement learning (RL). We use it to analyse how optimal stockpile management affects equilibrium prices and the dynamics of demand and supply. We test various RL algorithms and find that Soft Actor Critic (SAC) exhibits superior performance in the GasRL environment: multiple objectives of storage operators - including profitability, robust market clearing and price stabilisation - are successfully achieved. Moreover, the equilibrium price dynamics induced by SAC-derived optimal policies have characteristics, such as volatility and seasonality, that closely match those of real-world prices. Remarkably, this adherence to the historical distribution of prices is obtained without explicitly calibrating the model to price data. We show how the simulator can be used to assess the effects of EU-mandated minimum storage thresholds. We find that such thresholds have a positive effect on market resilience against unanticipated shifts in the distribution of supply shocks. For example, with unusually large shocks, market disruptions are averted more often if a threshold is in place.
We introduce GasRL, a simulator that couples a calibrated representation of the natural gas market with a model of storage-operator policies trained with deep reinforcement learning (RL). We use it to analyse how optimal stockpile management affects equilibrium prices and the dynamics of demand and supply. We test various RL algorithms and find that Soft Actor Critic (SAC) exhibits superior performance in the GasRL environment: multiple objectives of storage operators - including profitability, robust market clearing and price stabilisation - are successfully achieved. Moreover, the equilibrium price dynamics induced by SAC-derived optimal policies have characteristics, such as volatility and seasonality, that closely match those of real-world prices. Remarkably, this adherence to the historical distribution of prices is obtained without explicitly calibrating the model to price data. We show how the simulator can be used to assess the effects of EU-mandated minimum storage thresholds. We find that such thresholds have a positive effect on market resilience against unanticipated shifts in the distribution of supply shocks. For example, with unusually large shocks, market disruptions are averted more often if a threshold is in place.
AI Summary
- The GasRL simulator effectively couples a calibrated natural gas market representation with a deep reinforcement learning (RL) model of a monopolistic storage operator, providing a novel tool for market and regulatory analysis. [3]
- The Soft Actor Critic (SAC) algorithm demonstrates superior performance and learning stability in the GasRL environment compared to other state-of-the-art RL schemes, robustly achieving high rewards and multi-objective optimization. [3]
- GasRL: A simulator that integrates a calibrated stochastic model of the natural gas market with a deep reinforcement learning agent representing a monopolistic storage operator. [3]
- Storage Operator (RL Agent): A price-setting economic agent with market power, trained using deep reinforcement learning to optimize gas storage operations for profitability, market clearing, price stability, and regulatory compliance. [3]
- Soft Actor Critic (SAC): An off-policy maximum entropy deep reinforcement learning algorithm, identified as the most effective and stable for training the GasRL storage operator agent. [3]
- SAC-derived optimal policies endogenously generate realistic equilibrium price dynamics, including volatility and seasonality, that closely match real-world prices without explicit calibration to historical price data. [2]
- The RL agent successfully balances multiple objectives: maximizing profitability, ensuring robust market clearing (eliminating failures), stabilizing prices (minimizing volatility), and complying with regulatory refilling mandates. [2]
- EU-mandated minimum storage thresholds (e.g., 83% by November) significantly improve market resilience against unanticipated shifts in supply shock volatility, leading to fewer market disruptions during adverse events. [2]
- Implementing storage thresholds incurs trade-offs: while enhancing market resilience, they result in reduced profitability for the storage operator and a slight increase in price volatility, with no measurable impact on the average price level. [2]
- Market Environment (GasRL component): A component of GasRL that reproduces the main characteristics of a national gas market (e.g., Italian market), including sticky demand/supply, seasonal variations, and persistent stochastic shocks. [2]
US Army DEVCOM Army Res
Why we think this paper is great for you:
You will find this paper highly relevant as it explores the application of deep learning to optimize complex network routing, showcasing another practical use case for your preferred learning paradigms. It addresses challenges in heterogeneous environments.
You will find this paper highly relevant as it explores the application of deep learning to optimize complex network routing, showcasing another practical use case for your preferred learning paradigms. It addresses challenges in heterogeneous environments.
Abstract
Due to the rapid growth of heterogeneous wireless networks (HWNs), where devices with diverse communication technologies coexist, there is increasing demand for efficient and adaptive multi-hop routing with multiple data flows. Traditional routing methods, designed for homogeneous environments, fail to address the complexity introduced by links consisting of multiple technologies, frequency-dependent fading, and dynamic topology changes. In this paper, we propose a deep reinforcement learning (DRL)-based routing framework using deep Q-networks (DQN) to establish routes between multiple source-destination pairs in HWNs by enabling each node to jointly select a communication technology, a subband, and a next hop relay that maximizes the rate of the route. Our approach incorporates channel and interference-aware neighbor selection approaches to improve decision-making beyond conventional distance-based heuristics. We further evaluate the robustness and generalizability of the proposed method under varying network dynamics, including node mobility, changes in node density, and the number of data flows. Simulation results demonstrate that our DRL-based routing framework significantly enhances scalability, adaptability, and end-to-end throughput in complex HWN scenarios.
Due to the rapid growth of heterogeneous wireless networks (HWNs), where devices with diverse communication technologies coexist, there is increasing demand for efficient and adaptive multi-hop routing with multiple data flows. Traditional routing methods, designed for homogeneous environments, fail to address the complexity introduced by links consisting of multiple technologies, frequency-dependent fading, and dynamic topology changes. In this paper, we propose a deep reinforcement learning (DRL)-based routing framework using deep Q-networks (DQN) to establish routes between multiple source-destination pairs in HWNs by enabling each node to jointly select a communication technology, a subband, and a next hop relay that maximizes the rate of the route. Our approach incorporates channel and interference-aware neighbor selection approaches to improve decision-making beyond conventional distance-based heuristics. We further evaluate the robustness and generalizability of the proposed method under varying network dynamics, including node mobility, changes in node density, and the number of data flows. Simulation results demonstrate that our DRL-based routing framework significantly enhances scalability, adaptability, and end-to-end throughput in complex HWN scenarios.
Shanghai AI Laboratory
Why we think this paper is great for you:
This work directly addresses the evaluation of sophisticated agent capabilities, which is central to your focus on agentic systems. It provides a benchmark for understanding how agents reason and operate tools.
This work directly addresses the evaluation of sophisticated agent capabilities, which is central to your focus on agentic systems. It provides a benchmark for understanding how agents reason and operate tools.
Abstract
The frontier of visual reasoning is shifting toward models like OpenAI o3, which can intelligently create and operate tools to transform images for problem-solving, also known as thinking-\textit{with}-images in chain-of-thought. Yet existing benchmarks fail to fully capture this advanced capability. Even Visual Search, the most common benchmark for current thinking-\textit{with}-images methods, tests only basic operations such as localization and cropping, offering little insight into more complex, dynamic, and tool-dependent reasoning. We introduce \textbf{TIR-Bench}, a comprehensive benchmark for evaluating agentic thinking-with-images across 13 diverse tasks, each requiring novel tool use for image processing and manipulation in chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from leading open-sourced and proprietary models to those with explicit tool-use augmentation. Results show that TIR-Bench is universally challenging, and strong performance requires genuine thinking-with-images capabilities. Finally, we present a pilot study comparing direct versus agentic fine-tuning.
The frontier of visual reasoning is shifting toward models like OpenAI o3, which can intelligently create and operate tools to transform images for problem-solving, also known as thinking-\textit{with}-images in chain-of-thought. Yet existing benchmarks fail to fully capture this advanced capability. Even Visual Search, the most common benchmark for current thinking-\textit{with}-images methods, tests only basic operations such as localization and cropping, offering little insight into more complex, dynamic, and tool-dependent reasoning. We introduce \textbf{TIR-Bench}, a comprehensive benchmark for evaluating agentic thinking-with-images across 13 diverse tasks, each requiring novel tool use for image processing and manipulation in chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from leading open-sourced and proprietary models to those with explicit tool-use augmentation. Results show that TIR-Bench is universally challenging, and strong performance requires genuine thinking-with-images capabilities. Finally, we present a pilot study comparing direct versus agentic fine-tuning.
Ariel University, Israel
Why we think this paper is great for you:
This paper delves into the critical security aspects of multi-agent communication, a vital area for anyone interested in the robust deployment of agentic AI. It offers a comparative analysis of protocols in these systems.
This paper delves into the critical security aspects of multi-agent communication, a vital area for anyone interested in the robust deployment of agentic AI. It offers a comparative analysis of protocols in these systems.
Abstract
Multi-agent systems (MAS) powered by artificial intelligence (AI) are increasingly foundational to complex, distributed workflows. Yet, the security of their underlying communication protocols remains critically under-examined. This paper presents the first empirical, comparative security analysis of the official CORAL implementation and a high-fidelity, SDK-based ACP implementation, benchmarked against a literature-based evaluation of A2A. Using a 14 point vulnerability taxonomy, we systematically assess their defenses across authentication, authorization, integrity, confidentiality, and availability. Our results reveal a pronounced security dichotomy: CORAL exhibits a robust architectural design, particularly in its transport-layer message validation and session isolation, but suffers from critical implementation-level vulnerabilities, including authentication and authorization failures at its SSE gateway. Conversely, ACP's architectural flexibility, most notably its optional JWS enforcement, translates into high-impact integrity and confidentiality flaws. We contextualize these findings within current industry trends, highlighting that existing protocols remain insufficiently secure. As a path forward, we recommend a hybrid approach that combines CORAL's integrated architecture with ACP's mandatory per-message integrity guarantees, laying the groundwork for resilient, next-generation agent communications.
Multi-agent systems (MAS) powered by artificial intelligence (AI) are increasingly foundational to complex, distributed workflows. Yet, the security of their underlying communication protocols remains critically under-examined. This paper presents the first empirical, comparative security analysis of the official CORAL implementation and a high-fidelity, SDK-based ACP implementation, benchmarked against a literature-based evaluation of A2A. Using a 14 point vulnerability taxonomy, we systematically assess their defenses across authentication, authorization, integrity, confidentiality, and availability. Our results reveal a pronounced security dichotomy: CORAL exhibits a robust architectural design, particularly in its transport-layer message validation and session isolation, but suffers from critical implementation-level vulnerabilities, including authentication and authorization failures at its SSE gateway. Conversely, ACP's architectural flexibility, most notably its optional JWS enforcement, translates into high-impact integrity and confidentiality flaws. We contextualize these findings within current industry trends, highlighting that existing protocols remain insufficiently secure. As a path forward, we recommend a hybrid approach that combines CORAL's integrated architecture with ACP's mandatory per-message integrity guarantees, laying the groundwork for resilient, next-generation agent communications.
Fudan University, MAP
Why we think this paper is great for you:
This framework tackles significant challenges in training robust learning models, offering insights into self-improving mechanisms that are crucial for advancing the field. It explores how to overcome issues like policy over-specialization.
This framework tackles significant challenges in training robust learning models, offering insights into self-improving mechanisms that are crucial for advancing the field. It explores how to overcome issues like policy over-specialization.
Abstract
While Reinforcement Learning for Verifiable Rewards (RLVR) is powerful for training large reasoning models, its training dynamics harbor a critical challenge: RL overfitting, where models gain training rewards but lose generalization. Our analysis reveals this is driven by policy over-specialization and catastrophic forgetting of diverse solutions generated during training. Standard optimization discards this valuable inter-step policy diversity. To address this, we introduce RLoop, a self-improving framework built on iterative policy initialization. RLoop transforms the standard training process into a virtuous cycle: it first uses RL to explore the solution space from a given policy, then filters the successful trajectories to create an expert dataset. This dataset is used via Rejection-sampling Fine-Tuning (RFT) to refine the initial policy, creating a superior starting point for the next iteration. This loop of exploration and exploitation via iterative re-initialization effectively converts transient policy variations into robust performance gains. Our experiments show RLoop mitigates forgetting and substantially improves generalization, boosting average accuracy by 9% and pass@32 by over 15% compared to vanilla RL.
While Reinforcement Learning for Verifiable Rewards (RLVR) is powerful for training large reasoning models, its training dynamics harbor a critical challenge: RL overfitting, where models gain training rewards but lose generalization. Our analysis reveals this is driven by policy over-specialization and catastrophic forgetting of diverse solutions generated during training. Standard optimization discards this valuable inter-step policy diversity. To address this, we introduce RLoop, a self-improving framework built on iterative policy initialization. RLoop transforms the standard training process into a virtuous cycle: it first uses RL to explore the solution space from a given policy, then filters the successful trajectories to create an expert dataset. This dataset is used via Rejection-sampling Fine-Tuning (RFT) to refine the initial policy, creating a superior starting point for the next iteration. This loop of exploration and exploitation via iterative re-initialization effectively converts transient policy variations into robust performance gains. Our experiments show RLoop mitigates forgetting and substantially improves generalization, boosting average accuracy by 9% and pass@32 by over 15% compared to vanilla RL.
University of Virginia
Why we think this paper is great for you:
For a deeper understanding of the foundational principles, this paper formalizes the theory behind core learning algorithms. It provides rigorous proofs for the convergence of fundamental methods.
For a deeper understanding of the foundational principles, this paper formalizes the theory behind core learning algorithms. It provides rigorous proofs for the convergence of fundamental methods.
Abstract
In this paper, we formalize the almost sure convergence of $Q$-learning and linear temporal difference (TD) learning with Markovian samples using the Lean 4 theorem prover based on the Mathlib library. $Q$-learning and linear TD are among the earliest and most influential reinforcement learning (RL) algorithms. The investigation of their convergence properties is not only a major research topic during the early development of the RL field but also receives increasing attention nowadays. This paper formally verifies their almost sure convergence in a unified framework based on the Robbins-Siegmund theorem. The framework developed in this work can be easily extended to convergence rates and other modes of convergence. This work thus makes an important step towards fully formalizing convergent RL results. The code is available at https://github.com/ShangtongZhang/rl-theory-in-lean.
In this paper, we formalize the almost sure convergence of $Q$-learning and linear temporal difference (TD) learning with Markovian samples using the Lean 4 theorem prover based on the Mathlib library. $Q$-learning and linear TD are among the earliest and most influential reinforcement learning (RL) algorithms. The investigation of their convergence properties is not only a major research topic during the early development of the RL field but also receives increasing attention nowadays. This paper formally verifies their almost sure convergence in a unified framework based on the Robbins-Siegmund theorem. The framework developed in this work can be easily extended to convergence rates and other modes of convergence. This work thus makes an important step towards fully formalizing convergent RL results. The code is available at https://github.com/ShangtongZhang/rl-theory-in-lean.
Universit Roma Tre, Ban
Why we think this paper is great for you:
This paper is a strong match due to its practical application of advanced learning methods to a complex real-world problem. It exemplifies the kind of impactful solutions you are interested in.
This paper is a strong match due to its practical application of advanced learning methods to a complex real-world problem. It exemplifies the kind of impactful solutions you are interested in.
Abstract
We introduce GasRL, a simulator that couples a calibrated representation of the natural gas market with a model of storage-operator policies trained with deep reinforcement learning (RL). We use it to analyse how optimal stockpile management affects equilibrium prices and the dynamics of demand and supply. We test various RL algorithms and find that Soft Actor Critic (SAC) exhibits superior performance in the GasRL environment: multiple objectives of storage operators - including profitability, robust market clearing and price stabilisation - are successfully achieved. Moreover, the equilibrium price dynamics induced by SAC-derived optimal policies have characteristics, such as volatility and seasonality, that closely match those of real-world prices. Remarkably, this adherence to the historical distribution of prices is obtained without explicitly calibrating the model to price data. We show how the simulator can be used to assess the effects of EU-mandated minimum storage thresholds. We find that such thresholds have a positive effect on market resilience against unanticipated shifts in the distribution of supply shocks. For example, with unusually large shocks, market disruptions are averted more often if a threshold is in place.
We introduce GasRL, a simulator that couples a calibrated representation of the natural gas market with a model of storage-operator policies trained with deep reinforcement learning (RL). We use it to analyse how optimal stockpile management affects equilibrium prices and the dynamics of demand and supply. We test various RL algorithms and find that Soft Actor Critic (SAC) exhibits superior performance in the GasRL environment: multiple objectives of storage operators - including profitability, robust market clearing and price stabilisation - are successfully achieved. Moreover, the equilibrium price dynamics induced by SAC-derived optimal policies have characteristics, such as volatility and seasonality, that closely match those of real-world prices. Remarkably, this adherence to the historical distribution of prices is obtained without explicitly calibrating the model to price data. We show how the simulator can be used to assess the effects of EU-mandated minimum storage thresholds. We find that such thresholds have a positive effect on market resilience against unanticipated shifts in the distribution of supply shocks. For example, with unusually large shocks, market disruptions are averted more often if a threshold is in place.
AI Summary
- The GasRL simulator effectively couples a calibrated natural gas market representation with a deep reinforcement learning (RL) model of a monopolistic storage operator, providing a novel tool for market and regulatory analysis. [3]
- The Soft Actor Critic (SAC) algorithm demonstrates superior performance and learning stability in the GasRL environment compared to other state-of-the-art RL schemes, robustly achieving high rewards and multi-objective optimization. [3]
- GasRL: A simulator that integrates a calibrated stochastic model of the natural gas market with a deep reinforcement learning agent representing a monopolistic storage operator. [3]
- Storage Operator (RL Agent): A price-setting economic agent with market power, trained using deep reinforcement learning to optimize gas storage operations for profitability, market clearing, price stability, and regulatory compliance. [3]
- Soft Actor Critic (SAC): An off-policy maximum entropy deep reinforcement learning algorithm, identified as the most effective and stable for training the GasRL storage operator agent. [3]
- SAC-derived optimal policies endogenously generate realistic equilibrium price dynamics, including volatility and seasonality, that closely match real-world prices without explicit calibration to historical price data. [2]
- The RL agent successfully balances multiple objectives: maximizing profitability, ensuring robust market clearing (eliminating failures), stabilizing prices (minimizing volatility), and complying with regulatory refilling mandates. [2]
- EU-mandated minimum storage thresholds (e.g., 83% by November) significantly improve market resilience against unanticipated shifts in supply shock volatility, leading to fewer market disruptions during adverse events. [2]
- Implementing storage thresholds incurs trade-offs: while enhancing market resilience, they result in reduced profitability for the storage operator and a slight increase in price volatility, with no measurable impact on the average price level. [2]
- Market Environment (GasRL component): A component of GasRL that reproduces the main characteristics of a national gas market (e.g., Italian market), including sticky demand/supply, seasonal variations, and persistent stochastic shocks. [2]
Help us improve your experience!
This project is on its early stages your feedback can be pivotal on the future of the project.
Let us know what you think about this week's papers and suggestions!
Give Feedback