Six October arXiv papers advance agent RL: GPPO, STEPGATE, SAG, LiRA, MoSDOT, CASTLE

A wave of agent RL preprints in mid-October

Between 5 and 13 October 2026, six preprints appeared on arXiv addressing distinct challenges in agent reinforcement learning. The papers span single-agent policy optimization, small-model agent design, long-horizon decision making, multi-agent constraint sharing, offline MARL distillation, and fully decentralized MARL. Together they represent a coordinated advance across the agent RL stack.

GPPO: Generalizing PPO through f-divergence

GPPO (Generalized Proximal Policy Optimization) (arXiv:2610.13015v1, 9 Oct) by Zhou et al. (Tsinghua AIR, ByteDance) identifies that PPO's clipping proxy is geometrically degenerate--it cannot adapt resistance to policy shifts, causing premature clipping. GPPO establishes a generalized f-divergence trust-region theory with monotonic improvement guarantees. Different f-divergences induce distinct geometries: KL recovers PPO, total variation yields rigid clipping, and chi-squared enables zero-curvature updates. The paper introduces novel estimator topologies for more faithful divergence estimation and shows GPPO unifies PPO, TRPO, and mirror-descent variants. Experiments on LLM reasoning tasks demonstrate improved exploration and sample efficiency.

STEPGATE: Uncertainty-aware step-level handoff for SLM agents

STEPGATE (arXiv:2610.07816v1, 6 Oct) by authors from Tsinghua and collaborators addresses the small-language-model (SLM) agent problem: SLMs are efficient locally but fail on structured tool errors. Existing routers select a model once per query; STEPGATE scores each local SLM action and selectively escalates challenging steps to a stronger model. On a 52-task BFCL-derived held-out split, Qwen2.5-1.5B/7B pair achieves 82.7% task success with 30.8% escalation, vs. 67.3% local-only and 75.4% random escalation (33.8% escalation). In multi-turn evaluation, STEPGATE reaches 69.0% trajectory success and 84.0% action success using only 30.0% cloud actions. The framework uses conformal prediction for calibrated uncertainty estimates and minimizes handoff context.

SAG: Gated critique for cost-aware long-horizon agents

SAG (Self-improving Agent with Gated critique) (arXiv:2610.07335v1, 5 Oct) tackles the token-cost/reliability trade-off in long-horizon LLM agents. Always-on critique is expensive; SAG introduces a lightweight gating mechanism that estimates critique utility via action-level ambiguity signals--global entropy and local top-2 margin over admissible actions--approximating Value of Information (VoI). The actor selectively invokes critique only when expected benefit justifies cost. Online bootstrapped self-improvement lets the actor internalize critic-assisted behaviors. On ALFWorld, SAG lifts task success from 24.6% (ReAct) to 78.4% at comparable token budget, a 3.1x normalized token efficiency gain. A 7B actor with 3B critic matches 14B actor without critique.

LiRA: Learning responsibility for shared multi-agent constraints

LiRA (Lagrangian Responsibility Allocation) (arXiv:2610.07491v1, 5 Oct) by Cao & Li (MIT, UT Austin) addresses shared cost budgets in multi-agent RL. A common Lagrange multiplier enforces the aggregate constraint but does not determine penalty allocation across agents. LiRA learns each agent's share of the multiplier by optimizing social welfare over a finite training horizon. The multiplier enforces the budget; responsibility shares redistribute influence without modifying rewards or constraints. For convex games, varying shares induces a smooth family of normalized generalized Nash equilibria. Across CityLearn, MABIM, Harvest, and MetaDrive (3-400 agents), LiRA improves average social welfare by up to 29% over uniform and agent-specific multiplier baselines, with costs within budget and inventory violations reduced.

MoSDOT: Preserving coordination modes in offline MARL distillation

MoSDOT (Mode-Support Semi-Discrete Optimal Transport) (arXiv:2610.10087v1, 12 Oct) addresses a failure mode in offline MARL: standard flow-based teachers pair noise with replay targets independently, so nearby noise samples route toward conflicting coordination modes. The teacher then produces samples between valid modes, and distillation loss regresses each actor onto the conditional mean, propagating the error. MoSDOT summarizes multimodal replay into a finite mode support with prescribed capacities and uses conditional semi-discrete optimal transport to assign each noise sample to a single mode before teacher training. A shared-randomness variant isolates the strict-product execution gap. On controlled diagnostics and offline MARL benchmarks, MoSDOT improves endpoint quality and routing consistency, particularly on datasets with multimodal joint behavior.

CASTLE: Counterfactual world models for decentralized MARL

CASTLE (Counterfactual Action-conditioned Semantic Tokens for Local Execution) (arXiv:2610.07704v1, 6 Oct) by Martinez et al. (Fordham, CityU HK, IBM Research) targets fully decentralized MARL where each agent acts on local information only. Scalar rewards are ambiguous--poor returns may stem from ineffective ego action, incompatible teammate response, or effective opponent response. CASTLE introduces two frozen world models queried online: a Local Dynamics World Model (pre-trained on local trajectories) summarizing local dynamics and partial observability, and a Semantic-Social World Model predicting short-horizon task and social consequences for each candidate ego action, trained from counterfactual simulator rollouts. During online execution, both models provide in-context guidance to an independent PPO policy. Across 30 seeds on Tag, Spread, and Adversary (multi-particle environments), CASTLE achieves the highest mean final score, exceeding the strongest baseline by 10.67, 6.46, and 0.33 normalized points respectively.

Complementary but independent advances

The six papers address different layers of the agent RL stack and do not form a single unified framework. GPPO operates at the policy optimization level; STEPGATE at the model routing level; SAG at the critique invocation level; LiRA at the multi-agent constraint level; MoSDOT at the offline distillation level; CASTLE at the decentralized guidance level. No paper claims to solve the others' problems. Experimental scopes differ: GPPO on LLM reasoning; STEPGATE on BFCL tool use; SAG on ALFWorld/WebShop/SciWorld; LiRA on CityLearn/MABIM/Harvest/MetaDrive; MoSDOT on offline MARL benchmarks; CASTLE on multi-particle environments. No cross-paper reproduction or integration is reported.

Researchers and practitioners should evaluate relevant advances for their stack

Teams working on LLM policy optimization should test GPPO's f-divergence family. SLM agent builders should evaluate STEPGATE's step-level routing against query-level routing. Long-horizon agent developers should benchmark SAG's gated critique against always-on and no-critique baselines. Multi-agent constraint problems should consider LiRA's responsibility allocation. Offline MARL researchers should compare MoSDOT's mode-preserving distillation against standard flow-based teachers. Decentralized MARL practitioners should test CASTLE's counterfactual guidance against independent PPO baselines.