Δτ-Bench drives agents across six evolving tool ecosystems while Steward maintains skills

Static tool suites freeze the map. Δτ-Bench redraws it six times while the agent is still mid-task. Qiao and colleagues at Zhejiang University and University College London posted arXiv:2610.34238 on 25 October 2026 with an executable benchmark that treats tool ecosystems as moving systems, then propose Steward as the operational layer that keeps skills from rotting when familiar tools vanish.

The paper's motivating picture is familiar to anyone who has shipped an MCP or plugin fleet: services independently evolve, documentation drifts, overlapping tools appear, and a skill that worked last month now calls a removed endpoint. Existing agent benches mostly freeze the tool surface for the duration of evaluation. Δτ-Bench makes that freeze the variable under test.

Six capability-preserving transforms reshape the tool frontier

Δτ-Bench models structural evolution of large tool ecosystems. Independently evolving services continually reshape how tasks can be accomplished: familiar tools may disappear or be replaced, while more efficient alternatives emerge amid overlapping functionality and noisy documentation. Against that moving capability frontier, the benchmark asks whether agents can discover efficient tool-use paths and revise accumulated procedural knowledge in time.

The authors generate six successive ecosystem states through capability-preserving transforms. The transforms keep tasks solvable in principle while changing which tools and paths remain valid. That design separates "the task became impossible" from "the agent clung to a stale skill." The suite comprises 1,588 validated multi-turn tasks across 42 domains.

Steward is the companion systems claim. It is an ecosystem manager that jointly manages tools and procedural knowledge through incremental skill maintenance and dynamic provisioning during task execution. Where prior skill-learning methods treat tools as a mostly static library, Steward treats provisioning and skill repair as first-class runtime work.

Strongest harnesses clear about 42.3% of moving-ecosystem tasks

Primary outcome is task success under successive ecosystem versions. Even the strongest evaluated model or harness reached only about 42.3% task success. Named figures in the evaluation: Codex-GPT-5.5 at 42.27% and Gemini-3.6-Flash at 41.65%. Those ceilings sit on a suite built for multi-turn tool use under churn, so the absolute level is the story as much as the ranking.

Skill-learning methods that help on a fixed tool set failed to transfer consistently as the ecosystem evolved. Initial gains did not reliably become better adaptation on later versions. That gap is the empirical warrant for Steward.

Steward recorded the highest success rate on every ecosystem version, averaging 39.04% against SkillOpt at 36.36%. From version 2 onward it also posted the lowest skill-maintenance cost; baselines consumed 1.2x to 5.5x the skill-maintenance tokens.

Recovery and adoption moved with those gains. IDR and DAR each rose by roughly four to five percentage points on the authors' harness and model set. Read the pair as evidence that joint tool-and-skill management can raise success while cutting maintenance tokens on this suite.

Steward's loop is two coupled jobs. Incremental skill maintenance revises procedural knowledge when a tool path breaks or a cheaper path appears. Dynamic provisioning brings tools into the working set during task execution rather than assuming a frozen catalogue. The benchmark's six states are the pressure test for whether those jobs keep success from collapsing after a transform.

SkillOpt and peer skill methods face the same six states

Fairness for the moving-ecosystem claim rests on shared tasks and shared transforms. Every compared agent or harness walks the same 1,588-task, 42-domain corpus and the same six successive states. Capability-preserving transforms keep the comparison about adaptation rather than about tasks that quietly become unsolvable.

Steward's comparison to SkillOpt and other skill baselines uses the same version schedule, so average success rate and maintenance-token ratios are within-suite. The paper's own strongest model/harness ceiling (~42.3%) remains the external reference for how hard the suite is before Steward's managerial loop is applied.

Cross-vendor API fairness limits still apply. Named model strings (Codex-GPT-5.5, Gemini-3.6-Flash) reflect the authors' evaluation windows and harness wiring. Re-ranking under a different tool-call schema, temperature, or step budget would need a fresh run. Absolute success percentages should be cited with the suite version and harness, the same way any agent leaderboard row is.

Live SaaS deprecations sit outside the transform generator

The published transforms are capability-preserving by construction. They do not reproduce arbitrary vendor deprecations, broken OAuth, or region-locked APIs that make tasks truly impossible. Buyer fleets that lose a tool without a substitute remain outside the measured regime.

Dollar cost of Steward in production (engineering time, review queues, incident load) is unset. The paper reports skill-maintenance tokens relative to baselines, which is a compute proxy inside the benchmark loop.

Human-in-the-loop approval policies, enterprise IAM, and secret rotation during ecosystem churn are outside the reported metrics. Multi-tenant isolation when dynamic provisioning installs new tools mid-task is likewise unset in the abstract and summary results cited here.

Whether Steward's IDR/DAR gains hold on proprietary internal tool graphs with fewer than 42 domains, or on MCP server fleets with different documentation noise, remains unsettled until those graphs are plugged into the same versioned protocol.

zjunlp ships Δτ-Bench code and dataset on GitHub

Paper: arXiv:2610.34238, "Δτ-Bench: Pushing Agents Toward a Moving Capability," posted 25 October 2026; comment line marks the work as in progress.

Code and dataset ship in the authors' public repository named in Sources. The repository is the public reproduction surface named by the authors for the benchmark and Steward-related artefacts. Licence terms should be read from the repository LICENSE file at pin time; SPDX details are not restated in the arXiv abstract.

Authors listed on the arXiv entry include Shuofei Qiao, Xuanguang Pan, Yijun Chen, Wendi Feng, Bin Wu, Runnan Fang, Yuqi Zhu, Shumin Deng, and Emine Yilmaz, with ZJU/UCL affiliation context in the assignment record for this piece.

Score agents on versioned tool graphs before promoting skill caches

Δτ-Bench earns a place when your agent product accumulates procedural skills against a tool ecosystem that actually churns: plugin marketplaces, MCP catalogues, internal SOA fleets with overlapping endpoints. Run at least two successive ecosystem states before you declare a skill-learning method production-ready. Steward-style incremental maintenance is the path when your metrics show skill caches going stale while token spend on full re-learning stays high.

If your tools are a sealed, version-pinned set for the lifetime of each customer workflow and you already pass a static tool bench at acceptable rates, moving-ecosystem stress can wait until a deprecation calendar appears.

For Monday, clone the public Δτ-Bench repository named in Sources, pin a commit, and reproduce the strongest-harness ceiling on your API keys. Then compare your skill store's success and maintenance tokens across two ecosystem versions with and without an incremental maintenance loop. Treat a single frozen-tool score as incomplete evidence for ops agents.

Frozen tool suites give way to Steward-managed ecosystems

Prior agent tool benches (MCP-Bench lineage, Toolathlon-style suites, and related static catalogues cited across 2025-2026 agent evaluation) mostly hold the tool surface fixed while varying models, prompts, or memory. Δτ-Bench inherits the executable multi-turn tool-use framing and then adds successive ecosystem states via capability-preserving transforms.

Skill-learning and skill-optimisation baselines (SkillOpt among them in the paper's tables) supply the immediate comparison class: methods that improve procedural knowledge on a relatively stable tool library. Steward's claim is organisational: jointly manage tools and skills with incremental maintenance and dynamic provisioning so gains survive churn.

Relation to fault-injection work on MCP paths is complementary. Fault injectors stress unreliable tools; Δτ-Bench stresses evolving tool identity and documentation. An operator programme can need both: resilience when a tool fails, and adaptation when a tool is replaced.

The Record takeaway is structural. Static tool benches miss ecosystem churn. Steward is the operational answer this paper proposes, with public code to pressure-test that answer on the published 1,588-task suite.