Tsinghua’s AKV lets agents drop and repos KV cache without rewriting trajectories

Long agent trajectories burn money in the KV cache long before they burn the context window. Tsinghua's Cache Engineering for Agents paper (arXiv:2610.28577) argues the fix is to edit the cache directly, and it ships AKV so serving systems can keep up.

Context engineering trims or summarises history. Editing an early message changes the trajectory prefix, invalidates later cached states, and forces a re-prefill of everything retained. Harnesses therefore wait until a length threshold, then summarise aggressively so the re-prefill stays short, which risks deleting the middle rounds that still matter. Cache engineering keeps the text history and removes only selected KV entries from future attention, reusing retained states without recomputation.

Drop, repos, and a serving engine that remembers operation history

Beside ordinary append, the paper formalises two cache operations. Drop excludes selected KV entries from future attention. Repos adjusts position encodings on retained keys (via RoPE rotations in the described design) so position indices need not march with the logical trajectory until they hit the model limit even after the cache has shrunk. Agents and harnesses interleave these ops with append to shape the active cache over a run.

History dependence is the systems catch. Under standard prefix caching, the token prefix determines the cached state. Under cache engineering, the same tokens and retained positions can yield different states depending on when drops and appends happened. AKV is the co-designed serving engine: a stateless interface compatible with existing API-style serving, while internally tracking stateful cache reuse, eviction, and restoration with an extended radix tree and segment-batch attention that reconstructs missing states in one prefill pass. Agents submit messages plus cache-operation instructions without tracking which KV blocks still sit on the server.

Agent bills are already mostly cache reads

The authors report that KV cache reads took 82.4% of total API cost in their motivating breakdown. As trajectories stretch across hundreds of rounds, storage and I/O dominate, and attended length or positional range can exceed what the model handles reliably. Prior eviction and compression work mostly used low-level signals; AKV pushes high-level eviction decisions into the agent or harness while making cross-request reuse correct under operation histories. Recursive self-improvement (RSI) over cache policies is presented as a programmable consequence of exposing drop and repos as first-class ops.

Paper plus open repository, maturity still research-usable

Primary artefacts are arXiv:2610.28577 and the open AKV repository named in the abstract. Treat maturity as usable for research and internal serving experiments: the interface is deliberately API-shaped, yet production adoption still needs your model's RoPE assumptions, multi-tenant isolation rules, and exact agent harness to be validated. Licence text should be read from the repository LICENSE at pin time; the abstract points to the GitHub tree rather than a packaged release train.

Prefer AKV-shaped ops when summarisation is the expensive compromise

Cache-engineering policies earn their keep when tool-response bloat dominates cost, retained assistant reasoning still matters, and re-prefill after mid-trajectory edits is already the bill you are paying. The headline rolling-drop policy (keep reasoning KV and recent tool responses; drop older tool-response KV) improved average scores by 1.7 points and cut average cost by 27.7% versus conventional threshold-triggered summarisation across the paper's LLM and benchmark mix. With AKV, that policy reached up to about 1.2× output throughput and about 1.5× faster P95 request latency (paper: up to 1.18× throughput and 1.51× P95 speedup in the detailed claim). RSI-refined policies saved an additional 7.1% cost over expert-designed policies on held-out tasks.

Threshold summarisation or a simple trim still fit when your serving stack cannot track operation histories, or when legal logging requires the served attention set to match the stored transcript byte-for-byte without cache-side omissions. In those fleets, keep the text edits and accept the re-prefill until the stack can honour drop and repos correctly.

Wrong drop timing and opaque active context

Drop the wrong spans and the model loses tool results it still needs; the retained cache may still carry ghost information from dropped history, which helps efficiency and complicates debugging. Repos mistakes can desynchronise position encodings from what the checkpoint expects. Serving bugs that ignore operation history will silently reuse the wrong KV for an identical token prefix. Operators who only monitor token counts will miss that active cache length has diverged from trajectory length. Benchmark gains are paper-reported averages across three LLMs and four agent benchmarks; reproduce on your harness before booking the 27.7% saving.

Measure cache ops on your harness before you book the saving

Stand up a rolling-drop policy against a slice of production-like trajectories, compare cost and task success to your current threshold summariser, and keep operation-history logging on until the active cache length matches what you intend. The open Monday question is whether your serving stack can distinguish identical token prefixes that arrived through different drop histories.