MiRL trains terminal agents against recurring misbehaviours instead of whole-task RL alone

Terminal agents burn rollouts on the same wrong shell moves across many tasks. MiRL treats those recurring failures as the training unit.

Runchu Tian, Zaid Khan, Zun Wang, Justin Chih-Yao Chen, and Mohit Bansal post Misbehavior-driven RL on arXiv as 2610.35697. The recipe is simple enough to steal: profile a policy's wrong steps until you can name generalisable misbehaviours, restart RL from the prefixes where those misbehaviours appear, and pay reward only when the rollout both passes the tests and avoids the targeted failure.

Misbehaviour prefixes become the RL start state

MiRL sits in the error-targeted learning line for long-horizon terminal work. The authors first collect wrong steps from a base policy, then cluster them into misbehaviours that recur across environments rather than one-off bugs on a single task. Training then starts rollouts at those prefixes. The reward is conjunctive: tests must pass, and the named misbehaviour must stay absent.

That design differs from whole-task RL that samples from the task start and hopes rare failures appear often enough to learn from. On Qwen3.5-4B, eight misbehaviours alone explain 52.1% of wrong steps in the paper's accounting. After MiRL training, the policy repeats the measured misbehaviours 43% to 57% less often, on average, on unseen prefixes. The reduction is the main operational claim: fewer repeats of the same bad shell habit after you have named it.

Head-to-head, MiRL reports 51.8% on Terminal-Bench Lite against 47.4% for TMax-4B, and 21.3% against 20.6% on Terminal-Bench 2.1, while consuming 35% of TMax's rollouts. Those scores are the paper's evaluation on the stated benches. Reproducing them means pinning the same base model, misbehaviour set, and rollout budget.

Terminal benches still reward volume of whole-task rollouts

Frontier labs and indie harnesses keep pouring RL into shell agents because Terminal-Bench and its lite variants have become the public scoreboard for "can the model finish the job in a container." Whole-task RL scales with compute. It also spends most of that compute on trajectories that never hit the failure modes that dominate wrong steps.

MiRL arrives as an efficiency argument with a debugging taste. If eight named habits already cover half the wrong steps, operators can spend fewer rollouts by starting where those habits fire. The paper's own framing is error-targeted learning applied to multi-turn terminal work, filling a gap between single-turn error-targeted methods and agent failure analyses that only evaluate and debug.

Public paper plus GitHub recipe; maturity stays research-usable

Primary artefacts: the arXiv HTML and PDF for 2610.35697, plus the public MiRL repository named in Sources. Licence terms follow whatever LICENSE file the repo ships at the commit you pin; confirm before commercial redistribution.

The method is usable for teams already running terminal RL on Qwen-class bases and willing to build a misbehaviour profiler. Any particular eight-misbehaviour catalogue stays provisional until you re-profile on your own tool traces, freeze the taxonomy, and lock reward checks plus the eval harness for your containers.

Reach for MiRL once wrong steps cluster into named habits

MiRL-style training earns its keep when traces already show the same wrong shell patterns across many tasks and the RL budget is the bottleneck. Whole-task RL baselines still fit when failures are diffuse, environment-specific, or too rare to profile into stable prefixes. Classical supervised repair of tool wrappers is the cheaper path when the apparent "misbehaviour" is really a missing API or a broken sandbox rather than a policy habit.

If you already ship a TMax-class terminal policy, treat MiRL as a second-stage corrector: keep the base recipe, add a profiler, and spend the saved rollouts on harder environments.

Bad taxonomies, reward gaming, and bench-only habits

A weak profiler invents misbehaviour labels that fail to generalise, so RL starts from noise. A reward that checks only the named habit can teach the policy to dodge that habit while inventing a sibling failure one step later. Bench gains on Terminal-Bench Lite can overfit the paper's eight labels without transferring to your internal suite. Unseen-prefix gains of 43-57% describe the measured misbehaviours; they do not promise fewer failures of every other kind.

Operators who skip the "tests pass AND misbehaviour avoided" conjunction will train agents that look cleaner in traces while still shipping broken patches. Transfer across base models also stays open: the headline numbers ride on Qwen3.5-4B, so a different backbone needs its own wrong-step profile before you inherit the eight-label catalogue.

Profile your wrong steps before you buy another rollout batch

If you train terminal agents this quarter, pull the MiRL paper and repo, then run a one-week audit: cluster wrong steps on a held-out environment set, name the top recurring habits, and only then decide whether whole-task RL or misbehaviour-prefix RL is the cheaper path. Pin the Qwen3.5-4B comparison as a method check, and keep your own suite as the ship gate.

Whether eight labels cover half your production wrong steps remains unsettled until you profile that corpus.