Giving models the tools they were trained with can make them worse

A few nights ago, lying in bed, I had what felt like a galaxy-brain idea. I dictated it into my watch so I would still remember it in the morning.

Every frontier model learns to solve coding tasks with tools through reinforcement learning (RL), and I assume that training happens inside its vendor’s own coding agent: Anthropic’s models in Claude Code, and OpenAI’s models in Codex. That is a big assumption, since the labs do not publish their training environments, but it makes business sense: they sell those agents, so they have every reason to make their models work best in them. Every other harness, like pi, opencode, or my own MindRoom, gives the model a set of tools of its own design for running commands and editing files. The models are usually smart enough to figure those out, but not always. Sometimes a model assumes an edit tool works like the one it was trained with, and uses it wrong. This summer, that happened to Claude Opus 4.8 in pi, which I come back to below.

So the idea was simple: in MindRoom, give each model the same shell and file-editing tools as its native harness, and switch them automatically depending on which model is answering. Claude would see Claude Code’s Bash, Read, Edit, and Write. GPT would see Codex’s exec_command and apply_patch. And because the smallest agents do surprisingly well, I would go one step further: a minimal agent whose only tool is the shell each model knows from its training.

I was beyond excited. I could not really share that excitement locally, because my wife did not care, so I am sharing it with the internet instead.

Then I measured it against MindRoom’s own tools, and the familiar ones did not make the models more efficient. With only Codex’s tools, GPT-6 Astra did 84% more work, taking more steps and reading and writing more along the way, and used 59% more tokens. With Claude Code’s tools, Claude Sonnet 5.5 did 15% more work, and Claude only came out a few tokens cheaper because the descriptions I wrote for those tools are slightly shorter than MindRoom’s. Only one model clearly worked more efficiently with its own tools: Claude Opus 4.8, the model from the pi story. The models seem to generalize beyond the environment they were trained in, and what still matters is how much the agent sends with every request.

What a harness is #

“Harness” gets thrown around a lot, but there is no magic in it. Every coding agent talks to the model the same way: each request contains a system prompt, the definition of every tool the model may call, and the conversation so far. When the model answers with a tool call, the harness runs the tool, adds the result to the conversation, and sends everything again.

So from the model’s side, a harness is three things: its system prompt, its tool definitions, and what comes back when a tool runs. The first two are just text, so they are easy to copy, and the tool definitions are effectively the harness’s API: change them, and the model sees a different harness. The third is code, which takes more work to copy.

Because every request resends everything, a task’s tokens come in two parts. The fixed part is what the agent sends before anything happens: its system prompt, its tool definitions, and the task, which every request resends. The work is everything the steps add on top: the model’s replies, the tool results, and the growing conversation that every later request resends too. When I say below that a model did more work, I mean that second part.

Why I believed it #

Two things convinced me: how well the smallest agents do, and what goes wrong when a model meets unfamiliar tools.

Smaller agents do better #

Pi made its name by being small. When Mario Zechner introduced pi last November, its system prompt and tool definitions together came in below 1,000 tokens, and it had four tools: read, write, edit, and bash. He also wrote that “pi does not and will not support MCP”, the Model Context Protocol through which most agents load outside tools. And he pointed to Terminus 2, a minimal agent from the Terminal-Bench team, which was “holding its own against agents with far more sophisticated tooling”.

In August, DeepSeek released DeepSeek Harness, which was all over Hacker News and the local AI subreddits I read. Its standard setup comes with a full set of tools. But for benchmarks, DeepSeek points to its minimal profile. In early September, that profile had exactly one tool, a persistent bash, and a one-line system prompt: “You are a helpful software engineer assistant.” (Ten days later, it already had a second tool, working_directory; staying that small seems to be hard.) That is what inspired the minimal mode I added to MindRoom in September, which gives an agent a short prompt and a single bash tool.

Models should know their own tools best #

The case that got me thinking about this was Claude Opus 4.8 in pi. In July, a pi user reported that about 20% of its edits failed in some sessions. Pi’s edit tool takes a list of replacements, and Opus 4.8 kept adding made-up fields to them, like in_file, matchCase, or newText2. Claude Sonnet 5 did it too, while Opus 4.7 and older Claude models never did in the same tests.

Armin Ronacher, one of pi’s maintainers, dug into it and compared it with what Claude Code does with the tool calls it receives. It turns out that Claude Code quietly repairs a lot of them:

• It accepts several names for the same argument, like old_str and old_string.
• It parses an argument that arrives as a string when it should be an object.
• It fixes broken \uXXXX escapes in strings.
• It drops fields it does not know.
• When it cannot parse a call at all, it asks the model to try again.

His hypothesis is that RL itself might be the cause. If a model is trained in a harness that absorbs these mistakes, a slightly wrong tool call still completes the task and still gets rewarded, so nothing teaches the model not to make it. Pi now ignores unknown fields too, just like Claude Code.¹

Then pi itself strayed from its minimal roots. Pi 1.0 came out on October 1 and added MCP after all. The Register called it a 180. On Hacker News and r/LocalLLaMA, people worried that “Pi’s days as a nice minimal agent TUI are numbered”. What struck me was how the maintainers explained it. Armin Ronacher wrote that the models “are trained on their respective harnesses and we’re not here to fight their behavior”, and Mario Zechner that “we follow what the models are trained on”. That is exactly the native-tools half of my plan.

My plan combined both: a minimal agent, with each model’s native shell as its only tool. Terminal-Bench also said something I chose to ignore: agents like Terminus 2 do well with tools no model was trained on. I did not believe that part. In opencode and pi, I had repeatedly seen models get the edit tool wrong, so I assumed the benchmarks did not carry over to real work. My belief that a model works best in the harness it was trained in was that strong: I expected the native tools to help, and to help most in a minimal agent with nothing but a shell.

What I built #

MindRoom is my open-source platform for AI agents that live in Matrix chat rooms, where several agents can work together. Its agents get shell and file tools under MindRoom’s own names, like run_shell_command, read_file, and edit_file. In this pull request, I added what I call tool dialects, which show a model those tools in the shape of the coding agent it was trained in.

Inside MindRoom, nothing changes: approvals, hooks, and the stored history keep MindRoom’s names, and the translation only happens on the way to and from the model. That gives MindRoom a property I like a lot: a conversation can switch from Claude to GPT halfway through, and each model sees the earlier tool calls in its own shape, as if it had made them in its own harness.

Matching the native harness exactly turned out to be a stretch, though. Codex has no tool for reading files, so in the pull request, GPT keeps MindRoom’s read_file, and both keep MindRoom’s grep, find_files, and ls, which makes each native set a mix. For the main comparison, I removed those extra tools, so each model saw only the tools of its own harness. The tools have the same names and arguments as in Claude Code and Codex, but I wrote short descriptions of my own, because Claude Code’s are not openly licensed. What they return is MindRoom’s output, and the system prompt is MindRoom’s too. So keep in mind for everything below: I tested the names and shapes of the tools, not the whole harness.

How I measured it #

I used 12 terminal tasks from an evaluation suite I built for MindRoom, like writing a report from a CSV file, finding the commit that broke a check, or fixing failing tests.² A hidden check graded each result, and every setup ran every task three times. Almost every run passed, so the interesting number is how many tokens the agent needed to get there, cached input included, and cheaper in what follows means fewer tokens.³ The tool comparisons ran through MindRoom’s OpenAI-compatible API; minimal mode, which only works in chat, ran through Matrix. I tested five current models, Claude Opus 5.5 and Sonnet 5.5, and GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna, and later five older ones. The current Claude models ran with adaptive thinking, where the model decides when to think. I called the GPT models from MindRoom through OpenAI’s Codex backend, the endpoint the Codex CLI uses, so no run used Codex itself; with no reasoning effort set, the backend picks one.

Native tools don’t make today’s models more efficient #

To make the comparison fair, I gave each model only the tools of its own harness, against MindRoom’s matching set.⁴ For Claude, that is a shell and tools to read, edit, and write files; for GPT, a shell and Codex’s apply_patch, against MindRoom’s tools to edit and write files.

Claude got 6 to 13% cheaper with Claude Code’s tools, but only because the definitions I wrote for them are about 480 tokens shorter than MindRoom’s, and every request resends them; the tools themselves saved nothing, and Sonnet 5.5 even did 15% more work with them. It never even touched the file tools: Opus 5.5 and Sonnet 5.5 did everything in the shell, with either set.

GPT-6 Astra and Sol got much more expensive, while Luna barely changed. With only Codex’s tools, GPT-6 Astra did 84% more work and used 59% more tokens. It called apply_patch 30 times, as a separate step for edits it otherwise made inside its shell commands, and every extra step resends the whole conversation so far. I expected the opposite: these are the tools I assumed these models were trained on, so I thought they would need fewer and more confident steps. The mixed set, the lighter rows, points the same way.

The tool mistakes I wanted to prevent did not happen with MindRoom’s tools at all: with its matching set, none of the five models made a single tool error. The only errors came from the mixed set: next to Codex’s tools, Luna had to guess how MindRoom’s read_file resolves paths, and got it wrong seven times.

Older models: only Opus 4.8 did better with them #

The pi story made me wonder whether this used to be different, since Opus 4.8 struggled with pi’s edit tool while Opus 4.7 did not. So I ran the same comparisons on older models that are still available: Claude Opus 4.7, Opus 4.8, and Sonnet 5, and GPT-5.5 and GPT-5.6 Sol.

With only each harness’s own tools, the older models did not prefer their native tools either, except for one. The exception is the model from the pi story: Opus 4.8 did 17% less work and used 19% fewer tokens with Claude Code’s tools, reading files through Bash instead of with a separate tool. With the mixed set, the older Claude models seemed to prefer Claude Code’s tools, using 22 to 36% fewer tokens, but that came from the extra tools the mixed set keeps from MindRoom, grep, find_files, and ls: next to MindRoom’s other tools, those made the older Claude models take more steps and read more, and next to Claude Code’s tools, the models hardly touched them. And on the pi story itself: Opus 4.8 and Sonnet 5, the two models that made up fields in pi, made 111 edits with MindRoom’s edit_file, which takes a single path, old text, and new text, and none of them failed.

With only each harness’s own tools, the GPT-5 models did about the same with either set, and unlike GPT-6 Astra and Sol, they barely used apply_patch. So reaching for its native editing tool, and paying for it, is new in that generation.

Of the ten models I tested, familiar tools clearly made only one more efficient, the one that started it. The others did as well or better with tools they were presumably not trained on: they seem to generalize beyond the environment they were trained in.

What does matter: what the agent sends #

My next guess was the other half of my plan, a smaller agent. Every request to the model resends the system prompt and the definition of every tool, so an agent with a shorter prompt and fewer tools sends less with every step.

Fewer tools #

Fewer tools did help. With only MindRoom’s shell tools and none of its file tools, the agent used about a third fewer tokens and passed about as many runs. A shell-only agent is also where I still expected the native tools to matter most, because every step goes through the one tool whose shape the model knows best. They did not: GPT used the same number of tokens with Codex’s shell tools, and Claude saved only what its shorter descriptions save.

Minimal mode: one shell tool and a 12-line prompt #

MindRoom’s minimal mode, modeled on DeepSeek’s minimal profile, takes this all the way. The agent gets a 12-line prompt and a single bash tool instead of its normal prompt and tools. Its instructions, skills, context files, and memory move behind a command-line program inside that shell, which it can call when it needs them. Minimal mode only works in chat, so I ran this comparison in Matrix conversations, against the same agent in standard mode with only its shell tools. A chat adds context about the room to every request, but even so, the minimal agent used 21 to 52% fewer tokens than the shell-only agent through MindRoom’s API.

The difference starts before the model does anything. For Claude Sonnet 5.5, the prompt and tool definitions sent with every request shrink from 4,285 tokens to 661.⁵

Each bar splits a task’s tokens into the fixed part, the prompt and tool definitions resent with every request, and the work the model did on top of it. The minimal agent used 58 to 76% fewer tokens than the same agent in standard mode with only its shell tools, for every model. Most of that is input the provider had cached and bills at a fraction of the price, so the bill shrinks less than the token count does: counting only uncached input and output, the minimal agent used 16 to 42% fewer tokens. And the model did not pay for it with extra work: the dark part of the bars stayed about the same or shrank, except for Luna’s. I had also hoped that the smaller agent would need less to do the task itself, but only GPT-6 Astra and Sol did, with slightly fewer steps and about a quarter less output; Claude worked exactly the same in both modes.

These tasks are short, three or four requests each, which is why the prompt weighs so much. I tried to make them harder so they would take more steps, but the models were so good that they only took about 50% more requests, and for the two models I tried, Claude Sonnet 5.5 and GPT-6 Astra, the pattern stayed the same: Sonnet did the same work in both modes, and Astra less in minimal mode. For the models that save steps, I see no reason why longer tasks would erase that gain: every step saved also saves resending the conversation, so on long tasks it counts for more. In a long session, the growing conversation dominates and the prompt’s share of the tokens shrinks, but its saving per request stays: about 3,600 tokens for Claude and 2,200 for GPT. And a real agent’s prompt carries far more instructions, skills, and memory than this test agent’s, so it saves more per request, although fetching them through the shell when needed costs requests of its own.

Minimal mode with each model’s native shell tool #

That leaves the combination I had in mind from the start: the minimal agent, with each model’s native shell tool as its only tool. I patched minimal mode so that its single tool appears as Claude Code’s Bash for Claude and as Codex’s exec_command for GPT, with the same 12-line prompt.⁶

It did not make the minimal agent better for any model. For Claude, it made little difference: Sonnet used about the same, and Opus a little more. For GPT, it was clearly worse, mostly because exec_command hands back a command’s whole output, and in an agent that sends so little else, a few long outputs weigh a lot. With its output cut to 100 lines, bash’s default and the light rows in the chart, only Sol still clearly used more; Astra’s and Luna’s smaller increases could be chance.

So the whole effect of the combination came from its minimal half: models that were trained with a carefully designed set of tools did best with one tool and almost no prompt, and the tool’s native shape did not make it better.

What this does and does not show #

• I tested the names and arguments of the tools, not whole harnesses. Claude Code and Codex also have their own tool descriptions, system prompts, and output formats, Codex has an interactive terminal, and RL trains all of that together. The full harness might still help today’s models; matching only the tools did not.
• The tasks are easy and short: 97% of all runs passed, with three to seven requests per task on average. Harder, longer tasks might separate the tool sets on success, not only on cost.
• With 36 runs per configuration, a 15 to 30% difference in tokens is clear, but a difference of one or two passed runs is not.
• I tune MindRoom’s agent setup on other tasks from the same 12 families, and a note on working habits from that tuning sits in the shell tool’s description. Both tool sets carry the same note, so it should not favor either.
• The confidence intervals cover run-to-run variation on these 12 tasks, not the choice of tasks. Resampling the tasks as well widens them: GPT-6 Astra’s +59%, for example, becomes +38 to +81%, and Sonnet 5.5’s 6% saving becomes indistinguishable from none. Astra and Sol still used more tokens with Codex’s tools on every one of the 12 tasks, and the minimal agent used fewer on every task, for every model.
• Most runs used each model’s standard reasoning mode, which differs between generations: adaptive thinking for Sonnet 5 and the 5.5 models, none for Opus 4.7 and 4.8, and the effort Codex picks for GPT. I tried other settings on four models: two current ones at low and high effort, and Opus 4.7 and 4.8 with adaptive thinking. The pattern held at every effort: Astra used 27 to 45% more tokens with Codex’s tools, and Sonnet 5.5 stayed slightly cheaper with Claude Code’s.

What happens to the pull request #

The pull request is still unmerged, and given these results, I am not sure I will merge it at all. For today’s models, the native tools either cost more, as Codex’s did for GPT-6 Astra and Sol, or, as Claude Code’s did for Claude, only saved the few tokens of the shorter descriptions I wrote for them, which MindRoom’s own tools could get too. The one model that clearly did better with its own tools, Claude Opus 4.8, is a generation behind. What I will do is use minimal mode a lot more, with MindRoom’s own bash.

Why this excites me #

Half of my plan was wrong, and I am happy about that. The minimal-harness, DeepSeek-style half held up; the native-tools half did not. The benchmarks I did not believe were right, at least for these models. It suggests that these models generalize much better than I gave them credit for. They were trained with one specific set of tools, and on these tasks they did just as well with tools they had presumably never seen in training, as long as those tools were simple, like an edit tool that takes one path, the old text, and the new text. Only Opus 4.8, the model that started all this, still did better with its native tools. What matters more than familiarity is how much the agent sends with every request: the prompt and the definition of every tool.

It also changes how I think about who gets to build the best agents. If models only worked well in the harness they were trained in, the labs would own the best agents by default. Instead, at least the skill of using unfamiliar tools seems to carry over to other harnesses. I suspect the same holds one level up, for how several agents work together. The labs presumably train their models in their own harness, but if what the models learn transfers this well, a harness built outside the labs, like MindRoom, could get just as good at letting agents collaborate. That is still a hunch, and testing it is what I want to do next.

Appendix: how I measured, and the numbers behind the charts #

All scripts, the tasks, and one row per trial, with its reasoning setting, are on GitHub next to this post’s source.