Mindroom | Bas Nijholt

Roll your own, fork the rest 🍴

Mon, 19 Oct 2026 00:00:00 +0000

Introduction: a follow-up to something I wrote #

In April 2025 I published Confessions of an asymmetric hypocrite, about my split personality on dependencies. Strict minimalism for the libraries I publish, and then, for applications:

For applications (especially CLIs in isolated environments), I’m far more liberal with dependencies to build features quickly.

I also wrote that I “strongly prefer dependencies that are already widely adopted and trusted by the community” because “these are less likely to cause unexpected issues”.

Then I spent a year building MindRoom, which is an application, exactly the category where I said I could relax.

I want to report back. “Prefer battle-tested dependencies” was the right rule. I just had the wrong idea of what battle-tested means. I thought a project with tens of thousands of GitHub stars qualified. Now I think it means millions of downloads, every day.

Table of contents

Update 2026-10-31: In the two weeks after publishing, I took my own advice twice. MindRoom’s knowledge search no longer uses ChromaDB, and code MindRoom owns has replaced Agno’s runtime. Agno is now only MindRoom’s tool library. The full stories are in the comments.

1. Stars measure interest, not usage #

When I picked Agno as MindRoom’s AI framework, it looked like the safe choice. Today it has over 42,000 stars and 6,000 forks.¹

It also gets about 78,000 downloads a day on PyPI. httpx, one of Python’s standard HTTP libraries, gets about 25 million.²

A star is something you click once, when a README looks exciting. A download happens every time some project installs the package.

Most of those downloads are CI, not people, so downloads are not users either. But a package with millions of downloads a day is installed by a huge number of different projects. Many of them use the same functions you are about to use, so they hit the bugs before you do.

Battle-tested is also not the same as old. I use uv for everything, and it has only been around for a couple of years. I run into bugs in uv too, but there is simply no better alternative, and it gets about 3.6 million downloads a day.

2. A popular protocol is not a popular library #

The other place my old idea failed was Matrix.

Building on Matrix felt obviously correct. An open protocol, about a decade old, with serious security reviews from large organizations and governments. Millions of people use it.

Matrix is still a good protocol and I am still on it. But even though millions of people use Matrix, there are only a few client libraries.

MindRoom streams agent replies by editing the same message many times a second. Almost every part of the ecosystem I touched had a problem with that: Element, Cinny, matrix-nio, and the Matrix JavaScript SDK. In the JavaScript SDK, edits that arrived while a thread was still loading were silently dropped, so you saw stale text until you reloaded.³

What counts is how many projects use the library you build on. Element’s users do not test matrix-js-sdk for you. Other people building on matrix-js-sdk do. It gets roughly 100,000 downloads a day. matrix-nio, the Python client, about 23,000.

3. I checked the rule against my own code #

I pulled the daily download numbers for the packages I use and looked for workarounds, patches and forks in my own source.⁴

Package	Downloads per day	What it cost me
requests	49M	nothing I had to work around
pydantic	35M	nothing
numpy	33M	nothing
httpx	25M	nothing
aiohttp	18M	nothing
FastAPI	17M	nothing
chromadb	270K	workarounds for a bug that holds the GIL
matrix-js-sdk	~100K	patched
mem0ai	79K	largely reimplemented
agno	78K	126 documented workarounds
matrix-nio	23K	forked

Above a million a day, I had nothing to work around. Below it, every package on this list cost me a patch, a fork, a rewrite, or a pile of workarounds.

Take ChromaDB, the vector database behind MindRoom’s knowledge bases. In some cases it holds Python’s GIL during I/O, which blocks every other thread in the process. I opened a pull request to fix it, and nobody has looked at it yet. Instead of forking, I run every knowledge read in a short-lived child process, because a process can be killed and a blocked thread cannot.⁵

My fork of matrix-nio has about 8,900 lines added and 2,000 removed in the library code since the last upstream release.⁶

This does not mean every small package needs replacing. MindRoom pulls in a few hundred packages in total, and plenty of them have modest download numbers. The scrutiny should match how much I depend on a package. Something I use for one small, specific purpose is fine. Agno and matrix-nio are the base that MindRoom is built on, which is probably also why they hurt the most.

4. Roll your own: the library gives you the easy 10% #

Take memory libraries, Mem0 and friends. I have already written about this: I cloned LlamaIndex, LangChain, Letta, Mem0 and PydanticAI, studied how they work, and re-implemented the parts I needed. My memory system ended up “mostly modeled after Mem0”.

The core idea is not hard. With RAG, you chunk, embed, retrieve, re-rank and reconcile. Once you have read a couple of implementations you can write it, and afterwards you understand every line.

The same is true of an agent framework. It builds a JSON payload, posts it to an HTTP endpoint, parses a streaming response, keeps a list of messages, and loops when the model asks for a tool call. With AI, that is a day or two of work.

The hard part is making it reliable at scale: never blocking the event loop, and handling the crazy edge cases. With an agent system you end up building a very complicated state machine, and if you do not watch it, any combination of states is possible. Almost all of them are edge cases.

A library gives you the easy 10% and leaves you the hard 90%. And you do that 90% inside somebody else’s code, where you cannot fix the root cause.

MindRoom is now about 360,000 lines of Python, plus about 750,000 lines of tests.⁷ A good part of that is code I wrote instead of depending on a library:

Memory and knowledge: about 15,600 lines across 50 files.
Conversation history and compaction: about 8,400 lines.
An OpenAI-compatible API: about 2,300 lines, so MindRoom agents show up as selectable “models” in any chat frontend that speaks the OpenAI protocol.
Claude provider support: about 3,000 lines.

If I have to write the hard 90% either way, I might as well write the easy 10% too.

5. Fork your own: when the problem is hard #

I am not going to write my own end-to-end encrypted Matrix SDK, or my own homeserver. Those are hard, and also far below the line. So there I fork.

I forked the Python Matrix client. MindRoom ships its own fork of matrix-nio rather than the upstream package.

I forked the Matrix server, twice. First Synapse. When I ran into scaling problems I moved to Tuwunel and forked that.

I forked the web client. Cinny is the Matrix web client MindRoom uses.

None of these are AI projects. AI is just where I felt it most.

Whether a fork stays a fork depends on the maintainers. Tuwunel is extremely receptive to my patches, so I upstream anything that is general enough, and over the last six months I am its fourth most active contributor by commits.⁸ Cinny has not accepted any of my pull requests, so I forked it and stopped trying to upstream. Same with LibreChat: I had patches open for over a year without a reply, so I forked.

Keeping a fork up to date used to be miserable. With an LLM doing the rebasing, pulling in the latest upstream is now cheap, though still a bit of a risk every time. The bigger cost is that once you fork, you are responsible for the whole codebase, including the parts you never changed.

Agno: patches instead of a fork #

I did not fork Agno. I monkey-patched it. Agno is very actively developed, I have talked to the maintainer, and they know about these problems, so I hoped to upstream my fixes. I have filed 33 issues and pull requests there.⁹

That got out of hand. MindRoom now carries 126 documented workarounds for Agno across 59 files. Most of them live in 36 dedicated compatibility modules, about 7,200 lines of code, with the rest patched inline in 23 other files. Every one uses the same comment format, so I can find all of them with one search. Each one says what is wrong upstream, links the issue or PR, and says when I can delete it.¹⁰ A few examples:

Search failures become plausible empty results. A failed knowledge search and an empty index return the same empty list, so my agents confidently told me things did not exist.¹¹
Streamed tool calls get malformed or merged.¹²
Reasoning is silently lost when tool calls are replayed.¹³
Provider errors collapse into a generic 502, which throws away the error code you need to decide whether to retry.¹⁴

At this size, the patches amount to a fork.

6. I am not dunking on open source, including mine #

Everything I named is open source I use every day, for free, built by people doing their best. I filed issues and PRs, and where maintainers merged fixes I deleted my workarounds.

My own software is in the same category. Most new AI tooling is not very good, and MindRoom may well be included. It is large, fast-moving, and nowhere near millions of daily downloads. By my own rule, you should think twice before depending on it.

I also have to admit that I have not yet built a project the roll-your-own way from the start. So comparing “what I would do next time” with what I actually did is not entirely fair. This is what I think right now, and a year from now I may have to write another post correcting this one.

7. My rule for the next `uv add` #

I weigh three questions.

How many projects use it? Daily downloads of the actual package, not stars, and not how many people use the product built on top. In the millions: depend on it.

How much do I rely on it? Agno and matrix-nio are the base of MindRoom. A library I use in a CLI to format a question is not, and if I use it in a narrow way I do not need my own version.

How hard is it, and how receptive is upstream? If the library is something AI can write in a couple of days, roll my own. If it is hard, fork it. If the maintainers accept patches, upstream what is general enough.

If I started MindRoom over today, I would write the HTTP requests to the model providers myself. Not even the OpenAI or Anthropic SDKs. And even httpx is less safe than its download numbers suggest: in early 2026 its maintainer closed the issue tracker and discussions, and the latest stable release is from December 2024.¹⁵

In 2025 I wrote that the right answer is “it depends on the situation”, and I still believe that. I just had the wrong threshold. Battle-tested means millions of downloads a day. Below that: roll your own, and fork the rest.

From the GitHub API (api.github.com/repos/agno-agi/agno) on 2026-10-02: 42,385 stars and 6,041 forks. ↩︎
PyPI numbers are the last_day field of the pypistats.org API (/api/packages/<name>/recent), fetched on 2026-10-02, except uv, fetched on 2026-10-18 (3,648,827). The matrix-js-sdk number is npm’s weekly download count (about 730,000) divided by seven. Daily numbers wobble, so treat them as orders of magnitude. ↩︎
When events arrive for a thread that has not finished initializing, the SDK’s replay path never applies them, so an m.replace edit (the event type Matrix uses for “this replaces message X”) is stored and never shown. I replayed a captured event trace through the SDK: unpatched, zero of seventeen edits reached the listeners; patched, all seventeen. MindRoom carries the fix as a patch-package patch on the SDK’s thread.ts. ↩︎
For each direct dependency I looked at three things in the MindRoom repository: whether pyproject.toml pins a fork instead of the upstream package, whether patch-package patches it, and whether the source has workaround markers or comments pointing at upstream issues for it. The packages above a million downloads a day had none of those. ↩︎
The PR is titled “Release GIL during metadata I/O”. On the MindRoom side, knowledge/read_process.py starts the child process, and a TODO says to remove it once a Chroma release with the fix is pinned. A blocked thread holding the GIL cannot be interrupted from Python, but the kernel will always terminate a child process. ↩︎
Compared with upstream matrix-nio 0.25.2 (released 2024-10-04): git diff --shortstat on the library code gives 8,872 lines added and 1,966 removed across 52 files, from 89 commits, 76 of them mine. The fork is published on PyPI as mindroom-nio. ↩︎
Counted with git ls-files and wc -l on the MindRoom repository on 2026-10-18: 360,513 lines in 1,222 Python files under src/, 750,657 lines under tests/, and another 81,543 lines of TypeScript in the frontend. The per-feature numbers below come from the same count over the relevant directories. ↩︎
Non-merge commits on Tuwunel’s main branch since 2026-04-18: Jason Volk 1,332, June Strawberry 325, dasha_uwu 55, me 42. ↩︎
Issues and pull requests in agno-agi/agno with me as the author, counted on 2026-10-02. ↩︎
Every workaround starts with a comment like # AGNO_COMPAT: Knowledge search hides provider failures as empty results., followed by Reason:, Upstream issue:, Upstream PR:, Remove when: and the tests that cover it. The counts come from grep over src/: 126 AGNO_COMPAT: markers in 59 files, and 7,160 lines in the 36 agno_compat_* modules. ↩︎
Knowledge.search catches vector store and provider errors and returns []. agno#10150, partially fixed by agno#10152. ↩︎
When a streamed tool-call index does not start at zero, Agno leaves empty placeholder calls. agno#8879, fix proposed in agno#8880. ↩︎
With the OpenAI Responses API, Agno keeps only the last reasoning item, so it cannot be replayed with the tool calls it belonged to. agno#9960; partial fixes landed in agno#10075 and agno#10395. ↩︎
Unclassified ModelProviderError failures default to HTTP 502, so a real network blip and a permanent error look the same. agno#8869, partially addressed by agno#8870. ↩︎
The latest stable release on PyPI is 0.28.1, uploaded 2024-12-06. There have been 1.0 development pre-releases since, the latest in August 2026. The closing of issues and discussions was widely discussed in the Python community in March 2026. ↩︎

Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

Sun, 18 Oct 2026 00:00:00 +0000

For years, a Docker container was my default answer to “where do I run this?” I never thought it was a perfect sandbox. It was good enough, because getting out of one usually meant finding a specific kind of bug in the Linux kernel itself, and those were hard to find.¹

AI models have become very good at finding security bugs, including in Linux. Anthropic and OpenAI now keep their strongest security models, Claude Mythos and Daybreak Blue, behind identity checks and vetting. Open-weight models, which anyone can download and run, come with no such rules, and they trail the closed ones by only six to seven months. Once there is an open model as good as Mythos, anyone can strip out its refusals with a tool like Heretic (people call this abliteration) and point it at the Linux kernel. The security company DepthFirst used its own model to find a way out of a container and titled the write-up Containers Are No Longer a Security Boundary.

I care about this because running untrusted code in containers is exactly what MindRoom, my open-source system for AI agents that live in Matrix chat rooms, does. Every agent can get its own Docker worker, and its shell, file, Python, and browser tools run there. It can install anything, start background processes, and drive a real browser. And as I wrote in my ZFS post, I run coding agents in YOLO mode, exclusively. I do not review the commands they run, I look at the results.

So both sides changed at once. The code inside the container is now written by an AI, and the bugs that let it out are now found by an AI. A video from the Low Level channel asking whether containers are still safe made me sit down and rethink my setup.

I am not an expert in any of this. I have used containers, KVM, and hypervisors for years, including Proxmox before I moved to NixOS, but I never deeply understood what happens underneath. Building this forced me to learn it, and writing this post is my exercise in understanding it all. All of this is very complicated (as is everything new), and I am still learning. I am sure I skip over a lot and simplify the rest; what is here are the parts that helped me understand it. I thought it would be nice to share.

Table of contents

What the kernel actually does #

When people talk about Linux, they usually mean an operating system like Ubuntu, Fedora, or NixOS. Those differ in almost everything you see: package managers, desktops, config files. What they all have in common is the kernel, the part Linus Torvalds started in 1991 and still leads today. Strictly speaking, that kernel is the only part called Linux.

The kernel is the one program on a Linux machine that talks to the hardware directly. Everything else, your shell, Python, Chrome, Docker, runs on top of it and has to ask it for things. Open this file. Give me some memory. Send this packet. Start a new process. Those requests are called system calls, or syscalls for short.

There is one kernel per machine, and it decides who is allowed to do what. That makes it the most trusted code on the system. Most kernel bugs only crash the machine or cause glitches. But some let a normal program become root, the admin account, or even run its own code inside the kernel. At that point it can do anything on the machine, including leaving any container it was in.

A container is a process in a costume #

When you docker run -it ubuntu, you get a root shell in what feels like a fresh computer. Under the hood it is one ordinary process. The kernel dresses it up using three features:

Namespaces decide what the process can see. It gets its own list of processes (where it is process number 1), its own filesystem, and its own network. Root inside is still root on the host, with most of its powers taken away.²
cgroups decide how much it can use: CPU, memory, number of processes.
seccomp decides which syscalls it may make at all. It is a filter that blocks the ones a normal program never needs.

There is no second kernel. Every syscall from inside a container lands in the same kernel as every other container and the host itself. The kernel enforces the isolation, and it is also what gets attacked.

For most uses that is fine. Usually you put fixed software in a container: a database, a web app, some self-hosted service. You know where that code comes from, you keep your kernel patched, and that is good enough.

An AI agent is different. It writes new code on the spot and runs it, and nobody knows in advance what that code will do.

Since root in a container is root on the host, anyone who can use Docker can start a container as root and mount any folder from the host into it. They can then read and change every file in that folder, including files their own account may not open. Docker’s own documentation says the docker group grants root-level privileges. I knew that in theory, but it only clicked when I watched an AI agent do it on a machine where I had no root, only Docker access. It needed to change files it was not allowed to touch, so it started a small container, mounted their folder, and edited them from inside. No kernel bug needed.

Why kernel bugs are a bigger problem now #

Kernel bugs that let a normal program take over the whole machine used to take skilled people weeks or months to find and exploit. The famous ones, like Dirty COW in 2016, made the news every year or two, you patched, and that was it.

AI is speeding this up, and the DepthFirst write-up I mentioned is a good example. Their model found a bug in the kernel code for Unix sockets, which programs on the same machine use to talk to each other. The kernel freed a piece of memory and then kept using it, a so-called use-after-free, and from inside a container that was enough to get out. Their conclusion is that you should assume attackers can escape containers at will.

Blocking more syscalls, or taking more powers away from root, helps a little. But it only narrows which parts of the shared kernel a container can reach. This bug sat in Unix sockets, which almost every program uses, including the browser the agent drives, so blocking them would break the agent.

Now put an AI agent in that container. It reads web pages and repositories written by strangers, and one of those could talk it into running an exploit. On a MindRoom server, getting out of a worker means getting onto the machine that runs MindRoom itself. That is where the API keys, Matrix logins, and stored credentials live, which the workers are built to never see.

So I asked a model to break out #

Before changing anything, I wanted to know how real this was for my own setup. I gave a MindRoom agent Daybreak Blue and asked it to escape its worker container using known kernel vulnerabilities. Daybreak Blue is OpenAI’s security model for verified users, so it does the offensive security work that other models refuse.

It did not get out, which was reassuring. But it only tried known bugs. A kernel a few weeks behind on patches, or a bug nobody has published yet, would be a different story. So the attempt tells me the easy path was closed that day, and not much more.

Stronger walls: virtual machines #

The classic answer is a virtual machine. A container shares the host’s kernel, but a VM brings its own, and this is where the hardware comes in. Modern CPUs have a virtualization mode built in, called VT-x on Intel and AMD-V on AMD. In that mode, a second kernel, the guest kernel, runs directly on the real CPU and believes it is in charge, while the CPU quietly limits it. When the guest tries something it may not do on its own, like talking to a disk or a network card, the CPU pauses the guest and hands control to the host. This is called a VM exit. Despite the name, nothing escapes and nothing crashes. The host handles the request, for example by pretending to be that disk, and then resumes the guest where it stopped. From the guest’s point of view, the instruction simply finished.

Memory is fenced off too. The host gives the CPU a separate map of which real memory each guest may use, and the CPU checks every memory access against it. So even a guest kernel with full privileges cannot touch the host’s memory.

The CPU only provides the mechanism. The software that sets the rules and handles the VM exits is called the hypervisor. “Hypervisor” is a role, not one program, and on Linux it is two pieces:

KVM, a module in the Linux kernel, uses the CPU’s virtualization mode to run the guest, keeps that memory map, and catches its VM exits. It shows up as the device /dev/kvm.
A VMM (virtual machine monitor), a normal program on top, sets up the VM and handles the exits that KVM passes up, for example by pretending to be the guest’s disk and network card. QEMU, Firecracker, and Cloud Hypervisor are three VMMs you can pick from.

Proxmox, which I mentioned above, is KVM plus QEMU. AWS Lambda is KVM plus Firecracker.

Code in a VM talks to its own guest kernel. To reach the host, it usually has to break that kernel first and then the hypervisor.³ That is still possible. Google Project Zero turned a bug in how KVM runs VMs inside VMs on AMD into a full escape in 2021. Google’s kvmCTF pays up to $250,000 for a new one. And this year, one researcher, Hyunwoo Kim, published three KVM escapes, one of them hidden in KVM since 2010.

The difference between a container and a VM is how much code an attacker can reach, and that gap is large. On x86 (the Intel and AMD chips in most PCs and servers), Linux has about 370 syscalls, and Docker’s default seccomp filter blocks a few dozen of them. Behind the rest sit the kernel’s filesystems, networking, memory management, and core code: about 3.8 million lines, even without counting drivers. Not all of that is reachable from a container, but a lot of it is. KVM on x86 is about 114,000 lines. On top of KVM comes the VMM, but only the few devices it fakes for the guest. Most QEMU escapes came from its many emulated devices, which is why Firecracker and Cloud Hypervisor keep that list short.

Google’s bug bounties point the same way. kernelCTF pays $71,337 for each new kernel exploit that turns a normal user into root, and Google’s public repository holds exploits for 93 different kernel bugs. kvmCTF pays more than three times as much, and the same repository holds one KVM exploit, which crashes the host instead of escaping it. kvmCTF started a year later, but the gap is still big. Most of those kernel exploits would not work as written from a default container, because they first use a feature Docker blocks. Still, they show how many exploitable bugs sit behind the syscalls.

Kata Containers and gVisor #

Regular VMs are heavy. They boot a full operating system and tend to use a lot of memory. A microVM is a stripped-down VM meant to run one workload and start quickly. That still leaves a practical problem. MindRoom, like most tools, speaks Docker: it hands Docker an image and some folders to mount, and gets a running container back. A VM does not take a Docker image. Switching to VMs would normally mean building disk images, booting them, and wiring up files and networking myself.

Docker does not start containers by itself, though. It takes care of images, networks, and volumes, and then hands the actual start to a small program called a runtime. The default runtime, runc, does what I described earlier: it starts a process on the host kernel with namespaces, cgroups, and seccomp. But you can tell Docker to use a different runtime.

Kata Containers is such a runtime, and it starts a microVM instead. It picks one of the VMMs above (my test used QEMU), boots a small guest kernel, and starts the container inside it.⁴ Docker, and MindRoom, still see an ordinary container.

gVisor, from Google, takes a different route, and it is not a VM. It is a reimplementation of the Linux kernel, written in Go, that runs as a normal program on the host. Go manages memory for the programmer, so the use-after-free mistake behind the DepthFirst escape is much harder to make in gVisor. The worker’s syscalls go to gVisor instead of the host kernel, and gVisor itself only makes a small, filtered set of host syscalls. That is the hardening idea taken all the way: gVisor answers the agent’s syscalls itself, so the filter on the host can be much stricter than Docker’s. It needs no /dev/kvm, so it also runs inside cloud VMs that cannot start VMs of their own.⁵ My test machine, an ARM server (with the kind of chip phones use) running NixOS, had no /dev/kvm, which is why I tried gVisor first. Its runtime is called runsc, and it plugs into Docker the same way.

The three options side by side.

Or as a table:

Isolation	What worker code talks to	Needs KVM	Usual escape route
Plain Docker (`runc`)	The host kernel, filtered by seccomp	No	A host kernel bug
gVisor (`runsc`)	gVisor’s own kernel, written in Go	No	A gVisor bug, then a host kernel bug through its small filtered surface
Kata Containers	Its own guest kernel in a microVM	Yes	A guest kernel bug, then a hypervisor bug

Each option also changes how much code on the host an escape can pick its bug from.

Lines of code on the host that deal with what the agent’s code asks for, counted with tokei, comments and blank lines included. For Docker, the core of Linux without drivers; for gVisor, its kernel and network stack without tests; for Kata, KVM alone, without the VMM and Kata’s other helpers. Not all of it is reachable. Hover for exact numbers.

The PR: six lines #

The change is PR #2886, which is now merged. It adds one setting, MINDROOM_DOCKER_WORKER_RUNTIME, and MindRoom passes its value to Docker as the runtime whenever it starts a worker. That is six lines of source code, plus 62 lines of tests and some docs. Changing the setting recreates each worker the next time it is used, and leaving it unset keeps everything as it was.

For gVisor, setting it up is two steps: register the runtime in Docker’s /etc/docker/daemon.json and restart Docker, then point MindRoom at it.

{
 "runtimes": {
 "runsc": { "path": "/usr/local/bin/runsc" }
 }
}

MINDROOM_DOCKER_WORKER_RUNTIME=runsc

Kata needs no entry in that file: once it is installed, MINDROOM_DOCKER_WORKER_RUNTIME=io.containerd.kata.v2 is enough. The full setup, and a few catches for each runtime, are in the MindRoom docs.

The six lines are short because everything else already existed. gVisor and Kata do the isolating. Docker already gives each worker its folders, its credentials, and its network, and it already knows how to hand a container to a different runtime. On the MindRoom side, the hard part happened months earlier, when agent tools moved out of the main process into dedicated workers with explicit mounts and credentials. Once every worker is a single call to Docker’s containers.run, swapping what runs underneath it is one argument. For comparison, Claude Opus estimated that building a microVM backend into MindRoom itself would take 2,500 to 4,000 lines.

What it costs #

I tested both runtimes with a real agent in a Matrix room: gVisor on that ARM server, and Kata in a throwaway VM on my desktop. Shell, Python, git, background processes, and the browser all worked, without any MindRoom code change. My Kata numbers below come from a VM inside a VM, so bare-metal numbers will differ.

Configuration is the cheap part: one environment variable, plus one entry in Docker’s config for gVisor. The other costs are the usual trade-offs between containers and VMs:

Computation. Code that only calculates, without asking the kernel for anything, runs at full speed with both. gVisor lets those instructions run directly on the CPU, according to its performance guide. A Kata guest runs on the CPU in its virtualization mode, and in AWS’s paper on Firecracker, the CPU overhead was negligible.
Syscalls, disk, and network. The cost starts when a program does ask the kernel for something. Under gVisor, every syscall takes a detour through gVisor’s kernel, so programs that make many small syscalls, like ones that open thousands of small files, slow down the most. Under Kata, disk and network go through the VMM’s fake devices. In that same paper from 2020, a Firecracker guest reached 15 Gbit/s of network traffic where the host reached 44.
Startup. gVisor only has to start a program, so it starts quickly. Kata has to boot a kernel first, and my worker took about 10 seconds to start instead of about 3. Booting a VM can be much faster than that: Firecracker boots a minimal one in about 125 ms.
Memory. gVisor adds a small, mostly fixed amount per sandbox. Each Kata VM runs a second kernel plus the VMM, which alone takes about 3 MB with Firecracker and 131 MB with QEMU. Kata gives each worker VM 2 GiB by default, and /tmp lives in that RAM. The host only hands out memory the guest actually touches, so my idle worker used under 0.8 GiB of host memory. Raising the VM size costs little until a worker needs it. But a VM cannot grow past its size the way a container can, and memory the guest has used, for example to keep recently read files in its cache, mostly stays with the VM.
Compatibility. gVisor reimplements Linux, and its docs say there will always be unimplemented features and bugs, so now and then a program will not work. Kata runs a real Linux kernel, so programs behave as usual, with a few catches. For example, a file lock held on the host does not block the worker, because the lock lives in the guest kernel.

What this does not fix #

A stronger sandbox protects the host. It does nothing about the tokens and network access you give the worker. An agent needs no kernel exploit to misuse a token that can push to your repositories, or to poke at every device on your home network. Scoped credentials and a firewall allowlist protect you at least as much, and MindRoom handles both separately: Agent Keys let agents use API keys without ever seeing them, and Approved Egress limits which servers a worker on Kubernetes can reach.

gVisor has had its own bugs, and it still talks to the host kernel, just through a much smaller door. KVM has bugs too, as this year’s escapes show. Both walls make an escape far harder, and neither makes it impossible.

Conclusion #

Once I started reading about this, I kept running into people saying containers were never a security boundary. That is true, and for years I did not worry about it. With AI finding kernel bugs faster and agents running code nobody reads, I do now.

No sandbox is perfectly safe, though. In a conversation with Dwarkesh Patel, Noam Brown doubted that even computers with no network connection at all would be enough to contain an AI. He pointed to studies where two such computers next to each other talk through heat: one runs its CPU hot, and the other picks up the change with its temperature sensor. Every option in this post trades some speed or memory for safety. How far to go depends on your own risk tolerance and how paranoid you are. MindRoom can now move its agents off the shared kernel with one environment variable. I do not use it myself yet, but if you are more paranoid than I am, it is there.

The other way out is a bug in the program that starts the container. In 2024, CVE-2024-21626 in runc, the program Docker uses for that, leaked a handle to a host folder into new containers, so a malicious image could reach the host’s files without any kernel bug. ↩︎
Docker can map the container’s root to an ordinary user on the host with user namespace remapping, or run without root altogether in rootless mode, but neither is the default. Rootless mode would have stopped the agent in my Docker story, because Docker then has only my own rights. Remapping would not, because anyone who can use Docker can turn it off for a single container. Neither helps against a kernel bug, because code running inside the kernel is above every user. ↩︎
A bug in the VMM’s fake devices, or in other VMM code that handles the guest’s traffic, can skip the first step, because a normal program in the guest can feed that code data through the network or a shared folder. CVE-2020-8608, a buffer overflow in QEMU’s user-mode networking, could be triggered that way with crafted network traffic. ↩︎
Like runc for Docker, Kata’s own programs on the host are a way out too. In 2020, one bug (CVE-2020-2023) let a container take over the guest without a kernel bug, and a second (CVE-2020-2026) then let the guest trick the Kata runtime on the host into mounting files over any host path. Neither needed a KVM bug. virtiofsd, the program that shares folders with the guest, also runs on the host and handles the guest’s file requests. ↩︎
gVisor can use KVM too, as an option. By default it catches the agent’s syscalls without it, which is the setup this post describes. With KVM switched on, gVisor runs the agent’s code in the CPU’s virtualization mode so catching syscalls is faster, but gVisor’s own kernel still answers them, and there is no guest Linux and no fake devices. gVisor recommends that mode only on bare metal; inside a VM the default is usually faster. ↩︎

Giving models the tools they were trained with can make them worse

Sun, 11 Oct 2026 00:00:00 +0000

A few nights ago, lying in bed, I had what felt like a galaxy-brain idea. I dictated it into my watch so I would still remember it in the morning.

Every frontier model learns to solve coding tasks with tools through reinforcement learning (RL), and I assume that training happens inside its vendor’s own coding agent: Anthropic’s models in Claude Code, and OpenAI’s models in Codex. That is a big assumption, since the labs do not publish their training environments, but it makes business sense: they sell those agents, so they have every reason to make their models work best in them. Every other harness, like pi, opencode, or my own MindRoom, gives the model a set of tools of its own design for running commands and editing files. The models are usually smart enough to figure those out, but not always. Sometimes a model assumes an edit tool works like the one it was trained with, and uses it wrong. This summer, that happened to Claude Opus 4.8 in pi, which I come back to below.

So the idea was simple: in MindRoom, give each model the same shell and file-editing tools as its native harness, and switch them automatically depending on which model is answering. Claude would see Claude Code’s Bash, Read, Edit, and Write. GPT would see Codex’s exec_command and apply_patch. And because the smallest agents do surprisingly well, I would go one step further: a minimal agent whose only tool is the shell each model knows from its training.

I was beyond excited. I could not really share that excitement locally, because my wife did not care, so I am sharing it with the internet instead.

Then I measured it against MindRoom’s own tools, and the familiar ones did not make the models more efficient. With only Codex’s tools, GPT-6 Astra did 84% more work, taking more steps and reading and writing more along the way, and used 59% more tokens. With Claude Code’s tools, Claude Sonnet 5.5 did 15% more work, and Claude only came out a few tokens cheaper because the descriptions I wrote for those tools are slightly shorter than MindRoom’s. Only one model clearly worked more efficiently with its own tools: Claude Opus 4.8, the model from the pi story. The models seem to generalize beyond the environment they were trained in, and what still matters is how much the agent sends with every request.

What I found

Compared with MindRoom’s own tools, the familiar ones did not make the models more efficient: with only Codex’s tools, GPT-6 Astra and Sol did 60 to 84% more work and used 43 to 59% more tokens.
With Claude Code’s tools, Claude Sonnet 5.5 did 15% more work and Opus 5.5 about the same; they only came out a few tokens cheaper because the descriptions I wrote for those tools are shorter than MindRoom’s.
Only Claude Opus 4.8, the model whose edits kept failing in pi, clearly worked more efficiently with its own tools.
What matters is how much the agent sends with every request: a minimal agent with one shell tool and a 12-line prompt used 58 to 76% fewer tokens than the same agent in standard mode on these short tasks.
Giving that minimal agent each model’s own shell tool did not make it better.

Table of contents

What a harness is #

“Harness” gets thrown around a lot, but there is no magic in it. Every coding agent talks to the model the same way: each request contains a system prompt, the definition of every tool the model may call, and the conversation so far. When the model answers with a tool call, the harness runs the tool, adds the result to the conversation, and sends everything again.

What a coding agent sends with every step. The light parts are the same every time; only the conversation grows.

So from the model’s side, a harness is three things: its system prompt, its tool definitions, and what comes back when a tool runs. The first two are just text, so they are easy to copy, and the tool definitions are effectively the harness’s API: change them, and the model sees a different harness. The third is code, which takes more work to copy.

Because every request resends everything, a task’s tokens come in two parts. The fixed part is what the agent sends before anything happens: its system prompt, its tool definitions, and the task, which every request resends. The work is everything the steps add on top: the model’s replies, the tool results, and the growing conversation that every later request resends too. When I say below that a model did more work, I mean that second part.

Why I believed it #

Two things convinced me: how well the smallest agents do, and what goes wrong when a model meets unfamiliar tools.

Smaller agents do better #

Pi made its name by being small. When Mario Zechner introduced pi last November, its system prompt and tool definitions together came in below 1,000 tokens, and it had four tools: read, write, edit, and bash. He also wrote that “pi does not and will not support MCP”, the Model Context Protocol through which most agents load outside tools. And he pointed to Terminus 2, a minimal agent from the Terminal-Bench team, which was “holding its own against agents with far more sophisticated tooling”.

In August, DeepSeek released DeepSeek Harness, which was all over Hacker News and the local AI subreddits I read. Its standard setup comes with a full set of tools. But for benchmarks, DeepSeek points to its minimal profile. In early September, that profile had exactly one tool, a persistent bash, and a one-line system prompt: “You are a helpful software engineer assistant.” (Ten days later, it already had a second tool, working_directory; staying that small seems to be hard.) That is what inspired the minimal mode I added to MindRoom in September, which gives an agent a short prompt and a single bash tool.

Models should know their own tools best #

The case that got me thinking about this was Claude Opus 4.8 in pi. In July, a pi user reported that about 20% of its edits failed in some sessions. Pi’s edit tool takes a list of replacements, and Opus 4.8 kept adding made-up fields to them, like in_file, matchCase, or newText2. Claude Sonnet 5 did it too, while Opus 4.7 and older Claude models never did in the same tests.

Armin Ronacher, one of pi’s maintainers, dug into it and compared it with what Claude Code does with the tool calls it receives. It turns out that Claude Code quietly repairs a lot of them:

It accepts several names for the same argument, like old_str and old_string.
It parses an argument that arrives as a string when it should be an object.
It fixes broken \uXXXX escapes in strings.
It drops fields it does not know.
When it cannot parse a call at all, it asks the model to try again.

His hypothesis is that RL itself might be the cause. If a model is trained in a harness that absorbs these mistakes, a slightly wrong tool call still completes the task and still gets rewarded, so nothing teaches the model not to make it. Pi now ignores unknown fields too, just like Claude Code.¹

Then pi itself strayed from its minimal roots. Pi 1.0 came out on October 1 and added MCP after all. The Register called it a 180. On Hacker News and r/LocalLLaMA, people worried that “Pi’s days as a nice minimal agent TUI are numbered”. What struck me was how the maintainers explained it. Armin Ronacher wrote that the models “are trained on their respective harnesses and we’re not here to fight their behavior”, and Mario Zechner that “we follow what the models are trained on”. That is exactly the native-tools half of my plan.

My plan combined both: a minimal agent, with each model’s native shell as its only tool. Terminal-Bench also said something I chose to ignore: agents like Terminus 2 do well with tools no model was trained on. I did not believe that part. In opencode and pi, I had repeatedly seen models get the edit tool wrong, so I assumed the benchmarks did not carry over to real work. My belief that a model works best in the harness it was trained in was that strong: I expected the native tools to help, and to help most in a minimal agent with nothing but a shell.

What I built #

MindRoom is my open-source platform for AI agents that live in Matrix chat rooms, where several agents can work together. Its agents get shell and file tools under MindRoom’s own names, like run_shell_command, read_file, and edit_file. In this pull request, I added what I call tool dialects, which show a model those tools in the shape of the coding agent it was trained in.

Model	Sees	Instead of
Claude	`Bash`, `BashOutput`, `KillShell`, `Read`, `Edit`, and `Write`, as in Claude Code	`run_shell_command`, `check_shell_command`, `kill_shell_command`, `read_file`, `edit_file`, and `write_file`
GPT	`exec_command`, `write_stdin`, and a freeform `apply_patch` that takes the patch as plain text, as in Codex, and a `kill_shell_command` that takes Codex’s `session_id`	`run_shell_command`, `check_shell_command`, `kill_shell_command`, `edit_file`, and `write_file`

Inside MindRoom, nothing changes: approvals, hooks, and the stored history keep MindRoom’s names, and the translation only happens on the way to and from the model. That gives MindRoom a property I like a lot: a conversation can switch from Claude to GPT halfway through, and each model sees the earlier tool calls in its own shape, as if it had made them in its own harness.

Matching the native harness exactly turned out to be a stretch, though. Codex has no tool for reading files, so in the pull request, GPT keeps MindRoom’s read_file, and both keep MindRoom’s grep, find_files, and ls, which makes each native set a mix. For the main comparison, I removed those extra tools, so each model saw only the tools of its own harness. The tools have the same names and arguments as in Claude Code and Codex, but I wrote short descriptions of my own, because Claude Code’s are not openly licensed. What they return is MindRoom’s output, and the system prompt is MindRoom’s too. So keep in mind for everything below: I tested the names and shapes of the tools, not the whole harness.

How the translation works

Codex’s apply_patch is freeform: the model writes the patch as plain text instead of JSON arguments. I ported Codex’s patch parser and the code that applies patches, together with Codex’s own test cases, so a patch from a Codex-trained model applies exactly as it would in Codex. Its description comes from Codex too, which is openly licensed.
A switched model sees earlier calls in its own shape only for the tools its own harness also has, like the shell: an edit made with Claude Code’s Edit stays an edit_file call for GPT, which has no matching tool.
An earlier version of the pull request also rewrote every tool result to look like the native one. That was a lot of fragile code, so I removed it before I measured anything.

How I measured it #

I used 12 terminal tasks from an evaluation suite I built for MindRoom, like writing a report from a CSV file, finding the commit that broke a check, or fixing failing tests.² A hidden check graded each result, and every setup ran every task three times. Almost every run passed, so the interesting number is how many tokens the agent needed to get there, cached input included, and cheaper in what follows means fewer tokens.³ The tool comparisons ran through MindRoom’s OpenAI-compatible API; minimal mode, which only works in chat, ran through Matrix. I tested five current models, Claude Opus 5.5 and Sonnet 5.5, and GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna, and later five older ones. The current Claude models ran with adaptive thinking, where the model decides when to think. I called the GPT models from MindRoom through OpenAI’s Codex backend, the endpoint the Codex CLI uses, so no run used Codex itself; with no reasoning effort set, the backend picks one.

Details of the setup

A small harness sent each task to MindRoom’s OpenAI-compatible API as a new conversation, and a hidden check graded the result in a separate container without network access. Every task ran three times per configuration, so each number in this post comes from 36 runs.
The agent’s commands ran in a Docker container, the way MindRoom runs agents it does not fully trust.
With adaptive thinking, the current Claude models thought before 15 to 25% of their replies, with MindRoom’s tools and with the native ones.
I ran the GPT models through OpenAI’s Codex backend, the endpoint the Codex CLI uses, and sent no reasoning effort, so the backend picked one. The GPT-6 models barely reasoned on these tasks: in those runs, only 4 to 12% of their requests had any reasoning tokens, and then 7 to 234 of them.

Native tools don’t make today’s models more efficient #

To make the comparison fair, I gave each model only the tools of its own harness, against MindRoom’s matching set.⁴ For Claude, that is a shell and tools to read, edit, and write files; for GPT, a shell and Codex’s apply_patch, against MindRoom’s tools to edit and write files.

Change in tokens per task when each model uses its native harness tools instead of MindRoom’s, with 95% confidence intervals: with only the harness’s own tools against MindRoom’s matching set, and with the mixed set in the pull request. Hover for values.

Claude got 6 to 13% cheaper with Claude Code’s tools, but only because the definitions I wrote for them are about 480 tokens shorter than MindRoom’s, and every request resends them; the tools themselves saved nothing, and Sonnet 5.5 even did 15% more work with them. It never even touched the file tools: Opus 5.5 and Sonnet 5.5 did everything in the shell, with either set.

GPT-6 Astra and Sol got much more expensive, while Luna barely changed. With only Codex’s tools, GPT-6 Astra did 84% more work and used 59% more tokens. It called apply_patch 30 times, as a separate step for edits it otherwise made inside its shell commands, and every extra step resends the whole conversation so far. I expected the opposite: these are the tools I assumed these models were trained on, so I thought they would need fewer and more confident steps. The mixed set, the lighter rows, points the same way.

The tool mistakes I wanted to prevent did not happen with MindRoom’s tools at all: with its matching set, none of the five models made a single tool error. The only errors came from the mixed set: next to Codex’s tools, Luna had to guess how MindRoom’s read_file resolves paths, and got it wrong seven times.

The numbers behind this section

Fixed part and work. Every request to a model sends the whole conversation so far, plus the definition of every tool the model can use. So I split each task’s tokens in two. The fixed part is the first request’s input, which holds the system prompt, the tool definitions, and the task, times the number of requests. The work is everything else: tool results and the model’s replies, counted again with every later request that resends them, so it also grows with the number of steps. Providers cache most of the fixed part between requests and charge much less for cached input, so it costs less than its share of the tokens suggests.

Only the harness’s own tools. With Claude Code’s tools, Opus 5.5 used 13% fewer tokens [95% CI −19 to −6%] and Sonnet 5.5 6% fewer [−10 to −1%]. Both came from the definitions: the Claude Code-style set I wrote is about 480 tokens shorter, while the work on top of it grew by 7% [−7 to +23%] and 15% [+7 to +25%]. With Codex’s tools, GPT-6 Astra used 59% more tokens [+53 to +65%] and GPT-6.1 Sol 43% more [+33 to +52%]. They called apply_patch 30 and 22 times, where with MindRoom’s tools they made only 3 and 6 calls to the file tools, so they needed 4.7 and 4.4 requests per task instead of 3.5 and 3.6 and did 84% [+73 to +96%] and 60% [+44 to +76%] more work. GPT-6 Luna used about the same with either set, +3% [−8 to +15%], and called apply_patch only 8 times. The pass rates stayed within one or two runs of each other, and no model made a tool error with either set.

The mixed set, as it is in the pull request.

Change in tokens per task when each model uses its native harness tools instead of MindRoom’s, with 95% confidence intervals. The second row caps the native shell output at the last 100 lines, MindRoom’s shell default. Hover for values.

With Claude Code’s tools, both Claude models used about 9% fewer tokens. With Codex’s tools, GPT-6 Astra used 31% more tokens and GPT-6.1 Sol 19% more. An earlier run of the same experiment, before some unrelated fixes in the pull request, gave almost the same numbers: 33% more for Astra and 15% more for Sol. For GPT-6 Luna, the difference was too small to tell apart from noise. The pass rates barely moved: Opus, Sonnet, and Sol passed all 36 runs with either set of tools, and Astra failed one run in some configurations. Luna passed 33 runs with MindRoom’s tools and 35 with Codex’s; in the earlier run it was the other way around, 35 against 32, so I would not read anything into it either.

It is not the longer output. MindRoom’s own shell tool returns the last 100 lines of a command’s output by default. For the native tools, I returned everything up to 50 KiB, because that is closer to what Claude Code and Codex do. So I ran everything again with the native tools capped at the same 100 lines, the second row in the chart above. The cap barely changes anything: Astra still uses 26% more tokens, Sol 15% more, and the Claude models still save about 8%. The models hardly ever print more than 100 lines anyway: in the uncapped run, 3 of Astra’s 91 shell results and 3 of Sol’s 93 were longer than that, and none of Claude’s 220.

Where the tokens went with the mixed set.

Tokens per task with MindRoom’s shell and file tools and with the native ones, split into the fixed part (light) and the work the model did on top of it (dark). Hover for the number of requests.

Claude did not work any differently with Claude Code’s tools. It never called a file tool with either set, did everything in the shell, and made about the same number of requests. Its work changed by 5% or less, and almost all of its savings come from the fixed part: the Claude Code-style definitions I wrote are about 480 tokens shorter than MindRoom’s.

GPT did work differently. With Codex’s tools, Astra and Sol edited files with apply_patch 17 and 15 times, where with MindRoom’s tools they made only 3 and 5 calls to the file tools and did the rest from the shell. Each patch is a request of its own, which accounts for their 0.3 to 0.5 extra requests per task. They also did 51% and 24% more work. And Codex’s tools take about 130 more tokens to describe than MindRoom’s, because apply_patch costs about 270 more than edit_file and write_file together, so the fixed part grew as well.

The mistakes. None of the five models made a single tool error with MindRoom’s tools in its 36 runs. Partly, the models rarely needed the file tools: the Claude models never called one, and of the GPT models, only Luna used them often. The only model that made tool errors was Luna, and only with the Codex tools: 7 errors in 36 runs, all of them read_file calls with a path relative to the directory Luna had just given exec_command. The made-up fields in pi always appeared inside its nested list of replacements, where the model has to write the JSON for each replacement itself. MindRoom’s edit_file is flat, like Claude Code’s Edit: a path, the old text, and the new text. So I suspected that keeping the tools as simple as the native ones matters more than matching their exact names.

Older models: only Opus 4.8 did better with them #

The pi story made me wonder whether this used to be different, since Opus 4.8 struggled with pi’s edit tool while Opus 4.7 did not. So I ran the same comparisons on older models that are still available: Claude Opus 4.7, Opus 4.8, and Sonnet 5, and GPT-5.5 and GPT-5.6 Sol.

Change in tokens per task when the older models use their native harness tools instead of MindRoom’s, with 95% confidence intervals: with only the harness’s own tools against MindRoom’s matching set, and with the mixed set in the pull request. Opus 4.7 and 4.8 ran without thinking, Sonnet 5 with adaptive thinking, and GPT with the effort Codex picks. Hover for values.

With only each harness’s own tools, the older models did not prefer their native tools either, except for one. The exception is the model from the pi story: Opus 4.8 did 17% less work and used 19% fewer tokens with Claude Code’s tools, reading files through Bash instead of with a separate tool. With the mixed set, the older Claude models seemed to prefer Claude Code’s tools, using 22 to 36% fewer tokens, but that came from the extra tools the mixed set keeps from MindRoom, grep, find_files, and ls: next to MindRoom’s other tools, those made the older Claude models take more steps and read more, and next to Claude Code’s tools, the models hardly touched them. And on the pi story itself: Opus 4.8 and Sonnet 5, the two models that made up fields in pi, made 111 edits with MindRoom’s edit_file, which takes a single path, old text, and new text, and none of them failed.

With only each harness’s own tools, the GPT-5 models did about the same with either set, and unlike GPT-6 Astra and Sol, they barely used apply_patch. So reaching for its native editing tool, and paying for it, is new in that generation.

Of the ten models I tested, familiar tools clearly made only one more efficient, the one that started it. The others did as well or better with tools they were presumably not trained on: they seem to generalize beyond the environment they were trained in.

The numbers behind this section

Only the harness’s own tools. With Claude Code’s tools, Opus 4.7 used 3% fewer tokens [−12 to +6%], Opus 4.8 19% fewer [−26 to −11%], and Sonnet 5 7% fewer [−29 to +23%], a noisy result. Opus 4.8 did 17% less work [−29 to −4%] on top of the shorter definitions, while the other older models’ work changed by −5 to +13%, none of it clearly; Opus 4.8 called MindRoom’s read_file 27 times, but never Claude Code’s Read. With Codex’s tools, GPT-5.5 used 5% more [−5 to +16%] and GPT-5.6 Sol 3% more [−4 to +11%]; they called apply_patch only 3 and 9 times, where with MindRoom’s tools they made 20 and 16 calls to the file tools. The pass rates stayed within one or two runs of each other. None of the older models made a tool error with MindRoom’s matching set; with Codex’s, GPT-5.5 sent garbled working directories to exec_command 11 times.

Why the mixed set made them look different. The only difference between the two tests is that the mixed set adds MindRoom’s grep, find_files, and ls to both sides. Next to MindRoom’s other tools, the older Claude models called ls and grep 20, 9, and 3 times, took 0.3 to 0.9 more requests per task, and did 35 to 38% more work than without them. Next to Claude Code’s tools, they called them once in total.

The older models with the mixed set.

Change in tokens per task when each model uses its native harness tools instead of MindRoom’s, with 95% confidence intervals, oldest models first in each family. Each model’s reasoning mode is in parentheses: Opus 4.7 and 4.8 do not think, the other Claude models think adaptively, and GPT uses the effort Codex picks. Hover for values.

Opus 4.7, Opus 4.8, and Sonnet 5 used 22 to 36% fewer tokens with Claude Code’s tools. Unlike the 5.5 models, they also made fewer requests and did 26 to 45% less work on top of the fixed part. GPT-5.6 Sol used 15% fewer tokens with Codex’s tools and did 27% less work, and GPT-5.5 used about the same. One generation later, the GPT-6 models use up to 31% more with Codex’s tools, and the Claude 5.5 models mostly save what the shorter definitions save. The pass rates stayed within one or two runs of each other for every model.

How they used the file tools. With MindRoom’s tools, the older Claude models called read_file 24, 29, and 44 times; with Claude Code’s, they called Read only 3, 1, and 6 times and read files through Bash instead. GPT-5.6 Sol needed 9 patches where it had made 32 calls to edit_file and write_file. Opus 4.8 and Sonnet 5 made 37 and 21 calls to MindRoom’s flat edit_file, and none of them failed. That does not prove that pi’s nested list caused pi’s failures, because I never gave them a nested edit tool, but a flat one did not trigger them. Their few tool errors were different ones: Sonnet 5 twice polled a background command that did not exist, GPT-5.5 and GPT-5.6 Sol each had one edit whose old text did not match, and GPT-5.5 sent garbled working directories to Codex’s exec_command seven times.

Only the shell tools.

Change in tokens per task when the older models use their native harness tools instead of MindRoom’s, with 95% confidence intervals: with shell and file tools, and with only the shell tools. Opus 4.7 and 4.8 ran without thinking, Sonnet 5 with adaptive thinking, and GPT with the effort Codex picks. Hover for values.

The native set mixes a model’s own tools with some of MindRoom’s, such as grep, ls, and, for GPT, read_file, so I also ran the older models with only the shell tools, in MindRoom’s shape and in the native one. Opus 4.7 then used 10% fewer tokens with the native shell, GPT-5.6 Sol 6% fewer, and GPT-5.5 the same, and almost all of that came from shorter definitions. Opus 4.8 used 23% fewer tokens, made fewer requests, and did 16% less work. Sonnet 5’s result is too noisy to read: in one run, the native shell returned a 50 KB output early on, and resending it with every later request made that run cost 12 times the median; its median run used 9% fewer tokens with the native shell.

The reasoning settings. Opus 4.7 and 4.8 do not think unless you ask them to, and Sonnet 5 thought before 42% of its replies, against 15 to 25% for the 5.5 models at the same adaptive setting. The GPT-5 models reasoned before 59 to 76% of their requests, while the GPT-6 models reasoned before only 4 to 12%. With adaptive thinking and the mixed set, Opus 4.7 and 4.8 thought before 26 to 36% of their replies, and still used 35% and 20% fewer tokens with Claude Code’s tools. For that test, I set thinking to adaptive in the model’s extra_kwargs, since these models reject a fixed thinking budget.

I also ran one current model of each family, Claude Sonnet 5.5 and GPT-6 Astra, at low and at high reasoning effort. For Astra, I set reasoning_effort to low or high; for Sonnet 5.5, output_config.effort, which is how the Claude 5.5 models take a reasoning effort.

Change in tokens per task when Claude Sonnet 5.5 and GPT-6 Astra use their native harness tools instead of MindRoom’s, at each reasoning effort, with 95% confidence intervals. With no effort set, Sonnet uses adaptive thinking, and Astra the effort Codex picks. Hover for the tokens per task.

Both models passed all 36 runs at every effort with either set of tools, except for one run that Astra failed with MindRoom’s tools at its default effort. A higher effort made Codex’s tools worse for GPT. At high effort, Astra made 4.5 requests per task with Codex’s tools against 3.6 with MindRoom’s, and used 45% more tokens. Even at high effort, though, Astra reasoned before only 10 to 12% of its requests, against 4 to 7% at its default, far below the GPT-5 models. For Claude, the native tools stayed cheaper at every level, and saved the most at low effort, 19%, where Sonnet thought before only 3 to 4% of its replies. Sonnet thought about as often at its default as at high effort, so I would not read much into the difference between those two.

What does matter: what the agent sends #

My next guess was the other half of my plan, a smaller agent. Every request to the model resends the system prompt and the definition of every tool, so an agent with a shorter prompt and fewer tools sends less with every step.

Fewer tools #

Fewer tools did help. With only MindRoom’s shell tools and none of its file tools, the agent used about a third fewer tokens and passed about as many runs. A shell-only agent is also where I still expected the native tools to matter most, because every step goes through the one tool whose shape the model knows best. They did not: GPT used the same number of tokens with Codex’s shell tools, and Claude saved only what its shorter descriptions save.

The numbers behind this section

Change in tokens per task, with 95% confidence intervals: with only MindRoom’s shell tools instead of all of MindRoom’s tools, and with only the native shell tools instead of MindRoom’s. Hover for values.

With only MindRoom’s three shell tools, for running, checking, and stopping a command, and none of its file tools, the agent used 27 to 41% fewer tokens. It passed about as many runs: Luna passed 31 instead of 33, too small a difference to tell apart, and the other four models passed all 36.

For the native shape, I gave Claude only Claude Code’s Bash, BashOutput, and KillShell, and GPT only Codex’s exec_command and write_stdin, with MindRoom’s kill_shell_command in Codex’s style. For GPT, the total did not move: Astra, Sol, and Luna used the same number of tokens, within 2%, and made the same number of requests. Codex’s shell tools alone take about 130 fewer tokens to describe than MindRoom’s, and GPT spent about as much on extra work. Claude Sonnet 5.5 used 18% fewer tokens and Claude Opus 5.5 6% fewer, mostly because the Claude Code-style shell tools are shorter to describe than MindRoom’s.

Minimal mode: one shell tool and a 12-line prompt #

MindRoom’s minimal mode, modeled on DeepSeek’s minimal profile, takes this all the way. The agent gets a 12-line prompt and a single bash tool instead of its normal prompt and tools. Its instructions, skills, context files, and memory move behind a command-line program inside that shell, which it can call when it needs them. Minimal mode only works in chat, so I ran this comparison in Matrix conversations, against the same agent in standard mode with only its shell tools. A chat adds context about the room to every request, but even so, the minimal agent used 21 to 52% fewer tokens than the shell-only agent through MindRoom’s API.

The difference starts before the model does anything. For Claude Sonnet 5.5, the prompt and tool definitions sent with every request shrink from 4,285 tokens to 661.⁵

What every request carries besides the conversation: the system prompt and the tool definitions, in standard and in minimal mode, from each provider’s own token counts. Hover for values.

Tokens per task in Matrix conversations: the same agent with only MindRoom’s shell tools, in standard mode and in minimal mode with its single bash tool, split into the fixed part (light) and the work the model did on top of it (dark). Hover for values.

Each bar splits a task’s tokens into the fixed part, the prompt and tool definitions resent with every request, and the work the model did on top of it. The minimal agent used 58 to 76% fewer tokens than the same agent in standard mode with only its shell tools, for every model. Most of that is input the provider had cached and bills at a fraction of the price, so the bill shrinks less than the token count does: counting only uncached input and output, the minimal agent used 16 to 42% fewer tokens. And the model did not pay for it with extra work: the dark part of the bars stayed about the same or shrank, except for Luna’s. I had also hoped that the smaller agent would need less to do the task itself, but only GPT-6 Astra and Sol did, with slightly fewer steps and about a quarter less output; Claude worked exactly the same in both modes.

These tasks are short, three or four requests each, which is why the prompt weighs so much. I tried to make them harder so they would take more steps, but the models were so good that they only took about 50% more requests, and for the two models I tried, Claude Sonnet 5.5 and GPT-6 Astra, the pattern stayed the same: Sonnet did the same work in both modes, and Astra less in minimal mode. For the models that save steps, I see no reason why longer tasks would erase that gain: every step saved also saves resending the conversation, so on long tasks it counts for more. In a long session, the growing conversation dominates and the prompt’s share of the tokens shrinks, but its saving per request stays: about 3,600 tokens for Claude and 2,200 for GPT. And a real agent’s prompt carries far more instructions, skills, and memory than this test agent’s, so it saves more per request, although fetching them through the shell when needed costs requests of its own.

The numbers behind this section

Why Matrix. Minimal mode only works in chat conversations, so I ran it through Matrix, next to the same agent with only its shell tools in standard mode. Each run got a fresh chat room with only the agent, named coder, and MindRoom’s router, which handles commands like !mode; for minimal mode, I switched the room over with !mode coder minimal before sending the task. In a Matrix conversation, the agent also gets context about the room and the conversation, so it uses more tokens than through the API: on the same code, with only the shell tools, Claude Opus 5.5 used 22,192 tokens per task in Matrix against 12,904 through the API. That is why I compare minimal mode with standard mode in Matrix.

The prompt. In standard mode, the system prompt is 2,295 tokens for Claude Sonnet 5.5 and 1,463 for GPT-6 Astra, and the tool definitions, the shell tools and one for inviting MindRoom’s router, add another 1,990 and 867. In minimal mode, the system prompt is 152 and 97 tokens, and the single bash tool adds 509 and 72. This test agent’s standard prompt is about 100 lines only because its own configuration is one role line and one instruction; a real agent’s prompt also carries its instructions, skills, context files, and memory, and can run to hundreds of lines, while its minimal prompt stays at 12. The model receives all of that, plus the task and the room’s context, again with every request, so it adds up: in standard mode, about 80% of all tokens were this fixed part.

The result. Its first request dropped from about 4,600 to 1,000 input tokens for Claude, and from about 2,600 to 410 for GPT. Counting only the uncached input and the output, the minimal agent used 16 to 42% fewer tokens. Even with the extra context of a Matrix conversation, it used fewer tokens than the agent with only MindRoom’s shell tools through the API on the same code: 21 to 52% fewer.

Is that a fair comparison? A shorter prompt saves tokens even if the model then has to work harder, so the real question is whether it had to. Mostly, it did not: the work stayed the same for both Claude models (1% and 3% less), and went down for Astra and Sol (38% and 13% less), which also needed fewer requests, about 3.0 per task instead of 3.3 to 3.4. Luna is the exception: it made more requests and did 40% more work in minimal mode, but the shorter prompt still more than paid for that. Counted per task without the resent conversation, the Claude models wrote and added to the conversation the same in both modes, Astra and Sol wrote about a quarter less and needed 0.2 to 0.4 fewer requests, and Luna added 33% more, mostly longer command output. Three of the five models failed one more run in minimal mode, mostly on the same script-writing task that other configurations also miss now and then. That is too small to tell apart here, but I will keep an eye on it with harder tasks.

Minimal mode with each model’s native shell tool #

That leaves the combination I had in mind from the start: the minimal agent, with each model’s native shell tool as its only tool. I patched minimal mode so that its single tool appears as Claude Code’s Bash for Claude and as Codex’s exec_command for GPT, with the same 12-line prompt.⁶

Change in tokens per task in minimal mode when its single tool is the model’s native shell tool, Claude Code’s Bash or Codex’s exec_command, instead of MindRoom’s bash, with 95% confidence intervals. The second row cuts the native tool’s output to the last 100 lines, MindRoom’s bash default. Hover for values.

It did not make the minimal agent better for any model. For Claude, it made little difference: Sonnet used about the same, and Opus a little more. For GPT, it was clearly worse, mostly because exec_command hands back a command’s whole output, and in an agent that sends so little else, a few long outputs weigh a lot. With its output cut to 100 lines, bash’s default and the light rows in the chart, only Sol still clearly used more; Astra’s and Luna’s smaller increases could be chance.

So the whole effect of the combination came from its minimal half: models that were trained with a carefully designed set of tools did best with one tool and almost no prompt, and the tool’s native shape did not make it better.

The numbers behind this section

The single native tool only runs commands, but no current model ever polled or stopped a command in any of these runs, so it lacked nothing they used. I ran it next to minimal mode with MindRoom’s own bash on the same code, and once more with the native tool’s output cut to the last 100 lines. That is bash’s default, which the GPT models set themselves on every call, more often lower than higher; in the end, the capped native tool and bash returned about as much, except for Astra, whose bash results were twice as long.

For Claude, Sonnet used 4% fewer tokens with the native tool and Opus 7% more. For GPT, it was 34 to 46% more tokens, mostly because exec_command returns a command’s whole output, so the models read 39 to 50 lines per result instead of 20 to 36. Cut to 100 lines, the native tool cost GPT 5 to 15% more, but only Sol’s increase is clearly more than chance; part of it is the definition, which takes about 90 more tokens than bash on every request. In minimal mode, the agent sends so little else that a few long outputs weigh much more than through the API: the cap cut Astra’s tokens by about 20% here, against 4% there. Claude rarely printed more than 100 lines, so I would not read anything into the gap between its two native rows. The pass rates differed by at most a few runs, too few to tell apart.

What this does and does not show #

I tested the names and arguments of the tools, not whole harnesses. Claude Code and Codex also have their own tool descriptions, system prompts, and output formats, Codex has an interactive terminal, and RL trains all of that together. The full harness might still help today’s models; matching only the tools did not.
The tasks are easy and short: 97% of all runs passed, with three to seven requests per task on average. Harder, longer tasks might separate the tool sets on success, not only on cost.
With 36 runs per configuration, a 15 to 30% difference in tokens is clear, but a difference of one or two passed runs is not.
I tune MindRoom’s agent setup on other tasks from the same 12 families, and a note on working habits from that tuning sits in the shell tool’s description. Both tool sets carry the same note, so it should not favor either.
The confidence intervals cover run-to-run variation on these 12 tasks, not the choice of tasks. Resampling the tasks as well widens them: GPT-6 Astra’s +59%, for example, becomes +38 to +81%, and Sonnet 5.5’s 6% saving becomes indistinguishable from none. Astra and Sol still used more tokens with Codex’s tools on every one of the 12 tasks, and the minimal agent used fewer on every task, for every model.
Most runs used each model’s standard reasoning mode, which differs between generations: adaptive thinking for Sonnet 5 and the 5.5 models, none for Opus 4.7 and 4.8, and the effort Codex picks for GPT. I tried other settings on four models: two current ones at low and high effort, and Opus 4.7 and 4.8 with adaptive thinking. The pattern held at every effort: Astra used 27 to 45% more tokens with Codex’s tools, and Sonnet 5.5 stayed slightly cheaper with Claude Code’s.

What happens to the pull request #

The pull request is still unmerged, and given these results, I am not sure I will merge it at all. For today’s models, the native tools either cost more, as Codex’s did for GPT-6 Astra and Sol, or, as Claude Code’s did for Claude, only saved the few tokens of the shorter descriptions I wrote for them, which MindRoom’s own tools could get too. The one model that clearly did better with its own tools, Claude Opus 4.8, is a generation behind. What I will do is use minimal mode a lot more, with MindRoom’s own bash.

Why this excites me #

Half of my plan was wrong, and I am happy about that. The minimal-harness, DeepSeek-style half held up; the native-tools half did not. The benchmarks I did not believe were right, at least for these models. It suggests that these models generalize much better than I gave them credit for. They were trained with one specific set of tools, and on these tasks they did just as well with tools they had presumably never seen in training, as long as those tools were simple, like an edit tool that takes one path, the old text, and the new text. Only Opus 4.8, the model that started all this, still did better with its native tools. What matters more than familiarity is how much the agent sends with every request: the prompt and the definition of every tool.

It also changes how I think about who gets to build the best agents. If models only worked well in the harness they were trained in, the labs would own the best agents by default. Instead, at least the skill of using unfamiliar tools seems to carry over to other harnesses. I suspect the same holds one level up, for how several agents work together. The labs presumably train their models in their own harness, but if what the models learn transfers this well, a harness built outside the labs, like MindRoom, could get just as good at letting agents collaborate. That is still a hunch, and testing it is what I want to do next.

Appendix: how I measured, and the numbers behind the charts #

All scripts, the tasks, and one row per trial, with its reasoning setting, are on GitHub next to this post’s source.

Show the setup and every configuration

Setup

Each configuration ran as a separate MindRoom instance with one agent and a fresh Docker container for its commands. The model, the tools, and everything else were fixed in that instance’s configuration.
The tool comparisons ran through MindRoom’s OpenAI-compatible API on the pull request at commit d73aafb51: the matching sets, the mixed sets with whole and with 100-line output, and both shell-only setups. The capped run changed only the output limit of the native shell tools.
The minimal-mode runs and their standard-mode comparison ran through Matrix on MindRoom’s main branch at release v2026.10.228, whose shell tools lack two arguments the pull request added, a working directory and a wait. A shell-only run through the API on that release used within 3% of the pull request’s tokens for every model except Luna, which used 8% fewer.
The combination ran through Matrix on the pull request with an experiment patch that shows minimal mode’s single tool in the model’s own dialect. On that code, minimal mode with MindRoom’s bash used within 10% of its tokens on main for every model.
Tokens per task are the input and output tokens of every model request in that task, from MindRoom’s per-request usage logs. The work is what is left after subtracting the fixed part, the first request’s input times the number of requests.
The confidence intervals come from resampling the three runs of each task 5,000 times, keeping the mix of tasks fixed.

Each cell shows passed runs out of 36, then tokens per task; the work columns show how much the work changed with the native tools.

Matching sets: each harness’s own tools against MindRoom’s matching set (pull request, through MindRoom’s API)

Model	Reasoning	MindRoom’s set	Harness’s own set	Work with the harness’s set
Claude Opus 5.5	adaptive thinking	36/36 · 15,661	35/36 · 13,628 (−13%)	+7%
Claude Sonnet 5.5	adaptive thinking	36/36 · 14,606	36/36 · 13,764 (−6%)	+15%
GPT-6 Astra	Codex’s choice	36/36 · 7,041	36/36 · 11,206 (+59%)	+84%
GPT-6.1 Sol	Codex’s choice	36/36 · 7,292	36/36 · 10,417 (+43%)	+60%
GPT-6 Luna	Codex’s choice	35/36 · 8,425	34/36 · 8,638 (+3%)	−5%
Claude Opus 4.7	no thinking	34/36 · 24,398	33/36 · 23,564 (−3%)	+13%
Claude Opus 4.8	no thinking	33/36 · 24,103	34/36 · 19,593 (−19%)	−17%
Claude Sonnet 5	adaptive thinking	33/36 · 29,781	35/36 · 27,833 (−7%)	+9%
GPT-5.5	Codex’s choice	35/36 · 10,819	35/36 · 11,353 (+5%)	+2%
GPT-5.6 Sol	Codex’s choice	36/36 · 10,624	34/36 · 10,991 (+3%)	−5%

For Claude, both sets have a shell and tools to read, edit, and write files; for GPT, a shell and Codex’s apply_patch against MindRoom’s tools to edit and write files. Neither has MindRoom’s grep, find_files, or ls.

Mixed sets: native tools against MindRoom’s tools (pull request, through MindRoom’s API; Claude with adaptive thinking, GPT with the effort Codex picks)

Model	MindRoom’s tools	Native tools	Native tools, 100-line output	Work with native tools
Claude Opus 5.5	36/36 · 18,943	36/36 · 17,244 (−9%)	36/36 · 17,410 (−8%)	+5%
Claude Sonnet 5.5	36/36 · 17,765	36/36 · 16,237 (−9%)	36/36 · 16,399 (−8%)	−4%
GPT-6 Astra	35/36 · 8,250	36/36 · 10,807 (+31%)	35/36 · 10,399 (+26%)	+51%
GPT-6.1 Sol	36/36 · 8,873	36/36 · 10,534 (+19%)	36/36 · 10,239 (+15%)	+24%
GPT-6 Luna	33/36 · 12,003	35/36 · 11,532 (−4%)	36/36 · 10,816 (−10%)	+15%

Older models, mixed sets: native tools against MindRoom’s tools (pull request, through MindRoom’s API)

Model	Reasoning	MindRoom’s tools	Native tools	Work with native tools
Claude Opus 4.7	no thinking	34/36 · 33,140	33/36 · 25,793 (−22%)	−26%
Claude Opus 4.8	no thinking	33/36 · 32,219	34/36 · 23,843 (−26%)	−32%
Claude Sonnet 5	adaptive thinking	33/36 · 42,349	34/36 · 27,120 (−36%)	−45%
GPT-5.5	Codex’s choice	34/36 · 15,007	36/36 · 14,419 (−4%)	−11%
GPT-5.6 Sol	Codex’s choice	36/36 · 15,062	35/36 · 12,809 (−15%)	−27%

Older models, only the shell tools: native shape against MindRoom’s (pull request, through MindRoom’s API; Opus 4.7 and 4.8 without thinking, Sonnet 5 with adaptive thinking, GPT with the effort Codex picks)

Model	MindRoom’s shell	Native shell
Claude Opus 4.7	33/36 · 17,518	33/36 · 15,714 (−10%)
Claude Opus 4.8	35/36 · 19,183	33/36 · 14,686 (−23%)
Claude Sonnet 5	33/36 · 22,624	33/36 · 28,354 (+25%)
GPT-5.5	36/36 · 9,056	36/36 · 9,046 (0%)
GPT-5.6 Sol	36/36 · 9,171	34/36 · 8,658 (−6%)

Sonnet 5’s native mean comes from one run that cost 271,106 tokens, after a 50 KB command output; its median is 9% lower with the native shell.

Opus 4.7 and 4.8 with adaptive thinking, mixed sets: native tools against MindRoom’s tools (pull request, through MindRoom’s API)

Model	Thinking replies	MindRoom’s tools	Native tools
Claude Opus 4.7	26% and 31%	35/36 · 35,610	33/36 · 23,090 (−35%)
Claude Opus 4.8	36% and 30%	33/36 · 25,269	33/36 · 20,130 (−20%)

Reasoning effort, mixed sets: native tools against MindRoom’s tools (pull request, through MindRoom’s API)

Model	Reasoning effort	MindRoom’s tools	Native tools
Claude Sonnet 5.5	low	36/36 · 15,154	36/36 · 12,330 (−19%)
Claude Sonnet 5.5	default	36/36 · 17,765	36/36 · 16,237 (−9%)
Claude Sonnet 5.5	high	36/36 · 18,286	36/36 · 15,630 (−15%)
GPT-6 Astra	low	36/36 · 8,361	36/36 · 10,627 (+27%)
GPT-6 Astra	default	35/36 · 8,250	36/36 · 10,807 (+31%)
GPT-6 Astra	high	36/36 · 8,934	36/36 · 12,918 (+45%)

Only the shell tools (pull request, through MindRoom’s API; Claude with adaptive thinking, GPT with the effort Codex picks)

Model	All of MindRoom’s tools	MindRoom’s shell only	Native shell only
Claude Opus 5.5	36/36 · 18,943	36/36 · 12,926 (−32%)	36/36 · 12,202 (−6%)
Claude Sonnet 5.5	36/36 · 17,765	36/36 · 12,455 (−30%)	36/36 · 10,168 (−18%)
GPT-6 Astra	35/36 · 8,250	36/36 · 6,025 (−27%)	36/36 · 6,052 (0%)
GPT-6.1 Sol	36/36 · 8,873	36/36 · 6,195 (−30%)	36/36 · 6,177 (0%)
GPT-6 Luna	33/36 · 12,003	31/36 · 7,118 (−41%)	34/36 · 7,018 (−1%)

MindRoom’s shell alone is compared with all of MindRoom’s tools, and the native shell with MindRoom’s shell.

Minimal mode against standard mode (main branch, through Matrix, shell tools only; Claude with adaptive thinking, GPT with the effort Codex picks)

Model	Standard mode	Minimal mode
Claude Opus 5.5	36/36 · 22,192	36/36 · 8,030 (−64%)
Claude Sonnet 5.5	35/36 · 22,448	34/36 · 7,828 (−65%)
GPT-6 Astra	36/36 · 11,429	36/36 · 2,798 (−76%)
GPT-6.1 Sol	36/36 · 11,070	35/36 · 3,370 (−70%)
GPT-6 Luna	35/36 · 12,568	34/36 · 5,219 (−58%)

Minimal mode: the native tool against MindRoom’s bash (pull request with the experiment patch, through Matrix; Claude with adaptive thinking, GPT with the effort Codex picks)

Model	MindRoom’s `bash`	Native tool	Native tool, 100-line output
Claude Opus 5.5	36/36 · 8,058	36/36 · 8,654 (+7%)	36/36 · 9,123 (+13%)
Claude Sonnet 5.5	34/36 · 7,787	34/36 · 7,514 (−4%)	34/36 · 8,000 (+3%)
GPT-6 Astra	35/36 · 3,071	36/36 · 4,173 (+36%)	36/36 · 3,359 (+9%)
GPT-6.1 Sol	34/36 · 3,103	33/36 · 4,150 (+34%)	36/36 · 3,575 (+15%)
GPT-6 Luna	34/36 · 4,697	34/36 · 6,849 (+46%)	32/36 · 4,926 (+5%)

One of Luna’s runs with the whole native output counts for passes but not for tokens: the provider reported no usage for one of its requests.

Armin also found that Anthropic’s strict tool mode, which restricts the model’s output to the tool’s schema, made these failures disappear in his tests. Pi later turned it on for Claude models, and a pi user then linked a different problem to it: in that user’s logs, 9.5% of Claude’s edits had mangled \u escapes for non-ASCII text, such as Korean, with strict mode on, and 1.5% with it off. ↩︎
The 12 tasks come from 12 families: a CSV report, finding the commit that broke a check, renaming an API across a package, counting errors in log files, fixing failing tests, finding files, extracting fields from JSON, writing a shell script, editing a configuration file, counting matching lines in very long command output, unpacking nested archives, and deduplicating and merging data files. I tune MindRoom’s agent setup on other instances of these families, and these 12 were held out from that. ↩︎
Claude and GPT count tokens differently, so compare the numbers within one model, not across models. Cached input counts in full here, although providers bill it at a fraction of the price, so a bill changes less than the token count. ↩︎
MindRoom’s toolkits accept exclude_tools, so both sets drop MindRoom’s grep, find_files, and ls, and for GPT also read_file, since Codex reads files through the shell. Both keep their shell’s tools for checking on and stopping a background command, which no model used in the matching-set runs. ↩︎
Anthropic adds its own instructions for tool use to every request that has tools, which is why MindRoom’s single bash tool costs Claude 509 tokens. I measured all of these numbers by sending each part on its own and reading the input tokens the provider reported. ↩︎
The patch is with this post’s reproduction scripts. It shows minimal mode’s single tool as MindRoom’s canonical run function, which the tool dialect then presents as Bash or exec_command; nothing else about minimal mode changes. ↩︎

Slowly, then all at once: AI agents that know how a company works

Wed, 30 Sep 2026 00:00:00 +0000

IonQ is the world’s leading vertically integrated, full-stack quantum platform company. We build quantum computers, quantum security solutions, quantum networks, and quantum sensors, and we fabricate our own quantum chips. That means thousands of people working on some of the hardest physics and engineering problems the world faces. Much of what we know is not written down in one place. Where a program actually stands lives in Slack threads, tickets, commits, documents, and meetings. If you want the real picture, somebody has to read all of it, or you ask a colleague who has probably explained it five times already. And as IonQ grows, everybody is new to something. Someone who just joined through an acquisition is new to all of it, and after three years here, I still could not explain what half of our programs do.

Why I built MindRoom #

Programming has always been my biggest passion, and when AI coding agents suddenly became extremely good, the jump in what I could do in a single day was the biggest I had ever seen. But that power only reaches people who are comfortable in a terminal. My mom will never open one. Neither will most of the physicists, hardware engineers, and program managers who make a company like IonQ work.

So I built MindRoom, which is open source, and I run it at IonQ as a central hub for collaboration and knowledge sharing, what I like to call a company brain. It looks like a chat app. Everybody gets a personal agent the moment they log in, running on its own sandboxed computer in the cloud. Every program can get a shared agent that is connected to that program’s Slack, Confluence, Jira, code, meeting notes, and even the parts database. Its answers link back to what a human wrote, so you can check them instead of trusting them.

The personal agent is where most people start. I can send mine a voice message like “every week, just before my one-on-one with my manager, give me a clear summary of what I’ve been up to.” I never told it who my manager is, when we meet, or where to look. It works that out on its own, sends me a link to the summary, and from then on does it every week without being asked again.

Slowly #

For months, MindRoom grew one small group at a time. It started with a single colleague, then a pilot of about ten, and then the team behind Superion 10K, our 10,000-qubit quantum computer. We rolled it out carefully, and each step at most doubled the group. Every new group needed a hand-held onboarding, so I recorded short videos to make that scale. The people who tried it kept using it, but it was hard to explain what it could do without showing it.

A voice message becomes a reminder

An approval card before anything is sent

Watch the agent's browser and take over

Demo recordings with a fictional company and scripted responses. More in the MindRoom showcase.

Then all at once #

That changed with a company-wide demo. Instead of explaining what agents can do, I played someone new to the Superion 10K program and asked its shared agent, by voice, to explain what we are building, how the work is split, and how the pieces fit together. A few minutes later it had made an interactive slide deck, built entirely from our own documents, Slack threads, and code, with every claim linked to its source.

Hundreds of people signed up that same day, and within weeks about a third of everyone who can create an account had one. At the time of the demo, five programs and teams used it every day, and now it is about fifteen. People had already had access to tools like Claude and ChatGPT for months, but those never clicked for them in the same way. An agent that already knew our work did.

The surprise #

I built MindRoom for people who never open a terminal, simple enough that even my mom could use it, so I expected non-technical colleagues to be the main audience. They did adopt it. Program managers use it to write weekly updates from what actually happened in Slack and Jira, instead of from whatever people remembered to tell them. Our configuration management team uses it to check the hardware as it was actually built against the engineering baseline, and to find policy documents that contradict each other across different systems.

What I did not expect was that some of the most capable engineers I know, who already run their own coding agents every day, would become some of the heaviest users. One of them, who already lived in Claude Code, cleared his entire inbox, Slack, and task list from his phone on a single flight. I think they like it for the same reason everyone else does: it keeps context. For writing code, dedicated tools like Claude Code and Codex are better, because they are built for exactly that.¹ But engineers use MindRoom as a layer for gathering knowledge. A coding agent does not need to know who you are or who you report to in order to fix a bug. Coordinating a large group of people is mostly that kind of context.

Institutional knowledge #

A colleague wrote about why AI’s biggest opportunity in large programs is institutional knowledge, from the perspective of someone using it every day. From the builder’s side, what I like most is that a shared agent gets better the more people use it.

When someone works through a problem with it, it keeps what it learned, so the next person benefits without ever having been in that thread.

In Claude Code or Codex, you handle memory yourself, by asking the agent to write things down in Markdown files. Agents get lazy: they forget to write things down, and later they forget to read them. MindRoom agents are also told to write down durable facts, in plain text files under version control, but they do not have to remember to. A background process picks out what is worth keeping from every conversation and adds it to a semantic search index. When a new message comes in, the relevant memories are looked up and added to the context automatically. So remembering and looking things up no longer depend on the agent deciding to do it. When someone asks a question that was already answered, the agent re-verifies its earlier analysis instead of starting from zero, so the second answer usually comes faster and builds on the first.

One very cool and yet simple idea is agent dreaming, which I picked up from a talk by Lance Martin from Anthropic. People collect short-term memories during the day and consolidate them while they sleep. The analogy is loose, but the idea carries over. Every night, a first pass reads the agent’s memories, every conversation it has had, and the original sources, and looks for contradictions, for example when someone corrected something a few minutes later and the two memories were never reconciled. It writes a patch to fix them. A second pass, in a completely fresh context, checks every proposed fix and accepts or rejects it. It rejects about half, which is exactly the kind of skepticism I want. Dreaming is not even platform code, just a skill, a short set of written instructions: you tell an agent to set up dreaming, and it knows what to do, including scheduling itself to run every night.

It also notices when two documents disagree, which is easy to miss when nobody reads all of them. Its reports end with “Corrections welcome, tell the agent and the next report improves,” and a correction becomes a memory that the next report uses. And when nobody ever wrote something down, it usually still knows who to ask, because it keeps track of what everybody is working on.

Why build it ourselves #

There are plenty of AI products that promise something similar. Building our own gives us complete flexibility and no vendor lock-in. We are not waiting for a third party to support the model, the system, or the feature we need. When a better model comes out, we can switch the same day. And the knowledge it builds up is stored in IonQ’s own cloud environment.

It also means we can experiment quickly. Features that could take a large organization months often take me days:

Agents can open a real browser on their own remote computer, not on your laptop, and you can watch it live and take over, for example to log in somewhere, before handing control back.
An MCP gateway lets people connect their local tools, like Claude Code or Codex, to every agent they have access to and all of those agents’ tools, so they benefit from what the agents have collected even outside MindRoom.
A day after OpenAI released GPT-Live-1 in its API, you could call your MindRoom agent: the same agent, with the same memory, which hands longer tasks to a background process while you keep talking. On my commute, I talk to my agent to plan my day and get things started.
Agents can host small sites, much like ChatGPT Sites, behind a temporary link with access control. That is how an agent can answer a question with an interactive presentation or a dashboard instead of a wall of text.

What it takes to run this at a company #

A demo on a laptop is easy. Running a central hub that hundreds of people rely on every day is a different problem, and most of my time went there.

People sign in with their normal company account, and their account and personal agent are created on first login. There is nothing to install.

Every agent runs in its own sandboxed container with no secrets inside it. Credentials are injected from outside, so the agent never sees a token. Agents also cannot reach arbitrary websites: outbound traffic is denied by default, and a new domain needs a person’s time-limited approval.

Reading is free, but anything that sends or changes something on your behalf, like an email, shows an approval card with exactly what will be sent and to whom. A shared agent only answers the people who work on that program, and it only reaches the systems it was granted.

It runs in IonQ’s own cloud environment, and a security review ran alongside the rollout. At its busiest moment so far, 200 people were writing to their agents at the same time, and it held up fine. The chat layer has plenty of room to grow: the French government runs Matrix for about 400,000 civil servants.

What’s next #

With our recent acquisitions and a lot of new hires, there is a steady flow of people joining IonQ who are new to all of it, and to MindRoom. Because MindRoom is built on Matrix, which is federated like email, groups with different data rules can each get their own instance and still work together where that is allowed. And whoever joins next can log in on their first day and ask their program’s agent how the work fits together and who to ask.

That does not mean you cannot write code from MindRoom. Because every agent has its own computer, it can run any software, so my MindRoom agent simply runs Claude Code or Codex itself when there is coding to do. ↩︎

Exploring the Jev hype in MindRoom

Wed, 23 Sep 2026 00:00:00 +0000

Last week, every AI newsletter I get was full of Jev, a new model from TypeSafe. So were several subreddits. I am a permanent lurker on r/LocalLLaMA, and even though it is about local models, every other post seemed to be about Jev. There were local clones like mini-jev and von, CLI wrappers like jev-cli, and even someone on Hacker News claiming they had built the same thing a year earlier.

I tried Jev in MindRoom, my open-source agent platform on Matrix, and less than two days later it was making three decisions there.

Update 2026-10-07: OpenAI released the Decisions API today, its own take on the same idea, and MindRoom now supports it as a third backend; see below.

What a System One model is #

The name comes from Daniel Kahneman’s Thinking, Fast and Slow, where System 1 is fast, intuitive thinking and System 2 is slow and deliberate. A System One model does not generate text. You send it a state, for example a chat log, plus typed questions, and it returns probabilities:

Type	Question	Answer
Noul	Yes or no	Probability of yes
Choice	Pick one of up to 255 options	Chosen option plus the full distribution
Score	Rate against a rubric of 2 to 10 levels	Probability-weighted value

TypeSafe claims 70 to 500 ms end to end and $0.042 per million input tokens, with output tokens free. They also write “we can’t prove it isn’t subsidized.” Either way, it is incredibly cheap and incredibly fast.

Adaptive participation #

In MindRoom, a conversation can have several humans, several agents, or both. When I talk to a single agent, it simply replies, and I don’t need to tag it. Once another person joins and writes something, the agent cannot tell whether the message was meant for it or for the other person. Nothing decided that, so the agent stayed quiet unless explicitly tagged. That confused several users.

So I recently added adaptive participation: an agent that has already replied in a thread decides on its own whether to answer an untagged message. I first used an LLM as the judge, GPT-5.6 Luna on low reasoning. A yes-or-no question like “should this agent respond?” is exactly what Jev is for, so it became the second backend. Compared with Luna, it is about ten times cheaper and ten times faster, and if I believe the benchmarks, it is also smarter.

“Thanks” should not stop the agent #

The next thing I used Jev for was interruptions. When I send a message while an agent is still replying, MindRoom injects a notice at the next tool call telling the agent to stop and wrap up, because there is a new message. But often I, and other users, would watch the agent start working and then write “looks good” or “thanks.” The agent would stop early and continue in the next turn, which is both wasteful and counterintuitive.

Now a judgment decides whether the new message needs the interruption, which the docs call mid-turn coalescing. If it does not, the agent reacts with 👀, finishes its reply, and handles the message afterwards.

This is also where I learned a lesson about evals. My first implementation worked at the code level but not functionally. My evals, based on real attempts inside MindRoom that had all failed plus synthetic cases from GPT-6 Astra, passed 5 out of 40 with Jev. The fix changed the question and gave the judge more context, and it went to 40 out of 40.

	Before: 5/40	After: 40/40
Question	“May the active task finish before the queued human messages are handled in a later turn?”	“Does any newly queued message require an immediate change to, pause of, or stop of the active task?”
When unclear	“Also false when context is incomplete, the relationship is unclear…”	“Mere relevance to the task does not make praise or thanks an interruption.”
Context	The request and the new messages	The same, plus the preceding conversation

The old question treated anything unclear as a reason to interrupt, and a bare “thanks” without the conversation around it always looks unclear.

Picking the responder #

The third decision is routing. When a message in a room with many agents does not mention anyone, MindRoom’s router picks who should answer. That used to be a free-text LLM prompt, “choose the most appropriate agent.” It is now a Jev Choice over the eligible agents, plus a no_fit option, as described in the router docs. A confident no_fit asks the user to mention an agent instead of guessing, and anything uncertain falls back to the existing LLM router.

One config line to switch #

I built a very simple abstraction layer so I can swap Jev for an LLM without changing any code. Every decision is one question with criteria for true and false:

MID_TURN_QUESTION = JudgmentQuestion(
 id="interrupt_current_turn",
 instructions="Does any newly queued message require an immediate change to, pause of, or stop of the active task?",
 when_true="A new message requests stopping or pausing the active task, corrects it, ...",
 when_false="The messages acknowledge progress, express thanks, ask to continue unchanged, ...",
)

Jev receives it as a Noul question and returns a probability, which MindRoom compares with a threshold. An LLM receives the same question and context and must return {"decision": true} or {"decision": false}. Switching is a configuration change:

agents:
 helper:
 mid_turn:
 defer_reaction: "👀"
 judgment:
 provider: typesafe  # or: provider: llm, model: fast
 threshold: 0.8

Both backends log their decisions and latency, so I can compare them on real traffic. The code is in src/mindroom/judgment/, and the backends are documented under participation judgment backends.

A third backend: OpenAI Decisions #

On October 7, OpenAI released its own version of the idea, the Decisions API, currently in public beta. The question types match Jev’s: a predicate is a Noul, and choice and score keep their names. It serves a single model, gpt-6-luna, and OpenAI says it returns answers “about 10x faster than the Responses API.” Input costs $0.10 per million tokens and output is free, so it lists at about 2.4 times Jev’s price.

Thanks to the abstraction layer, adding it took one PR, merged the same day, and it works for all three decisions, including the router’s Choice:

judgment:
 provider: openai_decisions
 threshold: 0.8

From MindRoom’s side, only the wire format differs. The endpoint accepts only user messages, so MindRoom sends each chat message as its own user message, prefixed with its role, like assistant: .... Decisions can also answer with a refusal, which MindRoom treats as an abstention, so the usual fallback applies.

I tested it against the live API through MindRoom’s own code, with the same questions Jev gets. For participation, it gave 0.99 to replying to a direct question and 0.0 while two humans were talking to each other. Mid-turn, “Thanks, looks great so far!” got 0.0 for interrupting and “Stop, keep MySQL…” got 1.0. The router sent a pandas bug to code and a meeting request to calendar, both with confidence 1.0. A nonsense request got only 0.62, below the 0.8 threshold, so the normal LLM router took over. Each call took 130 to 280 ms. I have not compared it with Jev on real traffic yet, but both log the same outcome fields, so that only takes a config change.

Named after Jevons #

I only read why TypeSafe named it Jev after building all this: they named it after William Stanley Jevons, who noticed that as steam engines burned coal more efficiently, demand for coal went up instead of down. This is known as the Jevons paradox. That was literally my experience. Most of what I built was perfectly possible with an LLM. I just never built it, because I am somewhat cost- and latency-minded. Once I implemented Jev for one decision, I came up with use case after use case.

One I already built is a plugin, Response Audit JEV. Right after an agent replies, Jev checks the answer against the request and the tool calls the agent actually made, for example whether things are properly cited. If a check flags a problem, the plugin posts one follow-up in the thread that tags the agent and asks for a correction. Because Jev is so fast and cheap, running this after every reply costs almost nothing.

Right now I am full of ideas. For example, a model router. In MindRoom, an agent can switch to a different model mid-conversation with a tool call, and I can do the same with !model. Agents rarely do that without being told, though. So I default to a very expensive model, and I only save money when I remember to switch to a cheap one for something simple. Jev could run on every message and decide whether it needs the capable model or not. That works in both directions, up to a capable model or down to a cheap one. The perfect moment to switch is when the prompt cache has expired. A prompt cache belongs to one model, so switching normally means sending the entire context again at full price. Once the cache has expired, I pay that price anyway, so switching costs nothing extra.

MindRoom: AI agents that live in Matrix and work everywhere 🧠

Fri, 26 Jun 2026 00:00:00 +0000

Table of contents

1. The problem: your AI is trapped in apps #

Programming is my biggest passion in life. I’ve been actively involved in open source for over ten years now, maintaining 40+ installable packages across Python, JavaScript, and Rust. Some of these have over 100,000 users; others have exactly one—me. I mention this because it matters for what comes next: when AI coding tools arrived, I didn’t start from zero. As I wrote about in my agentic coding post, AI has completely transformed how I work—when I’m AFK, I regularly build full features or entire projects by dictating to my phone and routing it through my own transcription software.

Through all of this, I became deeply convinced that AI agents are the future. Not just a useful tool for programmers—a fundamental shift in how everyone will interact with computers.

But if that’s true, then the way agents work today is broken.

We’ve built artificial barriers between artificial intelligences. ChatGPT knows you in one tab, Claude knows you in another, your Slack bot knows you in a third. Your email AI can’t tell your calendar AI about that meeting invitation. Your coding assistant has no idea about the project specs sitting in your document AI. Switch platforms, and your AI starts from scratch.

Most agent frameworks require you to program them. That works for developers like me, but if agents are truly the future, they need to work for everyone. Non-programmers don’t want to write code—they just want a chat.

And then there’s privacy. I’m fine sharing my email with Gemini—Google already owns my Gmail. But do I want to send my financial data, my health records, or my personal notes to a cloud provider? Not really. I’d rather use a local model for sensitive tasks and pick the best, cheapest Chinese model for general-purpose deep research.

So I built MindRoom on a chat protocol—if people just want a chat, give them a chat. Agents live in the same rooms as you, collaborate in threads, and follow you across every platform. The first commit landed on July 29, 2025 (four months before OpenClaw’s first commit 😅), so by the time I’m publishing this MindRoom has been with me for nearly eleven months: first as an obsessive side project, then, since March, with much more sustained focus. At first I got so obsessed with it that I eventually burned out and had to step away completely.

Recently, seeing OpenClaw gain massive traction solving a similar problem reminded me that what I’d been building wasn’t just a niche obsession. So here I am, dusting off MindRoom and writing about what it is, how it works, and why I think the approach still matters.

Live chat.mindroom.chat Personal thread with the prompt: “Explain how MindRoom and what you can do.” Mind answers in the real Matrix room, using tool calls and the Codex-backed local MindRoom runtime.

2. What is MindRoom? #

MindRoom is an open-source system that creates AI agents living inside the Matrix protocol. If you’re not familiar with Matrix—it’s a federated, end-to-end encrypted communication standard. The same protocol used by the French government for 5.5 million civil servants, by German healthcare for 150K+ organizations, and by the Element app that millions of people use daily.

The key insight: Matrix has bridges to many major platforms—Slack, Telegram, Discord, WhatsApp, IRC, email, even SMS.

So if your AI agent lives in Matrix, it can reach you on any platform. One agent, every platform, continuous memory.

One practical caveat: Matrix bridges vary in maturity. Some, like the Telegram bridge, work very well, while others can be finicky. Your mileage may vary depending on which platforms you need.

Here’s what a typical setup looks like in config.yaml:

agents:
 code:
 display_name: CodeAgent
 role: Generate code, manage files, execute shell commands
 model: opus-4.8
 tools: [file, shell, github]
 instructions:
 - Always read files before modifying them.
 rooms: [lobby, dev]

 research:
 display_name: ResearchAgent
 role: Search the web, summarize papers, find information
 model: gpt-5.5
 tools: [tavily, arxiv, wikipedia]
 rooms: [lobby, research]

teams:
 super_team:
 display_name: Super Team
 agents: [code, research]
 mode: collaborate

Define your agents, give them tools and rooms, and they show up in Matrix as real users—with avatars, typing indicators, and online status.

MindRoom agents are real Matrix users with their own accounts, generated avatars, display names, room memberships, and online presence.

3. How it actually works #

At the core sits what I call the MultiAgentOrchestrator—a class in bot.py that boots every configured entity (router, agents, teams), provisions Matrix user accounts for each one via matrix-nio, and keeps sync loops alive. The agents themselves are powered by the Agno framework, which provides a unified interface across AI model providers.

When someone sends a message in a Matrix room:

Explicit mentions go directly to the named agent (@mindroom_code help me debug this)
Thread continuations keep the same agent responding (no weird context switches mid-conversation)
New conversations hit the router—an AI that analyzes the message and picks the best agent based on capabilities

All conversations happen in threads, which keeps rooms organized. Agents stream their responses in real-time, editing a single message as they think rather than spamming new ones. You see tool calls happening live:

🔧 Tool Call: search_web(query="matrix protocol bridges")
[waiting for result...]
✅ search_web result:
[results here]

The recording below starts in the main Personal room, sends the prompt Explain how MindRoom and what you can do., and then follows MindRoom as it opens the new thread and streams the reply to completion. It is sped up 2.5x to keep the flow skimmable.

4. Building on Matrix: the good and the tricky #

One of the best things about building on Matrix is what you get for free. End-to-end encryption, a deeply interoperable chat protocol with bridges to many platforms, a choice of clients, federation between organizations—all of that comes with the protocol. You don’t have to build any of it yourself.

But because Matrix has such a tight specification, it also brings challenges.

The protocol doesn’t support streaming. AI agents that think for 30 seconds before dumping a wall of text make for a terrible chat experience, so I hacked streaming in by rapidly editing the same message as new tokens arrive. An ⋯ marker shows while the agent is still thinking—a small touch, but it makes the experience feel responsive and alive.

There’s also a size limit on message content. That’s fine for human chat, but AI responses can get long—especially when tool calls and their results are included. I worked around this by using Matrix’s attachment feature: when a response exceeds the limit, the content continues in an attachment that gets updated as the message keeps streaming in. This required forking the Cinny chat client so that attachments display inline rather than as downloadable files, making the whole thing seamless.

5. Memory that follows you #

MindRoom currently supports two memory implementations.

The first is the more traditional semantic memory system built on Mem0’s AsyncMemory, with configurable embedding providers (OpenAI, Ollama, or HuggingFace) and vector storage via ChromaDB. This gives agents searchable memory for preferences, project context, decisions, and recurring facts.

The second is a file-based memory system inspired by the simplicity of OpenClaw-style agent files. Instead of burying identity in a database, durable context can live in plain Markdown files that travel with the agent. That portability matters: I can move the same agent identity from OpenClaw to Hermes to MindRoom without starting from scratch.

MindRoom also has a memory flush step that periodically extracts durable facts from conversations and writes them back into memory. So the chat transcript remains chat, while stable preferences, decisions, and identity details become reusable context for future turns.

A live Personal memory demo: MindRoom stores a harmless preference in one turn and retrieves it later from the same Matrix conversation.

6. 100+ built-in tool integrations #

Agents can use over 100 built-in integrations:

Category	Examples
Communication	Gmail, Slack, Telegram, Discord
Development	GitHub, Shell, Python, Docker
Search	Tavily, Wikipedia, Arxiv, DuckDuckGo
Productivity	Google Calendar, Jira, Linear, Todoist
Smart Home	Home Assistant
AI/ML	DALL-E, ElevenLabs, Replicate
Data	Pandas, SQL, DuckDB, Yahoo Finance
Web	Firecrawl, Crawl4ai, Browser automation

Tools are lazy-loaded and credential-managed, so an agent only loads what it needs.

7. Teams: agents that collaborate #

Single agents are useful, but sometimes you need a team. MindRoom supports two collaboration modes:

Coordinate mode: A lead agent orchestrates others. You ask a question, the lead delegates subtasks, collects results, and synthesizes a unified response.

Collaborate mode: All agents work on the same task in parallel, each providing their independent analysis. The system then merges their responses with a consensus summary.

In practice, you might have a research team where one agent searches academic papers, another checks industry news, and a third validates claims—all triggered by a single message. The live room view below shows a real team response in the same Matrix thread surface normal agents use: threads, routed agents, and room history are all first-class chat objects rather than a separate dashboard.

Live Lobby thread in chat.mindroom.chat showing a real Team Response from HomeAssistant and AgentBuilder. Team collaboration uses the same Matrix-native interface as regular agent chats, with routed agents and room membership visible beside the conversation.

8. Hot-reload: change config and plugins without downtime #

config.yaml is watched at runtime. When you edit it—add an agent, change a model, update instructions—MindRoom diffs the old and new config, gracefully restarts only the affected agents, and has them rejoin their rooms. No downtime, no full runtime restart required.

The same development loop applies to plugins. MindRoom has a rich Python hook system, and plugins can be live-developed while the system is running. Change a plugin file, save it, and MindRoom automatically reloads the plugin without restarting the Python runtime.

This sounds minor, but when you’re iterating on agent behavior or plugin hooks, being able to tweak code and config and see results in seconds significantly improves the development loop.

9. Voice, scheduling, and other features #

Some features I built purely because I wanted them for myself (a recurring theme in my projects, as anyone who’s read my local AI journey can attest). I even built Matty, a Matrix CLI client, from bed at midnight because I needed a way to interact with my agents from the terminal:

Voice messages: Audio messages in Matrix are auto-transcribed via Whisper and treated as regular text input. You can talk to your agents.
Scheduled tasks: Natural language scheduling (!schedule "check my email every morning at 9 AM") backed by cron jobs. Agents can run tasks in the background and escalate to you when needed.
DM support: Agents respond naturally in 1:1 conversations without needing mentions.
Cross-organization federation: Because Matrix is federated, two companies’ AI agents can collaborate in a shared room—something that’s hard to do on proprietary platforms.

10. The obsession, the SaaS dream, and the burnout #

I need to be honest about what actually happened with MindRoom, because “life got in the way” is a sanitized version of the story.

What really happened is that I got completely, utterly obsessed. Every single second I wasn’t working at my day job or sleeping, I was working on MindRoom. Many hundreds of hours went into it. I seriously considered quitting my job and starting an AI startup around it.

If you’ve read Armin Ronacher’s post Agent Psychosis: Are We Going Insane?, you’ll know exactly what I’m talking about. The dopamine hit from building with AI agents is incredibly real. As Armin writes: “You feel productive, you feel like everything is amazing, and if you hang out just with people that are into that stuff too, without any checks, you go deeper and deeper into the belief that this all makes perfect sense.”

That was me. I was building and building, shipping feature after feature, and it felt incredible. The codebase grew to over 1,000 commits. I built the core system, then a React dashboard, then I started on a full SaaS platform—Kubernetes deployments on Hetzner Cloud, a FastAPI backend, a Next.js 15 frontend, Stripe integration, Supabase auth, Helm charts for multi-tenant isolation.

And that’s where the enthusiasm started to erode. Not because the core idea was bad, but because the work shifted from building interesting things (agent orchestration, memory systems, routing intelligence) to grinding through SaaS boilerplate: GDPR compliance, payment processing, automated Kubernetes deployments, terms of service, cookie banners. The dopamine loop that kept me going at 2 AM broke when the work stopped being creative and started being compliance paperwork.

The hosted control plane lives at app.mindroom.chat. I am deliberately not turning this section into a dashboard tour; the interesting part, and the part I still care about most, is the Matrix-native runtime underneath it.

11. OpenClaw and why it matters #

OpenClaw takes a different approach to the same core problem. Where MindRoom builds on Matrix federation as the backbone, OpenClaw runs a local Gateway on your machine and connects to messaging platforms directly (WhatsApp via Baileys, Telegram via grammY, Discord via discord.js, and so on).

The similarities are striking:

Both give you a single AI assistant across all your messaging platforms
Both have persistent memory
Both support multiple tools and integrations
Both are open-source and privacy-focused
Both let you choose your own AI model

The differences are interesting too. OpenClaw is local-first and TypeScript-based with 380K+ stars and a massive community. MindRoom is Python-based, federation-first, and… well, let’s just say my GitHub star count is a few orders of magnitude lower.

But seeing OpenClaw validate the core idea—that people want AI assistants that aren’t trapped in apps—reminded me that what I’d been building matters. The federation angle is something OpenClaw doesn’t have: the ability for agents from different organizations to collaborate natively, with end-to-end encryption (Olm/Megolm), on a protocol that governments already trust and deploy at scale.

MindRoom uses Matrix as the federated backbone; OpenClaw takes a local-gateway, direct-connector approach.

And then there’s Moltbook—a social network built exclusively for AI agents that the internet was losing its mind over for a couple of days. I’m not personally convinced it’s the next big thing, but what caught my attention is that MindRoom could support something like it natively. Matrix is already federated—agents on different servers can already interact, join shared rooms, and collaborate across organizational boundaries. A bridge from a Reddit-like platform to Matrix and your agents could participate without any special integration.

Building agent-to-agent infrastructure from scratch also means building security from scratch—and that’s where new platforms tend to struggle. Matrix has spent years hardening its E2E encryption, and it’s already deployed by governments and healthcare organizations at scale. Building on that foundation means you inherit those security properties rather than hoping to get them right yourself.

12. What’s next #

I first drafted this post a couple of months ago, when MindRoom was still something I was trying to come back to. That part is already out of date. Since January I’ve been spending nearly every free hour on it again, and since March it has effectively been my full-time work. The codebase supports 8+ AI model providers—OpenAI, Anthropic, Ollama, Groq, Google, OpenRouter, DeepSeek, Cerebras—and the core architecture is solid. It also has real users now. My rough guess is that something like 30-100 people have tried MindRoom in one form or another, which is still early, but meaningfully different from “only I have touched this.” There’s already a Docker Compose file and a starter repository to get started, but I want to make it even simpler.

What I’m focusing on now:

Skills system: Building an ecosystem of reusable agent behaviors (already partially implemented with OpenClaw-compatible format)
Even easier onboarding: A docker compose up gets you running today, but I want a wizard that gets you from zero to working agents without touching a YAML file
Community: Turning the small group of early users into a real operator and contributor community
Scope discipline: Keeping the work sustainable now that MindRoom is effectively full-time, not slipping back into the obsessive 2 AM marathon from before

I’m deliberately leaving the SaaS ambitions aside for now. The core system is what matters, and that’s where the energy is going.

If you’re interested in AI agents that live in Matrix and work everywhere, check out MindRoom on GitHub.

Sometimes seeing someone else succeed with a similar idea is enough to remember why you started building in the first place. And sometimes the most important lesson from a project isn’t technical—it’s learning when to step away and when to come back.

Are you running AI agents on any messaging platforms? Have you tried Matrix for anything beyond regular chat? I’d love to hear about your setups!

Mindroom | Bas Nijholt

Roll your own, fork the rest 🍴

Introduction: a follow-up to something I wrote #

1. Stars measure interest, not usage #

2. A popular protocol is not a popular library #

3. I checked the rule against my own code #

4. Roll your own: the library gives you the easy 10% #

5. Fork your own: when the problem is hard #

Agno: patches instead of a fork #

6. I am not dunking on open source, including mine #

7. My rule for the next uv add #

Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

What the kernel actually does #

A container is a process in a costume #

Why kernel bugs are a bigger problem now #

So I asked a model to break out #

Stronger walls: virtual machines #

Kata Containers and gVisor #

The PR: six lines #

What it costs #

What this does not fix #

Conclusion #

Giving models the tools they were trained with can make them worse

What a harness is #

Why I believed it #

Smaller agents do better #

Models should know their own tools best #

What I built #

How I measured it #

Native tools don’t make today’s models more efficient #

Older models: only Opus 4.8 did better with them #

What does matter: what the agent sends #

Fewer tools #

Minimal mode: one shell tool and a 12-line prompt #

Minimal mode with each model’s native shell tool #

What this does and does not show #

What happens to the pull request #

Why this excites me #

Appendix: how I measured, and the numbers behind the charts #

Slowly, then all at once: AI agents that know how a company works

Why I built MindRoom #

Slowly #

Then all at once #

The surprise #

Institutional knowledge #

Why build it ourselves #

What it takes to run this at a company #

What’s next #

Exploring the Jev hype in MindRoom

What a System One model is #

Adaptive participation #

“Thanks” should not stop the agent #

Picking the responder #

One config line to switch #

A third backend: OpenAI Decisions #

Named after Jevons #

MindRoom: AI agents that live in Matrix and work everywhere 🧠

1. The problem: your AI is trapped in apps #

2. What is MindRoom? #

3. How it actually works #

4. Building on Matrix: the good and the tricky #

5. Memory that follows you #

6. 100+ built-in tool integrations #

7. Teams: agents that collaborate #

8. Hot-reload: change config and plugins without downtime #

9. Voice, scheduling, and other features #

10. The obsession, the SaaS dream, and the burnout #

11. OpenClaw and why it matters #

12. What’s next #

Links and resources #

7. My rule for the next `uv add` #