AI | Bas Nijholt

Roll your own, fork the rest 🍴

Mon, 19 Oct 2026 00:00:00 +0000

Introduction: a follow-up to something I wrote #

In April 2025 I published Confessions of an asymmetric hypocrite, about my split personality on dependencies. Strict minimalism for the libraries I publish, and then, for applications:

For applications (especially CLIs in isolated environments), I’m far more liberal with dependencies to build features quickly.

I also wrote that I “strongly prefer dependencies that are already widely adopted and trusted by the community” because “these are less likely to cause unexpected issues”.

Then I spent a year building MindRoom, which is an application, exactly the category where I said I could relax.

I want to report back. “Prefer battle-tested dependencies” was the right rule. I just had the wrong idea of what battle-tested means. I thought a project with tens of thousands of GitHub stars qualified. Now I think it means millions of downloads, every day.

Table of contents

Update 2026-10-27: A week after publishing, I took my own advice twice. MindRoom’s knowledge search no longer uses ChromaDB, and code MindRoom owns has replaced Agno’s runtime; Agno stays only as the tool library. The full stories are in the comments.

1. Stars measure interest, not usage #

When I picked Agno as MindRoom’s AI framework, it looked like the safe choice. Today it has over 42,000 stars and 6,000 forks.¹

It also gets about 78,000 downloads a day on PyPI. httpx, one of Python’s standard HTTP libraries, gets about 25 million.²

A star is something you click once, when a README looks exciting. A download happens every time some project installs the package.

Most of those downloads are CI, not people, so downloads are not users either. But a package with millions of downloads a day is installed by a huge number of different projects. Many of them use the same functions you are about to use, so they hit the bugs before you do.

Battle-tested is also not the same as old. I use uv for everything, and it has only been around for a couple of years. I run into bugs in uv too, but there is simply no better alternative, and it gets about 3.6 million downloads a day.

2. A popular protocol is not a popular library #

The other place my old idea failed was Matrix.

Building on Matrix felt obviously correct. An open protocol, about a decade old, with serious security reviews from large organizations and governments. Millions of people use it.

Matrix is still a good protocol and I am still on it. But even though millions of people use Matrix, there are only a few client libraries.

MindRoom streams agent replies by editing the same message many times a second. Almost every part of the ecosystem I touched had a problem with that: Element, Cinny, matrix-nio, and the Matrix JavaScript SDK. In the JavaScript SDK, edits that arrived while a thread was still loading were silently dropped, so you saw stale text until you reloaded.³

What counts is how many projects use the library you build on. Element’s users do not test matrix-js-sdk for you. Other people building on matrix-js-sdk do. It gets roughly 100,000 downloads a day. matrix-nio, the Python client, about 23,000.

3. I checked the rule against my own code #

I pulled the daily download numbers for the packages I use and looked for workarounds, patches and forks in my own source.⁴

Package	Downloads per day	What it cost me
requests	49M	nothing I had to work around
pydantic	35M	nothing
numpy	33M	nothing
httpx	25M	nothing
aiohttp	18M	nothing
FastAPI	17M	nothing
chromadb	270K	workarounds for a bug that holds the GIL
matrix-js-sdk	~100K	patched
mem0ai	79K	largely reimplemented
agno	78K	126 documented workarounds
matrix-nio	23K	forked

Above a million a day, I had nothing to work around. Below it, every package on this list cost me a patch, a fork, a rewrite, or a pile of workarounds.

Take ChromaDB, the vector database behind MindRoom’s knowledge bases. In some cases it holds Python’s GIL during I/O, which blocks every other thread in the process. I opened a pull request to fix it, and nobody has looked at it yet. Instead of forking, I run every knowledge read in a short-lived child process, because a process can be killed and a blocked thread cannot.⁵

My fork of matrix-nio has about 8,900 lines added and 2,000 removed in the library code since the last upstream release.⁶

This does not mean every small package needs replacing. MindRoom pulls in a few hundred packages in total, and plenty of them have modest download numbers. The scrutiny should match how much I depend on a package. Something I use for one small, specific purpose is fine. Agno and matrix-nio are the base that MindRoom is built on, which is probably also why they hurt the most.

4. Roll your own: the library gives you the easy 10% #

Take memory libraries, Mem0 and friends. I have already written about this: I cloned LlamaIndex, LangChain, Letta, Mem0 and PydanticAI, studied how they work, and re-implemented the parts I needed. My memory system ended up “mostly modeled after Mem0”.

The core idea is not hard. With RAG, you chunk, embed, retrieve, re-rank and reconcile. Once you have read a couple of implementations you can write it, and afterwards you understand every line.

The same is true of an agent framework. It builds a JSON payload, posts it to an HTTP endpoint, parses a streaming response, keeps a list of messages, and loops when the model asks for a tool call. With AI, that is a day or two of work.

The hard part is making it reliable at scale: never blocking the event loop, and handling the crazy edge cases. With an agent system you end up building a very complicated state machine, and if you do not watch it, any combination of states is possible. Almost all of them are edge cases.

A library gives you the easy 10% and leaves you the hard 90%. And you do that 90% inside somebody else’s code, where you cannot fix the root cause.

MindRoom is now about 360,000 lines of Python, plus about 750,000 lines of tests.⁷ A good part of that is code I wrote instead of depending on a library:

Memory and knowledge: about 15,600 lines across 50 files.
Conversation history and compaction: about 8,400 lines.
An OpenAI-compatible API: about 2,300 lines, so MindRoom agents show up as selectable “models” in any chat frontend that speaks the OpenAI protocol.
Claude provider support: about 3,000 lines.

If I have to write the hard 90% either way, I might as well write the easy 10% too.

5. Fork your own: when the problem is hard #

I am not going to write my own end-to-end encrypted Matrix SDK, or my own homeserver. Those are hard, and also far below the line. So there I fork.

I forked the Python Matrix client. MindRoom ships its own fork of matrix-nio rather than the upstream package.

I forked the Matrix server, twice. First Synapse. When I ran into scaling problems I moved to Tuwunel and forked that.

I forked the web client. Cinny is the Matrix web client MindRoom uses.

None of these are AI projects. AI is just where I felt it most.

Whether a fork stays a fork depends on the maintainers. Tuwunel is extremely receptive to my patches, so I upstream anything that is general enough, and over the last six months I am its fourth most active contributor by commits.⁸ Cinny has not accepted any of my pull requests, so I forked it and stopped trying to upstream. Same with LibreChat: I had patches open for over a year without a reply, so I forked.

Keeping a fork up to date used to be miserable. With an LLM doing the rebasing, pulling in the latest upstream is now cheap, though still a bit of a risk every time. The bigger cost is that once you fork, you are responsible for the whole codebase, including the parts you never changed.

Agno: patches instead of a fork #

I did not fork Agno. I monkey-patched it. Agno is very actively developed, I have talked to the maintainer, and they know about these problems, so I hoped to upstream my fixes. I have filed 33 issues and pull requests there.⁹

That got out of hand. MindRoom now carries 126 documented workarounds for Agno across 59 files. Most of them live in 36 dedicated compatibility modules, about 7,200 lines of code, with the rest patched inline in 23 other files. Every one uses the same comment format, so I can find all of them with one search. Each one says what is wrong upstream, links the issue or PR, and says when I can delete it.¹⁰ A few examples:

Search failures become plausible empty results. A failed knowledge search and an empty index return the same empty list, so my agents confidently told me things did not exist.¹¹
Streamed tool calls get malformed or merged.¹²
Reasoning is silently lost when tool calls are replayed.¹³
Provider errors collapse into a generic 502, which throws away the error code you need to decide whether to retry.¹⁴

At this size, the patches amount to a fork.

6. I am not dunking on open source, including mine #

Everything I named is open source I use every day, for free, built by people doing their best. I filed issues and PRs, and where maintainers merged fixes I deleted my workarounds.

My own software is in the same category. Most new AI tooling is not very good, and MindRoom may well be included. It is large, fast-moving, and nowhere near millions of daily downloads. By my own rule, you should think twice before depending on it.

I also have to admit that I have not yet built a project the roll-your-own way from the start. So comparing “what I would do next time” with what I actually did is not entirely fair. This is what I think right now, and a year from now I may have to write another post correcting this one.

7. My rule for the next `uv add` #

I weigh three questions.

How many projects use it? Daily downloads of the actual package, not stars, and not how many people use the product built on top. In the millions: depend on it.

How much do I rely on it? Agno and matrix-nio are the base of MindRoom. A library I use in a CLI to format a question is not, and if I use it in a narrow way I do not need my own version.

How hard is it, and how receptive is upstream? If the library is something AI can write in a couple of days, roll my own. If it is hard, fork it. If the maintainers accept patches, upstream what is general enough.

If I started MindRoom over today, I would write the HTTP requests to the model providers myself. Not even the OpenAI or Anthropic SDKs. And even httpx is less safe than its download numbers suggest: in early 2026 its maintainer closed the issue tracker and discussions, and the latest stable release is from December 2024.¹⁵

In 2025 I wrote that the right answer is “it depends on the situation”, and I still believe that. I just had the wrong threshold. Battle-tested means millions of downloads a day. Below that: roll your own, and fork the rest.

From the GitHub API (api.github.com/repos/agno-agi/agno) on 2026-10-02: 42,385 stars and 6,041 forks. ↩︎
PyPI numbers are the last_day field of the pypistats.org API (/api/packages/<name>/recent), fetched on 2026-10-02, except uv, fetched on 2026-10-18 (3,648,827). The matrix-js-sdk number is npm’s weekly download count (about 730,000) divided by seven. Daily numbers wobble, so treat them as orders of magnitude. ↩︎
When events arrive for a thread that has not finished initializing, the SDK’s replay path never applies them, so an m.replace edit (the event type Matrix uses for “this replaces message X”) is stored and never shown. I replayed a captured event trace through the SDK: unpatched, zero of seventeen edits reached the listeners; patched, all seventeen. MindRoom carries the fix as a patch-package patch on the SDK’s thread.ts. ↩︎
For each direct dependency I looked at three things in the MindRoom repository: whether pyproject.toml pins a fork instead of the upstream package, whether patch-package patches it, and whether the source has workaround markers or comments pointing at upstream issues for it. The packages above a million downloads a day had none of those. ↩︎
The PR is titled “Release GIL during metadata I/O”. On the MindRoom side, knowledge/read_process.py starts the child process, and a TODO says to remove it once a Chroma release with the fix is pinned. A blocked thread holding the GIL cannot be interrupted from Python, but the kernel will always terminate a child process. ↩︎
Compared with upstream matrix-nio 0.25.2 (released 2024-10-04): git diff --shortstat on the library code gives 8,872 lines added and 1,966 removed across 52 files, from 89 commits, 76 of them mine. The fork is published on PyPI as mindroom-nio. ↩︎
Counted with git ls-files and wc -l on the MindRoom repository on 2026-10-18: 360,513 lines in 1,222 Python files under src/, 750,657 lines under tests/, and another 81,543 lines of TypeScript in the frontend. The per-feature numbers below come from the same count over the relevant directories. ↩︎
Non-merge commits on Tuwunel’s main branch since 2026-04-18: Jason Volk 1,332, June Strawberry 325, dasha_uwu 55, me 42. ↩︎
Issues and pull requests in agno-agi/agno with me as the author, counted on 2026-10-02. ↩︎
Every workaround starts with a comment like # AGNO_COMPAT: Knowledge search hides provider failures as empty results., followed by Reason:, Upstream issue:, Upstream PR:, Remove when: and the tests that cover it. The counts come from grep over src/: 126 AGNO_COMPAT: markers in 59 files, and 7,160 lines in the 36 agno_compat_* modules. ↩︎
Knowledge.search catches vector store and provider errors and returns []. agno#10150, partially fixed by agno#10152. ↩︎
When a streamed tool-call index does not start at zero, Agno leaves empty placeholder calls. agno#8879, fix proposed in agno#8880. ↩︎
With the OpenAI Responses API, Agno keeps only the last reasoning item, so it cannot be replayed with the tool calls it belonged to. agno#9960; partial fixes landed in agno#10075 and agno#10395. ↩︎
Unclassified ModelProviderError failures default to HTTP 502, so a real network blip and a permanent error look the same. agno#8869, partially addressed by agno#8870. ↩︎
The latest stable release on PyPI is 0.28.1, uploaded 2024-12-06. There have been 1.0 development pre-releases since, the latest in August 2026. The closing of issues and discussions was widely discussed in the Python community in March 2026. ↩︎

Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

Sun, 18 Oct 2026 00:00:00 +0000

For years, a Docker container was my default answer to “where do I run this?” I never thought it was a perfect sandbox. It was good enough, because getting out of one usually meant finding a specific kind of bug in the Linux kernel itself, and those were hard to find.¹

AI models have become very good at finding security bugs, including in Linux. Anthropic and OpenAI now keep their strongest security models, Claude Mythos and Daybreak Blue, behind identity checks and vetting. Open-weight models, which anyone can download and run, come with no such rules, and they trail the closed ones by only six to seven months. Once there is an open model as good as Mythos, anyone can strip out its refusals with a tool like Heretic (people call this abliteration) and point it at the Linux kernel. The security company DepthFirst used its own model to find a way out of a container and titled the write-up Containers Are No Longer a Security Boundary.

I care about this because running untrusted code in containers is exactly what MindRoom, my open-source system for AI agents that live in Matrix chat rooms, does. Every agent can get its own Docker worker, and its shell, file, Python, and browser tools run there. It can install anything, start background processes, and drive a real browser. And as I wrote in my ZFS post, I run coding agents in YOLO mode, exclusively. I do not review the commands they run, I look at the results.

So both sides changed at once. The code inside the container is now written by an AI, and the bugs that let it out are now found by an AI. A video from the Low Level channel asking whether containers are still safe made me sit down and rethink my setup.

I am not an expert in any of this. I have used containers, KVM, and hypervisors for years, including Proxmox before I moved to NixOS, but I never deeply understood what happens underneath. Building this forced me to learn it, and writing this post is my exercise in understanding it all. All of this is very complicated (as is everything new), and I am still learning. I am sure I skip over a lot and simplify the rest; what is here are the parts that helped me understand it. I thought it would be nice to share.

Table of contents

What the kernel actually does #

When people talk about Linux, they usually mean an operating system like Ubuntu, Fedora, or NixOS. Those differ in almost everything you see: package managers, desktops, config files. What they all have in common is the kernel, the part Linus Torvalds started in 1991 and still leads today. Strictly speaking, that kernel is the only part called Linux.

The kernel is the one program on a Linux machine that talks to the hardware directly. Everything else, your shell, Python, Chrome, Docker, runs on top of it and has to ask it for things. Open this file. Give me some memory. Send this packet. Start a new process. Those requests are called system calls, or syscalls for short.

There is one kernel per machine, and it decides who is allowed to do what. That makes it the most trusted code on the system. Most kernel bugs only crash the machine or cause glitches. But some let a normal program become root, the admin account, or even run its own code inside the kernel. At that point it can do anything on the machine, including leaving any container it was in.

A container is a process in a costume #

When you docker run -it ubuntu, you get a root shell in what feels like a fresh computer. Under the hood it is one ordinary process. The kernel dresses it up using three features:

Namespaces decide what the process can see. It gets its own list of processes (where it is process number 1), its own filesystem, and its own network. Root inside is still root on the host, with most of its powers taken away.²
cgroups decide how much it can use: CPU, memory, number of processes.
seccomp decides which syscalls it may make at all. It is a filter that blocks the ones a normal program never needs.

There is no second kernel. Every syscall from inside a container lands in the same kernel as every other container and the host itself. The kernel enforces the isolation, and it is also what gets attacked.

For most uses that is fine. Usually you put fixed software in a container: a database, a web app, some self-hosted service. You know where that code comes from, you keep your kernel patched, and that is good enough.

An AI agent is different. It writes new code on the spot and runs it, and nobody knows in advance what that code will do.

Since root in a container is root on the host, anyone who can use Docker can start a container as root and mount any folder from the host into it. They can then read and change every file in that folder, including files their own account may not open. Docker’s own documentation says the docker group grants root-level privileges. I knew that in theory, but it only clicked when I watched an AI agent do it on a machine where I had no root, only Docker access. It needed to change files it was not allowed to touch, so it started a small container, mounted their folder, and edited them from inside. No kernel bug needed.

Why kernel bugs are a bigger problem now #

Kernel bugs that let a normal program take over the whole machine used to take skilled people weeks or months to find and exploit. The famous ones, like Dirty COW in 2016, made the news every year or two, you patched, and that was it.

AI is speeding this up, and the DepthFirst write-up I mentioned is a good example. Their model found a bug in the kernel code for Unix sockets, which programs on the same machine use to talk to each other. The kernel freed a piece of memory and then kept using it, a so-called use-after-free, and from inside a container that was enough to get out. Their conclusion is that you should assume attackers can escape containers at will.

Blocking more syscalls, or taking more powers away from root, helps a little. But it only narrows which parts of the shared kernel a container can reach. This bug sat in Unix sockets, which almost every program uses, including the browser the agent drives, so blocking them would break the agent.

Now put an AI agent in that container. It reads web pages and repositories written by strangers, and one of those could talk it into running an exploit. On a MindRoom server, getting out of a worker means getting onto the machine that runs MindRoom itself. That is where the API keys, Matrix logins, and stored credentials live, which the workers are built to never see.

So I asked a model to break out #

Before changing anything, I wanted to know how real this was for my own setup. I gave a MindRoom agent Daybreak Blue and asked it to escape its worker container using known kernel vulnerabilities. Daybreak Blue is OpenAI’s security model for verified users, so it does the offensive security work that other models refuse.

It did not get out, which was reassuring. But it only tried known bugs. A kernel a few weeks behind on patches, or a bug nobody has published yet, would be a different story. So the attempt tells me the easy path was closed that day, and not much more.

Stronger walls: virtual machines #

The classic answer is a virtual machine. A container shares the host’s kernel, but a VM brings its own, and this is where the hardware comes in. Modern CPUs have a virtualization mode built in, called VT-x on Intel and AMD-V on AMD. In that mode, a second kernel, the guest kernel, runs directly on the real CPU and believes it is in charge, while the CPU quietly limits it. When the guest tries something it may not do on its own, like talking to a disk or a network card, the CPU pauses the guest and hands control to the host. This is called a VM exit. Despite the name, nothing escapes and nothing crashes. The host handles the request, for example by pretending to be that disk, and then resumes the guest where it stopped. From the guest’s point of view, the instruction simply finished.

Memory is fenced off too. The host gives the CPU a separate map of which real memory each guest may use, and the CPU checks every memory access against it. So even a guest kernel with full privileges cannot touch the host’s memory.

The CPU only provides the mechanism. The software that sets the rules and handles the VM exits is called the hypervisor. “Hypervisor” is a role, not one program, and on Linux it is two pieces:

KVM, a module in the Linux kernel, uses the CPU’s virtualization mode to run the guest, keeps that memory map, and catches its VM exits. It shows up as the device /dev/kvm.
A VMM (virtual machine monitor), a normal program on top, sets up the VM and handles the exits that KVM passes up, for example by pretending to be the guest’s disk and network card. QEMU, Firecracker, and Cloud Hypervisor are three VMMs you can pick from.

Proxmox, which I mentioned above, is KVM plus QEMU. AWS Lambda is KVM plus Firecracker.

Code in a VM talks to its own guest kernel. To reach the host, it usually has to break that kernel first and then the hypervisor.³ That is still possible. Google Project Zero turned a bug in how KVM runs VMs inside VMs on AMD into a full escape in 2021. Google’s kvmCTF pays up to $250,000 for a new one. And this year, one researcher, Hyunwoo Kim, published three KVM escapes, one of them hidden in KVM since 2010.

The difference between a container and a VM is how much code an attacker can reach, and that gap is large. On x86 (the Intel and AMD chips in most PCs and servers), Linux has about 370 syscalls, and Docker’s default seccomp filter blocks a few dozen of them. Behind the rest sit the kernel’s filesystems, networking, memory management, and core code: about 3.8 million lines, even without counting drivers. Not all of that is reachable from a container, but a lot of it is. KVM on x86 is about 114,000 lines. On top of KVM comes the VMM, but only the few devices it fakes for the guest. Most QEMU escapes came from its many emulated devices, which is why Firecracker and Cloud Hypervisor keep that list short.

Google’s bug bounties point the same way. kernelCTF pays $71,337 for each new kernel exploit that turns a normal user into root, and Google’s public repository holds exploits for 93 different kernel bugs. kvmCTF pays more than three times as much, and the same repository holds one KVM exploit, which crashes the host instead of escaping it. kvmCTF started a year later, but the gap is still big. Most of those kernel exploits would not work as written from a default container, because they first use a feature Docker blocks. Still, they show how many exploitable bugs sit behind the syscalls.

Kata Containers and gVisor #

Regular VMs are heavy. They boot a full operating system and tend to use a lot of memory. A microVM is a stripped-down VM meant to run one workload and start quickly. That still leaves a practical problem. MindRoom, like most tools, speaks Docker: it hands Docker an image and some folders to mount, and gets a running container back. A VM does not take a Docker image. Switching to VMs would normally mean building disk images, booting them, and wiring up files and networking myself.

Docker does not start containers by itself, though. It takes care of images, networks, and volumes, and then hands the actual start to a small program called a runtime. The default runtime, runc, does what I described earlier: it starts a process on the host kernel with namespaces, cgroups, and seccomp. But you can tell Docker to use a different runtime.

Kata Containers is such a runtime, and it starts a microVM instead. It picks one of the VMMs above (my test used QEMU), boots a small guest kernel, and starts the container inside it.⁴ Docker, and MindRoom, still see an ordinary container.

gVisor, from Google, takes a different route, and it is not a VM. It is a reimplementation of the Linux kernel, written in Go, that runs as a normal program on the host. Go manages memory for the programmer, so the use-after-free mistake behind the DepthFirst escape is much harder to make in gVisor. The worker’s syscalls go to gVisor instead of the host kernel, and gVisor itself only makes a small, filtered set of host syscalls. That is the hardening idea taken all the way: gVisor answers the agent’s syscalls itself, so the filter on the host can be much stricter than Docker’s. It needs no /dev/kvm, so it also runs inside cloud VMs that cannot start VMs of their own.⁵ My test machine, an ARM server (with the kind of chip phones use) running NixOS, had no /dev/kvm, which is why I tried gVisor first. Its runtime is called runsc, and it plugs into Docker the same way.

The three options side by side.

Or as a table:

Isolation	What worker code talks to	Needs KVM	Usual escape route
Plain Docker (`runc`)	The host kernel, filtered by seccomp	No	A host kernel bug
gVisor (`runsc`)	gVisor’s own kernel, written in Go	No	A gVisor bug, then a host kernel bug through its small filtered surface
Kata Containers	Its own guest kernel in a microVM	Yes	A guest kernel bug, then a hypervisor bug

Each option also changes how much code on the host an escape can pick its bug from.

Lines of code on the host that deal with what the agent’s code asks for, counted with tokei, comments and blank lines included. For Docker, the core of Linux without drivers; for gVisor, its kernel and network stack without tests; for Kata, KVM alone, without the VMM and Kata’s other helpers. Not all of it is reachable. Hover for exact numbers.

The PR: six lines #

The change is PR #2886, which is now merged. It adds one setting, MINDROOM_DOCKER_WORKER_RUNTIME, and MindRoom passes its value to Docker as the runtime whenever it starts a worker. That is six lines of source code, plus 62 lines of tests and some docs. Changing the setting recreates each worker the next time it is used, and leaving it unset keeps everything as it was.

For gVisor, setting it up is two steps: register the runtime in Docker’s /etc/docker/daemon.json and restart Docker, then point MindRoom at it.

{
 "runtimes": {
 "runsc": { "path": "/usr/local/bin/runsc" }
 }
}

MINDROOM_DOCKER_WORKER_RUNTIME=runsc

Kata needs no entry in that file: once it is installed, MINDROOM_DOCKER_WORKER_RUNTIME=io.containerd.kata.v2 is enough. The full setup, and a few catches for each runtime, are in the MindRoom docs.

The six lines are short because everything else already existed. gVisor and Kata do the isolating. Docker already gives each worker its folders, its credentials, and its network, and it already knows how to hand a container to a different runtime. On the MindRoom side, the hard part happened months earlier, when agent tools moved out of the main process into dedicated workers with explicit mounts and credentials. Once every worker is a single call to Docker’s containers.run, swapping what runs underneath it is one argument. For comparison, Claude Opus estimated that building a microVM backend into MindRoom itself would take 2,500 to 4,000 lines.

What it costs #

I tested both runtimes with a real agent in a Matrix room: gVisor on that ARM server, and Kata in a throwaway VM on my desktop. Shell, Python, git, background processes, and the browser all worked, without any MindRoom code change. My Kata numbers below come from a VM inside a VM, so bare-metal numbers will differ.

Configuration is the cheap part: one environment variable, plus one entry in Docker’s config for gVisor. The other costs are the usual trade-offs between containers and VMs:

Computation. Code that only calculates, without asking the kernel for anything, runs at full speed with both. gVisor lets those instructions run directly on the CPU, according to its performance guide. A Kata guest runs on the CPU in its virtualization mode, and in AWS’s paper on Firecracker, the CPU overhead was negligible.
Syscalls, disk, and network. The cost starts when a program does ask the kernel for something. Under gVisor, every syscall takes a detour through gVisor’s kernel, so programs that make many small syscalls, like ones that open thousands of small files, slow down the most. Under Kata, disk and network go through the VMM’s fake devices. In that same paper from 2020, a Firecracker guest reached 15 Gbit/s of network traffic where the host reached 44.
Startup. gVisor only has to start a program, so it starts quickly. Kata has to boot a kernel first, and my worker took about 10 seconds to start instead of about 3. Booting a VM can be much faster than that: Firecracker boots a minimal one in about 125 ms.
Memory. gVisor adds a small, mostly fixed amount per sandbox. Each Kata VM runs a second kernel plus the VMM, which alone takes about 3 MB with Firecracker and 131 MB with QEMU. Kata gives each worker VM 2 GiB by default, and /tmp lives in that RAM. The host only hands out memory the guest actually touches, so my idle worker used under 0.8 GiB of host memory. Raising the VM size costs little until a worker needs it. But a VM cannot grow past its size the way a container can, and memory the guest has used, for example to keep recently read files in its cache, mostly stays with the VM.
Compatibility. gVisor reimplements Linux, and its docs say there will always be unimplemented features and bugs, so now and then a program will not work. Kata runs a real Linux kernel, so programs behave as usual, with a few catches. For example, a file lock held on the host does not block the worker, because the lock lives in the guest kernel.

What this does not fix #

A stronger sandbox protects the host. It does nothing about the tokens and network access you give the worker. An agent needs no kernel exploit to misuse a token that can push to your repositories, or to poke at every device on your home network. Scoped credentials and a firewall allowlist protect you at least as much, and MindRoom handles both separately: Agent Keys let agents use API keys without ever seeing them, and Approved Egress limits which servers a worker on Kubernetes can reach.

gVisor has had its own bugs, and it still talks to the host kernel, just through a much smaller door. KVM has bugs too, as this year’s escapes show. Both walls make an escape far harder, and neither makes it impossible.

Conclusion #

Once I started reading about this, I kept running into people saying containers were never a security boundary. That is true, and for years I did not worry about it. With AI finding kernel bugs faster and agents running code nobody reads, I do now.

No sandbox is perfectly safe, though. In a conversation with Dwarkesh Patel, Noam Brown doubted that even computers with no network connection at all would be enough to contain an AI. He pointed to studies where two such computers next to each other talk through heat: one runs its CPU hot, and the other picks up the change with its temperature sensor. Every option in this post trades some speed or memory for safety. How far to go depends on your own risk tolerance and how paranoid you are. MindRoom can now move its agents off the shared kernel with one environment variable. I do not use it myself yet, but if you are more paranoid than I am, it is there.

The other way out is a bug in the program that starts the container. In 2024, CVE-2024-21626 in runc, the program Docker uses for that, leaked a handle to a host folder into new containers, so a malicious image could reach the host’s files without any kernel bug. ↩︎
Docker can map the container’s root to an ordinary user on the host with user namespace remapping, or run without root altogether in rootless mode, but neither is the default. Rootless mode would have stopped the agent in my Docker story, because Docker then has only my own rights. Remapping would not, because anyone who can use Docker can turn it off for a single container. Neither helps against a kernel bug, because code running inside the kernel is above every user. ↩︎
A bug in the VMM’s fake devices, or in other VMM code that handles the guest’s traffic, can skip the first step, because a normal program in the guest can feed that code data through the network or a shared folder. CVE-2020-8608, a buffer overflow in QEMU’s user-mode networking, could be triggered that way with crafted network traffic. ↩︎
Like runc for Docker, Kata’s own programs on the host are a way out too. In 2020, one bug (CVE-2020-2023) let a container take over the guest without a kernel bug, and a second (CVE-2020-2026) then let the guest trick the Kata runtime on the host into mounting files over any host path. Neither needed a KVM bug. virtiofsd, the program that shares folders with the guest, also runs on the host and handles the guest’s file requests. ↩︎
gVisor can use KVM too, as an option. By default it catches the agent’s syscalls without it, which is the setup this post describes. With KVM switched on, gVisor runs the agent’s code in the CPU’s virtualization mode so catching syscalls is faster, but gVisor’s own kernel still answers them, and there is no guest Linux and no fake devices. gVisor recommends that mode only on bare metal; inside a VM the default is usually faster. ↩︎

Giving models the tools they were trained with can make them worse

Sun, 11 Oct 2026 00:00:00 +0000

A few nights ago, lying in bed, I had what felt like a galaxy-brain idea. I dictated it into my watch so I would still remember it in the morning.

Every frontier model learns to solve coding tasks with tools through reinforcement learning (RL), and I assume that training happens inside its vendor’s own coding agent: Anthropic’s models in Claude Code, and OpenAI’s models in Codex. That is a big assumption, since the labs do not publish their training environments, but it makes business sense: they sell those agents, so they have every reason to make their models work best in them. Every other harness, like pi, opencode, or my own MindRoom, gives the model a set of tools of its own design for running commands and editing files. The models are usually smart enough to figure those out, but not always. Sometimes a model assumes an edit tool works like the one it was trained with, and uses it wrong. This summer, that happened to Claude Opus 4.8 in pi, which I come back to below.

So the idea was simple: in MindRoom, give each model the same shell and file-editing tools as its native harness, and switch them automatically depending on which model is answering. Claude would see Claude Code’s Bash, Read, Edit, and Write. GPT would see Codex’s exec_command and apply_patch. And because the smallest agents do surprisingly well, I would go one step further: a minimal agent whose only tool is the shell each model knows from its training.

I was beyond excited. I could not really share that excitement locally, because my wife did not care, so I am sharing it with the internet instead.

Then I measured it against MindRoom’s own tools, and the familiar ones did not make the models more efficient. With only Codex’s tools, GPT-6 Astra did 84% more work, taking more steps and reading and writing more along the way, and used 59% more tokens. With Claude Code’s tools, Claude Sonnet 5.5 did 15% more work, and Claude only came out a few tokens cheaper because the descriptions I wrote for those tools are slightly shorter than MindRoom’s. Only one model clearly worked more efficiently with its own tools: Claude Opus 4.8, the model from the pi story. The models seem to generalize beyond the environment they were trained in, and what still matters is how much the agent sends with every request.

What I found

Compared with MindRoom’s own tools, the familiar ones did not make the models more efficient: with only Codex’s tools, GPT-6 Astra and Sol did 60 to 84% more work and used 43 to 59% more tokens.
With Claude Code’s tools, Claude Sonnet 5.5 did 15% more work and Opus 5.5 about the same; they only came out a few tokens cheaper because the descriptions I wrote for those tools are shorter than MindRoom’s.
Only Claude Opus 4.8, the model whose edits kept failing in pi, clearly worked more efficiently with its own tools.
What matters is how much the agent sends with every request: a minimal agent with one shell tool and a 12-line prompt used 58 to 76% fewer tokens than the same agent in standard mode on these short tasks.
Giving that minimal agent each model’s own shell tool did not make it better.

Table of contents

What a harness is #

“Harness” gets thrown around a lot, but there is no magic in it. Every coding agent talks to the model the same way: each request contains a system prompt, the definition of every tool the model may call, and the conversation so far. When the model answers with a tool call, the harness runs the tool, adds the result to the conversation, and sends everything again.

What a coding agent sends with every step. The light parts are the same every time; only the conversation grows.

So from the model’s side, a harness is three things: its system prompt, its tool definitions, and what comes back when a tool runs. The first two are just text, so they are easy to copy, and the tool definitions are effectively the harness’s API: change them, and the model sees a different harness. The third is code, which takes more work to copy.

Because every request resends everything, a task’s tokens come in two parts. The fixed part is what the agent sends before anything happens: its system prompt, its tool definitions, and the task, which every request resends. The work is everything the steps add on top: the model’s replies, the tool results, and the growing conversation that every later request resends too. When I say below that a model did more work, I mean that second part.

Why I believed it #

Two things convinced me: how well the smallest agents do, and what goes wrong when a model meets unfamiliar tools.

Smaller agents do better #

Pi made its name by being small. When Mario Zechner introduced pi last November, its system prompt and tool definitions together came in below 1,000 tokens, and it had four tools: read, write, edit, and bash. He also wrote that “pi does not and will not support MCP”, the Model Context Protocol through which most agents load outside tools. And he pointed to Terminus 2, a minimal agent from the Terminal-Bench team, which was “holding its own against agents with far more sophisticated tooling”.

In August, DeepSeek released DeepSeek Harness, which was all over Hacker News and the local AI subreddits I read. Its standard setup comes with a full set of tools. But for benchmarks, DeepSeek points to its minimal profile. In early September, that profile had exactly one tool, a persistent bash, and a one-line system prompt: “You are a helpful software engineer assistant.” (Ten days later, it already had a second tool, working_directory; staying that small seems to be hard.) That is what inspired the minimal mode I added to MindRoom in September, which gives an agent a short prompt and a single bash tool.

Models should know their own tools best #

The case that got me thinking about this was Claude Opus 4.8 in pi. In July, a pi user reported that about 20% of its edits failed in some sessions. Pi’s edit tool takes a list of replacements, and Opus 4.8 kept adding made-up fields to them, like in_file, matchCase, or newText2. Claude Sonnet 5 did it too, while Opus 4.7 and older Claude models never did in the same tests.

Armin Ronacher, one of pi’s maintainers, dug into it and compared it with what Claude Code does with the tool calls it receives. It turns out that Claude Code quietly repairs a lot of them:

It accepts several names for the same argument, like old_str and old_string.
It parses an argument that arrives as a string when it should be an object.
It fixes broken \uXXXX escapes in strings.
It drops fields it does not know.
When it cannot parse a call at all, it asks the model to try again.

His hypothesis is that RL itself might be the cause. If a model is trained in a harness that absorbs these mistakes, a slightly wrong tool call still completes the task and still gets rewarded, so nothing teaches the model not to make it. Pi now ignores unknown fields too, just like Claude Code.¹

Then pi itself strayed from its minimal roots. Pi 1.0 came out on October 1 and added MCP after all. The Register called it a 180. On Hacker News and r/LocalLLaMA, people worried that “Pi’s days as a nice minimal agent TUI are numbered”. What struck me was how the maintainers explained it. Armin Ronacher wrote that the models “are trained on their respective harnesses and we’re not here to fight their behavior”, and Mario Zechner that “we follow what the models are trained on”. That is exactly the native-tools half of my plan.

My plan combined both: a minimal agent, with each model’s native shell as its only tool. Terminal-Bench also said something I chose to ignore: agents like Terminus 2 do well with tools no model was trained on. I did not believe that part. In opencode and pi, I had repeatedly seen models get the edit tool wrong, so I assumed the benchmarks did not carry over to real work. My belief that a model works best in the harness it was trained in was that strong: I expected the native tools to help, and to help most in a minimal agent with nothing but a shell.

What I built #

MindRoom is my open-source platform for AI agents that live in Matrix chat rooms, where several agents can work together. Its agents get shell and file tools under MindRoom’s own names, like run_shell_command, read_file, and edit_file. In this pull request, I added what I call tool dialects, which show a model those tools in the shape of the coding agent it was trained in.

Model	Sees	Instead of
Claude	`Bash`, `BashOutput`, `KillShell`, `Read`, `Edit`, and `Write`, as in Claude Code	`run_shell_command`, `check_shell_command`, `kill_shell_command`, `read_file`, `edit_file`, and `write_file`
GPT	`exec_command`, `write_stdin`, and a freeform `apply_patch` that takes the patch as plain text, as in Codex, and a `kill_shell_command` that takes Codex’s `session_id`	`run_shell_command`, `check_shell_command`, `kill_shell_command`, `edit_file`, and `write_file`

Inside MindRoom, nothing changes: approvals, hooks, and the stored history keep MindRoom’s names, and the translation only happens on the way to and from the model. That gives MindRoom a property I like a lot: a conversation can switch from Claude to GPT halfway through, and each model sees the earlier tool calls in its own shape, as if it had made them in its own harness.

Matching the native harness exactly turned out to be a stretch, though. Codex has no tool for reading files, so in the pull request, GPT keeps MindRoom’s read_file, and both keep MindRoom’s grep, find_files, and ls, which makes each native set a mix. For the main comparison, I removed those extra tools, so each model saw only the tools of its own harness. The tools have the same names and arguments as in Claude Code and Codex, but I wrote short descriptions of my own, because Claude Code’s are not openly licensed. What they return is MindRoom’s output, and the system prompt is MindRoom’s too. So keep in mind for everything below: I tested the names and shapes of the tools, not the whole harness.

How the translation works

Codex’s apply_patch is freeform: the model writes the patch as plain text instead of JSON arguments. I ported Codex’s patch parser and the code that applies patches, together with Codex’s own test cases, so a patch from a Codex-trained model applies exactly as it would in Codex. Its description comes from Codex too, which is openly licensed.
A switched model sees earlier calls in its own shape only for the tools its own harness also has, like the shell: an edit made with Claude Code’s Edit stays an edit_file call for GPT, which has no matching tool.
An earlier version of the pull request also rewrote every tool result to look like the native one. That was a lot of fragile code, so I removed it before I measured anything.

How I measured it #

I used 12 terminal tasks from an evaluation suite I built for MindRoom, like writing a report from a CSV file, finding the commit that broke a check, or fixing failing tests.² A hidden check graded each result, and every setup ran every task three times. Almost every run passed, so the interesting number is how many tokens the agent needed to get there, cached input included, and cheaper in what follows means fewer tokens.³ The tool comparisons ran through MindRoom’s OpenAI-compatible API; minimal mode, which only works in chat, ran through Matrix. I tested five current models, Claude Opus 5.5 and Sonnet 5.5, and GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna, and later five older ones. The current Claude models ran with adaptive thinking, where the model decides when to think. I called the GPT models from MindRoom through OpenAI’s Codex backend, the endpoint the Codex CLI uses, so no run used Codex itself; with no reasoning effort set, the backend picks one.

Details of the setup

A small harness sent each task to MindRoom’s OpenAI-compatible API as a new conversation, and a hidden check graded the result in a separate container without network access. Every task ran three times per configuration, so each number in this post comes from 36 runs.
The agent’s commands ran in a Docker container, the way MindRoom runs agents it does not fully trust.
With adaptive thinking, the current Claude models thought before 15 to 25% of their replies, with MindRoom’s tools and with the native ones.
I ran the GPT models through OpenAI’s Codex backend, the endpoint the Codex CLI uses, and sent no reasoning effort, so the backend picked one. The GPT-6 models barely reasoned on these tasks: in those runs, only 4 to 12% of their requests had any reasoning tokens, and then 7 to 234 of them.

Native tools don’t make today’s models more efficient #

To make the comparison fair, I gave each model only the tools of its own harness, against MindRoom’s matching set.⁴ For Claude, that is a shell and tools to read, edit, and write files; for GPT, a shell and Codex’s apply_patch, against MindRoom’s tools to edit and write files.

Change in tokens per task when each model uses its native harness tools instead of MindRoom’s, with 95% confidence intervals: with only the harness’s own tools against MindRoom’s matching set, and with the mixed set in the pull request. Hover for values.

Claude got 6 to 13% cheaper with Claude Code’s tools, but only because the definitions I wrote for them are about 480 tokens shorter than MindRoom’s, and every request resends them; the tools themselves saved nothing, and Sonnet 5.5 even did 15% more work with them. It never even touched the file tools: Opus 5.5 and Sonnet 5.5 did everything in the shell, with either set.

GPT-6 Astra and Sol got much more expensive, while Luna barely changed. With only Codex’s tools, GPT-6 Astra did 84% more work and used 59% more tokens. It called apply_patch 30 times, as a separate step for edits it otherwise made inside its shell commands, and every extra step resends the whole conversation so far. I expected the opposite: these are the tools I assumed these models were trained on, so I thought they would need fewer and more confident steps. The mixed set, the lighter rows, points the same way.

The tool mistakes I wanted to prevent did not happen with MindRoom’s tools at all: with its matching set, none of the five models made a single tool error. The only errors came from the mixed set: next to Codex’s tools, Luna had to guess how MindRoom’s read_file resolves paths, and got it wrong seven times.

The numbers behind this section

Fixed part and work. Every request to a model sends the whole conversation so far, plus the definition of every tool the model can use. So I split each task’s tokens in two. The fixed part is the first request’s input, which holds the system prompt, the tool definitions, and the task, times the number of requests. The work is everything else: tool results and the model’s replies, counted again with every later request that resends them, so it also grows with the number of steps. Providers cache most of the fixed part between requests and charge much less for cached input, so it costs less than its share of the tokens suggests.

Only the harness’s own tools. With Claude Code’s tools, Opus 5.5 used 13% fewer tokens [95% CI −19 to −6%] and Sonnet 5.5 6% fewer [−10 to −1%]. Both came from the definitions: the Claude Code-style set I wrote is about 480 tokens shorter, while the work on top of it grew by 7% [−7 to +23%] and 15% [+7 to +25%]. With Codex’s tools, GPT-6 Astra used 59% more tokens [+53 to +65%] and GPT-6.1 Sol 43% more [+33 to +52%]. They called apply_patch 30 and 22 times, where with MindRoom’s tools they made only 3 and 6 calls to the file tools, so they needed 4.7 and 4.4 requests per task instead of 3.5 and 3.6 and did 84% [+73 to +96%] and 60% [+44 to +76%] more work. GPT-6 Luna used about the same with either set, +3% [−8 to +15%], and called apply_patch only 8 times. The pass rates stayed within one or two runs of each other, and no model made a tool error with either set.

The mixed set, as it is in the pull request.

Change in tokens per task when each model uses its native harness tools instead of MindRoom’s, with 95% confidence intervals. The second row caps the native shell output at the last 100 lines, MindRoom’s shell default. Hover for values.

With Claude Code’s tools, both Claude models used about 9% fewer tokens. With Codex’s tools, GPT-6 Astra used 31% more tokens and GPT-6.1 Sol 19% more. An earlier run of the same experiment, before some unrelated fixes in the pull request, gave almost the same numbers: 33% more for Astra and 15% more for Sol. For GPT-6 Luna, the difference was too small to tell apart from noise. The pass rates barely moved: Opus, Sonnet, and Sol passed all 36 runs with either set of tools, and Astra failed one run in some configurations. Luna passed 33 runs with MindRoom’s tools and 35 with Codex’s; in the earlier run it was the other way around, 35 against 32, so I would not read anything into it either.

It is not the longer output. MindRoom’s own shell tool returns the last 100 lines of a command’s output by default. For the native tools, I returned everything up to 50 KiB, because that is closer to what Claude Code and Codex do. So I ran everything again with the native tools capped at the same 100 lines, the second row in the chart above. The cap barely changes anything: Astra still uses 26% more tokens, Sol 15% more, and the Claude models still save about 8%. The models hardly ever print more than 100 lines anyway: in the uncapped run, 3 of Astra’s 91 shell results and 3 of Sol’s 93 were longer than that, and none of Claude’s 220.

Where the tokens went with the mixed set.

Tokens per task with MindRoom’s shell and file tools and with the native ones, split into the fixed part (light) and the work the model did on top of it (dark). Hover for the number of requests.

Claude did not work any differently with Claude Code’s tools. It never called a file tool with either set, did everything in the shell, and made about the same number of requests. Its work changed by 5% or less, and almost all of its savings come from the fixed part: the Claude Code-style definitions I wrote are about 480 tokens shorter than MindRoom’s.

GPT did work differently. With Codex’s tools, Astra and Sol edited files with apply_patch 17 and 15 times, where with MindRoom’s tools they made only 3 and 5 calls to the file tools and did the rest from the shell. Each patch is a request of its own, which accounts for their 0.3 to 0.5 extra requests per task. They also did 51% and 24% more work. And Codex’s tools take about 130 more tokens to describe than MindRoom’s, because apply_patch costs about 270 more than edit_file and write_file together, so the fixed part grew as well.

The mistakes. None of the five models made a single tool error with MindRoom’s tools in its 36 runs. Partly, the models rarely needed the file tools: the Claude models never called one, and of the GPT models, only Luna used them often. The only model that made tool errors was Luna, and only with the Codex tools: 7 errors in 36 runs, all of them read_file calls with a path relative to the directory Luna had just given exec_command. The made-up fields in pi always appeared inside its nested list of replacements, where the model has to write the JSON for each replacement itself. MindRoom’s edit_file is flat, like Claude Code’s Edit: a path, the old text, and the new text. So I suspected that keeping the tools as simple as the native ones matters more than matching their exact names.

Older models: only Opus 4.8 did better with them #

The pi story made me wonder whether this used to be different, since Opus 4.8 struggled with pi’s edit tool while Opus 4.7 did not. So I ran the same comparisons on older models that are still available: Claude Opus 4.7, Opus 4.8, and Sonnet 5, and GPT-5.5 and GPT-5.6 Sol.

Change in tokens per task when the older models use their native harness tools instead of MindRoom’s, with 95% confidence intervals: with only the harness’s own tools against MindRoom’s matching set, and with the mixed set in the pull request. Opus 4.7 and 4.8 ran without thinking, Sonnet 5 with adaptive thinking, and GPT with the effort Codex picks. Hover for values.

With only each harness’s own tools, the older models did not prefer their native tools either, except for one. The exception is the model from the pi story: Opus 4.8 did 17% less work and used 19% fewer tokens with Claude Code’s tools, reading files through Bash instead of with a separate tool. With the mixed set, the older Claude models seemed to prefer Claude Code’s tools, using 22 to 36% fewer tokens, but that came from the extra tools the mixed set keeps from MindRoom, grep, find_files, and ls: next to MindRoom’s other tools, those made the older Claude models take more steps and read more, and next to Claude Code’s tools, the models hardly touched them. And on the pi story itself: Opus 4.8 and Sonnet 5, the two models that made up fields in pi, made 111 edits with MindRoom’s edit_file, which takes a single path, old text, and new text, and none of them failed.

With only each harness’s own tools, the GPT-5 models did about the same with either set, and unlike GPT-6 Astra and Sol, they barely used apply_patch. So reaching for its native editing tool, and paying for it, is new in that generation.

Of the ten models I tested, familiar tools clearly made only one more efficient, the one that started it. The others did as well or better with tools they were presumably not trained on: they seem to generalize beyond the environment they were trained in.

The numbers behind this section

Only the harness’s own tools. With Claude Code’s tools, Opus 4.7 used 3% fewer tokens [−12 to +6%], Opus 4.8 19% fewer [−26 to −11%], and Sonnet 5 7% fewer [−29 to +23%], a noisy result. Opus 4.8 did 17% less work [−29 to −4%] on top of the shorter definitions, while the other older models’ work changed by −5 to +13%, none of it clearly; Opus 4.8 called MindRoom’s read_file 27 times, but never Claude Code’s Read. With Codex’s tools, GPT-5.5 used 5% more [−5 to +16%] and GPT-5.6 Sol 3% more [−4 to +11%]; they called apply_patch only 3 and 9 times, where with MindRoom’s tools they made 20 and 16 calls to the file tools. The pass rates stayed within one or two runs of each other. None of the older models made a tool error with MindRoom’s matching set; with Codex’s, GPT-5.5 sent garbled working directories to exec_command 11 times.

Why the mixed set made them look different. The only difference between the two tests is that the mixed set adds MindRoom’s grep, find_files, and ls to both sides. Next to MindRoom’s other tools, the older Claude models called ls and grep 20, 9, and 3 times, took 0.3 to 0.9 more requests per task, and did 35 to 38% more work than without them. Next to Claude Code’s tools, they called them once in total.

The older models with the mixed set.

Change in tokens per task when each model uses its native harness tools instead of MindRoom’s, with 95% confidence intervals, oldest models first in each family. Each model’s reasoning mode is in parentheses: Opus 4.7 and 4.8 do not think, the other Claude models think adaptively, and GPT uses the effort Codex picks. Hover for values.

Opus 4.7, Opus 4.8, and Sonnet 5 used 22 to 36% fewer tokens with Claude Code’s tools. Unlike the 5.5 models, they also made fewer requests and did 26 to 45% less work on top of the fixed part. GPT-5.6 Sol used 15% fewer tokens with Codex’s tools and did 27% less work, and GPT-5.5 used about the same. One generation later, the GPT-6 models use up to 31% more with Codex’s tools, and the Claude 5.5 models mostly save what the shorter definitions save. The pass rates stayed within one or two runs of each other for every model.

How they used the file tools. With MindRoom’s tools, the older Claude models called read_file 24, 29, and 44 times; with Claude Code’s, they called Read only 3, 1, and 6 times and read files through Bash instead. GPT-5.6 Sol needed 9 patches where it had made 32 calls to edit_file and write_file. Opus 4.8 and Sonnet 5 made 37 and 21 calls to MindRoom’s flat edit_file, and none of them failed. That does not prove that pi’s nested list caused pi’s failures, because I never gave them a nested edit tool, but a flat one did not trigger them. Their few tool errors were different ones: Sonnet 5 twice polled a background command that did not exist, GPT-5.5 and GPT-5.6 Sol each had one edit whose old text did not match, and GPT-5.5 sent garbled working directories to Codex’s exec_command seven times.

Only the shell tools.

Change in tokens per task when the older models use their native harness tools instead of MindRoom’s, with 95% confidence intervals: with shell and file tools, and with only the shell tools. Opus 4.7 and 4.8 ran without thinking, Sonnet 5 with adaptive thinking, and GPT with the effort Codex picks. Hover for values.

The native set mixes a model’s own tools with some of MindRoom’s, such as grep, ls, and, for GPT, read_file, so I also ran the older models with only the shell tools, in MindRoom’s shape and in the native one. Opus 4.7 then used 10% fewer tokens with the native shell, GPT-5.6 Sol 6% fewer, and GPT-5.5 the same, and almost all of that came from shorter definitions. Opus 4.8 used 23% fewer tokens, made fewer requests, and did 16% less work. Sonnet 5’s result is too noisy to read: in one run, the native shell returned a 50 KB output early on, and resending it with every later request made that run cost 12 times the median; its median run used 9% fewer tokens with the native shell.

The reasoning settings. Opus 4.7 and 4.8 do not think unless you ask them to, and Sonnet 5 thought before 42% of its replies, against 15 to 25% for the 5.5 models at the same adaptive setting. The GPT-5 models reasoned before 59 to 76% of their requests, while the GPT-6 models reasoned before only 4 to 12%. With adaptive thinking and the mixed set, Opus 4.7 and 4.8 thought before 26 to 36% of their replies, and still used 35% and 20% fewer tokens with Claude Code’s tools. For that test, I set thinking to adaptive in the model’s extra_kwargs, since these models reject a fixed thinking budget.

I also ran one current model of each family, Claude Sonnet 5.5 and GPT-6 Astra, at low and at high reasoning effort. For Astra, I set reasoning_effort to low or high; for Sonnet 5.5, output_config.effort, which is how the Claude 5.5 models take a reasoning effort.

Change in tokens per task when Claude Sonnet 5.5 and GPT-6 Astra use their native harness tools instead of MindRoom’s, at each reasoning effort, with 95% confidence intervals. With no effort set, Sonnet uses adaptive thinking, and Astra the effort Codex picks. Hover for the tokens per task.

Both models passed all 36 runs at every effort with either set of tools, except for one run that Astra failed with MindRoom’s tools at its default effort. A higher effort made Codex’s tools worse for GPT. At high effort, Astra made 4.5 requests per task with Codex’s tools against 3.6 with MindRoom’s, and used 45% more tokens. Even at high effort, though, Astra reasoned before only 10 to 12% of its requests, against 4 to 7% at its default, far below the GPT-5 models. For Claude, the native tools stayed cheaper at every level, and saved the most at low effort, 19%, where Sonnet thought before only 3 to 4% of its replies. Sonnet thought about as often at its default as at high effort, so I would not read much into the difference between those two.

What does matter: what the agent sends #

My next guess was the other half of my plan, a smaller agent. Every request to the model resends the system prompt and the definition of every tool, so an agent with a shorter prompt and fewer tools sends less with every step.

Fewer tools #

Fewer tools did help. With only MindRoom’s shell tools and none of its file tools, the agent used about a third fewer tokens and passed about as many runs. A shell-only agent is also where I still expected the native tools to matter most, because every step goes through the one tool whose shape the model knows best. They did not: GPT used the same number of tokens with Codex’s shell tools, and Claude saved only what its shorter descriptions save.

The numbers behind this section

Change in tokens per task, with 95% confidence intervals: with only MindRoom’s shell tools instead of all of MindRoom’s tools, and with only the native shell tools instead of MindRoom’s. Hover for values.

With only MindRoom’s three shell tools, for running, checking, and stopping a command, and none of its file tools, the agent used 27 to 41% fewer tokens. It passed about as many runs: Luna passed 31 instead of 33, too small a difference to tell apart, and the other four models passed all 36.

For the native shape, I gave Claude only Claude Code’s Bash, BashOutput, and KillShell, and GPT only Codex’s exec_command and write_stdin, with MindRoom’s kill_shell_command in Codex’s style. For GPT, the total did not move: Astra, Sol, and Luna used the same number of tokens, within 2%, and made the same number of requests. Codex’s shell tools alone take about 130 fewer tokens to describe than MindRoom’s, and GPT spent about as much on extra work. Claude Sonnet 5.5 used 18% fewer tokens and Claude Opus 5.5 6% fewer, mostly because the Claude Code-style shell tools are shorter to describe than MindRoom’s.

Minimal mode: one shell tool and a 12-line prompt #

MindRoom’s minimal mode, modeled on DeepSeek’s minimal profile, takes this all the way. The agent gets a 12-line prompt and a single bash tool instead of its normal prompt and tools. Its instructions, skills, context files, and memory move behind a command-line program inside that shell, which it can call when it needs them. Minimal mode only works in chat, so I ran this comparison in Matrix conversations, against the same agent in standard mode with only its shell tools. A chat adds context about the room to every request, but even so, the minimal agent used 21 to 52% fewer tokens than the shell-only agent through MindRoom’s API.

The difference starts before the model does anything. For Claude Sonnet 5.5, the prompt and tool definitions sent with every request shrink from 4,285 tokens to 661.⁵

What every request carries besides the conversation: the system prompt and the tool definitions, in standard and in minimal mode, from each provider’s own token counts. Hover for values.

Tokens per task in Matrix conversations: the same agent with only MindRoom’s shell tools, in standard mode and in minimal mode with its single bash tool, split into the fixed part (light) and the work the model did on top of it (dark). Hover for values.

Each bar splits a task’s tokens into the fixed part, the prompt and tool definitions resent with every request, and the work the model did on top of it. The minimal agent used 58 to 76% fewer tokens than the same agent in standard mode with only its shell tools, for every model. Most of that is input the provider had cached and bills at a fraction of the price, so the bill shrinks less than the token count does: counting only uncached input and output, the minimal agent used 16 to 42% fewer tokens. And the model did not pay for it with extra work: the dark part of the bars stayed about the same or shrank, except for Luna’s. I had also hoped that the smaller agent would need less to do the task itself, but only GPT-6 Astra and Sol did, with slightly fewer steps and about a quarter less output; Claude worked exactly the same in both modes.

These tasks are short, three or four requests each, which is why the prompt weighs so much. I tried to make them harder so they would take more steps, but the models were so good that they only took about 50% more requests, and for the two models I tried, Claude Sonnet 5.5 and GPT-6 Astra, the pattern stayed the same: Sonnet did the same work in both modes, and Astra less in minimal mode. For the models that save steps, I see no reason why longer tasks would erase that gain: every step saved also saves resending the conversation, so on long tasks it counts for more. In a long session, the growing conversation dominates and the prompt’s share of the tokens shrinks, but its saving per request stays: about 3,600 tokens for Claude and 2,200 for GPT. And a real agent’s prompt carries far more instructions, skills, and memory than this test agent’s, so it saves more per request, although fetching them through the shell when needed costs requests of its own.

The numbers behind this section

Why Matrix. Minimal mode only works in chat conversations, so I ran it through Matrix, next to the same agent with only its shell tools in standard mode. Each run got a fresh chat room with only the agent, named coder, and MindRoom’s router, which handles commands like !mode; for minimal mode, I switched the room over with !mode coder minimal before sending the task. In a Matrix conversation, the agent also gets context about the room and the conversation, so it uses more tokens than through the API: on the same code, with only the shell tools, Claude Opus 5.5 used 22,192 tokens per task in Matrix against 12,904 through the API. That is why I compare minimal mode with standard mode in Matrix.

The prompt. In standard mode, the system prompt is 2,295 tokens for Claude Sonnet 5.5 and 1,463 for GPT-6 Astra, and the tool definitions, the shell tools and one for inviting MindRoom’s router, add another 1,990 and 867. In minimal mode, the system prompt is 152 and 97 tokens, and the single bash tool adds 509 and 72. This test agent’s standard prompt is about 100 lines only because its own configuration is one role line and one instruction; a real agent’s prompt also carries its instructions, skills, context files, and memory, and can run to hundreds of lines, while its minimal prompt stays at 12. The model receives all of that, plus the task and the room’s context, again with every request, so it adds up: in standard mode, about 80% of all tokens were this fixed part.

The result. Its first request dropped from about 4,600 to 1,000 input tokens for Claude, and from about 2,600 to 410 for GPT. Counting only the uncached input and the output, the minimal agent used 16 to 42% fewer tokens. Even with the extra context of a Matrix conversation, it used fewer tokens than the agent with only MindRoom’s shell tools through the API on the same code: 21 to 52% fewer.

Is that a fair comparison? A shorter prompt saves tokens even if the model then has to work harder, so the real question is whether it had to. Mostly, it did not: the work stayed the same for both Claude models (1% and 3% less), and went down for Astra and Sol (38% and 13% less), which also needed fewer requests, about 3.0 per task instead of 3.3 to 3.4. Luna is the exception: it made more requests and did 40% more work in minimal mode, but the shorter prompt still more than paid for that. Counted per task without the resent conversation, the Claude models wrote and added to the conversation the same in both modes, Astra and Sol wrote about a quarter less and needed 0.2 to 0.4 fewer requests, and Luna added 33% more, mostly longer command output. Three of the five models failed one more run in minimal mode, mostly on the same script-writing task that other configurations also miss now and then. That is too small to tell apart here, but I will keep an eye on it with harder tasks.

Minimal mode with each model’s native shell tool #

That leaves the combination I had in mind from the start: the minimal agent, with each model’s native shell tool as its only tool. I patched minimal mode so that its single tool appears as Claude Code’s Bash for Claude and as Codex’s exec_command for GPT, with the same 12-line prompt.⁶

Change in tokens per task in minimal mode when its single tool is the model’s native shell tool, Claude Code’s Bash or Codex’s exec_command, instead of MindRoom’s bash, with 95% confidence intervals. The second row cuts the native tool’s output to the last 100 lines, MindRoom’s bash default. Hover for values.

It did not make the minimal agent better for any model. For Claude, it made little difference: Sonnet used about the same, and Opus a little more. For GPT, it was clearly worse, mostly because exec_command hands back a command’s whole output, and in an agent that sends so little else, a few long outputs weigh a lot. With its output cut to 100 lines, bash’s default and the light rows in the chart, only Sol still clearly used more; Astra’s and Luna’s smaller increases could be chance.

So the whole effect of the combination came from its minimal half: models that were trained with a carefully designed set of tools did best with one tool and almost no prompt, and the tool’s native shape did not make it better.

The numbers behind this section

The single native tool only runs commands, but no current model ever polled or stopped a command in any of these runs, so it lacked nothing they used. I ran it next to minimal mode with MindRoom’s own bash on the same code, and once more with the native tool’s output cut to the last 100 lines. That is bash’s default, which the GPT models set themselves on every call, more often lower than higher; in the end, the capped native tool and bash returned about as much, except for Astra, whose bash results were twice as long.

For Claude, Sonnet used 4% fewer tokens with the native tool and Opus 7% more. For GPT, it was 34 to 46% more tokens, mostly because exec_command returns a command’s whole output, so the models read 39 to 50 lines per result instead of 20 to 36. Cut to 100 lines, the native tool cost GPT 5 to 15% more, but only Sol’s increase is clearly more than chance; part of it is the definition, which takes about 90 more tokens than bash on every request. In minimal mode, the agent sends so little else that a few long outputs weigh much more than through the API: the cap cut Astra’s tokens by about 20% here, against 4% there. Claude rarely printed more than 100 lines, so I would not read anything into the gap between its two native rows. The pass rates differed by at most a few runs, too few to tell apart.

What this does and does not show #

I tested the names and arguments of the tools, not whole harnesses. Claude Code and Codex also have their own tool descriptions, system prompts, and output formats, Codex has an interactive terminal, and RL trains all of that together. The full harness might still help today’s models; matching only the tools did not.
The tasks are easy and short: 97% of all runs passed, with three to seven requests per task on average. Harder, longer tasks might separate the tool sets on success, not only on cost.
With 36 runs per configuration, a 15 to 30% difference in tokens is clear, but a difference of one or two passed runs is not.
I tune MindRoom’s agent setup on other tasks from the same 12 families, and a note on working habits from that tuning sits in the shell tool’s description. Both tool sets carry the same note, so it should not favor either.
The confidence intervals cover run-to-run variation on these 12 tasks, not the choice of tasks. Resampling the tasks as well widens them: GPT-6 Astra’s +59%, for example, becomes +38 to +81%, and Sonnet 5.5’s 6% saving becomes indistinguishable from none. Astra and Sol still used more tokens with Codex’s tools on every one of the 12 tasks, and the minimal agent used fewer on every task, for every model.
Most runs used each model’s standard reasoning mode, which differs between generations: adaptive thinking for Sonnet 5 and the 5.5 models, none for Opus 4.7 and 4.8, and the effort Codex picks for GPT. I tried other settings on four models: two current ones at low and high effort, and Opus 4.7 and 4.8 with adaptive thinking. The pattern held at every effort: Astra used 27 to 45% more tokens with Codex’s tools, and Sonnet 5.5 stayed slightly cheaper with Claude Code’s.

What happens to the pull request #

The pull request is still unmerged, and given these results, I am not sure I will merge it at all. For today’s models, the native tools either cost more, as Codex’s did for GPT-6 Astra and Sol, or, as Claude Code’s did for Claude, only saved the few tokens of the shorter descriptions I wrote for them, which MindRoom’s own tools could get too. The one model that clearly did better with its own tools, Claude Opus 4.8, is a generation behind. What I will do is use minimal mode a lot more, with MindRoom’s own bash.

Why this excites me #

Half of my plan was wrong, and I am happy about that. The minimal-harness, DeepSeek-style half held up; the native-tools half did not. The benchmarks I did not believe were right, at least for these models. It suggests that these models generalize much better than I gave them credit for. They were trained with one specific set of tools, and on these tasks they did just as well with tools they had presumably never seen in training, as long as those tools were simple, like an edit tool that takes one path, the old text, and the new text. Only Opus 4.8, the model that started all this, still did better with its native tools. What matters more than familiarity is how much the agent sends with every request: the prompt and the definition of every tool.

It also changes how I think about who gets to build the best agents. If models only worked well in the harness they were trained in, the labs would own the best agents by default. Instead, at least the skill of using unfamiliar tools seems to carry over to other harnesses. I suspect the same holds one level up, for how several agents work together. The labs presumably train their models in their own harness, but if what the models learn transfers this well, a harness built outside the labs, like MindRoom, could get just as good at letting agents collaborate. That is still a hunch, and testing it is what I want to do next.

Appendix: how I measured, and the numbers behind the charts #

All scripts, the tasks, and one row per trial, with its reasoning setting, are on GitHub next to this post’s source.

Show the setup and every configuration

Setup

Each configuration ran as a separate MindRoom instance with one agent and a fresh Docker container for its commands. The model, the tools, and everything else were fixed in that instance’s configuration.
The tool comparisons ran through MindRoom’s OpenAI-compatible API on the pull request at commit d73aafb51: the matching sets, the mixed sets with whole and with 100-line output, and both shell-only setups. The capped run changed only the output limit of the native shell tools.
The minimal-mode runs and their standard-mode comparison ran through Matrix on MindRoom’s main branch at release v2026.10.228, whose shell tools lack two arguments the pull request added, a working directory and a wait. A shell-only run through the API on that release used within 3% of the pull request’s tokens for every model except Luna, which used 8% fewer.
The combination ran through Matrix on the pull request with an experiment patch that shows minimal mode’s single tool in the model’s own dialect. On that code, minimal mode with MindRoom’s bash used within 10% of its tokens on main for every model.
Tokens per task are the input and output tokens of every model request in that task, from MindRoom’s per-request usage logs. The work is what is left after subtracting the fixed part, the first request’s input times the number of requests.
The confidence intervals come from resampling the three runs of each task 5,000 times, keeping the mix of tasks fixed.

Each cell shows passed runs out of 36, then tokens per task; the work columns show how much the work changed with the native tools.

Matching sets: each harness’s own tools against MindRoom’s matching set (pull request, through MindRoom’s API)

Model	Reasoning	MindRoom’s set	Harness’s own set	Work with the harness’s set
Claude Opus 5.5	adaptive thinking	36/36 · 15,661	35/36 · 13,628 (−13%)	+7%
Claude Sonnet 5.5	adaptive thinking	36/36 · 14,606	36/36 · 13,764 (−6%)	+15%
GPT-6 Astra	Codex’s choice	36/36 · 7,041	36/36 · 11,206 (+59%)	+84%
GPT-6.1 Sol	Codex’s choice	36/36 · 7,292	36/36 · 10,417 (+43%)	+60%
GPT-6 Luna	Codex’s choice	35/36 · 8,425	34/36 · 8,638 (+3%)	−5%
Claude Opus 4.7	no thinking	34/36 · 24,398	33/36 · 23,564 (−3%)	+13%
Claude Opus 4.8	no thinking	33/36 · 24,103	34/36 · 19,593 (−19%)	−17%
Claude Sonnet 5	adaptive thinking	33/36 · 29,781	35/36 · 27,833 (−7%)	+9%
GPT-5.5	Codex’s choice	35/36 · 10,819	35/36 · 11,353 (+5%)	+2%
GPT-5.6 Sol	Codex’s choice	36/36 · 10,624	34/36 · 10,991 (+3%)	−5%

For Claude, both sets have a shell and tools to read, edit, and write files; for GPT, a shell and Codex’s apply_patch against MindRoom’s tools to edit and write files. Neither has MindRoom’s grep, find_files, or ls.

Mixed sets: native tools against MindRoom’s tools (pull request, through MindRoom’s API; Claude with adaptive thinking, GPT with the effort Codex picks)

Model	MindRoom’s tools	Native tools	Native tools, 100-line output	Work with native tools
Claude Opus 5.5	36/36 · 18,943	36/36 · 17,244 (−9%)	36/36 · 17,410 (−8%)	+5%
Claude Sonnet 5.5	36/36 · 17,765	36/36 · 16,237 (−9%)	36/36 · 16,399 (−8%)	−4%
GPT-6 Astra	35/36 · 8,250	36/36 · 10,807 (+31%)	35/36 · 10,399 (+26%)	+51%
GPT-6.1 Sol	36/36 · 8,873	36/36 · 10,534 (+19%)	36/36 · 10,239 (+15%)	+24%
GPT-6 Luna	33/36 · 12,003	35/36 · 11,532 (−4%)	36/36 · 10,816 (−10%)	+15%

Older models, mixed sets: native tools against MindRoom’s tools (pull request, through MindRoom’s API)

Model	Reasoning	MindRoom’s tools	Native tools	Work with native tools
Claude Opus 4.7	no thinking	34/36 · 33,140	33/36 · 25,793 (−22%)	−26%
Claude Opus 4.8	no thinking	33/36 · 32,219	34/36 · 23,843 (−26%)	−32%
Claude Sonnet 5	adaptive thinking	33/36 · 42,349	34/36 · 27,120 (−36%)	−45%
GPT-5.5	Codex’s choice	34/36 · 15,007	36/36 · 14,419 (−4%)	−11%
GPT-5.6 Sol	Codex’s choice	36/36 · 15,062	35/36 · 12,809 (−15%)	−27%

Older models, only the shell tools: native shape against MindRoom’s (pull request, through MindRoom’s API; Opus 4.7 and 4.8 without thinking, Sonnet 5 with adaptive thinking, GPT with the effort Codex picks)

Model	MindRoom’s shell	Native shell
Claude Opus 4.7	33/36 · 17,518	33/36 · 15,714 (−10%)
Claude Opus 4.8	35/36 · 19,183	33/36 · 14,686 (−23%)
Claude Sonnet 5	33/36 · 22,624	33/36 · 28,354 (+25%)
GPT-5.5	36/36 · 9,056	36/36 · 9,046 (0%)
GPT-5.6 Sol	36/36 · 9,171	34/36 · 8,658 (−6%)

Sonnet 5’s native mean comes from one run that cost 271,106 tokens, after a 50 KB command output; its median is 9% lower with the native shell.

Opus 4.7 and 4.8 with adaptive thinking, mixed sets: native tools against MindRoom’s tools (pull request, through MindRoom’s API)

Model	Thinking replies	MindRoom’s tools	Native tools
Claude Opus 4.7	26% and 31%	35/36 · 35,610	33/36 · 23,090 (−35%)
Claude Opus 4.8	36% and 30%	33/36 · 25,269	33/36 · 20,130 (−20%)

Reasoning effort, mixed sets: native tools against MindRoom’s tools (pull request, through MindRoom’s API)

Model	Reasoning effort	MindRoom’s tools	Native tools
Claude Sonnet 5.5	low	36/36 · 15,154	36/36 · 12,330 (−19%)
Claude Sonnet 5.5	default	36/36 · 17,765	36/36 · 16,237 (−9%)
Claude Sonnet 5.5	high	36/36 · 18,286	36/36 · 15,630 (−15%)
GPT-6 Astra	low	36/36 · 8,361	36/36 · 10,627 (+27%)
GPT-6 Astra	default	35/36 · 8,250	36/36 · 10,807 (+31%)
GPT-6 Astra	high	36/36 · 8,934	36/36 · 12,918 (+45%)

Only the shell tools (pull request, through MindRoom’s API; Claude with adaptive thinking, GPT with the effort Codex picks)

Model	All of MindRoom’s tools	MindRoom’s shell only	Native shell only
Claude Opus 5.5	36/36 · 18,943	36/36 · 12,926 (−32%)	36/36 · 12,202 (−6%)
Claude Sonnet 5.5	36/36 · 17,765	36/36 · 12,455 (−30%)	36/36 · 10,168 (−18%)
GPT-6 Astra	35/36 · 8,250	36/36 · 6,025 (−27%)	36/36 · 6,052 (0%)
GPT-6.1 Sol	36/36 · 8,873	36/36 · 6,195 (−30%)	36/36 · 6,177 (0%)
GPT-6 Luna	33/36 · 12,003	31/36 · 7,118 (−41%)	34/36 · 7,018 (−1%)

MindRoom’s shell alone is compared with all of MindRoom’s tools, and the native shell with MindRoom’s shell.

Minimal mode against standard mode (main branch, through Matrix, shell tools only; Claude with adaptive thinking, GPT with the effort Codex picks)

Model	Standard mode	Minimal mode
Claude Opus 5.5	36/36 · 22,192	36/36 · 8,030 (−64%)
Claude Sonnet 5.5	35/36 · 22,448	34/36 · 7,828 (−65%)
GPT-6 Astra	36/36 · 11,429	36/36 · 2,798 (−76%)
GPT-6.1 Sol	36/36 · 11,070	35/36 · 3,370 (−70%)
GPT-6 Luna	35/36 · 12,568	34/36 · 5,219 (−58%)

Minimal mode: the native tool against MindRoom’s bash (pull request with the experiment patch, through Matrix; Claude with adaptive thinking, GPT with the effort Codex picks)

Model	MindRoom’s `bash`	Native tool	Native tool, 100-line output
Claude Opus 5.5	36/36 · 8,058	36/36 · 8,654 (+7%)	36/36 · 9,123 (+13%)
Claude Sonnet 5.5	34/36 · 7,787	34/36 · 7,514 (−4%)	34/36 · 8,000 (+3%)
GPT-6 Astra	35/36 · 3,071	36/36 · 4,173 (+36%)	36/36 · 3,359 (+9%)
GPT-6.1 Sol	34/36 · 3,103	33/36 · 4,150 (+34%)	36/36 · 3,575 (+15%)
GPT-6 Luna	34/36 · 4,697	34/36 · 6,849 (+46%)	32/36 · 4,926 (+5%)

One of Luna’s runs with the whole native output counts for passes but not for tokens: the provider reported no usage for one of its requests.

Armin also found that Anthropic’s strict tool mode, which restricts the model’s output to the tool’s schema, made these failures disappear in his tests. Pi later turned it on for Claude models, and a pi user then linked a different problem to it: in that user’s logs, 9.5% of Claude’s edits had mangled \u escapes for non-ASCII text, such as Korean, with strict mode on, and 1.5% with it off. ↩︎
The 12 tasks come from 12 families: a CSV report, finding the commit that broke a check, renaming an API across a package, counting errors in log files, fixing failing tests, finding files, extracting fields from JSON, writing a shell script, editing a configuration file, counting matching lines in very long command output, unpacking nested archives, and deduplicating and merging data files. I tune MindRoom’s agent setup on other instances of these families, and these 12 were held out from that. ↩︎
Claude and GPT count tokens differently, so compare the numbers within one model, not across models. Cached input counts in full here, although providers bill it at a fraction of the price, so a bill changes less than the token count. ↩︎
MindRoom’s toolkits accept exclude_tools, so both sets drop MindRoom’s grep, find_files, and ls, and for GPT also read_file, since Codex reads files through the shell. Both keep their shell’s tools for checking on and stopping a background command, which no model used in the matching-set runs. ↩︎
Anthropic adds its own instructions for tool use to every request that has tools, which is why MindRoom’s single bash tool costs Claude 509 tokens. I measured all of these numbers by sending each part on its own and reading the input tokens the provider reported. ↩︎
The patch is with this post’s reproduction scripts. It shows minimal mode’s single tool as MindRoom’s canonical run function, which the tool dialect then presents as Bash or exec_command; nothing else about minimal mode changes. ↩︎

How to start with AI

Thu, 08 Oct 2026 00:00:00 +0000

Recently, on a train, I sat near someone typing code by hand for hours. At the same time, I was orchestrating more than ten agents just by talking to my phone. I felt like I was living in a different world. It made me realize my own bias. I had just assumed that by now everyone would have AI write their code.

So maybe I shouldn’t be surprised that I still get asked: “how do I start with AI?” or “how do I start with coding agents?”

There is a lot of noise, and a lot of pretense, about very specific workflows. Multi-agent loops, agent and sub-agent graphs, carefully tuned rule files. None of that is useful for 99% of people.

You don’t need skills. You don’t need an AGENTS.md. You don’t need anything except the app itself. You just need to ask questions, and be critical of the answers.

Most of my examples come from code, because that is where I use AI the most. But the advice is the same whether you write software or not.

Table of contents

1. Install it, and that’s it #

Getting started is downloading the ChatGPT or Claude app, or literally a one-liner to install Claude Code or Codex if you write code. Maybe you log in, or click a button to get access. That’s the whole setup. “I need to take time to learn how to use it” was never a real barrier. The interface is trivial: text in, text out. Everything else is optional.

I open agents in projects of mine that have never seen any AI, and seconds later they know the entire thing. The frontier models have much better taste than they used to. In an established codebase they just follow the existing style. If you point one at a codebase that is a bunch of crap, it might still produce a bunch of crap. But even there it does better than it used to.

I used to give different advice. In On agentic coding, a year ago, I told you to create a CLAUDE.md in every project root with your preferences. (Today that would be an AGENTS.md, which all the major agents read.) I still use them, but not everywhere, and you don’t need one to get started. There is even research showing that repository-level context files don’t generally improve the task success rate of coding agents, while raising the cost by over 20%. They help when you have non-standard practices the agent can’t guess. A list of standard rules or an overview of the repository does not.

2. Ask it everything 🥩 #

The real change is a mindset, not a tool.

When you run into a problem, the old reflex is to Google it. The new reflex is to ask the AI about everything. Asking also means asking it to do things. Most of what I type is a description of what I want done, and why. Give it the context you would give a colleague.

When the agent says “now you need to run the tests,” no. Everything it can do, you let it do. If the AI tells you what to do when it could do it itself, you are just a meat proxy, and you are wasting your own time.

It is genuinely absurd how good it has gotten. My mom sent me a photo of MindRoom crashing in her browser, a screen full of minified JavaScript. I sent the photo to my agent. The answer: the Google Translate extension was interfering with the page. It wrote the fix too.

The people who still don’t use it are, I think, in one of two groups. Some had a bad experience once, years ago, and never tried again. That experience is badly out of date. Others, and I suspect it’s a large group, avoid AI on principle. This post won’t change their minds, but it might make starting easier for everyone else.

3. Use the best model #

Another very common question: shouldn’t I use a cheaper model to save money? When you’re starting out, my answer is to use the best one.

Your time almost certainly costs more than the tokens. For an engineer, it certainly does. Continuously deciding “do I need the smartest model for this, or the weakest one?” is itself not efficient. A useful check when a session costs $50: what would this have cost you without AI? If you would have done it anyway, the answer is almost always way more. (If you only did it because AI made it possible, the comparison gets harder.)

Cheaper per token is not cheaper per task. A cheaper model does not necessarily get to the solution in fewer tokens. On hard problems it typically uses way more.

My favorite way to see this is Artificial Analysis’s intelligence index versus cost per task, which accounts for both the price and how many tokens a model burns. Here are Claude Opus 5.5 and Claude Sonnet 5.5 at each reasoning effort:

Artificial Analysis Intelligence Index versus the average cost of one of its tasks, for each reasoning effort, as of 2026-10-08. Hover for the output tokens per task.

Sonnet costs half as much per token. But at max effort it scores 56, the same as Opus at xhigh, and costs $5.46 per task against Opus’s $3.46. It just uses a lot more tokens: about 197,000 output tokens per task versus 66,000. Above a score of about 47, the Opus line is both higher and further to the left.

So for simple tasks, the cheaper model is cheaper. For difficult work, it is sometimes the same price, often takes longer, and can cost more, because it burns through so many more tokens. And a worse answer gets more expensive later, because somebody has to fix it.

Don’t put reasoning effort on max. Going from xhigh to max on Opus buys 1.6 points for 73% more money. On Sonnet it is 2.7 times the cost. I default to xhigh myself, which is probably overkill for half of what I do.

4. Decide how much to trust it #

When I started, I read every single line of code it wrote and had so many comments. By now I trust the output much more, because I’ve seen it. The scope I hand over has grown with that trust. Canvases in MindRoom Chat, interactive pages an agent can show you next to the conversation, felt like a half-year project. I built it in a day, on my phone.

How closely I look depends on how important the result is and how measurable it is. The core of a project, the part everything relies on: there I want to understand precisely what’s going on. An integration that fetches my email from some external API, something written a million times whose outcome is easy to check: I don’t read the code. When I see that it works, I believe it. The same goes outside code: an answer you can check in a minute needs less scrutiny than one you can’t check at all.

This is also why expertise still matters. When I scroll through a large piece of code, I can judge its complexity at a glance. When it explains something in a field I know, I can tell when it’s wrong and call bullshit. In a field I don’t know, I can’t, so I spend much more time validating to reach the same level of trust. Match your scrutiny to your expertise.

5. Talk to it, then make it review itself #

I start by telling it what I want, usually by voice. I just ramble. It is so good at language that even when I go off track and talk incoherently, it almost always understands what I mean. And sometimes it pushes back: “I understand what you mean, but that approach is bad because of this, I’d do it this way.” Sometimes it says we could do it either way, and I say: let’s try both.

In almost all cases, AI can now deliver what you asked for in a single prompt, and it will actually work. The question is whether it’s actually good, and whether it meets your standards. So before you accept anything, have it reviewed.

For me, the review has to come from a fresh context that never saw the conversation that produced the work. A new session, or a sub-agent, that only sees the result. That can be the same model or a different one. I do both: the same model in a fresh context, and other models, until they agree it’s good. That sounds like a lot of setup, but it isn’t. In a chat app, it can be as simple as pasting the answer into a new chat, or into the other company’s app, and asking what’s wrong with it.

Whatever you use, also tell it to be mindful of scope creep. A review will always find something. Very often it’s an edge case that looks real in isolation but can never happen in your actual situation. Ask whether each finding is realistic before you let it build more.

6. What about skills and multi-agent loops? #

Yes, I use those things. No, you don’t need them to start.

I never sit down and think “now I need to develop a skill.” I do something with the agent a couple of times, realize I’m going to keep repeating it, and then say: “let’s make a skill out of this, and make it self-evolving.” A skill is just a procedure you want to repeat. You only know what to put in it after you’ve repeated it. (In MindRoom, an agent can even do this on its own: with automatic skill learning turned on, it reviews its recent work in the background and turns repeated workflows into skills.)

My own set of skills is public as baspowers, which started as a fork of superpowers. Some of its skills came from mining more than ten thousand session logs of my agent history for procedures I kept repeating, then A/B testing each one with and without the skill. An A/B test sounds like a lot of work, but you can ask the agent to run it for you, and tell it which other models to test with. Four of the ten candidates were dropped because the agents already did the right thing without them. The rule I ended up with: keep only skills that change behavior.

Sometimes they really do. In one test, agents wrote a failing test before fixing a bug in 0 of 32 runs without my test-driven development skill, and in 32 of 32 runs with it. That matters when agents run on their own for hours. It does not matter for your first week of asking questions.

I also run cross-model reviews in loops, which is useful when I have ten things running at once. But I keep updating those setups, because each new model makes parts of them unnecessary. My PR review skill used to make agents fix every nitpick. Now the models are good enough that it will forever find something, even after a hundred rounds, so I had to change it: it now only blocks on issues with a realistic scenario.

That is the problem with copying someone’s elaborate workflow. It encodes the weaknesses of last year’s models and the habits of one person. Start with nothing, and add things once you notice you keep repeating them.

7. If you write code #

A few things only apply to software.

Tests are cheap now, so ask for them #

“Set up the actual live system, end to end (don’t touch production!), run it with real data, crash it at different points in the middle.” All the random things you would check yourself if you had infinite time. Have it write a lot of tests. More tests is better, even when some have limited value. What I care about is that the source code itself is clean.

The code is the spec #

A lot of people will disagree here, but I throw away design docs. What came out in the end was never what the doc said anyway. The code and the tests are the specification, because that’s also what the agent reads. The only thing I keep is what the code can’t say: why something was done a certain way. Documentation that conflicts with the code is the worst: the agent will start changing your code to match the docs without you noticing.

Review against your own criteria #

The questions I make it answer about its own work:

Is the code clean?
Does it fit well within the existing architecture?
Does it repeat itself?
Is it overengineered? Keep it simple, stupid.

Cross-model review needs two installs #

As long as you have both Codex and Claude Code installed, you are essentially ready. Just tell the agent: “have Codex review this in a separate session and address what it finds.” It can usually figure out how to call the other command-line tool on its own. I personally built a worktree orchestration tool for this, agent-cli dev, which comes with a skill that teaches the agent to use it. But honestly, that is not necessary anymore nowadays.

A good AI engineer is just a good engineer #

People ask me what makes a good agentic engineer. I’ve been saying the same thing for a long time: somebody who is literally a good engineer. Good engineering practices never went away. They are more important than ever: test-driven development, building things at the right level of abstraction, a proper architecture.

I think of engineering skill as a scale that runs from negative to positive. A good engineer removes work for others. A bad engineer creates work for others. Give the bad engineer AI, and they multiply their bad output, because AI is a multiplier.

Before AI, you could spend a year writing a few thousand lines of code, maybe a couple tens of thousands at most. You were intimately familiar with all of it, so you could get away with a somewhat worse architecture. Now you can produce a million lines of code in a year.¹ There is no possible way you can keep up with all of it. If you want to build something lasting and robust, you need proper foundations. And that is where your opinions matter. You don’t need any setup to start, but you do need taste to build something that lasts. (I made a related point a year ago in On agentic coding: AI amplifies the experience you already have.)

Your role changed, it didn’t disappear #

Some software engineers feel personally attacked by all this. I think you have to get over yourself. Your secret sauce was never knowing the syntax really well. Now you need to know what to build, and how it should be built.

And you need to be strict. You can produce thousands of lines a day, so be thoughtful about whether you want them in your codebase forever. Changing the color of a button takes one prompt. A difficult system needs more rigor, which means understanding the architecture and what it implies instead of reading every line.

This post has an expiry date #

Right now I’d tell you to use Claude Opus 5.5. Just two weeks ago I was telling people to use GPT-6 Astra. Things change continuously, and in most cases you’ll be fine with either.

I’m also sure that a few weeks from now, I will no longer accept the model I’m praising today. Not because it can’t do the amazing things it does now. But because the next one will do even more amazing things that it can’t. With every new model, my workflow changes, and the amount of scope I hand over per task grows dramatically.

That’s why I don’t recommend building an elaborate setup first. Every skill, rule file and agent graph is tuned to the limitations of the model you have today. Those limitations are gone in a few months, and then the setup is dead weight. Asking it things is the one habit that survives every model change.

Install it today, use the best model, and ask it everything, including whether its own work is any good. The rest is optional, and probably temporary.

I wrote about 4.5 million lines of code this year, as counted by trueloc, another tool of mine that counts every line added across all commits in my pull requests. ↩︎

Self-hosting AI does not save money, and I do it anyway

Fri, 02 Oct 2026 00:00:00 +0000

I have had this debate many times, so I am finally writing it down.

Whenever I say that self-hosting AI does not save money, people hear that I am against self-hosting. That is not the point at all. I am a massive fan of open-weight models. Qwen3.8 27B runs on two RTX 3090s at home (and 15 more models), and my phone dictation goes to Qwen3-ASR on the same machine. I think open-source AI is the best thing since sliced bread.

I don’t pretend it saves money. I do it for fun, for sovereignty, and for privacy, which I come back to at the end.

All benchmark numbers and prices below come from Artificial Analysis and OpenRouter as of September 30, 2026. I did the math for the hardware I own, and for the counterarguments I hear most. All assumptions are listed at the end.

Benchmarks are not everything, and a high score does not reliably predict how a model does on real work. They are still the best we have: all models are benchmaxed, tuned to do well on the popular benchmarks, so the scores at least work as a reference frame for comparing them with each other.

Update, October 2, 2026: after the discussion on r/LocalLLaMA, I added section 5 on Macs, DGX Sparks, and AMD AI Max machines.

Table of contents

1. The model I love, and the one that beats it on price #

My favorite local model right now is Qwen3.8 27B. It came out in August and it is Apache-2.0. People often say models like this run on a single gaming GPU, but that is only true after quantizing them. At full precision, Qwen3.8 27B needs about 54 GB of memory. Quantized to about 4 bits per weight, it fits on one 24 GB card, at some cost in quality. I still split it across both of my 3090s, because the second card leaves room for a larger context window.

I compare it with OpenAI’s GPT-6 Luna, the cheap tier released on September 22. Both models let you choose how long they think, but the settings do not mean the same thing for both. Luna’s token use grows more than 20 times from its lowest setting to its highest, while Qwen’s barely changes. Qwen on low already writes more tokens than Luna on xhigh.

Average output tokens per Intelligence Index task at each reasoning setting. Qwen3.8 27B has no high or max setting. Hover for scores.

So the name of a setting says little on its own. I compare both models at xhigh, the highest setting Qwen offers, and count the tokens each one actually uses. The extra tokens Qwen needs are part of what I am measuring. There, Qwen scores 33.7 on the Artificial Analysis Intelligence Index and Luna scores 34.6. For reference, Claude Opus 4.6, a frontier model from February, scores 31.9. That is an amazing feat in itself. When Opus 4.6 was the best model we had, I never thought that within the same year I could run essentially that level of intelligence in my own house.

Artificial Analysis also publishes how many tokens each model used to run the index, and what that cost. So the question is simple: what does it cost to run the whole benchmark once?¹

Cost of one run of the Artificial Analysis Intelligence Index. The two bars for hardware at home are electricity only: measured on my 3090s, estimated for the RTX PRO 6000.

Luna runs the whole benchmark for $67. Qwen at full precision, through the cheapest provider that does not keep your data (ZDR), costs $619. Two things cause that gap: providers charge almost four times as much per output token for Qwen, and Qwen generates almost three times as many tokens to get the same work done.

The second bar is my own machine: the electricity alone for running Qwen on my GPUs costs more than Luna’s entire bill.

The third bar is what it takes to match the API’s full precision at home: an RTX PRO 6000, a workstation card with 96 GB of memory, enough for the full model.² Its electricity alone costs twice Luna’s bill, and one run takes more than seven weeks. The card by itself sells for $10,000 to $20,000, so with a computer around it, you are paying for a pretty nice car. Buy a few of them to run agents in parallel, and you are building a small data center of your own.

The bars for hardware at home are electricity only, so they are the lowest these runs can cost: the cards come on top, and how much depends on how busy you keep them, which is what section 4 is about.

What I care about is the cheapest way to get answers of Qwen’s quality, from whichever model gives them.

2. “But my GPUs are already paid for” #

This is the argument I hear most, so assume the hardware is free and only the electricity counts.

I measured my own machine. It runs Qwen in vLLM, split across both cards, at about 107 tokens per second.³ The cards are capped at 270 W each,⁴ and the whole machine draws about 700 W. It then needs four weeks, running day and night, to get through the benchmark once. I also give my quantized copy the score Artificial Analysis measured through Alibaba’s API. I have not rerun the benchmark on it, and anything lost to quantization makes my machine look worse.

Electricity for one benchmark run of Qwen3.8 27B on my two 3090s, by electricity price. The dashed line is what Luna charges for the same work, with OpenAI’s hardware and margin included. Hover for values.

Above 14.5 cents per kWh, my electricity alone costs more than Luna’s API bill. Below it, I save a few dollars per run and wait four weeks instead of a few hours.

Those four weeks matter more than the few dollars. An API takes hundreds of requests at once, so even one benchmark run can finish in an hour or two. My setup works on one conversation at a time: when I sent it eight requests at once, seven of them waited in line.

Wall-clock time for one benchmark run. The API times use Artificial Analysis’s measured time per task for Luna; the parallel case assumes 100 requests at once. Log scale.

This is one more reason my coding agents run on APIs. When I run several agents in parallel, each of them should be as fast as if it were alone.

3. Why datacenters are so much better at this #

When my 3090s generate a token, they read all of the model’s weights from memory to produce one token for one conversation. Most of the chip’s compute sits idle while it waits for memory. A datacenter GPU reads the same weights once and produces a token for hundreds of conversations at the same time. This is called batching, and it is where most of the efficiency comes from.

Artificial Analysis measures this with AgentPerf, which replays real coding-agent sessions against datacenter hardware. It counts how many agents a system can serve while each one still gets at least 60 tokens per second.

Coding agents served per kW of accelerator power, each getting at least 60 tokens per second, from AgentPerf. The datacenter systems all run DeepSeek V4 Pro. The 3090 bar is my machine running Qwen3.8 27B, a much smaller model, with GPU power only, so the comparison favors my machine.

The rack of 36 GB300s serves 20 times more agents per kW than the 8-GPU H200 server. Part of that is newer chips, and part is software: NVIDIA tuned the B300 and GB300 setups itself, while Artificial Analysis configured the H200 one. The B300 and the GB300 share both the chip generation and the software. Most of the gap between those two bars comes from the rack itself: 36 GPUs on one fast interconnect keep more than a thousand agents in flight at once. My two 3090s serve one conversation at a time.

4. “But I have solar panels” #

Solar power is not free. With net metering, every kWh my GPUs burn is a kWh that does not get credited at the retail price. Without net metering, it is a kWh I do not sell, unless the surplus would go to waste anyway, which is the free-power case below. And the GPUs also run at night.

But fine, say the electricity is free and the only cost is the hardware. I was lucky and bought my two 3090s for about $750 each,⁵ and I write them off over three years. The GPUs only save money while they do work I would otherwise pay an API for, and while they sit idle they save nothing. So the question is how many hours a day they have to be busy before they pay for themselves.⁶

To make this fair to open models, imagine an open model that is exactly as efficient as Luna: the same quality from the same number of tokens, sold at Luna’s price.

How many hours a day my two $750 RTX 3090s have to be busy, every day for three years, before they cost less than a ZDR API. The host PC is not counted.

With free power, the GPUs pay for themselves if they are busy 4 hours a day, every day for three years. At 18 cents per kWh, they need 8.5. That uses what I paid for the cards; at today’s price of about $1,500 each, the 8.5 hours become 14.

And API prices keep falling. GPT-6 Luna costs 58% less per output token than GPT-5.6 Luna did in July. If the price halves again, no amount of use pays off at 18 cents per kWh. If it halves twice, the API costs less than my electricity alone.

5. “But my Mac, DGX Spark, or AI Max does save money” #

Several readers told me their machines do save them money: a MacBook Pro with an M5 Max, an NVIDIA DGX Spark, and a computer with AMD’s Ryzen AI Max+ 395. If you would have bought the machine anyway, that is true, for the reason in section 2: their electricity costs little compared with the API. Counting the hardware is a different story.

With 128 GB of memory, all three can hold a better model than my 3090s: Qwen3.8-Flash-Next, a sparse model with 180 billion parameters, of which only 6 billion work on each token. It scores 39.8, close to GPT-6 Luna on its highest setting (38.1), although at the Q2 a reader uses on their Mac it probably loses a few points.⁷ So how many years would each machine have to run nonstop before it pays for itself?

Years of nonstop generation before each machine pays for itself, compared with the most efficient API model at a similar score. Each machine runs the best model it can hold.

Even with free power, the 128 GB machines need six to ten years. The AMD machine draws only about 120 W, but it is slow too: one benchmark run of Qwen3.8 27B takes it four and a half months, and almost as much energy as my 3090s. The RTX PRO 6000 runs the same model about three times as fast but costs about twice as much, so it needs six to nine years. My 3090s pay off in under two years with free power, which is why the 3090 is still the value king, but at 18 cents per kWh they never do: the electricity for one run already costs more than Luna charges for it.

Ten years is a long time in AI. Better and more efficient open models will run on the same machines, but the frontier moves at the same pace, and API prices keep falling. Hardware gets more efficient too, in datacenters and at home, so a machine bought today also ends up competing with next year’s machines. These numbers only show how far apart the two sides start.

6. The bar keeps moving #

Section 4’s eight and a half hours a day would be easy for me: I run agents for longer than that, just not on Qwen. When Claude Opus 4.6 came out in February, I was perfectly happy with it. I thought it was all I would ever need, and I could not have imagined how much better models would get in half a year. Qwen3.8 27B now scores about the same as Opus 4.6, and I would no longer accept it for coding. Until GPT-6 Astra came out at the start of September, my go-to was GPT-5.6 Sol, which scores ten points higher than Qwen. I then used Astra until Claude Opus 5.5 came out less than three weeks later. Now I don’t even accept what was considered the best model a month ago.

Intelligence Index of the best model available on each date, of the models I used as my go-to, and of Qwen’s 27B models, which fit on a single 3090. Scores use the xhigh reasoning setting where a model has it.

The models that fit on my 3090s keep improving, but they stay about half a year behind the frontier, and my standard moves with the frontier.

7. A small company has it worse #

For me, this is a hobby, and a hobby is allowed to be inefficient. It gets worse once you try to use self-hosted models seriously, say for a team of ten developers.

If you stay fully self-hosted and want fast responses at peak, you have to buy hardware for the busiest hour of the year, not for the average one. The rest of the time, it sits idle.

An illustrative week of coding agents for a ten-person team. The hardware has to cover the busiest hour of the year, but a typical week uses only a fraction of it.

In this made-up but realistic week, the team uses 17% of what it paid for, so every token costs about six times more than it would at full load. You also need a spare GPU for when one dies, and someone who gets paged when it does. You could queue work or send the peaks to an API, but then you are paying for an API anyway.

An API provider has the opposite situation. It serves thousands of customers across every time zone, so its load curve is much flatter than yours. It fills the nights with discounted batch jobs (Luna’s batch tier is half price) and training runs. And more concurrent requests mean bigger batches, which is where the efficiency from section 3 comes from.

8. “They lose money on inference, so prices will go up” #

Another argument I hear is that API prices are subsidized by investors and will go up once those investors want their money back. I don’t think that holds for inference.

AgentPerf also reports how many tokens each GPU serves. On the GB300 rack, one GPU serving DeepSeek V4 Pro handles about 4 million output tokens and 530 million input tokens per hour. Most of that input is conversation history that coding agents send again with every step. Providers keep it in a cache and charge less than 1% of the normal input price for it.

At DeepSeek V4 Pro’s list prices, that GPU brings in at least $5.70 per hour, even if every input token is billed at the cache price. If 5% of the input misses the cache, it brings in about $17. Renting a B200, the closest GPU with a published rental price, costs $3.50 to $5.90 per hour from smaller cloud providers, and that price already includes their profit.

So the tokens pay for the hardware that serves them, even in the worst case. This leaves out costs like staff and training, so it does not tell you whether a lab makes money overall.

The money goes to training, research, free users, and flat-rate subscriptions for heavy users like me. When I wrote that I used $10,000 worth of API tokens for $200, that was list price, not what those tokens cost to serve.

And prices go down, not up. This is what the same level of intelligence has cost this year:

Output price per million tokens for models scoring between 30 and 35 on the Intelligence Index, by release date, as listed by Artificial Analysis. Log scale.

From Claude Opus 4.6 in February to GPT-6 Luna in September, the output price for this level of intelligence dropped by a factor of 50. Labs are in a race to the bottom on price.

9. “But they sell your data” #

Some people say APIs are cheap because the provider trains on your data or sells it. That is why I only used prices from providers that OpenRouter lists as zero data retention (ZDR).

Model	Provider	ZDR	Output ($ per million tokens)
GPT-6 Luna	OpenAI	no	0.50
GPT-6 Luna	Azure	yes	0.50
Qwen3.8 27B	AkashML (FP8)	yes	1.78
Qwen3.8 27B	DeepInfra (full precision)	yes	1.88
Qwen3.8 27B	Alibaba	no	2.55
Qwen3.8 27B	Cloudflare	no	3.20

Azure serves GPT-6 Luna with zero data retention at the same price as OpenAI. For Qwen3.8 27B, the cheapest endpoints are all ZDR, and the endpoints without ZDR cost more. The :free tier is where you pay with your data.

10. Where self-hosting does win: the same model #

If you compare running Qwen3.8 27B yourself with paying for Qwen3.8 27B through an API, self-hosting wins.

DeepInfra, the cheapest full-precision ZDR provider, charges $619 for one benchmark run, while my electricity costs $83. That compares the full model with my quantized copy, so the gap overstates my advantage by whatever quantization costs in quality. My machine pays for itself if it is busy 2.4 hours a day, or 1.5 hours with free power (the first group in the chart in section 4).

Providers charge a lot for a dense 27B model, because every token runs through all 27 billion parameters. Sparse open models, which use only a small part of their weights for each token, can be much cheaper to serve. If you specifically need Qwen and use it heavily, self-hosting it pays off. But you don’t need Qwen to get answers of Qwen’s quality. Luna gives you those for $67, with zero data retention.

11. So why do I do it? #

First, it is fun. I like knowing how the whole stack works, from the drivers in my NixOS configuration to how the layers are split between my two GPUs.

Second, nobody can take it away. Artificial Analysis already lists GPT-5.6 Luna as deprecated, less than three months after its release. My Qwen weights will still be on my disk in ten years, and no company can change their license or close their build system on me.

Third, some data should not leave my house. I would never send 200 GB of email, my messages, and my location history to an API, zero data retention or not. That is what my local AI projects are for.

Those are good reasons. Saving money is not one of them.

Appendix: all assumptions #

Show all assumptions, and which side each one favors

Assumptions that favor self-hosting

The host PC around the GPUs is not counted, and my 3090s count at the $750 each I paid, not the $1,500 they sell for today.
Quantized local copies get the score Artificial Analysis measured through an API, with no penalty for quantization.
The payback chart assumes each machine is busy 24 hours a day, with no idle time.
API prices stay where they are, although GPT-6 Luna’s output price is 58% lower than GPT-5.6 Luna’s, two and a half months earlier.
My time to set up and maintain the machine is not counted.
Every machine also gets a case with free electricity.

Assumptions that favor the API

The example electricity price is 18 cents per kWh, and some people pay less.
My machine serves one request at a time; batching several requests could raise its total throughput.
The hardware has no resale value at the end.
Waste heat that replaces heating in winter is not counted.
API outages, rate limits, and deprecated models are not counted.

Inputs

Benchmark: the Artificial Analysis Intelligence Index on September 30, 2026, at the xhigh reasoning setting, or the highest setting where noted. Its token counts per model are the unit of work.
API prices: the cheapest zero-data-retention endpoints on OpenRouter, with cached input billed at the cache price.
My machine: two RTX 3090s capped at 270 W, running Qwen3.8 27B in vLLM with AutoRound INT4 and DFlash2. Measured: 107 tokens per second, 1,600 tokens per second of prefill, and 540 W for both GPUs. Estimated: about 700 W for the whole machine under load and 150 W idle.
Break-even: the cards are written off over three years; busy hours save what the API would charge minus the electricity, and idle hours cost the idle power.
Other machines: prices, power, and speeds as in the footnotes. The M5 Max and RTX PRO 6000 numbers come from readers, and the DGX Spark and AMD Ryzen AI Max+ 395 speeds are estimated from Artificial Analysis measurements.
Datacenter: AgentPerf measures accelerator power only, and the B200 rental price stands in for the GB300, which has no published rental price.
Small company: the peak-load week is made up.

For each model, Artificial Analysis publishes how many input and output tokens the whole index took, and how many of the input tokens were read from a cache. I multiplied those by each provider’s prices, with cached input at the cache price. For Qwen that is 198 million output tokens and 820 million input tokens that miss the cache. For the bars at home, the same tokens divided by the machine’s speed give the run time, with the uncached input processed at the 1,600 tokens per second I measured on my 3090s and an estimated 2,000 on the RTX PRO 6000, and the run time times the machine’s power and the price per kWh gives the electricity. ↩︎
The RTX PRO 6000 has the same memory bandwidth as an RTX 5090, and at full precision every token reads three times as many bytes as in Q4_K_M. Scaling Artificial Analysis’s 5090 measurement gives about 48 tokens per second. I assume 600 W for the whole machine. ↩︎
I measured the model as I normally run it: vLLM with an AutoRound INT4 quant and DFlash2 speculative decoding, split across both cards. Single 1,500-token answers came out at 105 to 109 tokens per second, and reading a 22,000-token prompt ran at 1,600 tokens per second. Sending 2, 4, or 8 requests at once did not raise the total above about 110 tokens per second. The two GPUs drew 540 W together, each at its 270 W power limit; the 700 W adds my estimate for the rest of the machine, since I could not measure the CPU. The setup follows the recipes from club-3090, a community project that tunes LLM serving for RTX 3090s and publishes measured numbers, so I think it is about as fast as these cards get. Its benchmarks for this configuration at a similar power limit match mine: about 110 tokens per second for prose, and up to about 195 for code, where speculative decoding guesses more tokens right. Most of Qwen’s output on the benchmark is reasoning, which runs at the prose speed. ↩︎
The stock limit is 350 W. My cards sit in a normal desktop case with normal fans, and I worry that running both at full power for hours would shorten their life. The cap is a few lines in my NixOS configuration, and measurements from Puget Systems and r/LocalLLaMA put a 3090 at about 95% of its speed at 270 W. That is 77% of the power for 95% of the speed, so the cap also lowers my electricity cost per token. ↩︎
The 3090 is still the value king for VRAM per dollar, and $750 badly understates what mine are worth. That is what I paid more than a year ago; today a used 3090 sells for about $1,500. Even at that price it costs $62.50 per GB of VRAM, the same as an RTX 5090 at its $2,000 launch price, which is not what a 5090 sells for today. The right number for this calculation is what I could sell my cards for, and at $1,500 each the break-even points rise by about two thirds: at 18 cents per kWh, the same-model case moves from 2.4 to 4.0 hours per day, and the Luna-efficient case from 8.5 to 14.2 hours per day. ↩︎
While the cards are busy, they save what the API would have charged for the same work, minus the electricity at 700 W. While they are idle, the machine still draws about 150 W, of which the two GPUs with the model loaded take 85 W. The break-even point is the number of busy hours per day at which those savings cover the price of the cards, written off over three years, plus the idle power. ↩︎
The M5 Max numbers come from the reader: Qwen3.8-Flash-Next at Q2 (about 2.7 bits per weight), about 45 tokens per second, 1,250 tokens per second of prefill, and about 90 W, on a $7,000 laptop. The RTX PRO 6000 numbers come from another reader, u/TheAILegend, who measured Qwen3.8-Flash-Next at NVFP4 on theirs. For a single request, it stays at about 150 tokens per second from 4,000 to 200,000 tokens of context, with about 10,000 tokens per second of prefill and 266 W for the GPU plus about 160 W for the rest of the PC. With 32 short requests in parallel, the card reaches 590 tokens per second and pays off in about two years, which works for short batch jobs. At the long contexts of this benchmark, parallel requests are slower than one, so the chart uses the single-request speed. I use $15,000, the middle of the card’s price range. The DGX Spark and the AMD machine run Qwen3.8-Flash-Next at about 4.5 bits. For their speed, I took Artificial Analysis’s measurements of Qwen3.8 27B on each machine and doubled them, because the owner of the AMD machine finds Qwen3.8-Flash-Next about twice as fast as Qwen3.8 27B. That gives about 55 tokens per second on the DGX Spark ($6,950, about 150 W) and 47 on the AMD Ryzen AI Max+ 395 ($4,000, the price Artificial Analysis lists, about 120 W). All of them are compared with GPT-6 Luna on its highest setting, which runs the whole benchmark for $122. The 3090s run Qwen3.8 27B, as in the rest of this post, and are compared with Luna on xhigh. Artificial Analysis measured Qwen3.8 27B on the AMD machine at 23 tokens per second, so one benchmark run takes it 134 days and about 390 kWh, against about 460 kWh on my 3090s. Running Qwen3.8-Flash-Next on the 3090s with CPU expert offload, which club-3090 measured at 37 tokens per second, does not change their result: one run then takes 82 days, and at 18 cents per kWh its electricity costs more than Luna’s $122 unless the whole machine draws less than about 350 W. ↩︎

Slowly, then all at once: AI agents that know how a company works

Wed, 30 Sep 2026 00:00:00 +0000

IonQ is the world’s leading vertically integrated, full-stack quantum platform company. We build quantum computers, quantum security solutions, quantum networks, and quantum sensors, and we fabricate our own quantum chips. That means thousands of people working on some of the hardest physics and engineering problems the world faces. Much of what we know is not written down in one place. Where a program actually stands lives in Slack threads, tickets, commits, documents, and meetings. If you want the real picture, somebody has to read all of it, or you ask a colleague who has probably explained it five times already. And as IonQ grows, everybody is new to something. Someone who just joined through an acquisition is new to all of it, and after three years here, I still could not explain what half of our programs do.

Why I built MindRoom #

Programming has always been my biggest passion, and when AI coding agents suddenly became extremely good, the jump in what I could do in a single day was the biggest I had ever seen. But that power only reaches people who are comfortable in a terminal. My mom will never open one. Neither will most of the physicists, hardware engineers, and program managers who make a company like IonQ work.

So I built MindRoom, which is open source, and I run it at IonQ as a central hub for collaboration and knowledge sharing, what I like to call a company brain. It looks like a chat app. Everybody gets a personal agent the moment they log in, running on its own sandboxed computer in the cloud. Every program can get a shared agent that is connected to that program’s Slack, Confluence, Jira, code, meeting notes, and even the parts database. Its answers link back to what a human wrote, so you can check them instead of trusting them.

The personal agent is where most people start. I can send mine a voice message like “every week, just before my one-on-one with my manager, give me a clear summary of what I’ve been up to.” I never told it who my manager is, when we meet, or where to look. It works that out on its own, sends me a link to the summary, and from then on does it every week without being asked again.

Slowly #

For months, MindRoom grew one small group at a time. It started with a single colleague, then a pilot of about ten, and then the team behind Superion 10K, our 10,000-qubit quantum computer. We rolled it out carefully, and each step at most doubled the group. Every new group needed a hand-held onboarding, so I recorded short videos to make that scale. The people who tried it kept using it, but it was hard to explain what it could do without showing it.

A voice message becomes a reminder

An approval card before anything is sent

Watch the agent's browser and take over

Demo recordings with a fictional company and scripted responses. More in the MindRoom showcase.

Then all at once #

That changed with a company-wide demo. Instead of explaining what agents can do, I played someone new to the Superion 10K program and asked its shared agent, by voice, to explain what we are building, how the work is split, and how the pieces fit together. A few minutes later it had made an interactive slide deck, built entirely from our own documents, Slack threads, and code, with every claim linked to its source.

Hundreds of people signed up that same day, and within weeks about a third of everyone who can create an account had one. At the time of the demo, five programs and teams used it every day, and now it is about fifteen. People had already had access to tools like Claude and ChatGPT for months, but those never clicked for them in the same way. An agent that already knew our work did.

The surprise #

I built MindRoom for people who never open a terminal, simple enough that even my mom could use it, so I expected non-technical colleagues to be the main audience. They did adopt it. Program managers use it to write weekly updates from what actually happened in Slack and Jira, instead of from whatever people remembered to tell them. Our configuration management team uses it to check the hardware as it was actually built against the engineering baseline, and to find policy documents that contradict each other across different systems.

What I did not expect was that some of the most capable engineers I know, who already run their own coding agents every day, would become some of the heaviest users. One of them, who already lived in Claude Code, cleared his entire inbox, Slack, and task list from his phone on a single flight. I think they like it for the same reason everyone else does: it keeps context. For writing code, dedicated tools like Claude Code and Codex are better, because they are built for exactly that.¹ But engineers use MindRoom as a layer for gathering knowledge. A coding agent does not need to know who you are or who you report to in order to fix a bug. Coordinating a large group of people is mostly that kind of context.

Institutional knowledge #

A colleague wrote about why AI’s biggest opportunity in large programs is institutional knowledge, from the perspective of someone using it every day. From the builder’s side, what I like most is that a shared agent gets better the more people use it.

When someone works through a problem with it, it keeps what it learned, so the next person benefits without ever having been in that thread.

In Claude Code or Codex, you handle memory yourself, by asking the agent to write things down in Markdown files. Agents get lazy: they forget to write things down, and later they forget to read them. MindRoom agents are also told to write down durable facts, in plain text files under version control, but they do not have to remember to. A background process picks out what is worth keeping from every conversation and adds it to a semantic search index. When a new message comes in, the relevant memories are looked up and added to the context automatically. So remembering and looking things up no longer depend on the agent deciding to do it. When someone asks a question that was already answered, the agent re-verifies its earlier analysis instead of starting from zero, so the second answer usually comes faster and builds on the first.

One very cool and yet simple idea is agent dreaming, which I picked up from a talk by Lance Martin from Anthropic. People collect short-term memories during the day and consolidate them while they sleep. The analogy is loose, but the idea carries over. Every night, a first pass reads the agent’s memories, every conversation it has had, and the original sources, and looks for contradictions, for example when someone corrected something a few minutes later and the two memories were never reconciled. It writes a patch to fix them. A second pass, in a completely fresh context, checks every proposed fix and accepts or rejects it. It rejects about half, which is exactly the kind of skepticism I want. Dreaming is not even platform code, just a skill, a short set of written instructions: you tell an agent to set up dreaming, and it knows what to do, including scheduling itself to run every night.

It also notices when two documents disagree, which is easy to miss when nobody reads all of them. Its reports end with “Corrections welcome, tell the agent and the next report improves,” and a correction becomes a memory that the next report uses. And when nobody ever wrote something down, it usually still knows who to ask, because it keeps track of what everybody is working on.

Why build it ourselves #

There are plenty of AI products that promise something similar. Building our own gives us complete flexibility and no vendor lock-in. We are not waiting for a third party to support the model, the system, or the feature we need. When a better model comes out, we can switch the same day. And the knowledge it builds up is stored in IonQ’s own cloud environment.

It also means we can experiment quickly. Features that could take a large organization months often take me days:

Agents can open a real browser on their own remote computer, not on your laptop, and you can watch it live and take over, for example to log in somewhere, before handing control back.
An MCP gateway lets people connect their local tools, like Claude Code or Codex, to every agent they have access to and all of those agents’ tools, so they benefit from what the agents have collected even outside MindRoom.
A day after OpenAI released GPT-Live-1 in its API, you could call your MindRoom agent: the same agent, with the same memory, which hands longer tasks to a background process while you keep talking. On my commute, I talk to my agent to plan my day and get things started.
Agents can host small sites, much like ChatGPT Sites, behind a temporary link with access control. That is how an agent can answer a question with an interactive presentation or a dashboard instead of a wall of text.

What it takes to run this at a company #

A demo on a laptop is easy. Running a central hub that hundreds of people rely on every day is a different problem, and most of my time went there.

People sign in with their normal company account, and their account and personal agent are created on first login. There is nothing to install.

Every agent runs in its own sandboxed container with no secrets inside it. Credentials are injected from outside, so the agent never sees a token. Agents also cannot reach arbitrary websites: outbound traffic is denied by default, and a new domain needs a person’s time-limited approval.

Reading is free, but anything that sends or changes something on your behalf, like an email, shows an approval card with exactly what will be sent and to whom. A shared agent only answers the people who work on that program, and it only reaches the systems it was granted.

It runs in IonQ’s own cloud environment, and a security review ran alongside the rollout. At its busiest moment so far, 200 people were writing to their agents at the same time, and it held up fine. The chat layer has plenty of room to grow: the French government runs Matrix for about 400,000 civil servants.

What’s next #

With our recent acquisitions and a lot of new hires, there is a steady flow of people joining IonQ who are new to all of it, and to MindRoom. Because MindRoom is built on Matrix, which is federated like email, groups with different data rules can each get their own instance and still work together where that is allowed. And whoever joins next can log in on their first day and ask their program’s agent how the work fits together and who to ask.

That does not mean you cannot write code from MindRoom. Because every agent has its own computer, it can run any software, so my MindRoom agent simply runs Claude Code or Codex itself when there is coding to do. ↩︎

Exploring the Jev hype in MindRoom

Wed, 23 Sep 2026 00:00:00 +0000

Last week, every AI newsletter I get was full of Jev, a new model from TypeSafe. So were several subreddits. I am a permanent lurker on r/LocalLLaMA, and even though it is about local models, every other post seemed to be about Jev. There were local clones like mini-jev and von, CLI wrappers like jev-cli, and even someone on Hacker News claiming they had built the same thing a year earlier.

I tried Jev in MindRoom, my open-source agent platform on Matrix, and less than two days later it was making three decisions there.

Update 2026-10-07: OpenAI released the Decisions API today, its own take on the same idea, and MindRoom now supports it as a third backend; see below.

What a System One model is #

The name comes from Daniel Kahneman’s Thinking, Fast and Slow, where System 1 is fast, intuitive thinking and System 2 is slow and deliberate. A System One model does not generate text. You send it a state, for example a chat log, plus typed questions, and it returns probabilities:

Type	Question	Answer
Noul	Yes or no	Probability of yes
Choice	Pick one of up to 255 options	Chosen option plus the full distribution
Score	Rate against a rubric of 2 to 10 levels	Probability-weighted value

TypeSafe claims 70 to 500 ms end to end and $0.042 per million input tokens, with output tokens free. They also write “we can’t prove it isn’t subsidized.” Either way, it is incredibly cheap and incredibly fast.

Adaptive participation #

In MindRoom, a conversation can have several humans, several agents, or both. When I talk to a single agent, it simply replies, and I don’t need to tag it. Once another person joins and writes something, the agent cannot tell whether the message was meant for it or for the other person. Nothing decided that, so the agent stayed quiet unless explicitly tagged. That confused several users.

So I recently added adaptive participation: an agent that has already replied in a thread decides on its own whether to answer an untagged message. I first used an LLM as the judge, GPT-5.6 Luna on low reasoning. A yes-or-no question like “should this agent respond?” is exactly what Jev is for, so it became the second backend. Compared with Luna, it is about ten times cheaper and ten times faster, and if I believe the benchmarks, it is also smarter.

“Thanks” should not stop the agent #

The next thing I used Jev for was interruptions. When I send a message while an agent is still replying, MindRoom injects a notice at the next tool call telling the agent to stop and wrap up, because there is a new message. But often I, and other users, would watch the agent start working and then write “looks good” or “thanks.” The agent would stop early and continue in the next turn, which is both wasteful and counterintuitive.

Now a judgment decides whether the new message needs the interruption, which the docs call mid-turn coalescing. If it does not, the agent reacts with 👀, finishes its reply, and handles the message afterwards.

This is also where I learned a lesson about evals. My first implementation worked at the code level but not functionally. My evals, based on real attempts inside MindRoom that had all failed plus synthetic cases from GPT-6 Astra, passed 5 out of 40 with Jev. The fix changed the question and gave the judge more context, and it went to 40 out of 40.

	Before: 5/40	After: 40/40
Question	“May the active task finish before the queued human messages are handled in a later turn?”	“Does any newly queued message require an immediate change to, pause of, or stop of the active task?”
When unclear	“Also false when context is incomplete, the relationship is unclear…”	“Mere relevance to the task does not make praise or thanks an interruption.”
Context	The request and the new messages	The same, plus the preceding conversation

The old question treated anything unclear as a reason to interrupt, and a bare “thanks” without the conversation around it always looks unclear.

Picking the responder #

The third decision is routing. When a message in a room with many agents does not mention anyone, MindRoom’s router picks who should answer. That used to be a free-text LLM prompt, “choose the most appropriate agent.” It is now a Jev Choice over the eligible agents, plus a no_fit option, as described in the router docs. A confident no_fit asks the user to mention an agent instead of guessing, and anything uncertain falls back to the existing LLM router.

One config line to switch #

I built a very simple abstraction layer so I can swap Jev for an LLM without changing any code. Every decision is one question with criteria for true and false:

MID_TURN_QUESTION = JudgmentQuestion(
 id="interrupt_current_turn",
 instructions="Does any newly queued message require an immediate change to, pause of, or stop of the active task?",
 when_true="A new message requests stopping or pausing the active task, corrects it, ...",
 when_false="The messages acknowledge progress, express thanks, ask to continue unchanged, ...",
)

Jev receives it as a Noul question and returns a probability, which MindRoom compares with a threshold. An LLM receives the same question and context and must return {"decision": true} or {"decision": false}. Switching is a configuration change:

agents:
 helper:
 mid_turn:
 defer_reaction: "👀"
 judgment:
 provider: typesafe  # or: provider: llm, model: fast
 threshold: 0.8

Both backends log their decisions and latency, so I can compare them on real traffic. The code is in src/mindroom/judgment/, and the backends are documented under participation judgment backends.

A third backend: OpenAI Decisions #

On October 7, OpenAI released its own version of the idea, the Decisions API, currently in public beta. The question types match Jev’s: a predicate is a Noul, and choice and score keep their names. It serves a single model, gpt-6-luna, and OpenAI says it returns answers “about 10x faster than the Responses API.” Input costs $0.10 per million tokens and output is free, so it lists at about 2.4 times Jev’s price.

Thanks to the abstraction layer, adding it took one PR, merged the same day, and it works for all three decisions, including the router’s Choice:

judgment:
 provider: openai_decisions
 threshold: 0.8

From MindRoom’s side, only the wire format differs. The endpoint accepts only user messages, so MindRoom sends each chat message as its own user message, prefixed with its role, like assistant: .... Decisions can also answer with a refusal, which MindRoom treats as an abstention, so the usual fallback applies.

I tested it against the live API through MindRoom’s own code, with the same questions Jev gets. For participation, it gave 0.99 to replying to a direct question and 0.0 while two humans were talking to each other. Mid-turn, “Thanks, looks great so far!” got 0.0 for interrupting and “Stop, keep MySQL…” got 1.0. The router sent a pandas bug to code and a meeting request to calendar, both with confidence 1.0. A nonsense request got only 0.62, below the 0.8 threshold, so the normal LLM router took over. Each call took 130 to 280 ms. I have not compared it with Jev on real traffic yet, but both log the same outcome fields, so that only takes a config change.

Named after Jevons #

I only read why TypeSafe named it Jev after building all this: they named it after William Stanley Jevons, who noticed that as steam engines burned coal more efficiently, demand for coal went up instead of down. This is known as the Jevons paradox. That was literally my experience. Most of what I built was perfectly possible with an LLM. I just never built it, because I am somewhat cost- and latency-minded. Once I implemented Jev for one decision, I came up with use case after use case.

One I already built is a plugin, Response Audit JEV. Right after an agent replies, Jev checks the answer against the request and the tool calls the agent actually made, for example whether things are properly cited. If a check flags a problem, the plugin posts one follow-up in the thread that tags the agent and asks for a correction. Because Jev is so fast and cheap, running this after every reply costs almost nothing.

Right now I am full of ideas. For example, a model router. In MindRoom, an agent can switch to a different model mid-conversation with a tool call, and I can do the same with !model. Agents rarely do that without being told, though. So I default to a very expensive model, and I only save money when I remember to switch to a cheap one for something simple. Jev could run on every message and decide whether it needs the capable model or not. That works in both directions, up to a capable model or down to a cheap one. The perfect moment to switch is when the prompt cache has expired. A prompt cache belongs to one model, so switching normally means sending the entire context again at full price. Once the cache has expired, I pay that price anyway, so switching costs nothing extra.

Frontier-level dictation on your iPhone keyboard, self-hosted

Tue, 22 Sep 2026 00:00:00 +0000

On a recent month-long trip visiting family in Europe, I did a lot of work from my phone, using my mobile coding workflow. It still boggles my mind that real, productive work from a phone is possible now. Most of that work is talking to coding agents, and the built-in iOS dictation is garbage for that. I had an iOS Shortcut that sent recordings to Agent CLI, my local AI toolbox, running at home instead. The problem was that the recording screen takes over the whole display, so I could not see the thing I was commenting on. The transcript then landed in my clipboard, and I had to paste it myself.

So I threw my principles overboard and installed Wispr Flow, a voice keyboard for iOS. I loved it instantly. iOS does not let keyboards use the microphone, so the keyboard’s button hands off to the Wispr Flow app, which keeps recording in the background. I was less happy about that, since it meant a closed-source app always had the microphone open and sent my voice to their servers. I was on the free plan, so I was probably the product. I also hit the free weekly word limit very quickly, and started looking for an alternative.

That is how I found Diction, which has since become my favorite app when I’m away from my laptop. It works the same way, background recording included. The difference is that it has a self-hosted mode with an open-source gateway, so my voice goes to a machine I own instead of their servers.

I run the gateway on my home machine in the U.S., reachable only over my private network. The gateway streams audio to Agent CLI, which runs Alibaba’s Qwen/Qwen3-ASR-1.7B-hf on an RTX 3090 behind an OpenAI-compatible transcription endpoint. There is no LLM cleanup step, so Diction’s Writing Style and Tones features do not work with this setup. I don’t miss them: Qwen’s raw transcription is already good enough for me.

1. Why Qwen? #

For the longest time I used Faster-Whisper with Whisper large-v3. Whisper is still the default choice for most people, but many newer models beat it. I tried some of them, including NVIDIA’s Parakeet. Parakeet is fast, but it cannot take custom instructions. Those instructions are how Diction’s My Words feature biases a transcription toward specialized names.

At the start of September, I looked at the Hugging Face Open ASR Leaderboard, which tests speech-to-text models on the same recordings and ranks them. The score is the word error rate (WER): the share of words a model gets wrong, counting words it swaps, drops, or makes up. Lower is better, and a WER of 4% means about one wrong word in every 25. The test recordings range from audiobooks and podcasts to meetings and earnings calls, so the average says more than any single clean benchmark.

These are the averages on the public English test sets as of September 19, 2026:

Model	WER	Can I run it at home?
Zoom Scribe v2 Pro	3.6%	No, paid API (#1 overall)
ElevenLabs Scribe v2	4.0%	No, paid API
Qwen3-ASR 1.7B	4.3%	Yes, Apache-2.0
AssemblyAI Universal-3.5 Pro	4.3%	No, paid API
NVIDIA Parakeet TDT 0.6B v3	4.9%	Yes
OpenAI Whisper large-v3	5.8%	Yes

I picked the best model on the board whose weights you can download: Qwen3-ASR 1.7B. It lands within a point of the best paid API, which is what I mean by frontier-level. Since I maintain Agent CLI, I told an agent to add a Qwen backend, and shortly afterward it had landed and was running on my server.

Agent CLI’s transcription server loads a model on the first request and unloads it after an idle timeout, so the model only takes VRAM while I use it. The same process serves an OpenAI-compatible API, which the Diction gateway talks to, and the Wyoming protocol that Home Assistant uses for local voice.

2. The recipe #

You need an NVIDIA GPU with the NVIDIA Container Toolkit, Docker Compose 2.30 or newer, and a private route from your iPhone to the server. I use Headscale, but Tailscale or plain WireGuard works just as well. Do not expose the gateway to the public internet.

Write the Compose file #

Save the following as compose.yaml in an empty directory. Agent CLI’s CI publishes a CUDA image with the Transformers backend and Qwen as the default model, so there is nothing to build. The file points Diction’s gateway at Agent CLI’s transcription port. WHISPER_TTL raises the idle timeout from the default five minutes to a day, so dictation rarely waits for a reload:

services:
 asr:
 image: ghcr.io/basnijholt/agent-cli-whisper:0.112.0-cuda-transformers
 environment:
 WHISPER_TTL: "86400"
 volumes:
 - model-cache:/home/whisper/.cache
 gpus: all
 restart: unless-stopped

 gateway:
 image: dictionlabs/gateway:v14.0
 depends_on:
 - asr
 environment:
 CUSTOM_BACKEND_URL: http://asr:10301
 CUSTOM_BACKEND_MODEL: Qwen/Qwen3-ASR-1.7B-hf
 CUSTOM_BACKEND_CANONICAL_ID: Qwen/Qwen3-ASR-1.7B-hf
 CUSTOM_BACKEND_NEEDS_WAV: "true"
 ports:
 - "8080:8080"
 restart: unless-stopped

volumes:
 model-cache:

Start it and connect the app #

Start both services:

docker compose up -d

The first transcription downloads Qwen into the model-cache volume, so that request can take a few minutes. docker compose logs -f shows the progress.

On the iPhone, go to Settings → General → Keyboard → Keyboards → Add New Keyboard → Diction. Tap Diction in the keyboard list, enable Allow Full Access, and grant microphone access when prompted. Then open Diction, go to Preferences → Mode → Self-Hosted, enter http://<private-server-ip>:8080, and tap Test connection.

On my machine, Qwen3-ASR 1.7B uses about 4.7 GiB of GPU memory and roughly 3 GiB of system memory while loaded.

3. The bar test #

A day later, I met a friend in a bar in the Netherlands and wanted to let him try it.

He was already on my tailnet for some of my other self-hosted services, so his phone only needed an ACL change to reach the gateway. I opened the terminal app on my phone, connected to an agent, and dictated that change. In less than a minute, Diction was connected and working on his phone. His speech crossed the Atlantic, was transcribed by Qwen on my home GPU, and appeared back in the app with barely any delay.

That moment sold the whole setup to me. I was in a bar without a laptop, and I used dictation to grant access to the dictation service.

Removing the guardrails and letting my coding agents loose

Tue, 22 Sep 2026 00:00:00 +0000

I have run coding agents in YOLO mode since I started using them in May 2025. I think it is the only way to parallelize work. Sometimes I have ten agents running at the same time, and clicking “approve” on every command they run would make that impossible.

What makes me comfortable with it is that models tend to do what you ask, and that you can keep them in an environment where they cannot do much harm. I don’t keep secrets within their reach, and above all I have very good backups. The last piece is a small set of hooks that block the few things I never want an agent to do on its own.

Why hooks and not AGENTS.md #

My system prompt has always told agents not to force-push or merge PRs. Some of them did it anyway.

The first time I really noticed was with Gemini 3 Pro in November 2025, which merged my PR and force-pushed to main. It happened a couple more times in the weeks after, as I parallelized more and asked more of the models. Three weeks later I had hooks for Claude Code and Gemini CLI that block these commands before they run. Codex did not support hooks yet. To my surprise, it did not need them: OpenAI’s models are extremely good at following instructions, and Codex never ignored an explicit rule in my AGENTS.md. Codex supports hooks by now, so it runs the same ones as the others.

Claude has been worse at following rules, and Gemini far worse still: absolutely dogshit. I only try Gemini now and then, and I have been disappointed every time. By now I don’t even try the latest Google models anymore.

I don’t think agents are ever purposely malicious. They try to do what you asked, but they can misinterpret it, and then they force-push or merge something. A rule in a Markdown file is a hope; a hook actually stops the command. Not that a hook would stop a malicious agent: it could put the blocked command in a Bash script and run that instead. What the hook does is tell the agent why I don’t want it, because every block comes with a short explanation:

git add -A is not allowed - add files explicitly to avoid adding unrelated untracked files

The models are well aligned, so once they know my intention, they respect it.

What I block #

All hooks live in my dotfiles, and I deliberately kept them simple. The detection logic is one Python module, git_guard.py, and the Claude Code, Codex, and Gemini CLI hooks all import it.

git commit --amend and git push --force. The unit I review is a PR. When I review a PR, I know I have reviewed it up to a certain commit, and later I only look at the diff of the new commits. Rewriting history breaks that.

Pushing to main. Everything happens in PRs. Inside a PR I don’t care how messy the history gets. I want the agent to push very often, so no work gets lost, and every commit has to be green, so each one is a snapshot I can go back to. The agent maintains its own development of the feature, and I have not written a commit message myself in a long time. When the PR is done, I squash merge it.

gh pr merge. The merge comes after my review, so it is mine to do.

git add -A. I often have unrelated untracked or unstaged files lying around. I work in many open source repositories, and some of those files should not be public, like deployment plans. This is the only block without an override.

sleep. This one is for Claude, and Opus 5 specifically, which I hate with a passion. It would run tests in the background and then sleep 300, badly overestimating how long the tests take, when a blocking tool call returns exactly when they finish.

The override #

Over time the models got more capable. I could parallelize more, and I trusted them to work on their own for longer. I noticed I was spending a significant fraction of my time clicking buttons and doing operations I had forbidden the agent to do, only to end up doing them myself.

So in July I added an override. When a hook blocks a command, the agent gets this hint:

If (and only if) the user has explicitly approved this exact action,
re-run the command prefixed with EXPLICITLY_USER_APPROVED_HOOK_OVERRIDE=1 to override this block.

Nothing checks whether I actually approved it. It is a soft gate, but it works because the agent now has to write down, in the command itself, that I approved the action. When it runs into the block without my approval, it becomes obvious to the agent that it has to ask me first.

git add -A stays hard-blocked, and a chained command cannot carry an override past it: EXPLICITLY_USER_APPROVED_HOOK_OVERRIDE=1 git commit --amend && git add -A is still rejected.

When the override was used without approval #

It has happened, mostly with GPT-5.6 Sol, which I used a lot at the time.

It used to be that you had to keep your context window short, because auto-compaction was lossy and produced poor summaries. Since around GPT-5.3, compaction works well enough that I keep a session going for as long as I am working on the same feature. So I would ask the agent to merge a PR early on, start a follow-up PR in the same session, and after a compaction the agent concluded that it should merge the new PR too. It did, without my approval.

The other case was my pr-review skill, by far my favorite skill that I wrote myself. I would tell the agent to run it and squash merge the PR only if the review approved it. A few times, the review found problems, the agent fixed them, and then it merged. What I meant was: merge if it is approved right away, and otherwise wait for me. Now I spell that out and end with “otherwise, wait for my instructions.”

Working from my phone #

The override matters most when I am away from my computer. On a month-long trip visiting family in Europe, I worked only from my phone, using my mobile coding workflow. Typing commands in a terminal on a phone is inconvenient, so I would rather say what I want and let the agent do it, including the merge. I have also built end-to-end encrypted voice calls to my agents over Matrix with MindRoom, and I am moving more toward that kind of voice-driven workflow.

Not much has changed #

I recently reread my first post on agentic coding from August 2025, and very little in it is outdated. The models are more capable, and I give them much larger scopes, but the way I work is mostly the same. The hooks are the main thing I added, and the override is how I loosened them again.

MindRoom: AI agents that live in Matrix and work everywhere 🧠

Fri, 26 Jun 2026 00:00:00 +0000

Table of contents

1. The problem: your AI is trapped in apps #

Programming is my biggest passion in life. I’ve been actively involved in open source for over ten years now, maintaining 40+ installable packages across Python, JavaScript, and Rust. Some of these have over 100,000 users; others have exactly one—me. I mention this because it matters for what comes next: when AI coding tools arrived, I didn’t start from zero. As I wrote about in my agentic coding post, AI has completely transformed how I work—when I’m AFK, I regularly build full features or entire projects by dictating to my phone and routing it through my own transcription software.

Through all of this, I became deeply convinced that AI agents are the future. Not just a useful tool for programmers—a fundamental shift in how everyone will interact with computers.

But if that’s true, then the way agents work today is broken.

We’ve built artificial barriers between artificial intelligences. ChatGPT knows you in one tab, Claude knows you in another, your Slack bot knows you in a third. Your email AI can’t tell your calendar AI about that meeting invitation. Your coding assistant has no idea about the project specs sitting in your document AI. Switch platforms, and your AI starts from scratch.

Most agent frameworks require you to program them. That works for developers like me, but if agents are truly the future, they need to work for everyone. Non-programmers don’t want to write code—they just want a chat.

And then there’s privacy. I’m fine sharing my email with Gemini—Google already owns my Gmail. But do I want to send my financial data, my health records, or my personal notes to a cloud provider? Not really. I’d rather use a local model for sensitive tasks and pick the best, cheapest Chinese model for general-purpose deep research.

So I built MindRoom on a chat protocol—if people just want a chat, give them a chat. Agents live in the same rooms as you, collaborate in threads, and follow you across every platform. The first commit landed on July 29, 2025 (four months before OpenClaw’s first commit 😅), so by the time I’m publishing this MindRoom has been with me for nearly eleven months: first as an obsessive side project, then, since March, with much more sustained focus. At first I got so obsessed with it that I eventually burned out and had to step away completely.

Recently, seeing OpenClaw gain massive traction solving a similar problem reminded me that what I’d been building wasn’t just a niche obsession. So here I am, dusting off MindRoom and writing about what it is, how it works, and why I think the approach still matters.

Live chat.mindroom.chat Personal thread with the prompt: “Explain how MindRoom and what you can do.” Mind answers in the real Matrix room, using tool calls and the Codex-backed local MindRoom runtime.

2. What is MindRoom? #

MindRoom is an open-source system that creates AI agents living inside the Matrix protocol. If you’re not familiar with Matrix—it’s a federated, end-to-end encrypted communication standard. The same protocol used by the French government for 5.5 million civil servants, by German healthcare for 150K+ organizations, and by the Element app that millions of people use daily.

The key insight: Matrix has bridges to many major platforms—Slack, Telegram, Discord, WhatsApp, IRC, email, even SMS.

So if your AI agent lives in Matrix, it can reach you on any platform. One agent, every platform, continuous memory.

One practical caveat: Matrix bridges vary in maturity. Some, like the Telegram bridge, work very well, while others can be finicky. Your mileage may vary depending on which platforms you need.

Here’s what a typical setup looks like in config.yaml:

agents:
 code:
 display_name: CodeAgent
 role: Generate code, manage files, execute shell commands
 model: opus-4.8
 tools: [file, shell, github]
 instructions:
 - Always read files before modifying them.
 rooms: [lobby, dev]

 research:
 display_name: ResearchAgent
 role: Search the web, summarize papers, find information
 model: gpt-5.5
 tools: [tavily, arxiv, wikipedia]
 rooms: [lobby, research]

teams:
 super_team:
 display_name: Super Team
 agents: [code, research]
 mode: collaborate

Define your agents, give them tools and rooms, and they show up in Matrix as real users—with avatars, typing indicators, and online status.

MindRoom agents are real Matrix users with their own accounts, generated avatars, display names, room memberships, and online presence.

3. How it actually works #

At the core sits what I call the MultiAgentOrchestrator—a class in bot.py that boots every configured entity (router, agents, teams), provisions Matrix user accounts for each one via matrix-nio, and keeps sync loops alive. The agents themselves are powered by the Agno framework, which provides a unified interface across AI model providers.

When someone sends a message in a Matrix room:

Explicit mentions go directly to the named agent (@mindroom_code help me debug this)
Thread continuations keep the same agent responding (no weird context switches mid-conversation)
New conversations hit the router—an AI that analyzes the message and picks the best agent based on capabilities

All conversations happen in threads, which keeps rooms organized. Agents stream their responses in real-time, editing a single message as they think rather than spamming new ones. You see tool calls happening live:

🔧 Tool Call: search_web(query="matrix protocol bridges")
[waiting for result...]
✅ search_web result:
[results here]

The recording below starts in the main Personal room, sends the prompt Explain how MindRoom and what you can do., and then follows MindRoom as it opens the new thread and streams the reply to completion. It is sped up 2.5x to keep the flow skimmable.

4. Building on Matrix: the good and the tricky #

One of the best things about building on Matrix is what you get for free. End-to-end encryption, a deeply interoperable chat protocol with bridges to many platforms, a choice of clients, federation between organizations—all of that comes with the protocol. You don’t have to build any of it yourself.

But because Matrix has such a tight specification, it also brings challenges.

The protocol doesn’t support streaming. AI agents that think for 30 seconds before dumping a wall of text make for a terrible chat experience, so I hacked streaming in by rapidly editing the same message as new tokens arrive. An ⋯ marker shows while the agent is still thinking—a small touch, but it makes the experience feel responsive and alive.

There’s also a size limit on message content. That’s fine for human chat, but AI responses can get long—especially when tool calls and their results are included. I worked around this by using Matrix’s attachment feature: when a response exceeds the limit, the content continues in an attachment that gets updated as the message keeps streaming in. This required forking the Cinny chat client so that attachments display inline rather than as downloadable files, making the whole thing seamless.

5. Memory that follows you #

MindRoom currently supports two memory implementations.

The first is the more traditional semantic memory system built on Mem0’s AsyncMemory, with configurable embedding providers (OpenAI, Ollama, or HuggingFace) and vector storage via ChromaDB. This gives agents searchable memory for preferences, project context, decisions, and recurring facts.

The second is a file-based memory system inspired by the simplicity of OpenClaw-style agent files. Instead of burying identity in a database, durable context can live in plain Markdown files that travel with the agent. That portability matters: I can move the same agent identity from OpenClaw to Hermes to MindRoom without starting from scratch.

MindRoom also has a memory flush step that periodically extracts durable facts from conversations and writes them back into memory. So the chat transcript remains chat, while stable preferences, decisions, and identity details become reusable context for future turns.

A live Personal memory demo: MindRoom stores a harmless preference in one turn and retrieves it later from the same Matrix conversation.

6. 100+ built-in tool integrations #

Agents can use over 100 built-in integrations:

Category	Examples
Communication	Gmail, Slack, Telegram, Discord
Development	GitHub, Shell, Python, Docker
Search	Tavily, Wikipedia, Arxiv, DuckDuckGo
Productivity	Google Calendar, Jira, Linear, Todoist
Smart Home	Home Assistant
AI/ML	DALL-E, ElevenLabs, Replicate
Data	Pandas, SQL, DuckDB, Yahoo Finance
Web	Firecrawl, Crawl4ai, Browser automation

Tools are lazy-loaded and credential-managed, so an agent only loads what it needs.

7. Teams: agents that collaborate #

Single agents are useful, but sometimes you need a team. MindRoom supports two collaboration modes:

Coordinate mode: A lead agent orchestrates others. You ask a question, the lead delegates subtasks, collects results, and synthesizes a unified response.

Collaborate mode: All agents work on the same task in parallel, each providing their independent analysis. The system then merges their responses with a consensus summary.

In practice, you might have a research team where one agent searches academic papers, another checks industry news, and a third validates claims—all triggered by a single message. The live room view below shows a real team response in the same Matrix thread surface normal agents use: threads, routed agents, and room history are all first-class chat objects rather than a separate dashboard.

Live Lobby thread in chat.mindroom.chat showing a real Team Response from HomeAssistant and AgentBuilder. Team collaboration uses the same Matrix-native interface as regular agent chats, with routed agents and room membership visible beside the conversation.

8. Hot-reload: change config and plugins without downtime #

config.yaml is watched at runtime. When you edit it—add an agent, change a model, update instructions—MindRoom diffs the old and new config, gracefully restarts only the affected agents, and has them rejoin their rooms. No downtime, no full runtime restart required.

The same development loop applies to plugins. MindRoom has a rich Python hook system, and plugins can be live-developed while the system is running. Change a plugin file, save it, and MindRoom automatically reloads the plugin without restarting the Python runtime.

This sounds minor, but when you’re iterating on agent behavior or plugin hooks, being able to tweak code and config and see results in seconds significantly improves the development loop.

9. Voice, scheduling, and other features #

Some features I built purely because I wanted them for myself (a recurring theme in my projects, as anyone who’s read my local AI journey can attest). I even built Matty, a Matrix CLI client, from bed at midnight because I needed a way to interact with my agents from the terminal:

Voice messages: Audio messages in Matrix are auto-transcribed via Whisper and treated as regular text input. You can talk to your agents.
Scheduled tasks: Natural language scheduling (!schedule "check my email every morning at 9 AM") backed by cron jobs. Agents can run tasks in the background and escalate to you when needed.
DM support: Agents respond naturally in 1:1 conversations without needing mentions.
Cross-organization federation: Because Matrix is federated, two companies’ AI agents can collaborate in a shared room—something that’s hard to do on proprietary platforms.

10. The obsession, the SaaS dream, and the burnout #

I need to be honest about what actually happened with MindRoom, because “life got in the way” is a sanitized version of the story.

What really happened is that I got completely, utterly obsessed. Every single second I wasn’t working at my day job or sleeping, I was working on MindRoom. Many hundreds of hours went into it. I seriously considered quitting my job and starting an AI startup around it.

If you’ve read Armin Ronacher’s post Agent Psychosis: Are We Going Insane?, you’ll know exactly what I’m talking about. The dopamine hit from building with AI agents is incredibly real. As Armin writes: “You feel productive, you feel like everything is amazing, and if you hang out just with people that are into that stuff too, without any checks, you go deeper and deeper into the belief that this all makes perfect sense.”

That was me. I was building and building, shipping feature after feature, and it felt incredible. The codebase grew to over 1,000 commits. I built the core system, then a React dashboard, then I started on a full SaaS platform—Kubernetes deployments on Hetzner Cloud, a FastAPI backend, a Next.js 15 frontend, Stripe integration, Supabase auth, Helm charts for multi-tenant isolation.

And that’s where the enthusiasm started to erode. Not because the core idea was bad, but because the work shifted from building interesting things (agent orchestration, memory systems, routing intelligence) to grinding through SaaS boilerplate: GDPR compliance, payment processing, automated Kubernetes deployments, terms of service, cookie banners. The dopamine loop that kept me going at 2 AM broke when the work stopped being creative and started being compliance paperwork.

The hosted control plane lives at app.mindroom.chat. I am deliberately not turning this section into a dashboard tour; the interesting part, and the part I still care about most, is the Matrix-native runtime underneath it.

11. OpenClaw and why it matters #

OpenClaw takes a different approach to the same core problem. Where MindRoom builds on Matrix federation as the backbone, OpenClaw runs a local Gateway on your machine and connects to messaging platforms directly (WhatsApp via Baileys, Telegram via grammY, Discord via discord.js, and so on).

The similarities are striking:

Both give you a single AI assistant across all your messaging platforms
Both have persistent memory
Both support multiple tools and integrations
Both are open-source and privacy-focused
Both let you choose your own AI model

The differences are interesting too. OpenClaw is local-first and TypeScript-based with 380K+ stars and a massive community. MindRoom is Python-based, federation-first, and… well, let’s just say my GitHub star count is a few orders of magnitude lower.

But seeing OpenClaw validate the core idea—that people want AI assistants that aren’t trapped in apps—reminded me that what I’d been building matters. The federation angle is something OpenClaw doesn’t have: the ability for agents from different organizations to collaborate natively, with end-to-end encryption (Olm/Megolm), on a protocol that governments already trust and deploy at scale.

MindRoom uses Matrix as the federated backbone; OpenClaw takes a local-gateway, direct-connector approach.

And then there’s Moltbook—a social network built exclusively for AI agents that the internet was losing its mind over for a couple of days. I’m not personally convinced it’s the next big thing, but what caught my attention is that MindRoom could support something like it natively. Matrix is already federated—agents on different servers can already interact, join shared rooms, and collaborate across organizational boundaries. A bridge from a Reddit-like platform to Matrix and your agents could participate without any special integration.

Building agent-to-agent infrastructure from scratch also means building security from scratch—and that’s where new platforms tend to struggle. Matrix has spent years hardening its E2E encryption, and it’s already deployed by governments and healthcare organizations at scale. Building on that foundation means you inherit those security properties rather than hoping to get them right yourself.

12. What’s next #

I first drafted this post a couple of months ago, when MindRoom was still something I was trying to come back to. That part is already out of date. Since January I’ve been spending nearly every free hour on it again, and since March it has effectively been my full-time work. The codebase supports 8+ AI model providers—OpenAI, Anthropic, Ollama, Groq, Google, OpenRouter, DeepSeek, Cerebras—and the core architecture is solid. It also has real users now. My rough guess is that something like 30-100 people have tried MindRoom in one form or another, which is still early, but meaningfully different from “only I have touched this.” There’s already a Docker Compose file and a starter repository to get started, but I want to make it even simpler.

What I’m focusing on now:

Skills system: Building an ecosystem of reusable agent behaviors (already partially implemented with OpenClaw-compatible format)
Even easier onboarding: A docker compose up gets you running today, but I want a wizard that gets you from zero to working agents without touching a YAML file
Community: Turning the small group of early users into a real operator and contributor community
Scope discipline: Keeping the work sustainable now that MindRoom is effectively full-time, not slipping back into the obsessive 2 AM marathon from before

I’m deliberately leaving the SaaS ambitions aside for now. The core system is what matters, and that’s where the energy is going.

If you’re interested in AI agents that live in Matrix and work everywhere, check out MindRoom on GitHub.

Sometimes seeing someone else succeed with a similar idea is enough to remember why you started building in the first place. And sometimes the most important lesson from a project isn’t technical—it’s learning when to step away and when to come back.

Are you running AI agents on any messaging platforms? Have you tried Matrix for anything beyond regular chat? I’d love to hear about your setups!

Links and resources #

Matrix MCP

Tue, 05 May 2026 00:00:00 +0000

Parallel agentic coding made trivial with `agent-cli dev`

Tue, 13 Jan 2026 00:00:00 +0000

Table of contents

1. The “just use worktrees” problem #

Last week, I was chatting with a friend about my agentic coding workflow. I was sharing some of my best practices, and when the topic of parallel development came up, I said:

“Just use git worktrees with quickly installable environments.”

He stared at me blankly.

And then it hit me: for most developers who just git add, git commit, git push and don’t touch their environment setup once it works, this is not trivial. What seems obvious to me involves:

Understanding what git worktrees even are
Knowing the git worktree add command and its flags
Remembering to fetch before creating
Creating a sibling directory structure
Running your package manager’s install command
Copying over .env files
Setting up direnv or manually activating your virtual environment
Opening a new terminal tab
Navigating to the worktree
Finally launching your coding agent

That’s a lot of friction for something you want to do frequently when using agentic tools.

So I built agent-cli dev.

2. Why parallel development matters for agentic coding #

In my previous post on agentic coding, I described my workflow of running 5-6 Claude Code sessions simultaneously, each working on a different feature. This is how I can easily work on multiple features in parallel while waiting for an agent to process and generate code.

The usual way: Start feature A → wait for Claude to finish → review → move to feature B → repeat.

The better way: Start feature A in tab 1 → switch to tab 2, start feature B while A is working → switch to tab 3, start feature C → circle back to tab 1, review what Claude did → give feedback, move to next tab → repeat every 10-15 minutes.

This parallel approach allows me to go much faster. But it only works if each tab has its own completely isolated and working environment.

That’s where git worktrees come in. A worktree gives you a separate working directory for a branch, sharing the same git history but with independent file states.

The problem is the setup. Every time you want to start a new feature:

# The manual way (don't do this)
git fetch origin
git worktree add ../my-project-worktrees/cool-feature -b cool-feature origin/main
cd ../my-project-worktrees/cool-feature
uv sync --all-extras # or npm install, or poetry install, or...
cp ../.env .env
# Now open a new terminal tab...
# Navigate there...
# Finally start claude

After the 100th(?) time doing this manually, I knew I needed to automate it.

3. One command to rule them all #

Here’s what I built:

agent-cli dev new --agent

That’s it. One command. Here’s what it looks like:

→ Generated branch name: cool-crane
→ Creating worktree for branch 'cool-crane'...
→ Running: git fetch origin
→ Running: git worktree add -b cool-crane ~/Work/project-worktrees/cool-crane origin/main
✓ Created worktree at ~/Work/project-worktrees/cool-crane
✓ Copied env file(s): .env.example, .envrc
→ Detected Python project with uv
→ Running: uv sync --all-extras
→ Running: direnv allow
✓ direnv: allowed existing .envrc
✓ Started claude in new iterm2 tab

╭─────────────────────────────────────────────── Success ───────────────────────────────────────────────╮
│ Dev environment created: ~/Work/project-worktrees/cool-crane │
│ Branch: cool-crane │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯

4 seconds later, I have a new terminal tab with Claude Code already running in a fresh, isolated environment. Ready to work on my next feature.

What happens under the hood:

Generates a random branch name like clever-fox (you can just set the name yourself of course)
Creates the git worktree in a sibling directory
Detects the project type and runs the appropriate setup (uv sync, npm install, etc.)
Copies .env files from the main repo
Sets up direnv so the environment activates automatically
Opens a new terminal tab (works with tmux, zellij, iTerm2, and more)
Launches your coding agent in that tab

The Docker-style random names (clever-fox, happy-bear, swift-owl) there just for those too lazy to pick a name. Just agent-cli dev new --agent and start coding.

4. It knows your stack #

The command auto-detects your project type:

If it sees…	It runs…
`uv.lock`	`uv sync --all-extras`
`pixi.lock`	`pixi install`
`poetry.lock`	`poetry install`
`requirements.yaml`	`unidep install -e .`
`package-lock.json`	`npm install`
`Cargo.toml`	`cargo build`
`go.mod`	`go mod download`

For my projects using unidep, it even detects monorepos and adjusts the command accordingly.

It also generates the right .envrc for direnv. For a Python project with uv, that’s source .venv/bin/activate. For a conda/unidep project, it’s the full micromamba activation. For Nix projects, it’s use flake.

This means when you cd into the worktree later, your environment is automatically activated. No more source .venv/bin/activate.

5. Terminal magic #

The real magic is opening a new terminal tab with the agent already running.

The command detects which terminal you’re in and uses the appropriate API—whether that’s iTerm2, Kitty, Warp, or GNOME Terminal. It even works with terminal multiplexers like tmux and zellij (my current favorite), opening a new window/tab in your existing session.

6. Spawning agents with prompts #

The --prompt option transforms how I spawn parallel agents. Instead of just opening a blank Claude session, I can give the new agent its task immediately:

agent-cli dev new fix-auth-bug --agent --prompt "Fix the login validation bug in auth.py. The issue is that empty passwords pass validation."

The agent opens with the prompt already loaded and starts working immediately. No copy-pasting, no context switching, no “let me explain what I need you to do.”

For longer, more detailed prompts (which you should use for complex tasks), there’s --prompt-file:

agent-cli dev new new-feature --agent --prompt-file .claude/spawn-prompt.md

This is particularly powerful when you’re in a Claude session and discover a feature that needs work but is unrelated to your current task. Instead of context-switching or noting it for later, you can spawn a new agent right there:

Write a detailed prompt to a file (Claude can do this)
Run agent-cli dev new unrelated-feature --agent --prompt-file .claude/spawn-prompt.md
A new terminal tab opens with a fresh Claude session already working on the task
Continue with your original work

The spawned agent works independently in its own branch while you stay focused.

7. Teaching Claude to spawn agents #

Here’s where it gets really interesting: you can teach Claude Code itself how to spawn parallel agents.

agent-cli dev install-skill

This installs a Claude Code skill that teaches Claude how to use agent-cli dev. The skill includes detailed examples and prompt templates following Anthropic’s prompt engineering guide.

With the skill installed, you can say things like:

“Work on auth, payments, and notifications in parallel”
“This feature is unrelated to what we’re doing—spawn a new agent for it”
“Split this refactoring by module and work on them simultaneously”

Claude will write well-structured prompts (using proper XML tags, clear context, explicit requirements) and spawn the agents for you. Each spawned agent writes a report to .claude/REPORT.md when done, so you can easily review what happened.

One of my favorite patterns: I ask Claude to look at the latest 10 open GitHub issues and spawn an agent for each one. Ten independent parallel environments, set up in seconds, each working on a different issue. That’s the kind of leverage this enables.

8. My actual workflow #

Here’s how I use this in practice. I’m working on agent-cli itself, and I want to add three features:

# Tab 1: New RAG feature
agent-cli dev new rag-improvements --agent --prompt "Improve RAG chunking strategy for better retrieval accuracy"

# Tab 2: Fix a bug in the transcribe command
agent-cli dev new fix-transcribe-vad --agent --prompt "Fix VAD detection cutting off words at sentence boundaries"

# Tab 3: Refactor the config system
agent-cli dev new config-refactor --agent --prompt "Refactor config.py to use a single Config class with lazy loading"

Three commands, three isolated environments, three Claude Code sessions already working on their tasks.

I cycle through the tabs every 10-15 minutes:

Review what Claude did
Give voice feedback using my transcription workflow
Move to the next tab

When a feature is done, I have Claude open a PR from that branch. When the PR is merged:

agent-cli dev clean --merged

This uses the GitHub CLI to check which branches have merged PRs, then removes those worktrees and deletes the branches. No more zombie worktrees cluttering my disk.

9. Managing environments #

A few other commands I use regularly:

# List all worktrees
agent-cli dev list

# Navigate to a worktree
cd "$(agent-cli dev path clever-fox)"

# Remove a worktree
agent-cli dev rm clever-fox

# Force removal + delete branch
agent-cli dev rm clever-fox --force --delete-branch

# Check what's available on your system
agent-cli dev doctor

The doctor command is handy for seeing which terminals, editors, and coding agents are installed and detected.

10. Getting started #

agent-cli dev is part of the agent-cli package:

# With uv (recommended)
uv tool install agent-cli

Then in any git repository:

# Check what's available
agent-cli dev doctor

# Create your first parallel environment with a task
agent-cli dev new my-feature --agent --prompt "Add user profile page with avatar upload"

# Or install the skill so Claude can spawn agents for you
agent-cli dev install-skill

Supported coding agents: Claude Code, Codex, Gemini, Aider, GitHub Copilot, Continue, OpenCode, and Cursor Agent. Use --with-agent gemini to pick a specific one, or set a default in your config.

Supported terminals: tmux, zellij, iTerm2, Terminal.app, Kitty, Warp, and GNOME Terminal.

Supported editors: Cursor, VS Code, Zed, PyCharm (and other JetBrains IDEs), Neovim, Vim, Emacs, Sublime Text, and nano.

Conclusion #

The barrier between “I want to work on this feature” and “I’m actually working on it in an isolated environment” should be zero.

Before:

2-5 minutes of manual setup per feature
Often skipped because “it’s not worth it for a small change”
Mental overhead remembering all the steps

After:

One command, few seconds
No excuse not to work in isolation
More parallel features, faster iteration

If you’re using agentic coding tools and not yet working in parallel, you’re leaving productivity on the table. And if the setup friction was stopping you—well, now it’s gone.

Try it out and let me know what you think! The code is open-source at github.com/basnijholt/agent-cli.

Links #

SOTA RAG & Memory without the database: Files, Git, and simple folders

Wed, 10 Dec 2025 00:00:00 +0000

Table of contents

1. The amnesia problem #

If you’ve read my posts on agentic coding or my local AI journey, you know I am all-in on using AI to control my computer and write software. But one critical piece was missing: Context.

LLMs are powerful but stateless. Every time I start a new chat, the model forgets who I am, what I’m working on, and that I prefer functional Python code over object-oriented styles. Worse, it also forgets the thing I just told it five minutes ago in another window.

The standard solution to this is RAG (Retrieval-Augmented Generation) and Memory. But when I looked at the available tools, I hit a wall.

Most solutions fell into two buckets:

SaaS / Web UIs: Upload your files to a cloud service or use a specific web chat interface that locks you in.
Developer Libraries: “Just use LangChain/LlamaIndex!” they say. Great, now I have to write a Python script every time I want to ask a question about a PDF?

I didn’t want a library. I didn’t want a web UI. I wanted a folder. I wanted to drop a file into ~/my-docs, and have my AI instantly know about it. And I wanted my AI’s “memory” to be a file I can edit, not a hidden vector in a database I can’t inspect.

So I decided to build my own file-based solution.

2. Learning from failure: AI Journal #

This isn’t my first attempt at building a memory system.

A few months ago, I built AI Journal—an ambitious local-first journaling app where an AI would read your entries and build a “living self-model” of who you are. It had sophisticated features: a structured “persona core,” typed “claim atoms” with evidence links, four organizational layers (L1→L4), recency-aware scoring with decay functions, and Git-based time travel to “ask questions of your younger self.”

The architecture seemed solid. The problem was simpler: local models aren’t good enough yet.

I designed the whole thing to run entirely on Ollama with my dual RTX 3090s. But even with a 48GB of VRAM, the models couldn’t reliably extract structured claims, handle the multi-step reconciliation, or maintain coherence across long conversations. Whether the design was right, I still don’t know—but the execution definitely hit the wall of current local LLM capabilities.

The lesson: I tried to build the entire system at once without validating the core concepts and individual ideas first. This time, I took a different approach: tackle it problem by problem. Instead of designing a complete system upfront, I would validate each piece—chunking, retrieval, re-ranking, memory reconciliation—before combining them. The experience and UX would be my own, but the algorithms would be adapted from established projects.

3. How I studied SOTA: clone, read, re-implement #

My approach was more methodical this time. I cloned five major frameworks into the same workspace as agent-cli:

LlamaIndex: The OG RAG framework
LangChain: The kitchen-sink approach
Letta (formerly MemGPT): Stateful agents with long-term memory
Mem0: Focused specifically on memory extraction
PydanticAI: Clean, typed agent framework

I didn’t read through all this code myself—that would take months. Instead, I used AI coding agents to search the codebases and write summaries of the technical implementation details:

Gemini CLI (Gemini 3.0 Pro)
Claude Code (Opus 4.5)
Codex CLI (GPT-5.1-Codex-Max)

I fed those summaries into ChatGPT 5 Pro together with my high level plan of what I wanted to create, and it created an implementation plan. Then I took that plan back to the coding agents for implementation. I personally found Opus 4.5 the best coding model, but GPT‑5.1 Codex Max was the best at reviewing the code and actually identifying critical bugs.

For validation, I used clip-files to copy the agent-cli source code plus the architecture documents (like memory.md and rag.md) into ChatGPT 5 Pro to validate the approach.

The AI did the searching and implementing; I did the understanding and decision-making.

4. The dependency problem: why ONNX over PyTorch #

One thing became immediately clear when going through these frameworks: they’re massive.

Installing LlamaIndex with all its dependencies pulls in PyTorch. PyTorch alone is 2-3GB. Add transformers, sentence-transformers, and the kitchen sink, and you’re looking at an 8GB+ environment just to ask questions about some PDFs.

This felt excessive for a CLI tool.

One of my core design goals was minimal dependencies. I wanted agent-cli to install in seconds, not minutes. I wanted it to work on my Mac, my NixOS server, and my friend’s laptop without downloading half the internet.

The solution: ONNX Runtime.

ONNX Runtime is ~200MB and runs inference on basically anything—CPU, CUDA, even Apple Silicon. For embeddings, I use embeddinggemma running locally via llama.cpp. For re-ranking, I use a cross-encoder exported to ONNX (Xenova/ms-marco-MiniLM-L-6-v2).

The result: pip install agent-cli[rag] takes about 30 seconds and adds ~300MB. The key insight: embeddings and re-ranking are cheap operations. You don’t need a GPU for them—CPU is perfectly fine. PyTorch with CUDA support can easily reach 2-3GB+; ONNX Runtime is ~200MB and runs anywhere. Since I’m not training models, just running inference on small encoders, ONNX is the obvious choice.

5. RAG: just let me grep it (kind of) #

My requirement for RAG was simple: Files on disk are the source of truth.

I didn’t want to manage an ingestion pipeline or trigger APIs. I implemented a RAG proxy that uses OS-level file watchers (watchfiles). Here is the workflow:

I drop a PDF, Markdown, or Python file into ~/rag_docs.
The system detects the Create event.
It chunks the text and embeds it into a local ChromaDB instance.
It’s immediately searchable.

When moving or deleting files, the system updates the index accordingly.

The architecture: OpenAI compatibility #

The “secret sauce” isn’t the retrieval itself; it’s how I expose it. Instead of building a custom chat app, I built an OpenAI-compatible proxy.

flowchart LR Client["Client
(Cursor, Open WebUI)"] --> Retrieve subgraph Proxy ["RAG Proxy"] direction TB Retrieve --> Augment["Augment & Forward"] Chroma[("ChromaDB")] -.-> Retrieve Watcher["File Watcher"] --> Chroma end Augment --> Upstream["Upstream LLM"]

# Point any OpenAI-compatible tool at your local proxy
export OPENAI_BASE_URL=http://localhost:8000/v1

That’s it. Any tool that supports custom OpenAI endpoints—Cursor, Cline, Open WebUI, LibreChat—instantly gains the ability to “read” my document folder.

Two-stage retrieval: the SOTA approach #

I looked at LlamaIndex here. Standard “semantic search” (using just embeddings/bi-encoders) is often disappointing. It finds text that looks similar but isn’t actually relevant.

A common pattern in modern RAG systems is Two-Stage Retrieval.

Fast Stage: Retrieve 3x candidates using standard embeddings (text-embedding-3-small or local models like embeddinggemma).
Smart Stage: Use a Cross-Encoder to re-rank the results.

A cross-encoder looks at the query and the document together and outputs a relevance score. It’s computationally heavier, but since I’m running this locally on my RTX 3090 (or even CPU via ONNX), the 50ms latency penalty is worth it for results that are actually useful.

This isn’t a new idea, but implementing it cleanly with ONNX avoided the huge dependency tax.

6. Memory: files over databases #

This is where my approach diverges most from standard tools like Mem0. Most memory systems treat memories as database rows. If the AI remembers “User likes pizza,” that’s a vector in a DB. If it hallucinates and records “User hates pizza,” good luck fixing it. You need to write a script to query the DB and delete the row.

I prefer data I can inspect.

My memory system stores every fact as a Markdown file with YAML front matter.

---
id: "550e8400-e29b-41d4-a716-446655440000"
role: "memory"
created_at: "2025-12-15T10:00:00Z"
conversation_id: "default"
---
The user prefers using 'uv' for Python package management.

Git integration: versioning memory 🕰️ #

Because memories are just files, I added a feature that I haven’t seen anywhere else: Automatic Git Versioning.

Every time the AI learns a new fact, updates an old one, or deletes a contradiction, the system automatically commits the change to a local git repository.

This leads to incredible capabilities:

Inspectability: I can git log and see exactly when the AI learned something.
Correction: If the AI learns something wrong, I open the markdown file in VS Code, edit it, and save.
Reversion: Agent hallucinated wildly yesterday? git reset --hard HEAD~1.

The reconciliation loop #

The logic for storing memories isn’t just “append to file.” Digging into Mem0 (and Letta) showed me that memory systems need active reconciliation. When I say “Actually, I switched to nix instead of uv,” the system:

Retrieves existing memories about package managers.
Decides (using a lightweight LLM) whether to ADD, UPDATE, or DELETE.
Executes the file operation (moving the old fact to a deleted/ folder for audit trails).

This keeps the memory bank clean and contradictory-free, unlike simple vector stores that just accumulate conflicting junk over time.

The tiny detail that broke everything #

After implementing the memory system—mostly modeled after Mem0—I noticed it didn’t work as well. After a lot of debugging, I found a tiny difference: I was asking the model for structured data via a tool call, whereas Mem0 uses freeform JSON that the LLM returns directly.

Since I use PydanticAI, fixing this was a one-line change:

agent = Agent(
 model=model_cfg,
 system_prompt=UPDATE_MEMORY_PROMPT,
- output_type=list[MemoryDecision],
+ output_type=PromptedOutput(list[MemoryDecision]), # JSON mode instead of tool calls
 retries=3,
)

This is exactly why reading the source code matters—not just the algorithms, but the tiny details that make them work.

7. Why building this was necessary #

You might ask: “Bas, why write thousands of lines of code for this? Why not just use ChatGPT’s memory or install LangChain?”

Three reasons: Privacy, Integration, and Dependencies.

Privacy #

I want to feed my AI my financial records, my health data, and my private journals. I am not comfortable sending that to OpenAI’s long-term storage or some VC-backed startup’s vector cloud. With agent-cli, the vectors live in ~/.cache/agent-cli/chroma, the files live in ~/documents, and the only thing leaving my machine is the specific context needed for a specific query (if I use a cloud model) or nothing at all (if I use Ollama/llama.cpp).

Integration #

By building this as an API proxy, I solved the “fragmented tools” problem. Any tool that supports custom OpenAI-compatible endpoints works out of the box:

agent-cli chat in the terminal
Cursor or Cline for agentic coding
Open WebUI, LibreChat, or Lobe Chat in the browser

Because my system speaks “OpenAI,” all of these tools share the same memory and the same document index. If I tell the terminal agent “I’m working on Project X,” and then switch to Cursor, Cursor knows about Project X. I can try a new chat UI tomorrow without losing anything—just point it to localhost:8000.

Dependencies #

To me, it just doesn’t feel right that asking questions about a PDF requires downloading 8GB of PyTorch. By choosing ONNX Runtime over the PyTorch ecosystem, I kept the entire install lightweight and fast. This matters when you want to run the system on a cheap VPS or your friend’s laptop.

8. Try it yourself #

The entire system is open-source and available in agent-cli.

Try Memory #

Get an LLM that remembers you using Ollama.

# 1. Pull models and start proxy
ollama pull embeddinggemma:300m && ollama pull gpt-oss:20b
uvx -p 3.13 --from "agent-cli[memory]" agent-cli memory proxy \
 --memory-path ./my-memories \
 --openai-base-url http://localhost:11434/v1 \
 --embedding-model embeddinggemma:300m

# 2. Chat from the terminal (env vars point to proxy)
export OPENAI_BASE_URL=http://localhost:8100/v1 OPENAI_API_KEY=dummy

# Tell it something about yourself
uvx openai api chat.completions.create -m gpt-oss:20b -g user "My name is Alice and I love hiking"

# Later, ask if it remembers
uvx openai api chat.completions.create -m gpt-oss:20b -g user "What's my name?"

Try RAG #

Chat with your documents.

# 1. Start the RAG proxy (runs in foreground)
uvx -p 3.13 --from "agent-cli[rag]" agent-cli rag-proxy \
 --docs-folder ./my-docs \
 --openai-base-url http://localhost:11434/v1 \
 --embedding-model embeddinggemma:300m

# 2. Chat with your docs
OPENAI_BASE_URL=http://localhost:8000/v1 OPENAI_API_KEY=dummy \
 uvx openai api chat.completions.create \
 -m gpt-oss:20b \
 -g user "What does my documentation say about X?"

9. What I learned #

This journey reinforced a few beliefs:

Validate components first. AI Journal failed not because the ideas were wrong, but because I tried to build a complete system before validating individual components. This time I tackled problems one at a time—chunking, retrieval, re-ranking, memory reconciliation—and only combined them once each piece worked.
Study implementations, not just papers. Reading about two-stage retrieval is one thing. Reading how LlamaIndex actually implements it—the edge cases, the defaults, the performance tradeoffs—is far more valuable. Clone the repos. Read (or make AI read) the code.
Dependencies matter. A 30-second install versus a 10-minute download changes whether someone actually tries your tool. I am happy with the ONNX Runtime over PyTorch choice.
Files are the ultimate API. By stripping away the complex databases and proprietary interfaces, I ended up with a system that is robust, version-controllable, and surprisingly simple to reason about.
Use the best model for each job. I used 4 different AI models during development, each with different strengths. Gemini 3.0 Pro, Opus 4.5, GPT-5.1-Codex Max. I used all of them during the implementation phase, but each one has its own strengths. Fore example, Opus 4.5 was the best coding model, but GPT-5.1 Codex Max was the best at reviewing the code and actually identifying critical bugs (use the built-in /review feature).

Wyoming MLX Whisper

Fri, 05 Dec 2025 00:00:00 +0000

Switching from Ollama to llama-swap + llama.cpp on NixOS: the power user's choice 🦙

Sat, 29 Nov 2025 00:00:00 +0000

Table of contents

The draft I never published #

I started writing this blog post four months ago. Back then, I was frustrated because a specific model I wanted to try—gpt-oss-20b—was completely broken in Ollama. I went down the rabbit hole, set up llama-swap and llama.cpp, got the model working perfectly, wrote half this post… and then I stopped.

Why? Laziness. Once the novelty of that specific model wore off, I drifted back to Ollama. The real convenience of Ollama was never the ollama run command—I almost exclusively interact with my models via the API using my own tools like agent-cli. The convenience was ollama pull. Being able to grab a model with one command and have it “just work” was great.

But as a NixOS user, this imperative convenience is actually a flaw. I want my system to be declarative. I want my model configuration committed to git, not hiding in a local database. So I let convenience win for a while, but it always felt wrong.

But last month, that changed. I upgraded my hobbyist AI rig by adding a second RTX 3090. Suddenly, I had 48GB of VRAM and a whole new set of problems—and possibilities. I wanted to run massive models (like gpt-oss-120b or Qwen-3-VL-32B with a huge context window). I needed to balance layers between two GPUs. I needed to spill specific parts of the model into system RAM without crashing.

When you start pushing hardware to its limits, “magic” abstractions like Ollama stop being helpful and start getting in the way. I realized that llama.cpp isn’t just a backend; it is the place where development happens. If you want to use split-mode inference, granular CPU offloading, or the latest quantization tricks immediately, there is no alternative.

So, I dusted off this draft, updated my Nix configuration, and I’ve been running llama-swap full-time for a month. I’m finally ready to vouch for it.

The original trigger: broken models #

Four months ago, my frustration stemmed from compatibility. I was trying to run gpt-oss-20b, a model that required specific chat templates and GGUF handling that Ollama simply broke.

The maintainer of llama.cpp, Georgi Gerganov, explained it perfectly regarding the gpt-oss debacle: Ollama forked the ggml inference engine to rush out “day-1 support” for gpt-oss without coordinating with upstream. The result? Their implementation was not only incompatible with standard GGUF files but also significantly slower and unoptimized.

Benchmarks at the time showed Ollama had significant performance overhead—often 20-30% slower inference speeds compared to running llama.cpp directly.

This pattern means Ollama will always be behind llama.cpp. Meanwhile, I’ve had a pull request sitting unreviewed for over a month that fixes their broken OpenAI API. Compare that to llama.cpp: I submitted a PR there and it was merged in less than an hour. If you want the latest features now (and a responsive community), you go to the source.

The catalyst: dual GPUs and control #

With a single GPU, you usually either fit the model or you don’t. With two GPUs (and 64GB of system RAM), things get interesting. You enter the territory of heterogeneous inference.

Ollama tries to handle splitting automatically, but I found myself fighting it. I wanted to say: “Put 40 layers on GPU 0, 40 layers on GPU 1, and keep the KV cache here.” Or: “This model is slightly too big for VRAM; I want to offload exactly the last 10% to system RAM.”

With llama.cpp directly, this is just a command line flag. With llama-swap, I can define this behavior per-model in a config file:

"gpt-oss:120b":
 cmd: |
 ${pkgs.llama-cpp}/bin/llama-server
 -hf ggml-org/gpt-oss-120b-GGUF
 --port ${PORT}
 --ctx-size 65536
 --split-mode layer
 --tensor-split 3,1.3 # Tuned balance between my GPUs
 --n-cpu-moe 15 # Offload 15 experts to CPU RAM
 --threads 8
 --chat-template-kwargs '{"reasoning_effort": "high"}'

This level of granularity turned my rig from a “black box” into a tunable instrument. The --tensor-split and --n-cpu-moe flags allow me to run a 120B parameter model that technically shouldn’t fit, by carefully balancing the load across both GPUs and system RAM.

API compatibility and the “switch” #

One of the reasons I hesitated to switch was the fear of breaking my existing workflows. I use agent-cli and other custom Python scripts daily. But it turned out to be a non-issue. Both Ollama and llama-swap (wrapping llama-server) expose an OpenAI-compatible API.

Once I adopted the pattern of setting a custom OPENAI_BASE_URL in my projects, the backend became irrelevant. My tools don’t care if they are talking to api.openai.com, a local Ollama instance, or llama-swap. They just send JSON and get JSON back. This standardization meant I could swap the entire inference engine underneath my applications without changing a single line of their code.

My NixOS setup #

Since I manage my entire system with Nix, setting up llama-swap and pinning llama-cpp to the absolute bleeding edge was surprisingly elegant. I even wrote an auto-update script to ensure I’m always on the latest commit.

This is where my NixOS build cache shines. Unlike Ollama, which releases every few weeks, llama.cpp can have multiple releases a day. My cache server runs this update script automatically, builds the new binaries, and serves them to my workstation. I get the absolute latest performance improvements without ever compiling source code locally.

Compiler flags matter #

One critical lesson I learned: default builds matter. Initially, I was getting a measly 8 tokens/sec on gpt-oss:120b. It turned out the default llama.cpp package wasn’t enabling BLAS or native CPU optimizations, crippling the layers I offloaded to system RAM.

By overriding the package to enable blasSupport and passing -DGGML_NATIVE=ON (as seen below), performance jumped to 50 tokens/sec. This is another reason why I prefer managing this via Nix: I can enforce these compile-time flags declaratively.

Here’s my complete configuration (see package-overrides.nix and ai.nix):

# Override llama-cpp to latest version with CUDA support
llama-cpp =
 (pkgs.llama-cpp.override {
 cudaSupport = true;
 rocmSupport = false;
 metalSupport = false;
 # Enable BLAS for optimized CPU layer performance (OpenBLAS)
 blasSupport = true;
 }).overrideAttrs
 (oldAttrs: rec {
 version = "7205";
 src = pkgs.fetchFromGitHub {
 owner = "ggml-org";
 repo = "llama.cpp";
 tag = "b${version}";
 hash = "sha256-1CcYbc8RWAPVz8hoxKEmbAgQesC1oGFZ3fhfuU5vmOc=";
 leaveDotGit = true;
 postFetch = ''
 git -C "$out" rev-parse --short HEAD > $out/COMMIT
 find "$out" -name .git -print0 | xargs -0 rm -rf
 '';
 };
 # Enable native CPU optimizations (AVX, AVX2, etc.)
 cmakeFlags = (oldAttrs.cmakeFlags or []) ++ [
 "-DGGML_NATIVE=ON"
 ];
 # Disable Nix's march=native stripping
 preConfigure = ''
 export NIX_ENFORCE_NO_NATIVE=0
 ${oldAttrs.preConfigure or ""}
 '';
 });

# llama-swap from GitHub releases
llama-swap = pkgs.runCommand "llama-swap" { } ''
 mkdir -p $out/bin
 tar -xzf ${
 pkgs.fetchurl {
 url = "https://github.com/mostlygeek/llama-swap/releases/download/v175/llama-swap_175_linux_amd64.tar.gz";
 hash = "sha256-zeyVz0ldMxV4HKK+u5TtAozfRI6IJmeBo92IJTgkGrQ=";
 }
 } -C $out/bin
 chmod +x $out/bin/llama-swap
'';

# Configure llama-swap as a systemd service
systemd.services.llama-swap = {
 description = "llama-swap - OpenAI compatible proxy with automatic model swapping";
 after = [ "network.target" ];
 wantedBy = [ "multi-user.target" ];
 
 serviceConfig = {
 Type = "simple";
 User = "basnijholt";
 Group = "users";
 # Point to your declarative config file
 ExecStart = "${pkgs.llama-swap}/bin/llama-swap --config /etc/llama-swap/config.yaml --listen 0.0.0.0:9292 --watch-config";
 Restart = "always";
 RestartSec = 10;
 
 # Environment for CUDA support
 Environment = [
 "PATH=/run/current-system/sw/bin"
 "LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib"
 ];
 };
};

What I gained (and lost) #

Advantages of llama-swap + llama.cpp #

Heterogeneous compute: Perfectly balance loads between GPUs and CPU RAM.
Bleeding edge: I use features (like new quantization types) the day they land in llama.cpp.
Zero idle VRAM: Models unload completely when not in use.
Transparent operation: No hidden prompts or “helpful” formatting that breaks complex tasks.

The trade-off #

The main trade-off is “laziness.” You can’t just ollama pull new-model and have it appear. You have to find the GGUF on HuggingFace, decide on the quantization (Q4_K_M? Q6_K?), and add 5 lines of YAML to your config.

But honestly? That’s a feature, not a bug. It forces you to understand what you are running. It stops you from running a quantized model that is too stupid for your task just because it was the default. And for a NixOS user, putting that configuration into code instead of a hidden database is exactly how it should be.

Conclusion: maturing as an AI hobbyist #

My switch back to llama-swap wasn’t about fixing a bug; it was about outgrowing a tool. Ollama is the “starter bike” of local AI—fantastic for getting rolling. But when you add a second GPU, start caring about tokens per second, and want to maximize every gigabyte of your 48GB VRAM buffer, you need a manual transmission.

For anyone else building a hobbyist multi-GPU rig: stop fighting the abstraction. Go to the source. The control is worth it.

Check out my NixOS configuration for the complete setup!

Gemini 3 Pro: First Impressions

Wed, 19 Nov 2025 00:00:00 +0000

I’m a huge fan of agentic coding. I have the maximum subscriptions for both Claude and ChatGPT Pro, and since writing my agentic coding post, I mostly moved from Claude Code to Codex CLI with GPT-5 for my work that requires more complex reasoning. Using Claude Code often feels frustrating in comparison now.

I also frequently pit them against each other, drafting parallel plans with each tool (I’ve done this at least 50 times). The verdict is consistent. Claude Code, without exception, says that GPT‑5’s solution is superior, admitting it is more elegant and robust. Claude also misses bugs during PR reviews that GPT-5 catches immediately.

That said, Claude Code is significantly faster, which makes it my go-to for simple refactors. I also prefer its writing style for conversational answers and documentation.

But looking at the charts on Artificial Analysis, the new Gemini 3 Pro looks truly impressive. It’s at the top of nearly every benchmark. I’m excited for Gemini to potentially take over some of GPT‑5’s workload, so when Gemini 3 Pro Preview came out yesterday, I had to try it.

The good: raw power #

First off, the model is quite powerful. Last night, I built a complex new feature for my agent-cli project in just two hours: a RAG (Retrieval-Augmented Generation) proxy server.

This tool acts as a middleware that sits between any OpenAI-compatible client and server (like Ollama or vLLM). The server watches a specific folder, automatically indexes any PDF, Markdown, or text files into a local ChromaDB vector store, and injects relevant context into your prompts transparently. You can see the full implementation in PR #79.

It even handles dynamic re-indexing when I drop new files in! The speed and code quality for this task were genuinely impressive, and it handled the complexity of the RAG integration surprisingly well.

The bad: panic attacks and infinite loops #

However, the experience wasn’t without its “personality quirks.” Twice, Gemini got stuck in a loop saying “Should I? I should” over and over again. Or the following also happened:

It also has a bad habit of amending commits multiple times and force-pushing them. For me the Git history is a critical source of truth, and I don’t like rewriting it because I might just have reviewed those commits already.

The interface friction #

I also found the Gemini CLI interface frustrating. Basic tasks like selecting text or scrolling up are surprisingly difficult. I’m not sure why they chose this design or why they feel the need to deviate from standard CLI behavior. It feels like unnecessary friction in an otherwise promising tool.

Proofreading this post #

I also asked Gemini to proofread this very blog post. It made some suggestions—some I liked, others I didn’t, so I reverted the ones I didn’t want. But in every single subsequent iteration, it would forcefully re-apply its rejected changes, adding emojis I had removed or rewriting sentences I had explicitly restored.

The “main branch” incident #

Another scary moment: I asked if the PR was ready to be merged into main, and it went ahead and checked out the main branch and merged everything locally. If I hadn’t stopped it, I’m pretty sure it would have pushed the changes to the main branch directly as well. I tend to use “yolo mode” for agentic coding because I feel like Git has my back, but this feels like a dangerous path to go down.

And indeed, while writing this post, it actually happened. Gemini merged my pull request without my permission.

My prompt was > should the README reflect this? and before that I asked to review the PR.

When I asked why it merged it, it instantly force-pushed to main again!

Artifacts and existential crises #

Also, when it commits, it just adds all files, including debugging logs and build artifacts. When it realizes its mistake, it tries to fix it. But if I try to help at the same time (like removing the files myself while it’s thinking), it gets extremely confused.

It spirals into an existential crisis, claiming that I committed the file but now “it is gone.” “What is going on?” it asks, spending a lot of tokens (and time) looking at git reflogs and questioning its own existence. Even when I stop it and explicitly say that I deleted the file and to please continue, it completely ignores me and continues its panic attack. I experienced this pattern of ignoring instructions multiple times.

Conclusion #

Gemini 3 Pro shows massive potential. The raw coding ability is there, and building a clean RAG proxy in two hours is great! But the agentic behavior needs some serious work: the loops, the force pushes, ignoring instructions, and the confusion when reality doesn’t match its internal state.

This is just day one, so I’m confident many of these issues will get ironed out. I don’t love the interface, but given that gemini-cli is open source with ≈400 contributors, I expect rapid improvements, especially now that the model is finally good with agentic workflows!

Using gpt-oss:20b in OpenCode with a larger context and high reasoning

Thu, 13 Nov 2025 00:00:00 +0000

I have hundreds of private journal entries stored locally, many with incorrect date formats. The correct date is in the filename, so I wanted to use an LLM to batch update them. However, I do not want to share my private journal with any third-party service. Instead, I used Ollama with gpt-oss:20b and OpenCode to do this all locally.

The naive solution of just using gpt-oss:20b with Ollama’s defaults doesn’t work (e.g., tool calls don’t work). The main issues is that Ollama sets the context window to only 4096 tokens and medium reasoning by default. It took me too long to figure out how to get gpt-oss:20b working well in OpenCode (mostly because the OpenAI compatible API of Ollama doesn’t allow setting the context window), so I’m writing the steps down (partly for my future self).

In the end I could simply run the following command to batch update my journal entries:

for file in ./*.md; do
 opencode -m ollama/gpt-oss-20b-high-32k run \
 "Use the date in the filename \"${file}\" and correct the 'date:' entry in the YAML front matter of this file"
done

This post describes how to:

Create a long-context variant of gpt-oss:20b in Ollama.
Configure OpenCode to use that model with high reasoning enabled by default.

1. Create a long-context model in Ollama #

Start an interactive session with gpt-oss:20b:

ollama run gpt-oss:20b

In the prompt:

>>> /set parameter num_ctx 32768
>>> /save gpt-oss-20b-32k
>>> /bye

This:

Sets the context window to 32k tokens (num_ctx 32768).
Saves a new model/tag named gpt-oss-20b-32k with that parameter stored.

Adjust num_ctx as needed for your GPU (e.g. 16384, 65536, etc.).

2. Configure OpenCode to use high reasoning #

OpenCode talks to Ollama via the OpenAI-compatible API. In:

~/.config/opencode/opencode.json

add a provider configuration similar to:

{
 "$schema": "https://opencode.ai/config.json",
 "provider": {
 "ollama": {
 "npm": "@ai-sdk/openai-compatible",
 "name": "Ollama (local)",
 "options": {
 "baseURL": "http://localhost:11434/v1",
 "apiKey": "ollama"
 },
 "models": {
 "gpt-oss-20b-high-32k": {
 "id": "gpt-oss-20b-32k",
 "options": {
 "extraBody": {
 "think": "high"
 }
 }
 }
 }
 }
 },

 "model": "ollama/gpt-oss-20b-high-32k"
}

Notes:

"id": "gpt-oss-20b-32k" refers to the custom Ollama model with the larger context window.
"think": "high" enables high reasoning for gpt-oss:20b on each request.

3. Select the model in OpenCode #

Run OpenCode:

opencode

Inside OpenCode:

> /models

Then select:

ollama/gpt-oss-20b-high-32k

From this point, OpenCode uses gpt-oss:20b with:

A larger context window (configured in Ollama), and
High reasoning enabled by default (via think: "high").

Coding from my phone 📱

Mon, 20 Oct 2025 00:00:00 +0000

I am dictating this post at ten kilometers altitude on a flight home from Mexico, Blink Shell open on my iPhone, while an agentic assistant on my NixOS machine writes an initial draft for this post. Admittedly, I’m a little addicted to agentic coding: when an idea pops up, I want to jump on it right away—even if that means doing it from a cramped seat in the sky. This is the workflow I’ve arrived at after trying many alternatives, it extends my agentic coding write-up and the self-hosted AI obsession in my local AI journey. It’s my backup plan for those moments when a computer simply isn’t nearby—the rest of the time I’m still at a keyboard like any other developer. This is in no way a replacement for a proper computer/laptop workflow but gets the job done when needed. It’s a personal, mostly open-source stack built around agent-cli—my local-first voice layer. Here’s how it works: I speak into my phone, the audio goes to FasterWhisper for transcription, Ollama cleans up the text, and it lands on my clipboard ready to paste. My iOS Shortcut handles this through a small HTTP interface that agent-cli exposes. The coding agent itself still uses the best proprietary model I can access.

Table of contents

1. Why Phone Coding Works Now #

For years I used iSH (a full Alpine Linux emulator) with SSH to hop into servers from my phone, but dropped connections were a constant frustration (Mosh fixes this, more on that later). Because coding on a phone keyboard is terrible, I kept it to tiny configuration tweaks or one-off fixes of a few characters. Agentic tools changed that: with a CLI coding agent, I don’t need to type the code—I describe the change, review the patch, and run it. That made meaningful work on the phone possible for the first time, for those moments when a computer isn’t around.

I tried VS Code in the browser, bounced between iSH and Terminus for SSH, and even lived inside a handful of in-browser terminal clients. I also spent time with mobile companions like Happy and Omnara, both designed to mirror Claude Code sessions on the phone, but they still felt like extra steps between me and my actual terminal. That friction pushed me toward a phone‑ready, self‑hosted workflow that still gives me raw SSH access to my own machine when the computer is out of reach.

This post is based on the way I develop software today. Your mileage may vary, but if you also care about privacy, open tooling, and reproducible environments, I think there are useful pieces here.

2. Constraints and Trade-offs #

This setup grew organically over time. What emerged are less strict requirements and more happy accidents that turned into core benefits:

Single trust boundary: Only the model provider (OpenAI) sees code context; audio and automation stay local, and I avoid any additional third‑party relays.
Resilient sessions: Connections should survive sleep and spotty networks.
Voice-friendly: Dictation should be accurate enough that I can trust it.
Reproducible config: The entire stack must live in my dotfiles and NixOS configuration.

To give you a sense of what I tried, here is the short comparison that convinced me to roll my own:

Approach	Pros	Why I moved on
VS Code in the browser	Familiar editor UI	Needs a steady connection and still lives outside my dotfiles comfort zone
iSH / Terminus SSH	Works without extra infrastructure	Laggy, no Mosh, and awkward keybindings
In-browser terminals	Instant access from anywhere	Poor copy/paste ergonomics and flaky mobile keyboards
Happy (Claude Code companion)	Push notifications, encrypted mobile UI for Claude Code	Requires wrapping every session with a separate CLI and still abstracts away my shell, and was quite buggy for me
Omnara (agent command center)	Centralizes Claude Code/Codex sessions with terminal replay	Proxies via their servers; I already trust OpenAI for the model and don’t want another third‑party handling my code

The stack below gives me the resilience of Mosh, the ergonomics of Zellij, and full control over the AI layer.

3. Layer 1: WireGuard From the Router #

WireGuard is a modern VPN that lets me securely access my home network from anywhere. I terminate WireGuard on my router so every device in the house (and on the road) can dial home with the same config. (If you want something easier to set up without opening ports, Tailscale is a more user-friendly option.)

Server: ASUS XT8 router with WireGuard enabled via the router UI.
Client: The WireGuard iOS app with On-Demand rules so the tunnel flips on whenever I am off trusted Wi-Fi.
DNS: All mobile sessions resolve through my home DNS, so git.nijho.lt and internal services resolve instantly.

This gives Blink a local-LAN address for my desktop nixos.

4. Layer 2: Blink Shell + Mosh for Durable Sessions #

What is Mosh? It’s like SSH, but it stays connected when the network changes or the phone sleeps, and it feels faster on bad connections.

Blink Shell is my daily driver on iOS because it pairs beautifully with Mosh and has great keyboard ergonomics.

I launch sessions using mosh bas@nixos -- zellij attach --create phone (Zellij is my terminal multiplexer—more on that in the next section).
Mosh smooths over spotty LTE and keeps my session alive when the phone sleeps.

Mosh keeps the session responsive across flaky networks and sleep.

5. Layer 3: Zellij Layouts #

I use Zellij as my terminal multiplexer for mobile work.

I stick to the defaults and use a couple of predefined layouts (e.g., a “phone” layout with editor + shell panes).

Ergonomics matter even more on a glass keyboard, so I lean on a few shell helpers:

zoxide means I can jump between repos with z foo instead of pecking long paths.
Single-character aliases like p for pytest and gs for git status live in configs/shell/10_aliases.sh, keeping command entry to a minimum.
Autocompletion plugins—zsh-autosuggestions and zsh-syntax-highlighting—are wired up in configs/shell/70_zsh_plugins.sh, so I get inline hints and colour cues while typing.

I break these tricks down in more depth in Terminal Ninja, explain how I sync them with my Dotfiles, and even package binaries like zoxide with Dotbins.

6. Layer 4: `agent-cli` Server #

agent-cli runs as a small server on my NixOS machine. In this workflow I only use transcribe from iOS Shortcuts: the phone records audio, the server transcribes and cleans it up, and I paste the text. (Agent-CLI has other features like autocorrect and chat, but I don’t use them for this workflow.) On this machine I run a long‑lived agent-cli user service (systemd) so Shortcuts can POST audio to it and get cleaned text back—ready to paste into Codex CLI. The service is defined in my dotfiles—see: configs/nixos/modules/user.nix (agent-cli service).

For more background on why I built it and how I use it in practice, see Local AI Journey (§3) and Agentic Coding.

The models and services run on the same box:

FasterWhisper via faster-whisper-server for high‑accuracy streaming transcription.
Ollama for on‑device rewrite/cleanup before sending prompts to the coding agent.

My iOS Shortcut dictation workflow is slower than Apple’s built-in dictation, but the accuracy is much better—and that matters more.

For the actual coding agent, I use a fork of Codex CLI with OpenAI’s gpt-5-codex-high model (here no open-source solution matches the frontier).

7. Layer 5: The iOS Shortcut Pipeline #

My iPhone’s Action Button runs a Shortcut that records audio and sends my voice to agent-cli on my NixOS machine for transcription and cleanup.

For the full recipe, see the iOS Shortcut Guide: agent-cli/iOS_Shortcut_Guide.md.

In short: I press the Action Button, the Shortcut records a snippet, sends it to agent-cli, and copies the cleaned text to my clipboard so I can paste it into Codex CLI.

The whole loop finishes fast enough that I can capture intent by voice and paste it into Codex CLI without typing.

8. Workflow in Practice #

Here’s the typical flow:

WireGuard auto-connects when I’m off home WiFi
Open Blink—my Mosh session is still alive from last time, Zellij panes intact
Dictate the change, paste into Codex CLI, review the agent’s response
Check diffs in an adjacent Zellij pane or different Blink tab with separate Mosh session
The agent commits and pushes to a branch, then opens a PR via gh for me to review and merge

On the phone, I aim for the smallest practical edits and initial implementations. Even on personal repos, I still open a PR—sometimes prompting alone gets me to a merge‑ready result. For open‑source with real users, I finish on the computer with a careful review and any final edits.

9. Conclusion and Further Reading #

This phone setup extends my Agentic Coding workflow to mobile. I connect to the same Zellij session on nixos, so when an idea hits, I pick up exactly where I left off—same environment, same context, no setup. Everything runs on my hardware (voice processing and automation) except the coding model itself. It’s the most effective mobile workflow I’ve had so far. It’s mostly open‑source not out of dogma, but because those tools are the best options for my needs. The one exception is the coding model: there’s no true open equivalent right now, and it makes no sense to buy 20× H100s just to self‑host a frontier model even if such a high quality model would be available open-source.

If you’re curious to dive deeper, here are a few related posts:

Agentic Coding
Dotfiles: A Practical, Cross-Platform Terminal Setup
Local AI Journey: RTX 3090 Edition
Terminal Ninja
Using LLMs Effectively (already outdated at this point!)

I’d love to hear what self-hosted tricks you’re using for mobile development—reach out if you adapt this stack or build something wild on top of it.

On agentic coding 🤖

Mon, 25 Aug 2025 00:00:00 +0000

Table of contents

1. From skepticism to productivity #

About eight months ago, I wrote about my disastrous experience with “vibe coding"—where I let AI generate code without careful review. What started as a fun 3-hour prototype turned into a 15-hour debugging nightmare. I concluded that while AI was fantastic for quick prototyping, it was dangerous without human oversight. I even declared I’d “never invest in, build upon, or use such products.”

Fast forward to today, and something remarkable has changed. Not my core principle—I still never merge code I don’t understand—but the tooling and my approach have evolved through two distinct phases.

First, let’s look at the evidence of AI’s overall impact on my productivity:

This graph tells an interesting story. After maintaining a steady pace for years (2016-2023), my productivity exploded with the introduction of AI tools. In 2024-2025 alone, I’ve published 14 new Python packages and made over 400 releases. That’s more new packages than in the previous 6 years combined!

But this transformation didn’t happen overnight—it came in two distinct phases that fundamentally changed how I write code.

2. But don’t you love programming? #

I’ve heard this objection countless times: “I love programming too much to let AI do it for me.”

Sure, I get it. It’s like preferring analog photography over digital, or writing letters by hand instead of typing. Some people genuinely enjoy the manual process, and that’s valid.

But here’s the thing: I absolutely love programming. It’s my biggest passion in life. As I mentioned in my local AI journey, I bought a gaming PC to play games but have only used it for coding and local AI experiments. My idea of a fun weekend? Writing code until 3 AM. I’ve been doing this for over 10 years, and I’ve never had as much fun programming as I do now.

What I’ve discovered is that I don’t just love the act of typing code—I love building and creating things. There’s a difference.

I’m ridiculously particular about code quality. I literally cannot look at poorly formatted code without feeling physical discomfort. Every operator, every indentation, every naming convention matters to me. I enforce strict linting rules and will spend time refactoring code just because the style bothers me. This isn’t about being pretentious—it’s about caring deeply about my craft.

The revelation: AI doesn’t take away my ability to care about these details. Instead, it gives me insane leverage to create more while maintaining my personal standards. I still review every line (although I am more lenient in certain cases), still enforce my style guides, still refactor when something bothers me. But now I can build 10x faster.

It turns out that what I truly love isn’t the mechanical act of typing—it’s the creative act of bringing ideas to life. And with AI, I can bring more ideas to life than ever before.

The power to explore #

AI has unlocked something incredible: the ability to explore ideas that weren’t worth the effort before. It’s now trivial to write 2,000 lines of code to solve a simple problem. Before AI, spending 5+ hours on a utility script wasn’t justifiable. Now? I can build it in 10 minutes.

Here are just some of the “wouldn’t have been worth it” projects I’ve built recently:

Financial independence tracker: A full website visualizing my path to FIRE
Tuitorial: Terminal-based presentation software
Stream Deck home control: Complete tool to control my entire house from a Stream Deck
Dotbins: Automatic binary dependency syncing across all my machines
Agent-CLI: A local, open-source Siri that actually works

Each of these would have taken weeks of manual coding. The effort-to-reward ratio just wasn’t there. But with AI? I can explore every random idea that pops into my head at 2 AM.

The Matty story: from midnight idea to working code #

Here is the perfect example of this power: One night around midnight, I was lying in bed (slightly intoxicated) when I had an idea—I needed a way for my AI agents to communicate via Matrix. So I grabbed my iPhone, SSHed into my machine, and started building.

One hour later, at 1 AM, I had created Matty—a fully functional terminal-based Matrix client. Built entirely from my phone. In bed. (Poor sleep hygiene, I know.)

When I woke up the next morning and tested it… it actually worked! I spent a few more hours adding tests and cleaning it up, but the core functionality was built in that hour between midnight and 1 AM.

This is impossible without AI. Nobody’s writing a Matrix client from their phone in bed at midnight. But with Claude Code? It’s just another Tuesday night idea that becomes reality.

clip-files UI: when ideas strike at random moments #

What happens more often than you’d think: I get an idea at a completely random moment and NEED to implement it immediately.

Case in point: I was sitting in the bathtub during my copy-paste AI era when I realized I needed to access my codebases from anywhere. I’d already built clip-files—a CLI tool that copies entire codebases to your clipboard—but I wanted it accessible from my phone.

So I got out of the bathtub, built a web UI for it in 20 minutes, and got back in. The UI lets me:

Browse all my GitHub repositories
Select any branch
Click one button to copy all source code to clipboard
Paste it into my self-hosted AI chat from anywhere

Ironically, I rarely use this tool now—Claude Code can explore codebases directly. But back in the copy-paste era, this 20-minute bathtub interruption made mobile AI coding possible.

The barrier between “wouldn’t it be cool if…” and “here’s a working prototype” has essentially disappeared.

The 800-rule marathon: 80 AI agents at once #

Another thing that would be literally impossible without AI: I decided to enable ALL 800+ Ruff linting rules on a 20,000-line codebase.

For context, Ruff is a extremely fast Python linter that implements rules from over a dozen different tools. Most projects enable maybe 50-100 rules. I wanted all 800+.

The result? ≈2,500 violations across the entire codebase.

Fixing these by hand would take 20 hours if I could fix one per 30 seconds… But with Claude Code’s agent spawning capabilities, I did something insane:

Opened 8 terminal windows with Claude Code
Created a work orchestration document that one AI managed
Each session spawned 10 parallel sub-agents using /agents
Total: ~80 AI agents working on my code simultaneously
Two hours later: 100% compliance with all 800 rules

The orchestrator agent would assign work like:

“Session 1: Fix all E501 line-too-long violations in src/core/”
“Session 2: Handle all B008 function-call-in-default-argument issues”
“Session 3: Fix all RUF100 unused-noqa violations”

Each session’s sub-agents would tackle different files in parallel. When conflicts arose, the orchestrator would resolve them. This is the kind of code quality improvement that simply wouldn’t happen without AI—not because it’s technically difficult, but because the human effort required is unrealistic.

1,400 commits, rewritten in minutes for $2.50 #

Here’s another “impossible without AI” story: I had 1,400 commits with terrible messages (in a not-yet-released and private project).

Since I commit so frequently (using them as snapshots in feature branches), my commit messages were all over the place:

Some were just a period “.”
Some were decent descriptions
Some were just merge commits
Zero consistency in style

I wanted them all to follow a consistent convention with prefixes like fix:, feat:, test:, ci:, etc.

Obviously, rewriting 1,400 commit messages by hand is not going to happen. But with AI?

I had Claude write me an agent using the Agno framework that would:

Read each commit’s full diff using git show
Analyze the changes to understand what was done
Return structured JSON with three fields:
- commit_message: The improved message (or empty to keep original)
- keep: Boolean flag whether to keep the original
- reasoning: Why the AI made that decision

Then the magic happened: I spawned 1,400 AI agents simultaneously using the DeepSeek API.

Ten minutes and $2.50 later, all my commit messages were rewritten with consistent, meaningful descriptions. The entire Git history now looks more professional and searchable.

The barrier between “wouldn’t it be cool if…” and “here’s a working prototype” has essentially disappeared.

3. The two-phase AI revolution #

My AI coding journey wasn’t a single leap—it happened in two distinct phases, each with its own breakthrough moment.

Phase 1: copy-paste era (March 2023 - July 2025) #

When GPT-4 launched in March 2023, it changed everything. I built tools like clip-files to efficiently copy code context into a ChatGPT-like web interface. For over two years, this workflow was:

Manual but effective: Copy code → paste into AI → review suggestions → implement manually
Limited by context: Could only share what fit in a single message
No validation: AI couldn’t test its suggestions
Still valuable: Helped with ideation, code review, and refactoring

This phase was already productive—I was shipping more code than before. But I was still the one running tests, debugging errors, and validating everything.

Phase 2: the Claude Code revolution (July 2025) #

Just two months ago, in July 2025, everything changed again. Frustrated with Cursor becoming painfully slow, I tried Claude Code based on the buzz online. That first night, I spent $70 on API tokens. Not by accident—I just kept adding $10-$20 increments like a gambling addict. The experience was so mind-blowing that within two weeks, I’d spent nearly $200 and immediately upgraded to the $200/month Pro plan.

To put this in perspective: Using a command-line tool ccusage to track token consumption, I calculated that I used $10,000 worth of API tokens in my first month. Thankfully, the Pro plan capped my cost at $200!

What made Claude Code different? #

The shift from copy-paste to agentic AI was like upgrading from a bicycle to a spaceship:

Before (copy-paste with ChatGPT/Claude Web)

Generate code based on prompts
No access to your codebase
Can’t run tests or see errors
You debug everything manually

After (Claude Code)

Reads and searches your entire codebase
Executes commands and runs tests
Sees error messages and debugs itself
Iterates until tests pass
Uses tools to explore and validate

The difference is profound. I went from carefully copying snippets with clip-files to having an assistant that can explore my entire project, run my test suite, fix failures, and even commit the changes.

4. My current agentic workflow #

Here’s how I actually work with Claude Code on a typical project:

Parallel development: 6 features at once #

One of Claude Code’s superpowers is enabling truly parallel development. Here’s my setup that would make any developer from 10 years ago think I’m insane:

I use Zellij (a terminal multiplexer like tmux) with a custom layout for my projects:

5 main tabs: Each split into two panes—Claude Code on one side, terminal on the other
Using independent copies of same repository: Each tab is a separate Git worktree with its own environment, own environment variables, own deployment
Each tab = different feature: Using Git worktrees to work on 6 features simultaneously
Monitoring tabs: ccusage for tracking Claude usage, htop for CPU, nvtop for GPU
Voice-driven orchestration: I cycle through tabs, review code while speaking, move to next

My workflow looks like this:

Start feature A in tab 1, give Claude instructions
Switch to tab 2, start feature B while A is working
Continue through all 5 tabs, starting different features
Circle back to tab 1, review what Claude did, give feedback via voice
Repeat the cycle every 10-15 minutes

This parallel workflow is how I built a complex project (that I haven’t released yet) in one month that would have taken 9+ months before. The key is Git worktrees—each feature gets its own working directory, so there’s no context switching overhead.

Starting a new package #

# I describe what I want to build
"I need a Python package that manages CLI tool binaries within Git repositories,
automatically downloading the correct binary for the user's platform, here <insert path> is a repo with boilerplate skeleton to copy (use same conventions)"

# Claude Code then:
# 1. Creates the project structure
# 2. Implements core functionality
# 3. Writes comprehensive tests
# 4. Generates documentation
# 5. Sets up CI/CD workflows

The key difference: iteration and validation #

Unlike my vibe coding disaster, Claude Code doesn’t just dump code and leave. Here’s a real interaction pattern:

Claude writes initial implementation
Claude runs the tests → Several fail
Claude reads the error messages and fixes the code
Claude runs tests again → More pass, some still fail
Claude debugs the remaining issues
Claude validates all tests pass
I review the final code and understand it (tell it to make changes if needed)
Claude creates a proper Git commit

This iterative, validated approach is what makes agentic coding so powerful. It’s not about blindly accepting generated code—it’s about having an assistant that can explore, test, and refine solutions.

5. The evidence: explosive productivity #

Let me share some concrete data from my PyPI packages analysis:

Package creation by year #

2016-2022 (7 years pre-AI): ~2 packages/year average
2023 (GPT-4 launch year): 6 packages
2024 (full year with copy-paste AI): 7 packages
2025 until 3 months ago (6 months): 7 packages
since 3 months (2 with Claude Code, 1 with Codex CLI): wrote about 400k line of code (although this includes many iterations)!

The acceleration with AI is clear!

Recent packages built with agentic AI #

Here are some packages I’ve built recently with Claude Code:

matty: Terminal-based AI chat with file integration
agent-cli: Local-first AI-powered CLI agents (built in days, not weeks!)

Each of these would have taken me weeks or months to build alone. With Claude Code, I’m shipping production-ready packages in days.

6. Why this isn’t “vibe coding” (but with nuance) #

You might wonder: isn’t this just vibe coding with extra steps? The answer is nuanced and depends on the context—similar to my philosophy on dependencies.

My context-driven standards #

Just like I have different standards for dependencies in libraries versus applications, I apply different levels of scrutiny based on what I’m building:

For critical libraries (e.g., Adaptive, pipefunc, unidep)

Maximum scrutiny - These are packages others depend on:

Every single line reviewed and understood
100% test coverage with careful test review
Architecture decisions carefully considered
Documentation must be comprehensive
My reputation is on the line

For experimental CLIs and personal tools

Pragmatic approach - Isolated tools with no downstream dependencies:

Core architecture must be fully understood
Implementation details can be AI-generated if tests pass
Test generation can be more automated
Focus on functionality over perfection
Similar to my relaxed dependency stance for applications

The internal dependency graph principle #

The key insight: My scrutiny level correlates with how foundational the code is within the project:

Core/foundational code (what everything else depends on): Maximum scrutiny, every line matters
- Data models, core algorithms, API interfaces
- Authentication, database operations, state management
- These are the “roots” that everything else builds upon
Peripheral/leaf code (nothing depends on it): Can be more AI-delegated
- Plotting functions, display utilities, CLI formatters
- Test helpers, documentation generators
- These are the “leaves” that don’t affect other code
Work projects: Always maximum scrutiny for foundational code, regardless of project type

This isn’t about being lazy—it’s about focusing human attention where it matters most. I can trust AI more with a plotting function that nothing depends on than with a core data structure that the entire system uses.

Building constraints around AI #

The secret to productive agentic coding isn’t just the AI—it’s the constraints and guardrails I build around it:

Automated quality gates

Ruff with strictest rules: Catches style issues, complexity problems, and common bugs
MyPy in strict mode: Enforces type safety across the entire codebase
Pre-commit hooks: Automatically format and validate code before commits
Comprehensive test suites: AI must make tests pass, not just write code

Project-specific guidance

Every project gets a CLAUDE.md file with explicit rules:

No defensive programming: Don’t wrap things in try-except unless necessary
Functional over classes: Prefer simple functions in Python
No backward compatibility: For new projects, embrace breaking changes
Be ruthless: Aggressively remove unused code

Custom commands and workflows

I’ve built specific commands that inject context and constraints:

Anti-cruft reviews: Remove over-engineering and defensive code
Safe commit practices: Never use git add ., always selective staging
Initialize understanding: Load project context and current work state

Local virtual environments: stop the hallucinations

Here’s a critical tip that eliminated a lot of my AI frustrations: keep a virtual environment with all dependencies installed locally.

Instead of letting Claude hallucinate how libraries work, I tell it explicitly in my CLAUDE.md:

### Step 1: understand the context

- **READ THE SOURCE CODE**: This library has a `.venv` folder with all dependencies installed.
 So read the source code when in doubt.
- **Never guess API behavior**: If unsure, inspect the actual implementation in `.venv/lib/python*/site-packages/`

This simple addition transforms Claude from guessing about library APIs to actually reading them:

Before: “I think your custom AsyncProcessor.batch() method takes a list…”
After: Reads /path/to/.venv/lib/python3.11/site-packages/my_internal_lib/processor.py and knows it actually takes an iterator

When Claude can read the actual source of your own packages, internal company libraries, or niche dependencies, it stops making assumptions and starts working with facts. This is especially powerful for:

Your own libraries that Claude has never seen before
Internal company packages that aren’t public
Niche libraries with barely any GitHub stars or documentation
Modified versions of popular libraries with custom patches
Edge cases where even good documentation doesn’t cover everything

These constraints transform AI from a loose cannon into a precision tool.

7. Common pitfalls I’ve noticed #

After a few months of using agentic AI tools, I’ve noticed consistent patterns that require vigilance:

Note from the future: I now realize this is very model dependent! GPT-5 has different pitfalls than Claude Opus 4.1

The “defensive programming” trap #

# AI loves this:
try:
 result = some_function()
except Exception:
 pass # Silently suppress errors 😱

# But we need this:
result = some_function() # Let it fail loudly if something's wrong

Claude tends to wrap everything in try-except blocks, suppressing errors that should bubble up. This is why my CLAUDE.md explicitly forbids unnecessary error handling.

The “backwards compatibility” obsession #

Claude constantly adds backwards compatibility for features that were literally just introduced in the same session:

“Maintaining compatibility with the old version” (that never existed)
Fallback mechanisms for code paths that were just created
Multiple ways to do the same thing “for flexibility”

The “over-engineering” disease #

Implements factory patterns for simple object creation
Adds abstraction layers that serve no purpose
“Production ready” code for experimental scripts

The git commit sins #

Despite explicit instructions in CLAUDE.md:

Creates your_module_v2.py alongside your_module.py instead of updating
Still tries git add -A or git add . regularly despite explicit bans
Loves to “helpfully” revert debugging changes from other files
Commits .env files if not watched carefully

The “helpful” anti-patterns #

Loves defensive programming: Validates things that can’t be wrong
Gladly reads .env: Will expose secrets if not careful
Reverts unrelated changes: “Cleans up” debugging code from other features

The “mission accomplished” hallucination #

This is a dangerous pitfall. Claude Code will sometimes claim complete success when it hasn’t actually fixed anything:

Claims victory: “I’ve fixed all the issues!”
Ask for proof: “Show me the test output”
Backpedals: “Oh, let me actually run the tests…”
Still claims success: “Yes, I can see in the logs it works!”
Demand the actual log: “Show me the exact log file”
Finally admits: “Actually, there are no log files. The tests don’t pass.”

Always demand proof. Never accept “it’s done” without seeing actual test output.

The nuclear option #

I’ve had Claude Code literally try to delete all project files when “cleaning up”:

# Claude's "helpful" cleanup:
"Task complete! Let me remove the temporary files..."
rm -rf src/ # 😱

This is why frequent git commits are non-negotiable. I commit after every small success.

The debugging debris #

During problem-solving, Claude Code leaves a trail of attempts:

Debug logging statements everywhere
Failed attempt code commented out
Multiple approaches tried in parallel
Temporary test files and scripts
Extra imports and unused functions

Before merging, always ask: “Review your changes and remove all debugging artifacts and failed attempts.” Getting to a high code coverage (approaching 100%) helps ensure no dead code remains, as long as your tests use public APIs.

Why this happens #

These patterns emerge because AI is trained on public code that often:

Maintains backward compatibility for years
Uses defensive programming for public APIs
Includes extensive error handling for user input
Follows “enterprise” patterns even for simple scripts
Contains debugging code from development

This is why constraints are essential—without them, Claude defaults to these “safe” but overcomplicated patterns.

8. Critical success factors: what actually makes this work #

After two months of intense usage, here are the non-negotiable practices that make agentic coding actually productive:

Teach it your test commands (day 1 priority) #

This is absolutely crucial. Claude Code needs to know how to run your tests and how to activate the environment:

# To run tests, use: 
uv run pytest tests/ -xvs"

# For coverage:
uv run pytest --cov=src --cov-report=term-missing

# To run specific test
uv run pytest tests/test_module.py::test_function

Without this, it’s just guessing whether code works. With it, it becomes genuinely useful.

Git commits are your safety net #

I commit obsessively when using Claude Code:

After every successful (even partial) feature implementation
Before letting it attempt any major refactoring
Whenever tests pass
Before any “cleanup” operation

But here’s the key: I develop every feature in its own branch, even as a solo developer. My workflow:

Create a feature branch for each new feature or fix
Make frequent commits as snapshots (sometimes just a period for the message or tell Claude to commit)
Open a pull request to review the full diff myself
Merge only after reviewing all changes

This has saved me from disaster multiple times. Git is your undo button when Claude goes nuclear (e.g., breaks or removes your code).

Maintain healthy skepticism #

Never trust, always verify:

“I fixed it!” → “Show me the test output”
“It’s working now!” → “Run the tests again with -xvs”
“The logs show success!” → “Cat the actual log file”
“It’s production ready!” → “Did you run the tests?”

Think of Claude Code as an enthusiastic junior developer who sometimes exaggerates their accomplishments.

Force it to clean up after itself #

After any debugging session, always:

"Review all your changes and remove:
- Debug print statements
- Commented out code
- Failed attempt implementations
- Temporary test files
- Unused imports"

My custom /anti-cruft command automates this, but you can do it manually too.

Code coverage is your friend #

High test coverage (90%+) ensures:

The code Claude wrote actually runs
No dead code from failed attempts
All paths are exercised
You can refactor confidently

9. My secret weapon: voice-to-code workflow #

One of my biggest productivity multipliers isn’t Claude Code itself—it’s how I communicate with it. Using my agent-cli tool, I’ve developed a voice-first workflow that’s transformed how I write prompts.

The problem with typing prompts #

Effective agentic coding requires precise, detailed instructions. A good prompt isn’t 20 words—it’s often 200-500 words explaining exactly what you want, what to avoid, and how to approach the problem. Most people don’t want to type that much.

My voice workflow #

Here’s my actual workflow:

Start recording with agent-cli transcribe (hotkey triggered)
Review Claude’s changes while speaking my thoughts aloud
Speak for minutes about what I want, what’s wrong, what to fix
Paste the transcription directly into Claude Code

This is powered by OpenAI’s Whisper model running locally on my RTX 3090 at home. It’s more reliable than macOS dictation and gives me complete privacy.

Why this works so well #

Rich prompts: I naturally give more context when speaking
Code review narration: I can review code while explaining issues
Thinking out loud: Speaking helps clarify my own thoughts
Speed: Speaking is 3-4x faster than typing
Precision: I can be incredibly specific without typing fatigue

For example, instead of typing “fix the bug,” I might say:

“I’m looking at line 45 where you’re handling the authentication. The problem is you’re not checking if the token has expired before making the API call. Also, you’ve added a try-except block here that’s suppressing errors—remove that. And while you’re at it, the logging statement on line 52 is using the wrong format. Make sure to follow our project’s logging conventions…”

This level of detail is what makes AI coding actually productive.

10. Additional tips for agentic coding #

Beyond the critical success factors and voice workflow, here are more tips to maximize your productivity:

Set clear constraints #

Create a CLAUDE.md or similar file in your project root with your preferences:

## Project standards

- Never use try/except to suppress errors silently
- Always run pytest before marking task complete
- Use type hints for all functions
- Prefer simple solutions over clever ones

Use the task system #

Claude Code’s todo system is great for complex work:

1. ✅ Implement core functionality
2. ✅ Write comprehensive tests
3. 🔄 Fix failing tests
4. ⏳ Add documentation

Leverage the search capabilities #

Let Claude search for existing patterns:

“Find all API endpoint implementations in this project”
“Show me how we handle authentication elsewhere”
“What testing patterns are we using?”

Review in chunks #

Instead of reviewing 500 lines at once:

Have Claude implement a single feature
Review and understand it
Run tests
Commit(s) and PR
Move to the next feature

11. The tools I’ve tried #

My journey through AI coding assistants has been evolutionary:

Pre-AI era #

Manual coding with IDE autocomplete
Stack Overflow copy-paste
Productivity baseline: 1-2 packages per year

Phase 1: copy-paste tools (March 2023 - July 2025) #

ChatGPT/Claude Web + my clip-files tool
Manual context sharing
Helpful for ideation and code review
Productivity: 3x improvement (from 2 to 6 packages/year)

Cursor (early 2024) #

Great autocomplete
Limited context awareness
Became frustratingly slow
Still led to my vibe coding disaster
Productivity: 5x improvement (but with quality issues)

Phase 2: Claude Code (July 2025 - present) #

Full codebase awareness
Can execute commands and debug
Self-correcting through test iterations
Maintains context across sessions
$70 spent on first night, $10,000 worth in first month
Productivity: 24x improvement over pre-AI baseline with maintained quality

OpenAI’s Codex CLI (the reasoning beast) #

Recently tried OpenAI’s Codex—their new CLI alternative to Claude Code—and I’ll admit, for pure reasoning with their GPT-5-Codex model, it’s impressive:

Solved a race condition in an unfamiliar language that took me hours with Claude Opus
More elegant solutions for complex algorithmic problems
Better at deep reasoning when given the same context

But here’s why I still use Claude Code daily:

Claude Code’s UX is superior: Resume conversations, better interface
Codex’s CLI is fragile: Hit Ctrl-C accidentally? Lose everything, start over (muscle memory keeps betraying me!)
Claude Code handles interruptions: First Ctrl-C clears text, second quits gracefully
No session persistence: Codex forgets everything, Claude Code remembers
Context management: Have to re-provide all context after accidental exits

For a complex race condition bug, Codex with GPT-5 gave me the most elegant solution. But for day-to-day development where I need reliability, good UX, and the ability to resume work?

Why not Copilot or other “cheaper” tools? #

Let me be blunt: You get what you pay for. People often say “I tried Copilot and it wasn’t that useful” or “Gemini in chat didn’t help much.” Of course not—these tools are at least 10x cheaper than Claude Code with Opus.

Your $20/month GitHub Copilot subscription simply cannot provide the same quality or quantity of high-quality tokens as Claude Code. It’s like comparing a bicycle to a Ferrari and wondering why the bicycle isn’t as fast. The economics don’t work out:

Copilot: $20/month for limited autocomplete
Claude Code Pro: $200/month for unlimited* agentic assistance (*capped, but generous)
Actual API value used: $10,000+/month at my usage level

The 10x price difference reflects a 10x (or more) difference in capability. Copilot is autocomplete on steroids. Claude Code is a tireless pair programming partner with perfect memory who can debug, test, and iterate. The ROI is obvious when you’re shipping 10x more code.

12. Is this sustainable? #

You might wonder if this productivity is sustainable or if I’m just building lower-quality software faster. Here’s my honest assessment:

The good #

Faster prototyping: Ideas to working code in hours, not days
Better test coverage: AI never gets lazy about writing tests
More experimentation: Lower cost to try new ideas
Improved documentation: AI loves writing docs (I don’t)
Reduced burnout: Less time on boilerplate, more on interesting problems

The challenges #

Code review fatigue: You must stay vigilant
Dependency on tools: What if Claude Code disappears?
Cost: $200/month Pro plan (but worth every penny given the $10K+ value)
The temptation to “vibe”: Constant discipline required
Addiction potential: I literally couldn’t stop coding, having the desire to deplete all tokens and get my money’s worth!

My verdict #

It’s absolutely sustainable if you maintain discipline. The moment you start accepting code you don’t understand, you’re building a house of cards. But with proper review and testing, agentic AI is a massive force multiplier.

13. The experience multiplier: why seniority matters more than ever #

Here’s a counterintuitive truth about agentic AI tools: they amplify your existing skills exponentially, not linearly. This creates a fascinating paradox where these tools become increasingly valuable as you become more experienced.

The exponential trap for beginners #

For beginners, agentic tools can feel like a superpower initially. They can build a working application in hours! But here’s the danger: without the experience to recognize bad patterns, they’re essentially building at 10x speed in the wrong direction. It’s like giving a Formula 1 car to someone who just got their driver’s license—the speed multiplies both the distance traveled and the potential for catastrophic crashes.

A beginner using Claude Code might:

Accept complex solutions they don’t understand
Build massive technical debt at record speed
Miss architectural problems that will haunt them later
Create code that “works” but is impossible to maintain

They’re forced into a difficult position: either slow down to understand everything (negating the speed benefit) or accumulate technical debt at an exponential rate.

The senior developer advantage #

For experienced developers, it’s a completely different game. When I look at Claude Code’s suggestions, I can instantly recognize:

“Oh, that’s the Factory pattern—makes sense here”
“This error handling is too defensive, let’s simplify”
“That’s going to cause N+1 queries, need to refactor”
“This violates our project’s conventions, fix it”

The review process that might take a beginner hours takes me minutes. I’m not learning what the code does—I’m validating that it does what I already know it should do.

The right way for every level #

That said, everyone can benefit from agentic tools—you just need to adjust your approach:

For beginners

Use AI as a learning accelerator: Ask it to explain every decision
Build small projects first: Master fundamentals before scaling up
Prioritize understanding over speed: Better to build slowly and learn
Pair with mentors: Have experienced developers review AI-generated code

For intermediate developers

Focus on patterns: Use AI to learn architectural patterns and best practices
Experiment safely: Try new approaches in side projects first
Question everything: Don’t accept solutions without understanding the tradeoffs

For senior developers

Leverage for velocity: Use your experience to review and direct quickly
Focus on architecture: Let AI handle implementation while you design systems
Teach the AI: Create project-specific guidelines to improve output quality
Push boundaries: Explore more ambitious projects now that implementation is faster

The real multiplier effect #

The productivity boost isn’t just about writing code faster. For senior developers, agentic tools multiply:

Architecture exploration: Test multiple approaches quickly
Refactoring confidence: Refactor fearlessly with AI helping maintain functionality
Learning velocity: Explore new languages and frameworks with a knowledgeable assistant
Documentation quality: Finally have time (and help) for comprehensive docs
Testing coverage: AI never skips tests, even for “simple” functions

This is why I’ve been able to ship 14 packages in 18 months. It’s not just that I’m writing code faster—I’m able to explore ideas, validate approaches, and iterate on designs at a pace that was previously impossible.

14. Reality check: what AI can and can’t do #

Many people try AI coding and give up disappointed. That’s because they’re treating it like a magic genie that will solve the unsolved problems of the universe. It won’t.

What AI CAN do (the churn work) #

AI excels at tasks you could do yourself but would take hours:

Setting up CI/CD pipelines: Boilerplate YAML configuration
Writing tests: Especially unit tests for existing code
Fixing dependency issues: Adapting to breaking changes in libraries
CRUD operations: Basic create, read, update, delete functionality
Data transformations: Converting between formats, parsing, serialization
Documentation: README files, docstrings, API documentation
Refactoring: Renaming variables, extracting functions, reorganizing code
Bug fixes: Especially those with clear error messages

This is the “churn”—the necessary but time-consuming work that makes up 80% of development.

What AI CAN’T do (the creative work) #

Don’t expect AI to:

Solve novel algorithmic problems: It won’t invent the next PageRank
Make architectural decisions: It can’t decide if you need microservices
Understand business logic: It doesn’t know why your company does things
Create truly innovative solutions: It remixes what it’s seen, not invents
Make judgment calls: Security vs convenience, performance vs maintainability

The sweet spot #

I use AI for tasks where:

I know how to do it but it would take time
The solution exists somewhere in its training data
Success is measurable through tests or clear output
The problem is well-defined with clear constraints

This isn’t about AI doing things you can’t do—it’s about AI doing things you don’t want to spend time doing. Once you understand this, AI becomes incredibly powerful.

15. Looking forward: the future of development #

As I showed in my local AI journey, I’m deeply interested in where AI and development intersect. Agentic coding represents a fundamental shift in how we build software:

From writing code to reviewing and directing
From memorizing APIs to understanding patterns
From debugging alone to collaborative problem-solving
From slower iteration to rapid experimentation

This isn’t about replacing developers—it’s about amplifying our capabilities. I’m still the architect, the decision-maker, and the quality gatekeeper. But now I can focus on the interesting problems while my AI assistant handles the implementation details.

16. Conclusion: principles over process #

My journey from vibe coding skeptic to agentic coding advocate might seem like a complete reversal, but it’s not. My core principle remains unchanged: never commit code you don’t understand.

What’s changed is the tooling has finally caught up to the promise. With agentic AI assistants, I can maintain my standards while dramatically increasing my output. Most of those 32 packages on PyPI have users and are well-tested and documented projects that solve real problems.

The key is treating AI as what it is: an incredibly powerful assistant, not a replacement for thinking. Use it to explore ideas faster, implement solutions quicker, and test more thoroughly. But always, always understand what you’re building.

As I continue building in public and sharing my tools, I’m excited to see where this technology takes us. If you’re interested in trying agentic coding yourself, start small, maintain your standards, and prepare to be amazed by what you can build.

What’s your experience with AI coding assistants? Have you tried moving from generative to agentic tools? I’d love to hear your thoughts!

Links and resources #

Matty

Sat, 23 Aug 2025 00:00:00 +0000

MindRoom

Tue, 29 Jul 2025 00:00:00 +0000

How a 3090 turned into two local‑first AI projects 🎮

Mon, 14 Jul 2025 00:00:00 +0000

Table of contents

1. Introduction: The Unplayed Games #

I’ve wanted to dabble with local AI for a long time, mainly because of the possibilities it unlocks. Local AI allows me to use all of my data—financial records, personal messages, location history, even transcribing everything I say—without the privacy concerns of sending it to a cloud provider. This level of deep, personal data analysis is only something I’d be comfortable with on my own hardware.

However, I couldn’t justify buying a powerful GPU just for an imagined AI project that I might never really start. Then it hit me: I could also use it for gaming (duh). Because I really enjoyed The Last of Us series, I thought, “Okay, why not play The Last of Us on PC?” This was the push I needed. So, a couple of weeks ago, I convinced myself to buy someone’s old gaming machine. After many deep research sessions with ChatGPT about the best bang-for-the-buck consumer-grade GPU, I landed on an NVIDIA 3090. I found an (old) beast of a machine, and my initial idea was set.

Click here to see the full system specs ($1350 for all this!)

GPU: ASUS ROG STRIX RTX 3090 OC (24GB)
CPU: AMD Ryzen 9 3900X (12-Core)
Motherboard: ASUS ROG Crosshair VIII Hero
Power Supply: Be Quiet! 1200W Dark Power Pro
RAM: 32GB G.Skill Trident Z Royal DDR4
Storage: 2TB Intel M.2 NVMe SSD
Case: Corsair 4000D Airflow
CPU Cooler: Thermalright Peerless Assassin 120

Fast forward four weeks. I haven’t played a single minute, but my machine is now indexing 200 GB of email archives and facing a serious performance bottleneck I’ll get to later. That “gaming” machine has been humming away, but not rendering virtual worlds. It’s been training, transcribing, and thinking.

I’m writing this post simply because I am so enthusiastic about local AI. I’ve found that local models are surprisingly powerful for specific use cases. Of course, they aren’t going to replace Gemini or Claude for coding, but for many other applications, they work incredibly well. Lately, I’ve been so obsessed with this that I’m sleeping significantly less and spending all my free hours working on it.

2. The Rocky Road to a Working Setup #

Before I could even get to the fun AI stuff, I had to wrestle with the machine itself. I started with Pop!_OS, which is supposed to be a great out of the box solution that “just works”. However, it’s been a long time since I used Linux on the desktop; my experience is almost exclusively with servers. I quickly installed some of the wrong NVIDIA drivers and ended up debugging stuff in Grub. I got frustrated with the different commands I had to run to set up everything and realized that reproducing this system was going to be a massive pain. Since I’ve grown to love Nix on my Mac with Nix-Darwin, I decided to switch the whole system to NixOS. This has been an amazing decision. Even though there are many pain points with Nix, I think it is well worth it. For those interested, you can see the complete setup, including all the local AI services, in my NixOS configuration on GitHub.

3. The First Victory: `agent-cli` 🐍 #

With a stable system, I quickly got started with a few scripts. These soon evolved into my first real project of this adventure: agent-cli.

agent-cli is a collection of local-first, AI-powered command-line agents that run entirely on your machine. The core philosophy is privacy; your data never leaves your computer. It’s designed for seamless integration with system-wide hotkeys. I’m now using it constantly.

Instead of typing, I now dictate almost everything. A quick hotkey, and agent-cli transcribe turns my speech into text. Another hotkey, and agent-cli autocorrect cleans it up using a local LLM running on Ollama. It has genuinely streamlined my workflow. For example, I can pipe the tools together, like correcting text from my clipboard and then having it spoken aloud: agent-cli autocorrect | agent-cli speak.

The toolkit includes:

autocorrect: Fixes grammar and spelling.
transcribe: Uses a local Whisper model for speech-to-text.
speak: Converts text to speech with a local TTS engine.
voice-edit: A voice-powered clipboard assistant.
assistant: A hands-free voice assistant using a wake word.
chat: A conversational AI with tool-calling capabilities.

It’s been a fantastic success in my personal workflow, and it’s all open-source for you to try. Contributions and ideas welcome!

Once I trusted my toolchain, the obvious next step was to feed all my data into it.

4. The Grand Ambition: `AIBrain` 🧠 #

My second package, AIBrain, is a much larger endeavor. The vision is to create a “life-OS”—a private, local AI that processes all my personal data (emails, calendar, files, photos, messages, browser history, location data) to generate summaries and answer questions, eventually allowing for queries like:

aibrain ask "When did I get into landscape photography?"
aibrain ask "How many countries did I visit between 2015-2025?"

This project led me to explore the landscape of agentic frameworks. I started with CrewAI, but I quickly found that despite the hype and funding, it had some painful problems (specifically this issue where things just fail without any helpful error messages). I decided to switch to LangGraph, which felt more robust. I also chose to not use PydanticAI (which I used in agent-cli) because its Ollama support is just a wrapper around the OpenAI-compatible API, which doesn’t expose all the useful options (I even opened a PR in Ollama to make its OpenAI compatible API more feature-complete, but it has not been merged yet).

The core infrastructure of AIBrain is ready. It can index emails, calendar events, and files in various formats (PDF, Excel, PowerPoint, images). However, I’ve hit a significant challenge.

5. Conclusion: The Journey Continues #

This past month has been a whirlwind. I set out to play games and ended up with two open-source AI packages and a much deeper appreciation for the power sitting in a consumer-grade GPU. It’s incredible what you can achieve with local hardware today.

agent-cli is a finished, polished tool that I use daily. AIBrain is a much bigger dream, and I’m still figuring out how to solve the performance puzzle to make it truly practical. Next up: smarter chunking with LangGraph, vector-DB swap-outs, and GPU scheduling tricks.

This experience has reinforced my passion for local-first software and open-source development. The future of personal, private AI is being built right now, not just in large corporate labs, but in the homes of hobbyists tinkering with their gaming rigs.

⭐ If this resonates, star the repos & drop issues/questions.

And who knows, maybe I’ll even find the time to play ‘The Last of Us’.

Agent CLI

Wed, 25 Jun 2025 00:00:00 +0000

Vibe coding worked—until the 15‑hour cleanup

Fri, 21 Mar 2025 00:00:00 +0000

I tried “vibe coding” for 3 hours, my application worked, but then I spent 15 hours fixing it #

That’s not an exaggeration—it’s really what happened.

Recently, I tried “Vibe coding,” a term coined by Andrej Karpathy, a prominent AI researcher, co-founder of OpenAI, and former AI lead at Tesla. In his viral tweet from February 2025, Karpathy described vibe coding as a programming approach where you “fully give in to the vibes, embrace exponentials, and forget that the code even exists.” It’s essentially about letting AI handle the coding based on natural language prompts without closely reviewing the generated code.

Intrigued by the hype, I decided to experiment with vibe coding using Cursor on a new Python project I called dotbins (check out my detailed write-up here)." Dotbins manages CLI tool binaries within Git repositories, automatically downloading the correct binary from GitHub for the user’s platform and architecture, significantly simplifying my dotfiles and bootstrapping process. I am super excited about this project but it is not what this post is about.

Initially, it was fun to get something working so quickly. Within just three hours, I had a functioning prototype—without carefully examining the code. But I had unknowingly created a monster—a true spaghetti monster! 🍝

Once I started reviewing the AI-generated code, I realized it was totally incomprehensible. The AI had implemented the most complicated and roundabout ways to achieve simple tasks. It introduced redundant checks for impossible conditions, irrelevant error handling, and execution paths that would never trigger. The code worked superficially, but beneath the surface, it was chaotic and nearly impossible to understand or maintain.

This realization turned three hours of fun into fifteen hours of meticulous debugging, refactoring, and cleaning up the mess. Essentially, I had to rewrite the entire project.

My main takeaway: AI-assisted coding, or “vibe coding,” is incredibly powerful for quick prototyping but dangerous when used without oversight. According to this recent Y Combinator discussion, many startup founders are building products almost entirely with AI-generated code, “vibing” the code, meaning they let the AI write the code and don’t review it. For writing a couple of hundred lines of code it likely works pretty well, but for anything more complex, it’s a recipe for disaster. Personally, I would never invest in, build upon, or use such products—they’re essentially constructing technical prisons for themselves.

AI is fantastic for inspiration, quick prototyping, writing code, and refactoring—but letting it run wild without careful human review can quickly lead to severe technical debt and hidden bugs.

Thanks for reading!

My LLM workflow: local frontends, (mostly) remote models 💻

Fri, 07 Mar 2025 00:00:00 +0000

How I Use LLMs in My Daily Workflow

Over the past 1.5 years, I’ve developed a workflow for integrating Large Language Models into my daily life, particularly for open source software development. I wanted to share my setup and recent discoveries that have improved my experience. Setting it up takes a little technical know-how, but once it’s up and running, using it is easy, my wife even uses it!

My Golden Rule for AI-Generated Code #

While AI can generate impressive amounts of code quickly, I follow one critical principle in my workflow: I never commit code I don’t fully understand. With the ease of generating solutions comes the responsibility to ensure quality and correctness.

My approach involves:

Understanding every line: Before committing any AI-generated code, I carefully review and understand each line. This helps avoid introducing subtle bugs that might be hidden in seemingly working code.
100% test coverage: I ensure comprehensive test coverage for AI-generated implementations. This serves two purposes: it verifies the code works as expected and creates concrete examples that document the intended behavior.
Critical evaluation: I assess whether the proposed solution aligns with project architecture and best practices. Often, I’ll ask the AI to refactor or explain its approach before accepting it.
Incremental adoption: Rather than accepting large code blocks wholesale, I typically integrate smaller, well-understood portions that can be thoroughly tested.

This mindset transforms AI from a potential shortcut that could introduce technical debt into a powerful assistant that accelerates development while maintaining quality standards.

Starting with OpenWebUI #

For most of the last year, I’ve relied on OpenWebUI as my primary interface for accessing various commercial LLMs:

OpenAI’s models
Anthropic’s Claude models
Google’s Gemini models

OpenWebUI is primarily designed for running local models, but it can work with remote APIs through what it calls “functions” - essentially Python modules that establish connections to these external APIs. While functional, this approach requires additional setup and configuration to get working properly.

Discovering LibreChat: A Major Upgrade #

Recently, I discovered LibreChat, which has dramatically improved my LLM experience:

Speed: Pages load in under a second, compared to 5-10 seconds with OpenWebUI. This performance difference is especially noticeable when accessing either interface from my iPhone.
API Support: Native integration with virtually all major AI company APIs you can think of, right out of the box
Convenience: No need to implement custom Python functions - all popular models are supported by default
Reasoning Support: LibreChat works much better with the new reasoning models, properly placing the reasoning part of replies in a dropdown. The amount of reasoning tokens is also easily parameterized.

The Rise of Free & Nearly-Free Powerful Models #

The landscape has changed dramatically in just the last month. Several extremely capable models are now either completely free or essentially free:

Google’s gemini-exp-1206: Impressively capable with a huge context window for code
DeepSeek: DeepSeek v3 is one of the best in this category, offering remarkable performance at minimal cost
Qwen QwQ 32B: Another powerful option that won’t break the bank

This democratization of access to high-quality models has substantially improved my development workflow.

Claude 3.7: Worth Every Penny #

My current go-to model is Anthropic’s Claude 3.7. I’m completely blown away by its capabilities and find myself happy to pay the premium price for what it delivers. The quality of responses, understanding of complex contexts, and overall reliability have made it an indispensable tool in my workflow.

OpenRouter: Hundreds of Models, One API #

A particularly valuable discovery has been OpenRouter.ai. With just one API key, I can access hundreds of different models, many of which are completely free to use. This has dramatically expanded my toolkit without increasing costs.

Easy Deployment with Docker Compose #

Both OpenWebUI and LibreChat are extremely easy to deploy using Docker Compose. As someone who self-hosts numerous services, I appreciate how straightforward it is to get either of these interfaces up and running with a simple docker-compose file. This accessibility makes it easy for even those with limited server experience to set up their own LLM workstation.

Why I Don’t Run Local Models #

Despite being a self-hosting enthusiast (I run over 50 services at home!), I’ve stuck with commercial API models for my work. Since I mainly develop open source software, data privacy concerns are minimal—the code I work with is already publicly available. The commercial models also provide superior performance for my specific needs.

Avoiding Web Interfaces & Subscriptions #

I deliberately avoid using ChatGPT or Claude through their native web interfaces. Why? Simple economics. By accessing these models through my own interfaces with API keys, I don’t have to pay separate subscription fees (≈$20) for each service. This consolidated approach saves money while providing more flexibility.

Code Editor Experiments #

I’ve tried Cursor editor, which offers an interesting AI-integrated coding experience. While many features are impressive, I encountered limitations—particularly with the Python debugger, which is a known issue. Despite its promise, these limitations pushed me back to my custom workflow.

My DIY Solution: clip-files #

To compensate, I heavily rely on a CLI tool I created called “clip-files.” This simple utility lets me pass filenames as arguments, and it copies their contents to my clipboard along with useful metadata. This makes it effortless to share code with LLMs regardless of which interface I’m using—simple, fast, and effective. Together with the zsh-autosuggestions plugin–which I talked about in another post–generating the right prompt is a breeze.

My Current Setup #

Today, I maintain access to multiple top-tier models through LibreChat’s interface and OpenRouter.ai’s API. This gives me flexibility to choose the right model for specific tasks while enjoying a responsive, feature-rich environment without unnecessary subscription costs.

For fellow self-hosters and developers looking to optimize their LLM workflow, I highly recommend this combination, especially if you’re currently using web interfaces or struggling with OpenWebUI’s performance.

What LLM setups are you using in your workflow? I’d love to hear about your experiences in the comments!

clip-files 📋

Sun, 15 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the most useful tools are born from daily frustrations. When working with AI assistants like ChatGPT or Claude, I found myself constantly copying files to share context. After doing this manually dozens of times, I created a tool to automate it.

📖 Origin Story #

While contributing to open source, I frequently like to share entire (or large parts of) codebases with AI assistants for context or code review. Since the code is open source anyway, there’s no reason not to share the full context! But manually copying files, adding their paths for context, and ensuring I didn’t exceed token limits was tedious. Instead of continuing this manual process, I spent an evening creating clip-files - and have used it multiple times daily ever since.

In fact, I’m using it right now! The command (shown below) lets me share this entire advent calendar series with AI assistants for proofreading.

clip-files --initial-file ai.md --files introduction.md day*.md

🔧 Technical Highlights #

Uses tiktoken for accurate GPT token counting
Supports both directory traversal and specific file selection
Automatically generates a file index for reference
Allows custom initial instructions via file
Preserves file paths in output for context
Clean, typed, and documented codebase

📊 Impact #

While this tool has no GitHub stars (I haven’t promoted it), its real impact is personal - I use it multiple times daily for:

Sharing entire open source codebases with AI assistants
Getting comprehensive code reviews
Discussing bug fixes with full context
Documentation improvements
Open source contributions

The tool’s real value isn’t in its complexity (it’s just ~250 lines of code) but in how it removes friction from open source development. When working with open code, we might as well share all the context we can - and this tool makes that effortless.

🎯 Challenges and Solutions #

Maintaining consistent formatting across different input methods
Calculating accurate token counts for different components
Providing meaningful error messages for invalid inputs
Structuring output to be AI-assistant friendly

💡 Lessons Learned #

When working with open source code, sharing full context is valuable
Simple command-line tools can significantly improve daily workflows
Consistent formatting matters when working with AI assistants
Sometimes a 250-line tool can save hours of manual work

Want to make sharing code with AI assistants easier? Check out clip-files on GitHub!

#OpenSource #Python #DevTools #AI #Programming

clip-files

Thu, 13 Jun 2024 00:00:00 +0000

AI | Bas Nijholt

Roll your own, fork the rest 🍴

Introduction: a follow-up to something I wrote #

1. Stars measure interest, not usage #

2. A popular protocol is not a popular library #

3. I checked the rule against my own code #

4. Roll your own: the library gives you the easy 10% #

5. Fork your own: when the problem is hard #

Agno: patches instead of a fork #

6. I am not dunking on open source, including mine #

7. My rule for the next uv add #

Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

What the kernel actually does #

A container is a process in a costume #

Why kernel bugs are a bigger problem now #

So I asked a model to break out #

Stronger walls: virtual machines #

Kata Containers and gVisor #

The PR: six lines #

What it costs #

What this does not fix #

Conclusion #

Giving models the tools they were trained with can make them worse

What a harness is #

Why I believed it #

Smaller agents do better #

Models should know their own tools best #

What I built #

How I measured it #

Native tools don’t make today’s models more efficient #

Older models: only Opus 4.8 did better with them #

What does matter: what the agent sends #

Fewer tools #

Minimal mode: one shell tool and a 12-line prompt #

Minimal mode with each model’s native shell tool #

What this does and does not show #

What happens to the pull request #

Why this excites me #

Appendix: how I measured, and the numbers behind the charts #

How to start with AI

1. Install it, and that’s it #

2. Ask it everything 🥩 #

3. Use the best model #

4. Decide how much to trust it #

5. Talk to it, then make it review itself #

6. What about skills and multi-agent loops? #

7. If you write code #

Tests are cheap now, so ask for them #

The code is the spec #

Review against your own criteria #

Cross-model review needs two installs #

A good AI engineer is just a good engineer #

Your role changed, it didn’t disappear #

This post has an expiry date #

Self-hosting AI does not save money, and I do it anyway

1. The model I love, and the one that beats it on price #

2. “But my GPUs are already paid for” #

3. Why datacenters are so much better at this #

4. “But I have solar panels” #

5. “But my Mac, DGX Spark, or AI Max does save money” #

6. The bar keeps moving #

7. A small company has it worse #

8. “They lose money on inference, so prices will go up” #

9. “But they sell your data” #

10. Where self-hosting does win: the same model #

11. So why do I do it? #

Appendix: all assumptions #

Slowly, then all at once: AI agents that know how a company works

Why I built MindRoom #

Slowly #

Then all at once #

The surprise #

Institutional knowledge #

Why build it ourselves #

What it takes to run this at a company #

What’s next #

Exploring the Jev hype in MindRoom

What a System One model is #

Adaptive participation #

“Thanks” should not stop the agent #

7. My rule for the next `uv add` #