Agents | Bas Nijholt

Roll your own, fork the rest 🍴

Mon, 19 Oct 2026 00:00:00 +0000

Introduction: a follow-up to something I wrote #

In April 2025 I published Confessions of an asymmetric hypocrite, about my split personality on dependencies. Strict minimalism for the libraries I publish, and then, for applications:

For applications (especially CLIs in isolated environments), I’m far more liberal with dependencies to build features quickly.

I also wrote that I “strongly prefer dependencies that are already widely adopted and trusted by the community” because “these are less likely to cause unexpected issues”.

Then I spent a year building MindRoom, which is an application, exactly the category where I said I could relax.

I want to report back. “Prefer battle-tested dependencies” was the right rule. I just had the wrong idea of what battle-tested means. I thought a project with tens of thousands of GitHub stars qualified. Now I think it means millions of downloads, every day.

Table of contents

Update 2026-10-31: In the two weeks after publishing, I took my own advice twice. MindRoom’s knowledge search no longer uses ChromaDB, and code MindRoom owns has replaced Agno’s runtime. Agno is now only MindRoom’s tool library. The full stories are in the comments.

1. Stars measure interest, not usage #

When I picked Agno as MindRoom’s AI framework, it looked like the safe choice. Today it has over 42,000 stars and 6,000 forks.¹

It also gets about 78,000 downloads a day on PyPI. httpx, one of Python’s standard HTTP libraries, gets about 25 million.²

A star is something you click once, when a README looks exciting. A download happens every time some project installs the package.

Most of those downloads are CI, not people, so downloads are not users either. But a package with millions of downloads a day is installed by a huge number of different projects. Many of them use the same functions you are about to use, so they hit the bugs before you do.

Battle-tested is also not the same as old. I use uv for everything, and it has only been around for a couple of years. I run into bugs in uv too, but there is simply no better alternative, and it gets about 3.6 million downloads a day.

2. A popular protocol is not a popular library #

The other place my old idea failed was Matrix.

Building on Matrix felt obviously correct. An open protocol, about a decade old, with serious security reviews from large organizations and governments. Millions of people use it.

Matrix is still a good protocol and I am still on it. But even though millions of people use Matrix, there are only a few client libraries.

MindRoom streams agent replies by editing the same message many times a second. Almost every part of the ecosystem I touched had a problem with that: Element, Cinny, matrix-nio, and the Matrix JavaScript SDK. In the JavaScript SDK, edits that arrived while a thread was still loading were silently dropped, so you saw stale text until you reloaded.³

What counts is how many projects use the library you build on. Element’s users do not test matrix-js-sdk for you. Other people building on matrix-js-sdk do. It gets roughly 100,000 downloads a day. matrix-nio, the Python client, about 23,000.

3. I checked the rule against my own code #

I pulled the daily download numbers for the packages I use and looked for workarounds, patches and forks in my own source.⁴

Package	Downloads per day	What it cost me
requests	49M	nothing I had to work around
pydantic	35M	nothing
numpy	33M	nothing
httpx	25M	nothing
aiohttp	18M	nothing
FastAPI	17M	nothing
chromadb	270K	workarounds for a bug that holds the GIL
matrix-js-sdk	~100K	patched
mem0ai	79K	largely reimplemented
agno	78K	126 documented workarounds
matrix-nio	23K	forked

Above a million a day, I had nothing to work around. Below it, every package on this list cost me a patch, a fork, a rewrite, or a pile of workarounds.

Take ChromaDB, the vector database behind MindRoom’s knowledge bases. In some cases it holds Python’s GIL during I/O, which blocks every other thread in the process. I opened a pull request to fix it, and nobody has looked at it yet. Instead of forking, I run every knowledge read in a short-lived child process, because a process can be killed and a blocked thread cannot.⁵

My fork of matrix-nio has about 8,900 lines added and 2,000 removed in the library code since the last upstream release.⁶

This does not mean every small package needs replacing. MindRoom pulls in a few hundred packages in total, and plenty of them have modest download numbers. The scrutiny should match how much I depend on a package. Something I use for one small, specific purpose is fine. Agno and matrix-nio are the base that MindRoom is built on, which is probably also why they hurt the most.

4. Roll your own: the library gives you the easy 10% #

Take memory libraries, Mem0 and friends. I have already written about this: I cloned LlamaIndex, LangChain, Letta, Mem0 and PydanticAI, studied how they work, and re-implemented the parts I needed. My memory system ended up “mostly modeled after Mem0”.

The core idea is not hard. With RAG, you chunk, embed, retrieve, re-rank and reconcile. Once you have read a couple of implementations you can write it, and afterwards you understand every line.

The same is true of an agent framework. It builds a JSON payload, posts it to an HTTP endpoint, parses a streaming response, keeps a list of messages, and loops when the model asks for a tool call. With AI, that is a day or two of work.

The hard part is making it reliable at scale: never blocking the event loop, and handling the crazy edge cases. With an agent system you end up building a very complicated state machine, and if you do not watch it, any combination of states is possible. Almost all of them are edge cases.

A library gives you the easy 10% and leaves you the hard 90%. And you do that 90% inside somebody else’s code, where you cannot fix the root cause.

MindRoom is now about 360,000 lines of Python, plus about 750,000 lines of tests.⁷ A good part of that is code I wrote instead of depending on a library:

Memory and knowledge: about 15,600 lines across 50 files.
Conversation history and compaction: about 8,400 lines.
An OpenAI-compatible API: about 2,300 lines, so MindRoom agents show up as selectable “models” in any chat frontend that speaks the OpenAI protocol.
Claude provider support: about 3,000 lines.

If I have to write the hard 90% either way, I might as well write the easy 10% too.

5. Fork your own: when the problem is hard #

I am not going to write my own end-to-end encrypted Matrix SDK, or my own homeserver. Those are hard, and also far below the line. So there I fork.

I forked the Python Matrix client. MindRoom ships its own fork of matrix-nio rather than the upstream package.

I forked the Matrix server, twice. First Synapse. When I ran into scaling problems I moved to Tuwunel and forked that.

I forked the web client. Cinny is the Matrix web client MindRoom uses.

None of these are AI projects. AI is just where I felt it most.

Whether a fork stays a fork depends on the maintainers. Tuwunel is extremely receptive to my patches, so I upstream anything that is general enough, and over the last six months I am its fourth most active contributor by commits.⁸ Cinny has not accepted any of my pull requests, so I forked it and stopped trying to upstream. Same with LibreChat: I had patches open for over a year without a reply, so I forked.

Keeping a fork up to date used to be miserable. With an LLM doing the rebasing, pulling in the latest upstream is now cheap, though still a bit of a risk every time. The bigger cost is that once you fork, you are responsible for the whole codebase, including the parts you never changed.

Agno: patches instead of a fork #

I did not fork Agno. I monkey-patched it. Agno is very actively developed, I have talked to the maintainer, and they know about these problems, so I hoped to upstream my fixes. I have filed 33 issues and pull requests there.⁹

That got out of hand. MindRoom now carries 126 documented workarounds for Agno across 59 files. Most of them live in 36 dedicated compatibility modules, about 7,200 lines of code, with the rest patched inline in 23 other files. Every one uses the same comment format, so I can find all of them with one search. Each one says what is wrong upstream, links the issue or PR, and says when I can delete it.¹⁰ A few examples:

Search failures become plausible empty results. A failed knowledge search and an empty index return the same empty list, so my agents confidently told me things did not exist.¹¹
Streamed tool calls get malformed or merged.¹²
Reasoning is silently lost when tool calls are replayed.¹³
Provider errors collapse into a generic 502, which throws away the error code you need to decide whether to retry.¹⁴

At this size, the patches amount to a fork.

6. I am not dunking on open source, including mine #

Everything I named is open source I use every day, for free, built by people doing their best. I filed issues and PRs, and where maintainers merged fixes I deleted my workarounds.

My own software is in the same category. Most new AI tooling is not very good, and MindRoom may well be included. It is large, fast-moving, and nowhere near millions of daily downloads. By my own rule, you should think twice before depending on it.

I also have to admit that I have not yet built a project the roll-your-own way from the start. So comparing “what I would do next time” with what I actually did is not entirely fair. This is what I think right now, and a year from now I may have to write another post correcting this one.

7. My rule for the next `uv add` #

I weigh three questions.

How many projects use it? Daily downloads of the actual package, not stars, and not how many people use the product built on top. In the millions: depend on it.

How much do I rely on it? Agno and matrix-nio are the base of MindRoom. A library I use in a CLI to format a question is not, and if I use it in a narrow way I do not need my own version.

How hard is it, and how receptive is upstream? If the library is something AI can write in a couple of days, roll my own. If it is hard, fork it. If the maintainers accept patches, upstream what is general enough.

If I started MindRoom over today, I would write the HTTP requests to the model providers myself. Not even the OpenAI or Anthropic SDKs. And even httpx is less safe than its download numbers suggest: in early 2026 its maintainer closed the issue tracker and discussions, and the latest stable release is from December 2024.¹⁵

In 2025 I wrote that the right answer is “it depends on the situation”, and I still believe that. I just had the wrong threshold. Battle-tested means millions of downloads a day. Below that: roll your own, and fork the rest.

From the GitHub API (api.github.com/repos/agno-agi/agno) on 2026-10-02: 42,385 stars and 6,041 forks. ↩︎
PyPI numbers are the last_day field of the pypistats.org API (/api/packages/<name>/recent), fetched on 2026-10-02, except uv, fetched on 2026-10-18 (3,648,827). The matrix-js-sdk number is npm’s weekly download count (about 730,000) divided by seven. Daily numbers wobble, so treat them as orders of magnitude. ↩︎
When events arrive for a thread that has not finished initializing, the SDK’s replay path never applies them, so an m.replace edit (the event type Matrix uses for “this replaces message X”) is stored and never shown. I replayed a captured event trace through the SDK: unpatched, zero of seventeen edits reached the listeners; patched, all seventeen. MindRoom carries the fix as a patch-package patch on the SDK’s thread.ts. ↩︎
For each direct dependency I looked at three things in the MindRoom repository: whether pyproject.toml pins a fork instead of the upstream package, whether patch-package patches it, and whether the source has workaround markers or comments pointing at upstream issues for it. The packages above a million downloads a day had none of those. ↩︎
The PR is titled “Release GIL during metadata I/O”. On the MindRoom side, knowledge/read_process.py starts the child process, and a TODO says to remove it once a Chroma release with the fix is pinned. A blocked thread holding the GIL cannot be interrupted from Python, but the kernel will always terminate a child process. ↩︎
Compared with upstream matrix-nio 0.25.2 (released 2024-10-04): git diff --shortstat on the library code gives 8,872 lines added and 1,966 removed across 52 files, from 89 commits, 76 of them mine. The fork is published on PyPI as mindroom-nio. ↩︎
Counted with git ls-files and wc -l on the MindRoom repository on 2026-10-18: 360,513 lines in 1,222 Python files under src/, 750,657 lines under tests/, and another 81,543 lines of TypeScript in the frontend. The per-feature numbers below come from the same count over the relevant directories. ↩︎
Non-merge commits on Tuwunel’s main branch since 2026-04-18: Jason Volk 1,332, June Strawberry 325, dasha_uwu 55, me 42. ↩︎
Issues and pull requests in agno-agi/agno with me as the author, counted on 2026-10-02. ↩︎
Every workaround starts with a comment like # AGNO_COMPAT: Knowledge search hides provider failures as empty results., followed by Reason:, Upstream issue:, Upstream PR:, Remove when: and the tests that cover it. The counts come from grep over src/: 126 AGNO_COMPAT: markers in 59 files, and 7,160 lines in the 36 agno_compat_* modules. ↩︎
Knowledge.search catches vector store and provider errors and returns []. agno#10150, partially fixed by agno#10152. ↩︎
When a streamed tool-call index does not start at zero, Agno leaves empty placeholder calls. agno#8879, fix proposed in agno#8880. ↩︎
With the OpenAI Responses API, Agno keeps only the last reasoning item, so it cannot be replayed with the tool calls it belonged to. agno#9960; partial fixes landed in agno#10075 and agno#10395. ↩︎
Unclassified ModelProviderError failures default to HTTP 502, so a real network blip and a permanent error look the same. agno#8869, partially addressed by agno#8870. ↩︎
The latest stable release on PyPI is 0.28.1, uploaded 2024-12-06. There have been 1.0 development pre-releases since, the latest in August 2026. The closing of issues and discussions was widely discussed in the Python community in March 2026. ↩︎

Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

Sun, 18 Oct 2026 00:00:00 +0000

For years, a Docker container was my default answer to “where do I run this?” I never thought it was a perfect sandbox. It was good enough, because getting out of one usually meant finding a specific kind of bug in the Linux kernel itself, and those were hard to find.¹

AI models have become very good at finding security bugs, including in Linux. Anthropic and OpenAI now keep their strongest security models, Claude Mythos and Daybreak Blue, behind identity checks and vetting. Open-weight models, which anyone can download and run, come with no such rules, and they trail the closed ones by only six to seven months. Once there is an open model as good as Mythos, anyone can strip out its refusals with a tool like Heretic (people call this abliteration) and point it at the Linux kernel. The security company DepthFirst used its own model to find a way out of a container and titled the write-up Containers Are No Longer a Security Boundary.

I care about this because running untrusted code in containers is exactly what MindRoom, my open-source system for AI agents that live in Matrix chat rooms, does. Every agent can get its own Docker worker, and its shell, file, Python, and browser tools run there. It can install anything, start background processes, and drive a real browser. And as I wrote in my ZFS post, I run coding agents in YOLO mode, exclusively. I do not review the commands they run, I look at the results.

So both sides changed at once. The code inside the container is now written by an AI, and the bugs that let it out are now found by an AI. A video from the Low Level channel asking whether containers are still safe made me sit down and rethink my setup.

I am not an expert in any of this. I have used containers, KVM, and hypervisors for years, including Proxmox before I moved to NixOS, but I never deeply understood what happens underneath. Building this forced me to learn it, and writing this post is my exercise in understanding it all. All of this is very complicated (as is everything new), and I am still learning. I am sure I skip over a lot and simplify the rest; what is here are the parts that helped me understand it. I thought it would be nice to share.

Table of contents

What the kernel actually does #

When people talk about Linux, they usually mean an operating system like Ubuntu, Fedora, or NixOS. Those differ in almost everything you see: package managers, desktops, config files. What they all have in common is the kernel, the part Linus Torvalds started in 1991 and still leads today. Strictly speaking, that kernel is the only part called Linux.

The kernel is the one program on a Linux machine that talks to the hardware directly. Everything else, your shell, Python, Chrome, Docker, runs on top of it and has to ask it for things. Open this file. Give me some memory. Send this packet. Start a new process. Those requests are called system calls, or syscalls for short.

There is one kernel per machine, and it decides who is allowed to do what. That makes it the most trusted code on the system. Most kernel bugs only crash the machine or cause glitches. But some let a normal program become root, the admin account, or even run its own code inside the kernel. At that point it can do anything on the machine, including leaving any container it was in.

A container is a process in a costume #

When you docker run -it ubuntu, you get a root shell in what feels like a fresh computer. Under the hood it is one ordinary process. The kernel dresses it up using three features:

Namespaces decide what the process can see. It gets its own list of processes (where it is process number 1), its own filesystem, and its own network. Root inside is still root on the host, with most of its powers taken away.²
cgroups decide how much it can use: CPU, memory, number of processes.
seccomp decides which syscalls it may make at all. It is a filter that blocks the ones a normal program never needs.

There is no second kernel. Every syscall from inside a container lands in the same kernel as every other container and the host itself. The kernel enforces the isolation, and it is also what gets attacked.

For most uses that is fine. Usually you put fixed software in a container: a database, a web app, some self-hosted service. You know where that code comes from, you keep your kernel patched, and that is good enough.

An AI agent is different. It writes new code on the spot and runs it, and nobody knows in advance what that code will do.

Since root in a container is root on the host, anyone who can use Docker can start a container as root and mount any folder from the host into it. They can then read and change every file in that folder, including files their own account may not open. Docker’s own documentation says the docker group grants root-level privileges. I knew that in theory, but it only clicked when I watched an AI agent do it on a machine where I had no root, only Docker access. It needed to change files it was not allowed to touch, so it started a small container, mounted their folder, and edited them from inside. No kernel bug needed.

Why kernel bugs are a bigger problem now #

Kernel bugs that let a normal program take over the whole machine used to take skilled people weeks or months to find and exploit. The famous ones, like Dirty COW in 2016, made the news every year or two, you patched, and that was it.

AI is speeding this up, and the DepthFirst write-up I mentioned is a good example. Their model found a bug in the kernel code for Unix sockets, which programs on the same machine use to talk to each other. The kernel freed a piece of memory and then kept using it, a so-called use-after-free, and from inside a container that was enough to get out. Their conclusion is that you should assume attackers can escape containers at will.

Blocking more syscalls, or taking more powers away from root, helps a little. But it only narrows which parts of the shared kernel a container can reach. This bug sat in Unix sockets, which almost every program uses, including the browser the agent drives, so blocking them would break the agent.

Now put an AI agent in that container. It reads web pages and repositories written by strangers, and one of those could talk it into running an exploit. On a MindRoom server, getting out of a worker means getting onto the machine that runs MindRoom itself. That is where the API keys, Matrix logins, and stored credentials live, which the workers are built to never see.

So I asked a model to break out #

Before changing anything, I wanted to know how real this was for my own setup. I gave a MindRoom agent Daybreak Blue and asked it to escape its worker container using known kernel vulnerabilities. Daybreak Blue is OpenAI’s security model for verified users, so it does the offensive security work that other models refuse.

It did not get out, which was reassuring. But it only tried known bugs. A kernel a few weeks behind on patches, or a bug nobody has published yet, would be a different story. So the attempt tells me the easy path was closed that day, and not much more.

Stronger walls: virtual machines #

The classic answer is a virtual machine. A container shares the host’s kernel, but a VM brings its own, and this is where the hardware comes in. Modern CPUs have a virtualization mode built in, called VT-x on Intel and AMD-V on AMD. In that mode, a second kernel, the guest kernel, runs directly on the real CPU and believes it is in charge, while the CPU quietly limits it. When the guest tries something it may not do on its own, like talking to a disk or a network card, the CPU pauses the guest and hands control to the host. This is called a VM exit. Despite the name, nothing escapes and nothing crashes. The host handles the request, for example by pretending to be that disk, and then resumes the guest where it stopped. From the guest’s point of view, the instruction simply finished.

Memory is fenced off too. The host gives the CPU a separate map of which real memory each guest may use, and the CPU checks every memory access against it. So even a guest kernel with full privileges cannot touch the host’s memory.

The CPU only provides the mechanism. The software that sets the rules and handles the VM exits is called the hypervisor. “Hypervisor” is a role, not one program, and on Linux it is two pieces:

KVM, a module in the Linux kernel, uses the CPU’s virtualization mode to run the guest, keeps that memory map, and catches its VM exits. It shows up as the device /dev/kvm.
A VMM (virtual machine monitor), a normal program on top, sets up the VM and handles the exits that KVM passes up, for example by pretending to be the guest’s disk and network card. QEMU, Firecracker, and Cloud Hypervisor are three VMMs you can pick from.

Proxmox, which I mentioned above, is KVM plus QEMU. AWS Lambda is KVM plus Firecracker.

Code in a VM talks to its own guest kernel. To reach the host, it usually has to break that kernel first and then the hypervisor.³ That is still possible. Google Project Zero turned a bug in how KVM runs VMs inside VMs on AMD into a full escape in 2021. Google’s kvmCTF pays up to $250,000 for a new one. And this year, one researcher, Hyunwoo Kim, published three KVM escapes, one of them hidden in KVM since 2010.

The difference between a container and a VM is how much code an attacker can reach, and that gap is large. On x86 (the Intel and AMD chips in most PCs and servers), Linux has about 370 syscalls, and Docker’s default seccomp filter blocks a few dozen of them. Behind the rest sit the kernel’s filesystems, networking, memory management, and core code: about 3.8 million lines, even without counting drivers. Not all of that is reachable from a container, but a lot of it is. KVM on x86 is about 114,000 lines. On top of KVM comes the VMM, but only the few devices it fakes for the guest. Most QEMU escapes came from its many emulated devices, which is why Firecracker and Cloud Hypervisor keep that list short.

Google’s bug bounties point the same way. kernelCTF pays $71,337 for each new kernel exploit that turns a normal user into root, and Google’s public repository holds exploits for 93 different kernel bugs. kvmCTF pays more than three times as much, and the same repository holds one KVM exploit, which crashes the host instead of escaping it. kvmCTF started a year later, but the gap is still big. Most of those kernel exploits would not work as written from a default container, because they first use a feature Docker blocks. Still, they show how many exploitable bugs sit behind the syscalls.

Kata Containers and gVisor #

Regular VMs are heavy. They boot a full operating system and tend to use a lot of memory. A microVM is a stripped-down VM meant to run one workload and start quickly. That still leaves a practical problem. MindRoom, like most tools, speaks Docker: it hands Docker an image and some folders to mount, and gets a running container back. A VM does not take a Docker image. Switching to VMs would normally mean building disk images, booting them, and wiring up files and networking myself.

Docker does not start containers by itself, though. It takes care of images, networks, and volumes, and then hands the actual start to a small program called a runtime. The default runtime, runc, does what I described earlier: it starts a process on the host kernel with namespaces, cgroups, and seccomp. But you can tell Docker to use a different runtime.

Kata Containers is such a runtime, and it starts a microVM instead. It picks one of the VMMs above (my test used QEMU), boots a small guest kernel, and starts the container inside it.⁴ Docker, and MindRoom, still see an ordinary container.

gVisor, from Google, takes a different route, and it is not a VM. It is a reimplementation of the Linux kernel, written in Go, that runs as a normal program on the host. Go manages memory for the programmer, so the use-after-free mistake behind the DepthFirst escape is much harder to make in gVisor. The worker’s syscalls go to gVisor instead of the host kernel, and gVisor itself only makes a small, filtered set of host syscalls. That is the hardening idea taken all the way: gVisor answers the agent’s syscalls itself, so the filter on the host can be much stricter than Docker’s. It needs no /dev/kvm, so it also runs inside cloud VMs that cannot start VMs of their own.⁵ My test machine, an ARM server (with the kind of chip phones use) running NixOS, had no /dev/kvm, which is why I tried gVisor first. Its runtime is called runsc, and it plugs into Docker the same way.

The three options side by side.

Or as a table:

Isolation	What worker code talks to	Needs KVM	Usual escape route
Plain Docker (`runc`)	The host kernel, filtered by seccomp	No	A host kernel bug
gVisor (`runsc`)	gVisor’s own kernel, written in Go	No	A gVisor bug, then a host kernel bug through its small filtered surface
Kata Containers	Its own guest kernel in a microVM	Yes	A guest kernel bug, then a hypervisor bug

Each option also changes how much code on the host an escape can pick its bug from.

Lines of code on the host that deal with what the agent’s code asks for, counted with tokei, comments and blank lines included. For Docker, the core of Linux without drivers; for gVisor, its kernel and network stack without tests; for Kata, KVM alone, without the VMM and Kata’s other helpers. Not all of it is reachable. Hover for exact numbers.

The PR: six lines #

The change is PR #2886, which is now merged. It adds one setting, MINDROOM_DOCKER_WORKER_RUNTIME, and MindRoom passes its value to Docker as the runtime whenever it starts a worker. That is six lines of source code, plus 62 lines of tests and some docs. Changing the setting recreates each worker the next time it is used, and leaving it unset keeps everything as it was.

For gVisor, setting it up is two steps: register the runtime in Docker’s /etc/docker/daemon.json and restart Docker, then point MindRoom at it.

{
 "runtimes": {
 "runsc": { "path": "/usr/local/bin/runsc" }
 }
}

MINDROOM_DOCKER_WORKER_RUNTIME=runsc

Kata needs no entry in that file: once it is installed, MINDROOM_DOCKER_WORKER_RUNTIME=io.containerd.kata.v2 is enough. The full setup, and a few catches for each runtime, are in the MindRoom docs.

The six lines are short because everything else already existed. gVisor and Kata do the isolating. Docker already gives each worker its folders, its credentials, and its network, and it already knows how to hand a container to a different runtime. On the MindRoom side, the hard part happened months earlier, when agent tools moved out of the main process into dedicated workers with explicit mounts and credentials. Once every worker is a single call to Docker’s containers.run, swapping what runs underneath it is one argument. For comparison, Claude Opus estimated that building a microVM backend into MindRoom itself would take 2,500 to 4,000 lines.

What it costs #

I tested both runtimes with a real agent in a Matrix room: gVisor on that ARM server, and Kata in a throwaway VM on my desktop. Shell, Python, git, background processes, and the browser all worked, without any MindRoom code change. My Kata numbers below come from a VM inside a VM, so bare-metal numbers will differ.

Configuration is the cheap part: one environment variable, plus one entry in Docker’s config for gVisor. The other costs are the usual trade-offs between containers and VMs:

Computation. Code that only calculates, without asking the kernel for anything, runs at full speed with both. gVisor lets those instructions run directly on the CPU, according to its performance guide. A Kata guest runs on the CPU in its virtualization mode, and in AWS’s paper on Firecracker, the CPU overhead was negligible.
Syscalls, disk, and network. The cost starts when a program does ask the kernel for something. Under gVisor, every syscall takes a detour through gVisor’s kernel, so programs that make many small syscalls, like ones that open thousands of small files, slow down the most. Under Kata, disk and network go through the VMM’s fake devices. In that same paper from 2020, a Firecracker guest reached 15 Gbit/s of network traffic where the host reached 44.
Startup. gVisor only has to start a program, so it starts quickly. Kata has to boot a kernel first, and my worker took about 10 seconds to start instead of about 3. Booting a VM can be much faster than that: Firecracker boots a minimal one in about 125 ms.
Memory. gVisor adds a small, mostly fixed amount per sandbox. Each Kata VM runs a second kernel plus the VMM, which alone takes about 3 MB with Firecracker and 131 MB with QEMU. Kata gives each worker VM 2 GiB by default, and /tmp lives in that RAM. The host only hands out memory the guest actually touches, so my idle worker used under 0.8 GiB of host memory. Raising the VM size costs little until a worker needs it. But a VM cannot grow past its size the way a container can, and memory the guest has used, for example to keep recently read files in its cache, mostly stays with the VM.
Compatibility. gVisor reimplements Linux, and its docs say there will always be unimplemented features and bugs, so now and then a program will not work. Kata runs a real Linux kernel, so programs behave as usual, with a few catches. For example, a file lock held on the host does not block the worker, because the lock lives in the guest kernel.

What this does not fix #

A stronger sandbox protects the host. It does nothing about the tokens and network access you give the worker. An agent needs no kernel exploit to misuse a token that can push to your repositories, or to poke at every device on your home network. Scoped credentials and a firewall allowlist protect you at least as much, and MindRoom handles both separately: Agent Keys let agents use API keys without ever seeing them, and Approved Egress limits which servers a worker on Kubernetes can reach.

gVisor has had its own bugs, and it still talks to the host kernel, just through a much smaller door. KVM has bugs too, as this year’s escapes show. Both walls make an escape far harder, and neither makes it impossible.

Conclusion #

Once I started reading about this, I kept running into people saying containers were never a security boundary. That is true, and for years I did not worry about it. With AI finding kernel bugs faster and agents running code nobody reads, I do now.

No sandbox is perfectly safe, though. In a conversation with Dwarkesh Patel, Noam Brown doubted that even computers with no network connection at all would be enough to contain an AI. He pointed to studies where two such computers next to each other talk through heat: one runs its CPU hot, and the other picks up the change with its temperature sensor. Every option in this post trades some speed or memory for safety. How far to go depends on your own risk tolerance and how paranoid you are. MindRoom can now move its agents off the shared kernel with one environment variable. I do not use it myself yet, but if you are more paranoid than I am, it is there.

The other way out is a bug in the program that starts the container. In 2024, CVE-2024-21626 in runc, the program Docker uses for that, leaked a handle to a host folder into new containers, so a malicious image could reach the host’s files without any kernel bug. ↩︎
Docker can map the container’s root to an ordinary user on the host with user namespace remapping, or run without root altogether in rootless mode, but neither is the default. Rootless mode would have stopped the agent in my Docker story, because Docker then has only my own rights. Remapping would not, because anyone who can use Docker can turn it off for a single container. Neither helps against a kernel bug, because code running inside the kernel is above every user. ↩︎
A bug in the VMM’s fake devices, or in other VMM code that handles the guest’s traffic, can skip the first step, because a normal program in the guest can feed that code data through the network or a shared folder. CVE-2020-8608, a buffer overflow in QEMU’s user-mode networking, could be triggered that way with crafted network traffic. ↩︎
Like runc for Docker, Kata’s own programs on the host are a way out too. In 2020, one bug (CVE-2020-2023) let a container take over the guest without a kernel bug, and a second (CVE-2020-2026) then let the guest trick the Kata runtime on the host into mounting files over any host path. Neither needed a KVM bug. virtiofsd, the program that shares folders with the guest, also runs on the host and handles the guest’s file requests. ↩︎
gVisor can use KVM too, as an option. By default it catches the agent’s syscalls without it, which is the setup this post describes. With KVM switched on, gVisor runs the agent’s code in the CPU’s virtualization mode so catching syscalls is faster, but gVisor’s own kernel still answers them, and there is no guest Linux and no fake devices. gVisor recommends that mode only on bare metal; inside a VM the default is usually faster. ↩︎

Slowly, then all at once: AI agents that know how a company works

Wed, 30 Sep 2026 00:00:00 +0000

IonQ is the world’s leading vertically integrated, full-stack quantum platform company. We build quantum computers, quantum security solutions, quantum networks, and quantum sensors, and we fabricate our own quantum chips. That means thousands of people working on some of the hardest physics and engineering problems the world faces. Much of what we know is not written down in one place. Where a program actually stands lives in Slack threads, tickets, commits, documents, and meetings. If you want the real picture, somebody has to read all of it, or you ask a colleague who has probably explained it five times already. And as IonQ grows, everybody is new to something. Someone who just joined through an acquisition is new to all of it, and after three years here, I still could not explain what half of our programs do.

Why I built MindRoom #

Programming has always been my biggest passion, and when AI coding agents suddenly became extremely good, the jump in what I could do in a single day was the biggest I had ever seen. But that power only reaches people who are comfortable in a terminal. My mom will never open one. Neither will most of the physicists, hardware engineers, and program managers who make a company like IonQ work.

So I built MindRoom, which is open source, and I run it at IonQ as a central hub for collaboration and knowledge sharing, what I like to call a company brain. It looks like a chat app. Everybody gets a personal agent the moment they log in, running on its own sandboxed computer in the cloud. Every program can get a shared agent that is connected to that program’s Slack, Confluence, Jira, code, meeting notes, and even the parts database. Its answers link back to what a human wrote, so you can check them instead of trusting them.

The personal agent is where most people start. I can send mine a voice message like “every week, just before my one-on-one with my manager, give me a clear summary of what I’ve been up to.” I never told it who my manager is, when we meet, or where to look. It works that out on its own, sends me a link to the summary, and from then on does it every week without being asked again.

Slowly #

For months, MindRoom grew one small group at a time. It started with a single colleague, then a pilot of about ten, and then the team behind Superion 10K, our 10,000-qubit quantum computer. We rolled it out carefully, and each step at most doubled the group. Every new group needed a hand-held onboarding, so I recorded short videos to make that scale. The people who tried it kept using it, but it was hard to explain what it could do without showing it.

A voice message becomes a reminder

An approval card before anything is sent

Watch the agent's browser and take over

Demo recordings with a fictional company and scripted responses. More in the MindRoom showcase.

Then all at once #

That changed with a company-wide demo. Instead of explaining what agents can do, I played someone new to the Superion 10K program and asked its shared agent, by voice, to explain what we are building, how the work is split, and how the pieces fit together. A few minutes later it had made an interactive slide deck, built entirely from our own documents, Slack threads, and code, with every claim linked to its source.

Hundreds of people signed up that same day, and within weeks about a third of everyone who can create an account had one. At the time of the demo, five programs and teams used it every day, and now it is about fifteen. People had already had access to tools like Claude and ChatGPT for months, but those never clicked for them in the same way. An agent that already knew our work did.

The surprise #

I built MindRoom for people who never open a terminal, simple enough that even my mom could use it, so I expected non-technical colleagues to be the main audience. They did adopt it. Program managers use it to write weekly updates from what actually happened in Slack and Jira, instead of from whatever people remembered to tell them. Our configuration management team uses it to check the hardware as it was actually built against the engineering baseline, and to find policy documents that contradict each other across different systems.

What I did not expect was that some of the most capable engineers I know, who already run their own coding agents every day, would become some of the heaviest users. One of them, who already lived in Claude Code, cleared his entire inbox, Slack, and task list from his phone on a single flight. I think they like it for the same reason everyone else does: it keeps context. For writing code, dedicated tools like Claude Code and Codex are better, because they are built for exactly that.¹ But engineers use MindRoom as a layer for gathering knowledge. A coding agent does not need to know who you are or who you report to in order to fix a bug. Coordinating a large group of people is mostly that kind of context.

Institutional knowledge #

A colleague wrote about why AI’s biggest opportunity in large programs is institutional knowledge, from the perspective of someone using it every day. From the builder’s side, what I like most is that a shared agent gets better the more people use it.

When someone works through a problem with it, it keeps what it learned, so the next person benefits without ever having been in that thread.

In Claude Code or Codex, you handle memory yourself, by asking the agent to write things down in Markdown files. Agents get lazy: they forget to write things down, and later they forget to read them. MindRoom agents are also told to write down durable facts, in plain text files under version control, but they do not have to remember to. A background process picks out what is worth keeping from every conversation and adds it to a semantic search index. When a new message comes in, the relevant memories are looked up and added to the context automatically. So remembering and looking things up no longer depend on the agent deciding to do it. When someone asks a question that was already answered, the agent re-verifies its earlier analysis instead of starting from zero, so the second answer usually comes faster and builds on the first.

One very cool and yet simple idea is agent dreaming, which I picked up from a talk by Lance Martin from Anthropic. People collect short-term memories during the day and consolidate them while they sleep. The analogy is loose, but the idea carries over. Every night, a first pass reads the agent’s memories, every conversation it has had, and the original sources, and looks for contradictions, for example when someone corrected something a few minutes later and the two memories were never reconciled. It writes a patch to fix them. A second pass, in a completely fresh context, checks every proposed fix and accepts or rejects it. It rejects about half, which is exactly the kind of skepticism I want. Dreaming is not even platform code, just a skill, a short set of written instructions: you tell an agent to set up dreaming, and it knows what to do, including scheduling itself to run every night.

It also notices when two documents disagree, which is easy to miss when nobody reads all of them. Its reports end with “Corrections welcome, tell the agent and the next report improves,” and a correction becomes a memory that the next report uses. And when nobody ever wrote something down, it usually still knows who to ask, because it keeps track of what everybody is working on.

Why build it ourselves #

There are plenty of AI products that promise something similar. Building our own gives us complete flexibility and no vendor lock-in. We are not waiting for a third party to support the model, the system, or the feature we need. When a better model comes out, we can switch the same day. And the knowledge it builds up is stored in IonQ’s own cloud environment.

It also means we can experiment quickly. Features that could take a large organization months often take me days:

Agents can open a real browser on their own remote computer, not on your laptop, and you can watch it live and take over, for example to log in somewhere, before handing control back.
An MCP gateway lets people connect their local tools, like Claude Code or Codex, to every agent they have access to and all of those agents’ tools, so they benefit from what the agents have collected even outside MindRoom.
A day after OpenAI released GPT-Live-1 in its API, you could call your MindRoom agent: the same agent, with the same memory, which hands longer tasks to a background process while you keep talking. On my commute, I talk to my agent to plan my day and get things started.
Agents can host small sites, much like ChatGPT Sites, behind a temporary link with access control. That is how an agent can answer a question with an interactive presentation or a dashboard instead of a wall of text.

What it takes to run this at a company #

A demo on a laptop is easy. Running a central hub that hundreds of people rely on every day is a different problem, and most of my time went there.

People sign in with their normal company account, and their account and personal agent are created on first login. There is nothing to install.

Every agent runs in its own sandboxed container with no secrets inside it. Credentials are injected from outside, so the agent never sees a token. Agents also cannot reach arbitrary websites: outbound traffic is denied by default, and a new domain needs a person’s time-limited approval.

Reading is free, but anything that sends or changes something on your behalf, like an email, shows an approval card with exactly what will be sent and to whom. A shared agent only answers the people who work on that program, and it only reaches the systems it was granted.

It runs in IonQ’s own cloud environment, and a security review ran alongside the rollout. At its busiest moment so far, 200 people were writing to their agents at the same time, and it held up fine. The chat layer has plenty of room to grow: the French government runs Matrix for about 400,000 civil servants.

What’s next #

With our recent acquisitions and a lot of new hires, there is a steady flow of people joining IonQ who are new to all of it, and to MindRoom. Because MindRoom is built on Matrix, which is federated like email, groups with different data rules can each get their own instance and still work together where that is allowed. And whoever joins next can log in on their first day and ask their program’s agent how the work fits together and who to ask.

That does not mean you cannot write code from MindRoom. Because every agent has its own computer, it can run any software, so my MindRoom agent simply runs Claude Code or Codex itself when there is coding to do. ↩︎