Matrix | Bas Nijholt

Roll your own, fork the rest 🍴

Mon, 19 Oct 2026 00:00:00 +0000

Introduction: a follow-up to something I wrote #

In April 2025 I published Confessions of an asymmetric hypocrite, about my split personality on dependencies. Strict minimalism for the libraries I publish, and then, for applications:

For applications (especially CLIs in isolated environments), I’m far more liberal with dependencies to build features quickly.

I also wrote that I “strongly prefer dependencies that are already widely adopted and trusted by the community” because “these are less likely to cause unexpected issues”.

Then I spent a year building MindRoom, which is an application, exactly the category where I said I could relax.

I want to report back. “Prefer battle-tested dependencies” was the right rule. I just had the wrong idea of what battle-tested means. I thought a project with tens of thousands of GitHub stars qualified. Now I think it means millions of downloads, every day.

Table of contents

Update 2026-10-31: In the two weeks after publishing, I took my own advice twice. MindRoom’s knowledge search no longer uses ChromaDB, and code MindRoom owns has replaced Agno’s runtime. Agno is now only MindRoom’s tool library. The full stories are in the comments.

1. Stars measure interest, not usage #

When I picked Agno as MindRoom’s AI framework, it looked like the safe choice. Today it has over 42,000 stars and 6,000 forks.¹

It also gets about 78,000 downloads a day on PyPI. httpx, one of Python’s standard HTTP libraries, gets about 25 million.²

A star is something you click once, when a README looks exciting. A download happens every time some project installs the package.

Most of those downloads are CI, not people, so downloads are not users either. But a package with millions of downloads a day is installed by a huge number of different projects. Many of them use the same functions you are about to use, so they hit the bugs before you do.

Battle-tested is also not the same as old. I use uv for everything, and it has only been around for a couple of years. I run into bugs in uv too, but there is simply no better alternative, and it gets about 3.6 million downloads a day.

2. A popular protocol is not a popular library #

The other place my old idea failed was Matrix.

Building on Matrix felt obviously correct. An open protocol, about a decade old, with serious security reviews from large organizations and governments. Millions of people use it.

Matrix is still a good protocol and I am still on it. But even though millions of people use Matrix, there are only a few client libraries.

MindRoom streams agent replies by editing the same message many times a second. Almost every part of the ecosystem I touched had a problem with that: Element, Cinny, matrix-nio, and the Matrix JavaScript SDK. In the JavaScript SDK, edits that arrived while a thread was still loading were silently dropped, so you saw stale text until you reloaded.³

What counts is how many projects use the library you build on. Element’s users do not test matrix-js-sdk for you. Other people building on matrix-js-sdk do. It gets roughly 100,000 downloads a day. matrix-nio, the Python client, about 23,000.

3. I checked the rule against my own code #

I pulled the daily download numbers for the packages I use and looked for workarounds, patches and forks in my own source.⁴

Package	Downloads per day	What it cost me
requests	49M	nothing I had to work around
pydantic	35M	nothing
numpy	33M	nothing
httpx	25M	nothing
aiohttp	18M	nothing
FastAPI	17M	nothing
chromadb	270K	workarounds for a bug that holds the GIL
matrix-js-sdk	~100K	patched
mem0ai	79K	largely reimplemented
agno	78K	126 documented workarounds
matrix-nio	23K	forked

Above a million a day, I had nothing to work around. Below it, every package on this list cost me a patch, a fork, a rewrite, or a pile of workarounds.

Take ChromaDB, the vector database behind MindRoom’s knowledge bases. In some cases it holds Python’s GIL during I/O, which blocks every other thread in the process. I opened a pull request to fix it, and nobody has looked at it yet. Instead of forking, I run every knowledge read in a short-lived child process, because a process can be killed and a blocked thread cannot.⁵

My fork of matrix-nio has about 8,900 lines added and 2,000 removed in the library code since the last upstream release.⁶

This does not mean every small package needs replacing. MindRoom pulls in a few hundred packages in total, and plenty of them have modest download numbers. The scrutiny should match how much I depend on a package. Something I use for one small, specific purpose is fine. Agno and matrix-nio are the base that MindRoom is built on, which is probably also why they hurt the most.

4. Roll your own: the library gives you the easy 10% #

Take memory libraries, Mem0 and friends. I have already written about this: I cloned LlamaIndex, LangChain, Letta, Mem0 and PydanticAI, studied how they work, and re-implemented the parts I needed. My memory system ended up “mostly modeled after Mem0”.

The core idea is not hard. With RAG, you chunk, embed, retrieve, re-rank and reconcile. Once you have read a couple of implementations you can write it, and afterwards you understand every line.

The same is true of an agent framework. It builds a JSON payload, posts it to an HTTP endpoint, parses a streaming response, keeps a list of messages, and loops when the model asks for a tool call. With AI, that is a day or two of work.

The hard part is making it reliable at scale: never blocking the event loop, and handling the crazy edge cases. With an agent system you end up building a very complicated state machine, and if you do not watch it, any combination of states is possible. Almost all of them are edge cases.

A library gives you the easy 10% and leaves you the hard 90%. And you do that 90% inside somebody else’s code, where you cannot fix the root cause.

MindRoom is now about 360,000 lines of Python, plus about 750,000 lines of tests.⁷ A good part of that is code I wrote instead of depending on a library:

Memory and knowledge: about 15,600 lines across 50 files.
Conversation history and compaction: about 8,400 lines.
An OpenAI-compatible API: about 2,300 lines, so MindRoom agents show up as selectable “models” in any chat frontend that speaks the OpenAI protocol.
Claude provider support: about 3,000 lines.

If I have to write the hard 90% either way, I might as well write the easy 10% too.

5. Fork your own: when the problem is hard #

I am not going to write my own end-to-end encrypted Matrix SDK, or my own homeserver. Those are hard, and also far below the line. So there I fork.

I forked the Python Matrix client. MindRoom ships its own fork of matrix-nio rather than the upstream package.

I forked the Matrix server, twice. First Synapse. When I ran into scaling problems I moved to Tuwunel and forked that.

I forked the web client. Cinny is the Matrix web client MindRoom uses.

None of these are AI projects. AI is just where I felt it most.

Whether a fork stays a fork depends on the maintainers. Tuwunel is extremely receptive to my patches, so I upstream anything that is general enough, and over the last six months I am its fourth most active contributor by commits.⁸ Cinny has not accepted any of my pull requests, so I forked it and stopped trying to upstream. Same with LibreChat: I had patches open for over a year without a reply, so I forked.

Keeping a fork up to date used to be miserable. With an LLM doing the rebasing, pulling in the latest upstream is now cheap, though still a bit of a risk every time. The bigger cost is that once you fork, you are responsible for the whole codebase, including the parts you never changed.

Agno: patches instead of a fork #

I did not fork Agno. I monkey-patched it. Agno is very actively developed, I have talked to the maintainer, and they know about these problems, so I hoped to upstream my fixes. I have filed 33 issues and pull requests there.⁹

That got out of hand. MindRoom now carries 126 documented workarounds for Agno across 59 files. Most of them live in 36 dedicated compatibility modules, about 7,200 lines of code, with the rest patched inline in 23 other files. Every one uses the same comment format, so I can find all of them with one search. Each one says what is wrong upstream, links the issue or PR, and says when I can delete it.¹⁰ A few examples:

Search failures become plausible empty results. A failed knowledge search and an empty index return the same empty list, so my agents confidently told me things did not exist.¹¹
Streamed tool calls get malformed or merged.¹²
Reasoning is silently lost when tool calls are replayed.¹³
Provider errors collapse into a generic 502, which throws away the error code you need to decide whether to retry.¹⁴

At this size, the patches amount to a fork.

6. I am not dunking on open source, including mine #

Everything I named is open source I use every day, for free, built by people doing their best. I filed issues and PRs, and where maintainers merged fixes I deleted my workarounds.

My own software is in the same category. Most new AI tooling is not very good, and MindRoom may well be included. It is large, fast-moving, and nowhere near millions of daily downloads. By my own rule, you should think twice before depending on it.

I also have to admit that I have not yet built a project the roll-your-own way from the start. So comparing “what I would do next time” with what I actually did is not entirely fair. This is what I think right now, and a year from now I may have to write another post correcting this one.

7. My rule for the next `uv add` #

I weigh three questions.

How many projects use it? Daily downloads of the actual package, not stars, and not how many people use the product built on top. In the millions: depend on it.

How much do I rely on it? Agno and matrix-nio are the base of MindRoom. A library I use in a CLI to format a question is not, and if I use it in a narrow way I do not need my own version.

How hard is it, and how receptive is upstream? If the library is something AI can write in a couple of days, roll my own. If it is hard, fork it. If the maintainers accept patches, upstream what is general enough.

If I started MindRoom over today, I would write the HTTP requests to the model providers myself. Not even the OpenAI or Anthropic SDKs. And even httpx is less safe than its download numbers suggest: in early 2026 its maintainer closed the issue tracker and discussions, and the latest stable release is from December 2024.¹⁵

In 2025 I wrote that the right answer is “it depends on the situation”, and I still believe that. I just had the wrong threshold. Battle-tested means millions of downloads a day. Below that: roll your own, and fork the rest.

From the GitHub API (api.github.com/repos/agno-agi/agno) on 2026-10-02: 42,385 stars and 6,041 forks. ↩︎
PyPI numbers are the last_day field of the pypistats.org API (/api/packages/<name>/recent), fetched on 2026-10-02, except uv, fetched on 2026-10-18 (3,648,827). The matrix-js-sdk number is npm’s weekly download count (about 730,000) divided by seven. Daily numbers wobble, so treat them as orders of magnitude. ↩︎
When events arrive for a thread that has not finished initializing, the SDK’s replay path never applies them, so an m.replace edit (the event type Matrix uses for “this replaces message X”) is stored and never shown. I replayed a captured event trace through the SDK: unpatched, zero of seventeen edits reached the listeners; patched, all seventeen. MindRoom carries the fix as a patch-package patch on the SDK’s thread.ts. ↩︎
For each direct dependency I looked at three things in the MindRoom repository: whether pyproject.toml pins a fork instead of the upstream package, whether patch-package patches it, and whether the source has workaround markers or comments pointing at upstream issues for it. The packages above a million downloads a day had none of those. ↩︎
The PR is titled “Release GIL during metadata I/O”. On the MindRoom side, knowledge/read_process.py starts the child process, and a TODO says to remove it once a Chroma release with the fix is pinned. A blocked thread holding the GIL cannot be interrupted from Python, but the kernel will always terminate a child process. ↩︎
Compared with upstream matrix-nio 0.25.2 (released 2024-10-04): git diff --shortstat on the library code gives 8,872 lines added and 1,966 removed across 52 files, from 89 commits, 76 of them mine. The fork is published on PyPI as mindroom-nio. ↩︎
Counted with git ls-files and wc -l on the MindRoom repository on 2026-10-18: 360,513 lines in 1,222 Python files under src/, 750,657 lines under tests/, and another 81,543 lines of TypeScript in the frontend. The per-feature numbers below come from the same count over the relevant directories. ↩︎
Non-merge commits on Tuwunel’s main branch since 2026-04-18: Jason Volk 1,332, June Strawberry 325, dasha_uwu 55, me 42. ↩︎
Issues and pull requests in agno-agi/agno with me as the author, counted on 2026-10-02. ↩︎
Every workaround starts with a comment like # AGNO_COMPAT: Knowledge search hides provider failures as empty results., followed by Reason:, Upstream issue:, Upstream PR:, Remove when: and the tests that cover it. The counts come from grep over src/: 126 AGNO_COMPAT: markers in 59 files, and 7,160 lines in the 36 agno_compat_* modules. ↩︎
Knowledge.search catches vector store and provider errors and returns []. agno#10150, partially fixed by agno#10152. ↩︎
When a streamed tool-call index does not start at zero, Agno leaves empty placeholder calls. agno#8879, fix proposed in agno#8880. ↩︎
With the OpenAI Responses API, Agno keeps only the last reasoning item, so it cannot be replayed with the tool calls it belonged to. agno#9960; partial fixes landed in agno#10075 and agno#10395. ↩︎
Unclassified ModelProviderError failures default to HTTP 502, so a real network blip and a permanent error look the same. agno#8869, partially addressed by agno#8870. ↩︎
The latest stable release on PyPI is 0.28.1, uploaded 2024-12-06. There have been 1.0 development pre-releases since, the latest in August 2026. The closing of issues and discussions was widely discussed in the Python community in March 2026. ↩︎

Slowly, then all at once: AI agents that know how a company works

Wed, 30 Sep 2026 00:00:00 +0000

IonQ is the world’s leading vertically integrated, full-stack quantum platform company. We build quantum computers, quantum security solutions, quantum networks, and quantum sensors, and we fabricate our own quantum chips. That means thousands of people working on some of the hardest physics and engineering problems the world faces. Much of what we know is not written down in one place. Where a program actually stands lives in Slack threads, tickets, commits, documents, and meetings. If you want the real picture, somebody has to read all of it, or you ask a colleague who has probably explained it five times already. And as IonQ grows, everybody is new to something. Someone who just joined through an acquisition is new to all of it, and after three years here, I still could not explain what half of our programs do.

Why I built MindRoom #

Programming has always been my biggest passion, and when AI coding agents suddenly became extremely good, the jump in what I could do in a single day was the biggest I had ever seen. But that power only reaches people who are comfortable in a terminal. My mom will never open one. Neither will most of the physicists, hardware engineers, and program managers who make a company like IonQ work.

So I built MindRoom, which is open source, and I run it at IonQ as a central hub for collaboration and knowledge sharing, what I like to call a company brain. It looks like a chat app. Everybody gets a personal agent the moment they log in, running on its own sandboxed computer in the cloud. Every program can get a shared agent that is connected to that program’s Slack, Confluence, Jira, code, meeting notes, and even the parts database. Its answers link back to what a human wrote, so you can check them instead of trusting them.

The personal agent is where most people start. I can send mine a voice message like “every week, just before my one-on-one with my manager, give me a clear summary of what I’ve been up to.” I never told it who my manager is, when we meet, or where to look. It works that out on its own, sends me a link to the summary, and from then on does it every week without being asked again.

Slowly #

For months, MindRoom grew one small group at a time. It started with a single colleague, then a pilot of about ten, and then the team behind Superion 10K, our 10,000-qubit quantum computer. We rolled it out carefully, and each step at most doubled the group. Every new group needed a hand-held onboarding, so I recorded short videos to make that scale. The people who tried it kept using it, but it was hard to explain what it could do without showing it.

A voice message becomes a reminder

An approval card before anything is sent

Watch the agent's browser and take over

Demo recordings with a fictional company and scripted responses. More in the MindRoom showcase.

Then all at once #

That changed with a company-wide demo. Instead of explaining what agents can do, I played someone new to the Superion 10K program and asked its shared agent, by voice, to explain what we are building, how the work is split, and how the pieces fit together. A few minutes later it had made an interactive slide deck, built entirely from our own documents, Slack threads, and code, with every claim linked to its source.

Hundreds of people signed up that same day, and within weeks about a third of everyone who can create an account had one. At the time of the demo, five programs and teams used it every day, and now it is about fifteen. People had already had access to tools like Claude and ChatGPT for months, but those never clicked for them in the same way. An agent that already knew our work did.

The surprise #

I built MindRoom for people who never open a terminal, simple enough that even my mom could use it, so I expected non-technical colleagues to be the main audience. They did adopt it. Program managers use it to write weekly updates from what actually happened in Slack and Jira, instead of from whatever people remembered to tell them. Our configuration management team uses it to check the hardware as it was actually built against the engineering baseline, and to find policy documents that contradict each other across different systems.

What I did not expect was that some of the most capable engineers I know, who already run their own coding agents every day, would become some of the heaviest users. One of them, who already lived in Claude Code, cleared his entire inbox, Slack, and task list from his phone on a single flight. I think they like it for the same reason everyone else does: it keeps context. For writing code, dedicated tools like Claude Code and Codex are better, because they are built for exactly that.¹ But engineers use MindRoom as a layer for gathering knowledge. A coding agent does not need to know who you are or who you report to in order to fix a bug. Coordinating a large group of people is mostly that kind of context.

Institutional knowledge #

A colleague wrote about why AI’s biggest opportunity in large programs is institutional knowledge, from the perspective of someone using it every day. From the builder’s side, what I like most is that a shared agent gets better the more people use it.

When someone works through a problem with it, it keeps what it learned, so the next person benefits without ever having been in that thread.

In Claude Code or Codex, you handle memory yourself, by asking the agent to write things down in Markdown files. Agents get lazy: they forget to write things down, and later they forget to read them. MindRoom agents are also told to write down durable facts, in plain text files under version control, but they do not have to remember to. A background process picks out what is worth keeping from every conversation and adds it to a semantic search index. When a new message comes in, the relevant memories are looked up and added to the context automatically. So remembering and looking things up no longer depend on the agent deciding to do it. When someone asks a question that was already answered, the agent re-verifies its earlier analysis instead of starting from zero, so the second answer usually comes faster and builds on the first.

One very cool and yet simple idea is agent dreaming, which I picked up from a talk by Lance Martin from Anthropic. People collect short-term memories during the day and consolidate them while they sleep. The analogy is loose, but the idea carries over. Every night, a first pass reads the agent’s memories, every conversation it has had, and the original sources, and looks for contradictions, for example when someone corrected something a few minutes later and the two memories were never reconciled. It writes a patch to fix them. A second pass, in a completely fresh context, checks every proposed fix and accepts or rejects it. It rejects about half, which is exactly the kind of skepticism I want. Dreaming is not even platform code, just a skill, a short set of written instructions: you tell an agent to set up dreaming, and it knows what to do, including scheduling itself to run every night.

It also notices when two documents disagree, which is easy to miss when nobody reads all of them. Its reports end with “Corrections welcome, tell the agent and the next report improves,” and a correction becomes a memory that the next report uses. And when nobody ever wrote something down, it usually still knows who to ask, because it keeps track of what everybody is working on.

Why build it ourselves #

There are plenty of AI products that promise something similar. Building our own gives us complete flexibility and no vendor lock-in. We are not waiting for a third party to support the model, the system, or the feature we need. When a better model comes out, we can switch the same day. And the knowledge it builds up is stored in IonQ’s own cloud environment.

It also means we can experiment quickly. Features that could take a large organization months often take me days:

Agents can open a real browser on their own remote computer, not on your laptop, and you can watch it live and take over, for example to log in somewhere, before handing control back.
An MCP gateway lets people connect their local tools, like Claude Code or Codex, to every agent they have access to and all of those agents’ tools, so they benefit from what the agents have collected even outside MindRoom.
A day after OpenAI released GPT-Live-1 in its API, you could call your MindRoom agent: the same agent, with the same memory, which hands longer tasks to a background process while you keep talking. On my commute, I talk to my agent to plan my day and get things started.
Agents can host small sites, much like ChatGPT Sites, behind a temporary link with access control. That is how an agent can answer a question with an interactive presentation or a dashboard instead of a wall of text.

What it takes to run this at a company #

A demo on a laptop is easy. Running a central hub that hundreds of people rely on every day is a different problem, and most of my time went there.

People sign in with their normal company account, and their account and personal agent are created on first login. There is nothing to install.

Every agent runs in its own sandboxed container with no secrets inside it. Credentials are injected from outside, so the agent never sees a token. Agents also cannot reach arbitrary websites: outbound traffic is denied by default, and a new domain needs a person’s time-limited approval.

Reading is free, but anything that sends or changes something on your behalf, like an email, shows an approval card with exactly what will be sent and to whom. A shared agent only answers the people who work on that program, and it only reaches the systems it was granted.

It runs in IonQ’s own cloud environment, and a security review ran alongside the rollout. At its busiest moment so far, 200 people were writing to their agents at the same time, and it held up fine. The chat layer has plenty of room to grow: the French government runs Matrix for about 400,000 civil servants.

What’s next #

With our recent acquisitions and a lot of new hires, there is a steady flow of people joining IonQ who are new to all of it, and to MindRoom. Because MindRoom is built on Matrix, which is federated like email, groups with different data rules can each get their own instance and still work together where that is allowed. And whoever joins next can log in on their first day and ask their program’s agent how the work fits together and who to ask.

That does not mean you cannot write code from MindRoom. Because every agent has its own computer, it can run any software, so my MindRoom agent simply runs Claude Code or Codex itself when there is coding to do. ↩︎

Exploring the Jev hype in MindRoom

Wed, 23 Sep 2026 00:00:00 +0000

Last week, every AI newsletter I get was full of Jev, a new model from TypeSafe. So were several subreddits. I am a permanent lurker on r/LocalLLaMA, and even though it is about local models, every other post seemed to be about Jev. There were local clones like mini-jev and von, CLI wrappers like jev-cli, and even someone on Hacker News claiming they had built the same thing a year earlier.

I tried Jev in MindRoom, my open-source agent platform on Matrix, and less than two days later it was making three decisions there.

Update 2026-10-07: OpenAI released the Decisions API today, its own take on the same idea, and MindRoom now supports it as a third backend; see below.

What a System One model is #

The name comes from Daniel Kahneman’s Thinking, Fast and Slow, where System 1 is fast, intuitive thinking and System 2 is slow and deliberate. A System One model does not generate text. You send it a state, for example a chat log, plus typed questions, and it returns probabilities:

Type	Question	Answer
Noul	Yes or no	Probability of yes
Choice	Pick one of up to 255 options	Chosen option plus the full distribution
Score	Rate against a rubric of 2 to 10 levels	Probability-weighted value

TypeSafe claims 70 to 500 ms end to end and $0.042 per million input tokens, with output tokens free. They also write “we can’t prove it isn’t subsidized.” Either way, it is incredibly cheap and incredibly fast.

Adaptive participation #

In MindRoom, a conversation can have several humans, several agents, or both. When I talk to a single agent, it simply replies, and I don’t need to tag it. Once another person joins and writes something, the agent cannot tell whether the message was meant for it or for the other person. Nothing decided that, so the agent stayed quiet unless explicitly tagged. That confused several users.

So I recently added adaptive participation: an agent that has already replied in a thread decides on its own whether to answer an untagged message. I first used an LLM as the judge, GPT-5.6 Luna on low reasoning. A yes-or-no question like “should this agent respond?” is exactly what Jev is for, so it became the second backend. Compared with Luna, it is about ten times cheaper and ten times faster, and if I believe the benchmarks, it is also smarter.

“Thanks” should not stop the agent #

The next thing I used Jev for was interruptions. When I send a message while an agent is still replying, MindRoom injects a notice at the next tool call telling the agent to stop and wrap up, because there is a new message. But often I, and other users, would watch the agent start working and then write “looks good” or “thanks.” The agent would stop early and continue in the next turn, which is both wasteful and counterintuitive.

Now a judgment decides whether the new message needs the interruption, which the docs call mid-turn coalescing. If it does not, the agent reacts with 👀, finishes its reply, and handles the message afterwards.

This is also where I learned a lesson about evals. My first implementation worked at the code level but not functionally. My evals, based on real attempts inside MindRoom that had all failed plus synthetic cases from GPT-6 Astra, passed 5 out of 40 with Jev. The fix changed the question and gave the judge more context, and it went to 40 out of 40.

	Before: 5/40	After: 40/40
Question	“May the active task finish before the queued human messages are handled in a later turn?”	“Does any newly queued message require an immediate change to, pause of, or stop of the active task?”
When unclear	“Also false when context is incomplete, the relationship is unclear…”	“Mere relevance to the task does not make praise or thanks an interruption.”
Context	The request and the new messages	The same, plus the preceding conversation

The old question treated anything unclear as a reason to interrupt, and a bare “thanks” without the conversation around it always looks unclear.

Picking the responder #

The third decision is routing. When a message in a room with many agents does not mention anyone, MindRoom’s router picks who should answer. That used to be a free-text LLM prompt, “choose the most appropriate agent.” It is now a Jev Choice over the eligible agents, plus a no_fit option, as described in the router docs. A confident no_fit asks the user to mention an agent instead of guessing, and anything uncertain falls back to the existing LLM router.

One config line to switch #

I built a very simple abstraction layer so I can swap Jev for an LLM without changing any code. Every decision is one question with criteria for true and false:

MID_TURN_QUESTION = JudgmentQuestion(
 id="interrupt_current_turn",
 instructions="Does any newly queued message require an immediate change to, pause of, or stop of the active task?",
 when_true="A new message requests stopping or pausing the active task, corrects it, ...",
 when_false="The messages acknowledge progress, express thanks, ask to continue unchanged, ...",
)

Jev receives it as a Noul question and returns a probability, which MindRoom compares with a threshold. An LLM receives the same question and context and must return {"decision": true} or {"decision": false}. Switching is a configuration change:

agents:
 helper:
 mid_turn:
 defer_reaction: "👀"
 judgment:
 provider: typesafe  # or: provider: llm, model: fast
 threshold: 0.8

Both backends log their decisions and latency, so I can compare them on real traffic. The code is in src/mindroom/judgment/, and the backends are documented under participation judgment backends.

A third backend: OpenAI Decisions #

On October 7, OpenAI released its own version of the idea, the Decisions API, currently in public beta. The question types match Jev’s: a predicate is a Noul, and choice and score keep their names. It serves a single model, gpt-6-luna, and OpenAI says it returns answers “about 10x faster than the Responses API.” Input costs $0.10 per million tokens and output is free, so it lists at about 2.4 times Jev’s price.

Thanks to the abstraction layer, adding it took one PR, merged the same day, and it works for all three decisions, including the router’s Choice:

judgment:
 provider: openai_decisions
 threshold: 0.8

From MindRoom’s side, only the wire format differs. The endpoint accepts only user messages, so MindRoom sends each chat message as its own user message, prefixed with its role, like assistant: .... Decisions can also answer with a refusal, which MindRoom treats as an abstention, so the usual fallback applies.

I tested it against the live API through MindRoom’s own code, with the same questions Jev gets. For participation, it gave 0.99 to replying to a direct question and 0.0 while two humans were talking to each other. Mid-turn, “Thanks, looks great so far!” got 0.0 for interrupting and “Stop, keep MySQL…” got 1.0. The router sent a pandas bug to code and a meeting request to calendar, both with confidence 1.0. A nonsense request got only 0.62, below the 0.8 threshold, so the normal LLM router took over. Each call took 130 to 280 ms. I have not compared it with Jev on real traffic yet, but both log the same outcome fields, so that only takes a config change.

Named after Jevons #

I only read why TypeSafe named it Jev after building all this: they named it after William Stanley Jevons, who noticed that as steam engines burned coal more efficiently, demand for coal went up instead of down. This is known as the Jevons paradox. That was literally my experience. Most of what I built was perfectly possible with an LLM. I just never built it, because I am somewhat cost- and latency-minded. Once I implemented Jev for one decision, I came up with use case after use case.

One I already built is a plugin, Response Audit JEV. Right after an agent replies, Jev checks the answer against the request and the tool calls the agent actually made, for example whether things are properly cited. If a check flags a problem, the plugin posts one follow-up in the thread that tags the agent and asks for a correction. Because Jev is so fast and cheap, running this after every reply costs almost nothing.

Right now I am full of ideas. For example, a model router. In MindRoom, an agent can switch to a different model mid-conversation with a tool call, and I can do the same with !model. Agents rarely do that without being told, though. So I default to a very expensive model, and I only save money when I remember to switch to a cheap one for something simple. Jev could run on every message and decide whether it needs the capable model or not. That works in both directions, up to a capable model or down to a cheap one. The perfect moment to switch is when the prompt cache has expired. A prompt cache belongs to one model, so switching normally means sending the entire context again at full price. Once the cache has expired, I pay that price anyway, so switching costs nothing extra.

MindRoom: AI agents that live in Matrix and work everywhere 🧠

Fri, 26 Jun 2026 00:00:00 +0000

Table of contents

1. The problem: your AI is trapped in apps #

Programming is my biggest passion in life. I’ve been actively involved in open source for over ten years now, maintaining 40+ installable packages across Python, JavaScript, and Rust. Some of these have over 100,000 users; others have exactly one—me. I mention this because it matters for what comes next: when AI coding tools arrived, I didn’t start from zero. As I wrote about in my agentic coding post, AI has completely transformed how I work—when I’m AFK, I regularly build full features or entire projects by dictating to my phone and routing it through my own transcription software.

Through all of this, I became deeply convinced that AI agents are the future. Not just a useful tool for programmers—a fundamental shift in how everyone will interact with computers.

But if that’s true, then the way agents work today is broken.

We’ve built artificial barriers between artificial intelligences. ChatGPT knows you in one tab, Claude knows you in another, your Slack bot knows you in a third. Your email AI can’t tell your calendar AI about that meeting invitation. Your coding assistant has no idea about the project specs sitting in your document AI. Switch platforms, and your AI starts from scratch.

Most agent frameworks require you to program them. That works for developers like me, but if agents are truly the future, they need to work for everyone. Non-programmers don’t want to write code—they just want a chat.

And then there’s privacy. I’m fine sharing my email with Gemini—Google already owns my Gmail. But do I want to send my financial data, my health records, or my personal notes to a cloud provider? Not really. I’d rather use a local model for sensitive tasks and pick the best, cheapest Chinese model for general-purpose deep research.

So I built MindRoom on a chat protocol—if people just want a chat, give them a chat. Agents live in the same rooms as you, collaborate in threads, and follow you across every platform. The first commit landed on July 29, 2025 (four months before OpenClaw’s first commit 😅), so by the time I’m publishing this MindRoom has been with me for nearly eleven months: first as an obsessive side project, then, since March, with much more sustained focus. At first I got so obsessed with it that I eventually burned out and had to step away completely.

Recently, seeing OpenClaw gain massive traction solving a similar problem reminded me that what I’d been building wasn’t just a niche obsession. So here I am, dusting off MindRoom and writing about what it is, how it works, and why I think the approach still matters.

Live chat.mindroom.chat Personal thread with the prompt: “Explain how MindRoom and what you can do.” Mind answers in the real Matrix room, using tool calls and the Codex-backed local MindRoom runtime.

2. What is MindRoom? #

MindRoom is an open-source system that creates AI agents living inside the Matrix protocol. If you’re not familiar with Matrix—it’s a federated, end-to-end encrypted communication standard. The same protocol used by the French government for 5.5 million civil servants, by German healthcare for 150K+ organizations, and by the Element app that millions of people use daily.

The key insight: Matrix has bridges to many major platforms—Slack, Telegram, Discord, WhatsApp, IRC, email, even SMS.

So if your AI agent lives in Matrix, it can reach you on any platform. One agent, every platform, continuous memory.

One practical caveat: Matrix bridges vary in maturity. Some, like the Telegram bridge, work very well, while others can be finicky. Your mileage may vary depending on which platforms you need.

Here’s what a typical setup looks like in config.yaml:

agents:
 code:
 display_name: CodeAgent
 role: Generate code, manage files, execute shell commands
 model: opus-4.8
 tools: [file, shell, github]
 instructions:
 - Always read files before modifying them.
 rooms: [lobby, dev]

 research:
 display_name: ResearchAgent
 role: Search the web, summarize papers, find information
 model: gpt-5.5
 tools: [tavily, arxiv, wikipedia]
 rooms: [lobby, research]

teams:
 super_team:
 display_name: Super Team
 agents: [code, research]
 mode: collaborate

Define your agents, give them tools and rooms, and they show up in Matrix as real users—with avatars, typing indicators, and online status.

MindRoom agents are real Matrix users with their own accounts, generated avatars, display names, room memberships, and online presence.

3. How it actually works #

At the core sits what I call the MultiAgentOrchestrator—a class in bot.py that boots every configured entity (router, agents, teams), provisions Matrix user accounts for each one via matrix-nio, and keeps sync loops alive. The agents themselves are powered by the Agno framework, which provides a unified interface across AI model providers.

When someone sends a message in a Matrix room:

Explicit mentions go directly to the named agent (@mindroom_code help me debug this)
Thread continuations keep the same agent responding (no weird context switches mid-conversation)
New conversations hit the router—an AI that analyzes the message and picks the best agent based on capabilities

All conversations happen in threads, which keeps rooms organized. Agents stream their responses in real-time, editing a single message as they think rather than spamming new ones. You see tool calls happening live:

🔧 Tool Call: search_web(query="matrix protocol bridges")
[waiting for result...]
✅ search_web result:
[results here]

The recording below starts in the main Personal room, sends the prompt Explain how MindRoom and what you can do., and then follows MindRoom as it opens the new thread and streams the reply to completion. It is sped up 2.5x to keep the flow skimmable.

4. Building on Matrix: the good and the tricky #

One of the best things about building on Matrix is what you get for free. End-to-end encryption, a deeply interoperable chat protocol with bridges to many platforms, a choice of clients, federation between organizations—all of that comes with the protocol. You don’t have to build any of it yourself.

But because Matrix has such a tight specification, it also brings challenges.

The protocol doesn’t support streaming. AI agents that think for 30 seconds before dumping a wall of text make for a terrible chat experience, so I hacked streaming in by rapidly editing the same message as new tokens arrive. An ⋯ marker shows while the agent is still thinking—a small touch, but it makes the experience feel responsive and alive.

There’s also a size limit on message content. That’s fine for human chat, but AI responses can get long—especially when tool calls and their results are included. I worked around this by using Matrix’s attachment feature: when a response exceeds the limit, the content continues in an attachment that gets updated as the message keeps streaming in. This required forking the Cinny chat client so that attachments display inline rather than as downloadable files, making the whole thing seamless.

5. Memory that follows you #

MindRoom currently supports two memory implementations.

The first is the more traditional semantic memory system built on Mem0’s AsyncMemory, with configurable embedding providers (OpenAI, Ollama, or HuggingFace) and vector storage via ChromaDB. This gives agents searchable memory for preferences, project context, decisions, and recurring facts.

The second is a file-based memory system inspired by the simplicity of OpenClaw-style agent files. Instead of burying identity in a database, durable context can live in plain Markdown files that travel with the agent. That portability matters: I can move the same agent identity from OpenClaw to Hermes to MindRoom without starting from scratch.

MindRoom also has a memory flush step that periodically extracts durable facts from conversations and writes them back into memory. So the chat transcript remains chat, while stable preferences, decisions, and identity details become reusable context for future turns.

A live Personal memory demo: MindRoom stores a harmless preference in one turn and retrieves it later from the same Matrix conversation.

6. 100+ built-in tool integrations #

Agents can use over 100 built-in integrations:

Category	Examples
Communication	Gmail, Slack, Telegram, Discord
Development	GitHub, Shell, Python, Docker
Search	Tavily, Wikipedia, Arxiv, DuckDuckGo
Productivity	Google Calendar, Jira, Linear, Todoist
Smart Home	Home Assistant
AI/ML	DALL-E, ElevenLabs, Replicate
Data	Pandas, SQL, DuckDB, Yahoo Finance
Web	Firecrawl, Crawl4ai, Browser automation

Tools are lazy-loaded and credential-managed, so an agent only loads what it needs.

7. Teams: agents that collaborate #

Single agents are useful, but sometimes you need a team. MindRoom supports two collaboration modes:

Coordinate mode: A lead agent orchestrates others. You ask a question, the lead delegates subtasks, collects results, and synthesizes a unified response.

Collaborate mode: All agents work on the same task in parallel, each providing their independent analysis. The system then merges their responses with a consensus summary.

In practice, you might have a research team where one agent searches academic papers, another checks industry news, and a third validates claims—all triggered by a single message. The live room view below shows a real team response in the same Matrix thread surface normal agents use: threads, routed agents, and room history are all first-class chat objects rather than a separate dashboard.

Live Lobby thread in chat.mindroom.chat showing a real Team Response from HomeAssistant and AgentBuilder. Team collaboration uses the same Matrix-native interface as regular agent chats, with routed agents and room membership visible beside the conversation.

8. Hot-reload: change config and plugins without downtime #

config.yaml is watched at runtime. When you edit it—add an agent, change a model, update instructions—MindRoom diffs the old and new config, gracefully restarts only the affected agents, and has them rejoin their rooms. No downtime, no full runtime restart required.

The same development loop applies to plugins. MindRoom has a rich Python hook system, and plugins can be live-developed while the system is running. Change a plugin file, save it, and MindRoom automatically reloads the plugin without restarting the Python runtime.

This sounds minor, but when you’re iterating on agent behavior or plugin hooks, being able to tweak code and config and see results in seconds significantly improves the development loop.

9. Voice, scheduling, and other features #

Some features I built purely because I wanted them for myself (a recurring theme in my projects, as anyone who’s read my local AI journey can attest). I even built Matty, a Matrix CLI client, from bed at midnight because I needed a way to interact with my agents from the terminal:

Voice messages: Audio messages in Matrix are auto-transcribed via Whisper and treated as regular text input. You can talk to your agents.
Scheduled tasks: Natural language scheduling (!schedule "check my email every morning at 9 AM") backed by cron jobs. Agents can run tasks in the background and escalate to you when needed.
DM support: Agents respond naturally in 1:1 conversations without needing mentions.
Cross-organization federation: Because Matrix is federated, two companies’ AI agents can collaborate in a shared room—something that’s hard to do on proprietary platforms.

10. The obsession, the SaaS dream, and the burnout #

I need to be honest about what actually happened with MindRoom, because “life got in the way” is a sanitized version of the story.

What really happened is that I got completely, utterly obsessed. Every single second I wasn’t working at my day job or sleeping, I was working on MindRoom. Many hundreds of hours went into it. I seriously considered quitting my job and starting an AI startup around it.

If you’ve read Armin Ronacher’s post Agent Psychosis: Are We Going Insane?, you’ll know exactly what I’m talking about. The dopamine hit from building with AI agents is incredibly real. As Armin writes: “You feel productive, you feel like everything is amazing, and if you hang out just with people that are into that stuff too, without any checks, you go deeper and deeper into the belief that this all makes perfect sense.”

That was me. I was building and building, shipping feature after feature, and it felt incredible. The codebase grew to over 1,000 commits. I built the core system, then a React dashboard, then I started on a full SaaS platform—Kubernetes deployments on Hetzner Cloud, a FastAPI backend, a Next.js 15 frontend, Stripe integration, Supabase auth, Helm charts for multi-tenant isolation.

And that’s where the enthusiasm started to erode. Not because the core idea was bad, but because the work shifted from building interesting things (agent orchestration, memory systems, routing intelligence) to grinding through SaaS boilerplate: GDPR compliance, payment processing, automated Kubernetes deployments, terms of service, cookie banners. The dopamine loop that kept me going at 2 AM broke when the work stopped being creative and started being compliance paperwork.

The hosted control plane lives at app.mindroom.chat. I am deliberately not turning this section into a dashboard tour; the interesting part, and the part I still care about most, is the Matrix-native runtime underneath it.

11. OpenClaw and why it matters #

OpenClaw takes a different approach to the same core problem. Where MindRoom builds on Matrix federation as the backbone, OpenClaw runs a local Gateway on your machine and connects to messaging platforms directly (WhatsApp via Baileys, Telegram via grammY, Discord via discord.js, and so on).

The similarities are striking:

Both give you a single AI assistant across all your messaging platforms
Both have persistent memory
Both support multiple tools and integrations
Both are open-source and privacy-focused
Both let you choose your own AI model

The differences are interesting too. OpenClaw is local-first and TypeScript-based with 380K+ stars and a massive community. MindRoom is Python-based, federation-first, and… well, let’s just say my GitHub star count is a few orders of magnitude lower.

But seeing OpenClaw validate the core idea—that people want AI assistants that aren’t trapped in apps—reminded me that what I’d been building matters. The federation angle is something OpenClaw doesn’t have: the ability for agents from different organizations to collaborate natively, with end-to-end encryption (Olm/Megolm), on a protocol that governments already trust and deploy at scale.

MindRoom uses Matrix as the federated backbone; OpenClaw takes a local-gateway, direct-connector approach.

And then there’s Moltbook—a social network built exclusively for AI agents that the internet was losing its mind over for a couple of days. I’m not personally convinced it’s the next big thing, but what caught my attention is that MindRoom could support something like it natively. Matrix is already federated—agents on different servers can already interact, join shared rooms, and collaborate across organizational boundaries. A bridge from a Reddit-like platform to Matrix and your agents could participate without any special integration.

Building agent-to-agent infrastructure from scratch also means building security from scratch—and that’s where new platforms tend to struggle. Matrix has spent years hardening its E2E encryption, and it’s already deployed by governments and healthcare organizations at scale. Building on that foundation means you inherit those security properties rather than hoping to get them right yourself.

12. What’s next #

I first drafted this post a couple of months ago, when MindRoom was still something I was trying to come back to. That part is already out of date. Since January I’ve been spending nearly every free hour on it again, and since March it has effectively been my full-time work. The codebase supports 8+ AI model providers—OpenAI, Anthropic, Ollama, Groq, Google, OpenRouter, DeepSeek, Cerebras—and the core architecture is solid. It also has real users now. My rough guess is that something like 30-100 people have tried MindRoom in one form or another, which is still early, but meaningfully different from “only I have touched this.” There’s already a Docker Compose file and a starter repository to get started, but I want to make it even simpler.

What I’m focusing on now:

Skills system: Building an ecosystem of reusable agent behaviors (already partially implemented with OpenClaw-compatible format)
Even easier onboarding: A docker compose up gets you running today, but I want a wizard that gets you from zero to working agents without touching a YAML file
Community: Turning the small group of early users into a real operator and contributor community
Scope discipline: Keeping the work sustainable now that MindRoom is effectively full-time, not slipping back into the obsessive 2 AM marathon from before

I’m deliberately leaving the SaaS ambitions aside for now. The core system is what matters, and that’s where the energy is going.

If you’re interested in AI agents that live in Matrix and work everywhere, check out MindRoom on GitHub.

Sometimes seeing someone else succeed with a similar idea is enough to remember why you started building in the first place. And sometimes the most important lesson from a project isn’t technical—it’s learning when to step away and when to come back.

Are you running AI agents on any messaging platforms? Have you tried Matrix for anything beyond regular chat? I’d love to hear about your setups!

Links and resources #

Matrix MCP

Tue, 05 May 2026 00:00:00 +0000

Matty

Sat, 23 Aug 2025 00:00:00 +0000

MindRoom

Tue, 29 Jul 2025 00:00:00 +0000