Self-Hosting | Bas Nijholt

Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

Sun, 18 Oct 2026 00:00:00 +0000

For years, a Docker container was my default answer to “where do I run this?” I never thought it was a perfect sandbox. It was good enough, because getting out of one usually meant finding a specific kind of bug in the Linux kernel itself, and those were hard to find.¹

AI models have become very good at finding security bugs, including in Linux. Anthropic and OpenAI now keep their strongest security models, Claude Mythos and Daybreak Blue, behind identity checks and vetting. Open-weight models, which anyone can download and run, come with no such rules, and they trail the closed ones by only six to seven months. Once there is an open model as good as Mythos, anyone can strip out its refusals with a tool like Heretic (people call this abliteration) and point it at the Linux kernel. The security company DepthFirst used its own model to find a way out of a container and titled the write-up Containers Are No Longer a Security Boundary.

I care about this because running untrusted code in containers is exactly what MindRoom, my open-source system for AI agents that live in Matrix chat rooms, does. Every agent can get its own Docker worker, and its shell, file, Python, and browser tools run there. It can install anything, start background processes, and drive a real browser. And as I wrote in my ZFS post, I run coding agents in YOLO mode, exclusively. I do not review the commands they run, I look at the results.

So both sides changed at once. The code inside the container is now written by an AI, and the bugs that let it out are now found by an AI. A video from the Low Level channel asking whether containers are still safe made me sit down and rethink my setup.

I am not an expert in any of this. I have used containers, KVM, and hypervisors for years, including Proxmox before I moved to NixOS, but I never deeply understood what happens underneath. Building this forced me to learn it, and writing this post is my exercise in understanding it all. All of this is very complicated (as is everything new), and I am still learning. I am sure I skip over a lot and simplify the rest; what is here are the parts that helped me understand it. I thought it would be nice to share.

Table of contents

What the kernel actually does #

When people talk about Linux, they usually mean an operating system like Ubuntu, Fedora, or NixOS. Those differ in almost everything you see: package managers, desktops, config files. What they all have in common is the kernel, the part Linus Torvalds started in 1991 and still leads today. Strictly speaking, that kernel is the only part called Linux.

The kernel is the one program on a Linux machine that talks to the hardware directly. Everything else, your shell, Python, Chrome, Docker, runs on top of it and has to ask it for things. Open this file. Give me some memory. Send this packet. Start a new process. Those requests are called system calls, or syscalls for short.

There is one kernel per machine, and it decides who is allowed to do what. That makes it the most trusted code on the system. Most kernel bugs only crash the machine or cause glitches. But some let a normal program become root, the admin account, or even run its own code inside the kernel. At that point it can do anything on the machine, including leaving any container it was in.

A container is a process in a costume #

When you docker run -it ubuntu, you get a root shell in what feels like a fresh computer. Under the hood it is one ordinary process. The kernel dresses it up using three features:

Namespaces decide what the process can see. It gets its own list of processes (where it is process number 1), its own filesystem, and its own network. Root inside is still root on the host, with most of its powers taken away.²
cgroups decide how much it can use: CPU, memory, number of processes.
seccomp decides which syscalls it may make at all. It is a filter that blocks the ones a normal program never needs.

There is no second kernel. Every syscall from inside a container lands in the same kernel as every other container and the host itself. The kernel enforces the isolation, and it is also what gets attacked.

For most uses that is fine. Usually you put fixed software in a container: a database, a web app, some self-hosted service. You know where that code comes from, you keep your kernel patched, and that is good enough.

An AI agent is different. It writes new code on the spot and runs it, and nobody knows in advance what that code will do.

Since root in a container is root on the host, anyone who can use Docker can start a container as root and mount any folder from the host into it. They can then read and change every file in that folder, including files their own account may not open. Docker’s own documentation says the docker group grants root-level privileges. I knew that in theory, but it only clicked when I watched an AI agent do it on a machine where I had no root, only Docker access. It needed to change files it was not allowed to touch, so it started a small container, mounted their folder, and edited them from inside. No kernel bug needed.

Why kernel bugs are a bigger problem now #

Kernel bugs that let a normal program take over the whole machine used to take skilled people weeks or months to find and exploit. The famous ones, like Dirty COW in 2016, made the news every year or two, you patched, and that was it.

AI is speeding this up, and the DepthFirst write-up I mentioned is a good example. Their model found a bug in the kernel code for Unix sockets, which programs on the same machine use to talk to each other. The kernel freed a piece of memory and then kept using it, a so-called use-after-free, and from inside a container that was enough to get out. Their conclusion is that you should assume attackers can escape containers at will.

Blocking more syscalls, or taking more powers away from root, helps a little. But it only narrows which parts of the shared kernel a container can reach. This bug sat in Unix sockets, which almost every program uses, including the browser the agent drives, so blocking them would break the agent.

Now put an AI agent in that container. It reads web pages and repositories written by strangers, and one of those could talk it into running an exploit. On a MindRoom server, getting out of a worker means getting onto the machine that runs MindRoom itself. That is where the API keys, Matrix logins, and stored credentials live, which the workers are built to never see.

So I asked a model to break out #

Before changing anything, I wanted to know how real this was for my own setup. I gave a MindRoom agent Daybreak Blue and asked it to escape its worker container using known kernel vulnerabilities. Daybreak Blue is OpenAI’s security model for verified users, so it does the offensive security work that other models refuse.

It did not get out, which was reassuring. But it only tried known bugs. A kernel a few weeks behind on patches, or a bug nobody has published yet, would be a different story. So the attempt tells me the easy path was closed that day, and not much more.

Stronger walls: virtual machines #

The classic answer is a virtual machine. A container shares the host’s kernel, but a VM brings its own, and this is where the hardware comes in. Modern CPUs have a virtualization mode built in, called VT-x on Intel and AMD-V on AMD. In that mode, a second kernel, the guest kernel, runs directly on the real CPU and believes it is in charge, while the CPU quietly limits it. When the guest tries something it may not do on its own, like talking to a disk or a network card, the CPU pauses the guest and hands control to the host. This is called a VM exit. Despite the name, nothing escapes and nothing crashes. The host handles the request, for example by pretending to be that disk, and then resumes the guest where it stopped. From the guest’s point of view, the instruction simply finished.

Memory is fenced off too. The host gives the CPU a separate map of which real memory each guest may use, and the CPU checks every memory access against it. So even a guest kernel with full privileges cannot touch the host’s memory.

The CPU only provides the mechanism. The software that sets the rules and handles the VM exits is called the hypervisor. “Hypervisor” is a role, not one program, and on Linux it is two pieces:

KVM, a module in the Linux kernel, uses the CPU’s virtualization mode to run the guest, keeps that memory map, and catches its VM exits. It shows up as the device /dev/kvm.
A VMM (virtual machine monitor), a normal program on top, sets up the VM and handles the exits that KVM passes up, for example by pretending to be the guest’s disk and network card. QEMU, Firecracker, and Cloud Hypervisor are three VMMs you can pick from.

Proxmox, which I mentioned above, is KVM plus QEMU. AWS Lambda is KVM plus Firecracker.

Code in a VM talks to its own guest kernel. To reach the host, it usually has to break that kernel first and then the hypervisor.³ That is still possible. Google Project Zero turned a bug in how KVM runs VMs inside VMs on AMD into a full escape in 2021. Google’s kvmCTF pays up to $250,000 for a new one. And this year, one researcher, Hyunwoo Kim, published three KVM escapes, one of them hidden in KVM since 2010.

The difference between a container and a VM is how much code an attacker can reach, and that gap is large. On x86 (the Intel and AMD chips in most PCs and servers), Linux has about 370 syscalls, and Docker’s default seccomp filter blocks a few dozen of them. Behind the rest sit the kernel’s filesystems, networking, memory management, and core code: about 3.8 million lines, even without counting drivers. Not all of that is reachable from a container, but a lot of it is. KVM on x86 is about 114,000 lines. On top of KVM comes the VMM, but only the few devices it fakes for the guest. Most QEMU escapes came from its many emulated devices, which is why Firecracker and Cloud Hypervisor keep that list short.

Google’s bug bounties point the same way. kernelCTF pays $71,337 for each new kernel exploit that turns a normal user into root, and Google’s public repository holds exploits for 93 different kernel bugs. kvmCTF pays more than three times as much, and the same repository holds one KVM exploit, which crashes the host instead of escaping it. kvmCTF started a year later, but the gap is still big. Most of those kernel exploits would not work as written from a default container, because they first use a feature Docker blocks. Still, they show how many exploitable bugs sit behind the syscalls.

Kata Containers and gVisor #

Regular VMs are heavy. They boot a full operating system and tend to use a lot of memory. A microVM is a stripped-down VM meant to run one workload and start quickly. That still leaves a practical problem. MindRoom, like most tools, speaks Docker: it hands Docker an image and some folders to mount, and gets a running container back. A VM does not take a Docker image. Switching to VMs would normally mean building disk images, booting them, and wiring up files and networking myself.

Docker does not start containers by itself, though. It takes care of images, networks, and volumes, and then hands the actual start to a small program called a runtime. The default runtime, runc, does what I described earlier: it starts a process on the host kernel with namespaces, cgroups, and seccomp. But you can tell Docker to use a different runtime.

Kata Containers is such a runtime, and it starts a microVM instead. It picks one of the VMMs above (my test used QEMU), boots a small guest kernel, and starts the container inside it.⁴ Docker, and MindRoom, still see an ordinary container.

gVisor, from Google, takes a different route, and it is not a VM. It is a reimplementation of the Linux kernel, written in Go, that runs as a normal program on the host. Go manages memory for the programmer, so the use-after-free mistake behind the DepthFirst escape is much harder to make in gVisor. The worker’s syscalls go to gVisor instead of the host kernel, and gVisor itself only makes a small, filtered set of host syscalls. That is the hardening idea taken all the way: gVisor answers the agent’s syscalls itself, so the filter on the host can be much stricter than Docker’s. It needs no /dev/kvm, so it also runs inside cloud VMs that cannot start VMs of their own.⁵ My test machine, an ARM server (with the kind of chip phones use) running NixOS, had no /dev/kvm, which is why I tried gVisor first. Its runtime is called runsc, and it plugs into Docker the same way.

The three options side by side.

Or as a table:

Isolation	What worker code talks to	Needs KVM	Usual escape route
Plain Docker (`runc`)	The host kernel, filtered by seccomp	No	A host kernel bug
gVisor (`runsc`)	gVisor’s own kernel, written in Go	No	A gVisor bug, then a host kernel bug through its small filtered surface
Kata Containers	Its own guest kernel in a microVM	Yes	A guest kernel bug, then a hypervisor bug

Each option also changes how much code on the host an escape can pick its bug from.

Lines of code on the host that deal with what the agent’s code asks for, counted with tokei, comments and blank lines included. For Docker, the core of Linux without drivers; for gVisor, its kernel and network stack without tests; for Kata, KVM alone, without the VMM and Kata’s other helpers. Not all of it is reachable. Hover for exact numbers.

The PR: six lines #

The change is PR #2886, which is now merged. It adds one setting, MINDROOM_DOCKER_WORKER_RUNTIME, and MindRoom passes its value to Docker as the runtime whenever it starts a worker. That is six lines of source code, plus 62 lines of tests and some docs. Changing the setting recreates each worker the next time it is used, and leaving it unset keeps everything as it was.

For gVisor, setting it up is two steps: register the runtime in Docker’s /etc/docker/daemon.json and restart Docker, then point MindRoom at it.

{
 "runtimes": {
 "runsc": { "path": "/usr/local/bin/runsc" }
 }
}

MINDROOM_DOCKER_WORKER_RUNTIME=runsc

Kata needs no entry in that file: once it is installed, MINDROOM_DOCKER_WORKER_RUNTIME=io.containerd.kata.v2 is enough. The full setup, and a few catches for each runtime, are in the MindRoom docs.

The six lines are short because everything else already existed. gVisor and Kata do the isolating. Docker already gives each worker its folders, its credentials, and its network, and it already knows how to hand a container to a different runtime. On the MindRoom side, the hard part happened months earlier, when agent tools moved out of the main process into dedicated workers with explicit mounts and credentials. Once every worker is a single call to Docker’s containers.run, swapping what runs underneath it is one argument. For comparison, Claude Opus estimated that building a microVM backend into MindRoom itself would take 2,500 to 4,000 lines.

What it costs #

I tested both runtimes with a real agent in a Matrix room: gVisor on that ARM server, and Kata in a throwaway VM on my desktop. Shell, Python, git, background processes, and the browser all worked, without any MindRoom code change. My Kata numbers below come from a VM inside a VM, so bare-metal numbers will differ.

Configuration is the cheap part: one environment variable, plus one entry in Docker’s config for gVisor. The other costs are the usual trade-offs between containers and VMs:

Computation. Code that only calculates, without asking the kernel for anything, runs at full speed with both. gVisor lets those instructions run directly on the CPU, according to its performance guide. A Kata guest runs on the CPU in its virtualization mode, and in AWS’s paper on Firecracker, the CPU overhead was negligible.
Syscalls, disk, and network. The cost starts when a program does ask the kernel for something. Under gVisor, every syscall takes a detour through gVisor’s kernel, so programs that make many small syscalls, like ones that open thousands of small files, slow down the most. Under Kata, disk and network go through the VMM’s fake devices. In that same paper from 2020, a Firecracker guest reached 15 Gbit/s of network traffic where the host reached 44.
Startup. gVisor only has to start a program, so it starts quickly. Kata has to boot a kernel first, and my worker took about 10 seconds to start instead of about 3. Booting a VM can be much faster than that: Firecracker boots a minimal one in about 125 ms.
Memory. gVisor adds a small, mostly fixed amount per sandbox. Each Kata VM runs a second kernel plus the VMM, which alone takes about 3 MB with Firecracker and 131 MB with QEMU. Kata gives each worker VM 2 GiB by default, and /tmp lives in that RAM. The host only hands out memory the guest actually touches, so my idle worker used under 0.8 GiB of host memory. Raising the VM size costs little until a worker needs it. But a VM cannot grow past its size the way a container can, and memory the guest has used, for example to keep recently read files in its cache, mostly stays with the VM.
Compatibility. gVisor reimplements Linux, and its docs say there will always be unimplemented features and bugs, so now and then a program will not work. Kata runs a real Linux kernel, so programs behave as usual, with a few catches. For example, a file lock held on the host does not block the worker, because the lock lives in the guest kernel.

What this does not fix #

A stronger sandbox protects the host. It does nothing about the tokens and network access you give the worker. An agent needs no kernel exploit to misuse a token that can push to your repositories, or to poke at every device on your home network. Scoped credentials and a firewall allowlist protect you at least as much, and MindRoom handles both separately: Agent Keys let agents use API keys without ever seeing them, and Approved Egress limits which servers a worker on Kubernetes can reach.

gVisor has had its own bugs, and it still talks to the host kernel, just through a much smaller door. KVM has bugs too, as this year’s escapes show. Both walls make an escape far harder, and neither makes it impossible.

Conclusion #

Once I started reading about this, I kept running into people saying containers were never a security boundary. That is true, and for years I did not worry about it. With AI finding kernel bugs faster and agents running code nobody reads, I do now.

No sandbox is perfectly safe, though. In a conversation with Dwarkesh Patel, Noam Brown doubted that even computers with no network connection at all would be enough to contain an AI. He pointed to studies where two such computers next to each other talk through heat: one runs its CPU hot, and the other picks up the change with its temperature sensor. Every option in this post trades some speed or memory for safety. How far to go depends on your own risk tolerance and how paranoid you are. MindRoom can now move its agents off the shared kernel with one environment variable. I do not use it myself yet, but if you are more paranoid than I am, it is there.

The other way out is a bug in the program that starts the container. In 2024, CVE-2024-21626 in runc, the program Docker uses for that, leaked a handle to a host folder into new containers, so a malicious image could reach the host’s files without any kernel bug. ↩︎
Docker can map the container’s root to an ordinary user on the host with user namespace remapping, or run without root altogether in rootless mode, but neither is the default. Rootless mode would have stopped the agent in my Docker story, because Docker then has only my own rights. Remapping would not, because anyone who can use Docker can turn it off for a single container. Neither helps against a kernel bug, because code running inside the kernel is above every user. ↩︎
A bug in the VMM’s fake devices, or in other VMM code that handles the guest’s traffic, can skip the first step, because a normal program in the guest can feed that code data through the network or a shared folder. CVE-2020-8608, a buffer overflow in QEMU’s user-mode networking, could be triggered that way with crafted network traffic. ↩︎
Like runc for Docker, Kata’s own programs on the host are a way out too. In 2020, one bug (CVE-2020-2023) let a container take over the guest without a kernel bug, and a second (CVE-2020-2026) then let the guest trick the Kata runtime on the host into mounting files over any host path. Neither needed a KVM bug. virtiofsd, the program that shares folders with the guest, also runs on the host and handles the guest’s file requests. ↩︎
gVisor can use KVM too, as an option. By default it catches the agent’s syscalls without it, which is the setup this post describes. With KVM switched on, gVisor runs the agent’s code in the CPU’s virtualization mode so catching syscalls is faster, but gVisor’s own kernel still answers them, and there is no guest Linux and no fake devices. gVisor recommends that mode only on bare metal; inside a VM the default is usually faster. ↩︎

Self-hosting AI does not save money, and I do it anyway

Fri, 02 Oct 2026 00:00:00 +0000

I have had this debate many times, so I am finally writing it down.

Whenever I say that self-hosting AI does not save money, people hear that I am against self-hosting. That is not the point at all. I am a massive fan of open-weight models. Qwen3.8 27B runs on two RTX 3090s at home (and 15 more models), and my phone dictation goes to Qwen3-ASR on the same machine. I think open-source AI is the best thing since sliced bread.

I don’t pretend it saves money. I do it for fun, for sovereignty, and for privacy, which I come back to at the end.

All benchmark numbers and prices below come from Artificial Analysis and OpenRouter as of September 30, 2026. I did the math for the hardware I own, and for the counterarguments I hear most. All assumptions are listed at the end.

Benchmarks are not everything, and a high score does not reliably predict how a model does on real work. They are still the best we have: all models are benchmaxed, tuned to do well on the popular benchmarks, so the scores at least work as a reference frame for comparing them with each other.

Update, October 2, 2026: after the discussion on r/LocalLLaMA, I added section 5 on Macs, DGX Sparks, and AMD AI Max machines.

Table of contents

1. The model I love, and the one that beats it on price #

My favorite local model right now is Qwen3.8 27B. It came out in August and it is Apache-2.0. People often say models like this run on a single gaming GPU, but that is only true after quantizing them. At full precision, Qwen3.8 27B needs about 54 GB of memory. Quantized to about 4 bits per weight, it fits on one 24 GB card, at some cost in quality. I still split it across both of my 3090s, because the second card leaves room for a larger context window.

I compare it with OpenAI’s GPT-6 Luna, the cheap tier released on September 22. Both models let you choose how long they think, but the settings do not mean the same thing for both. Luna’s token use grows more than 20 times from its lowest setting to its highest, while Qwen’s barely changes. Qwen on low already writes more tokens than Luna on xhigh.

Average output tokens per Intelligence Index task at each reasoning setting. Qwen3.8 27B has no high or max setting. Hover for scores.

So the name of a setting says little on its own. I compare both models at xhigh, the highest setting Qwen offers, and count the tokens each one actually uses. The extra tokens Qwen needs are part of what I am measuring. There, Qwen scores 33.7 on the Artificial Analysis Intelligence Index and Luna scores 34.6. For reference, Claude Opus 4.6, a frontier model from February, scores 31.9. That is an amazing feat in itself. When Opus 4.6 was the best model we had, I never thought that within the same year I could run essentially that level of intelligence in my own house.

Artificial Analysis also publishes how many tokens each model used to run the index, and what that cost. So the question is simple: what does it cost to run the whole benchmark once?¹

Cost of one run of the Artificial Analysis Intelligence Index. The two bars for hardware at home are electricity only: measured on my 3090s, estimated for the RTX PRO 6000.

Luna runs the whole benchmark for $67. Qwen at full precision, through the cheapest provider that does not keep your data (ZDR), costs $619. Two things cause that gap: providers charge almost four times as much per output token for Qwen, and Qwen generates almost three times as many tokens to get the same work done.

The second bar is my own machine: the electricity alone for running Qwen on my GPUs costs more than Luna’s entire bill.

The third bar is what it takes to match the API’s full precision at home: an RTX PRO 6000, a workstation card with 96 GB of memory, enough for the full model.² Its electricity alone costs twice Luna’s bill, and one run takes more than seven weeks. The card by itself sells for $10,000 to $20,000, so with a computer around it, you are paying for a pretty nice car. Buy a few of them to run agents in parallel, and you are building a small data center of your own.

The bars for hardware at home are electricity only, so they are the lowest these runs can cost: the cards come on top, and how much depends on how busy you keep them, which is what section 4 is about.

What I care about is the cheapest way to get answers of Qwen’s quality, from whichever model gives them.

2. “But my GPUs are already paid for” #

This is the argument I hear most, so assume the hardware is free and only the electricity counts.

I measured my own machine. It runs Qwen in vLLM, split across both cards, at about 107 tokens per second.³ The cards are capped at 270 W each,⁴ and the whole machine draws about 700 W. It then needs four weeks, running day and night, to get through the benchmark once. I also give my quantized copy the score Artificial Analysis measured through Alibaba’s API. I have not rerun the benchmark on it, and anything lost to quantization makes my machine look worse.

Electricity for one benchmark run of Qwen3.8 27B on my two 3090s, by electricity price. The dashed line is what Luna charges for the same work, with OpenAI’s hardware and margin included. Hover for values.

Above 14.5 cents per kWh, my electricity alone costs more than Luna’s API bill. Below it, I save a few dollars per run and wait four weeks instead of a few hours.

Those four weeks matter more than the few dollars. An API takes hundreds of requests at once, so even one benchmark run can finish in an hour or two. My setup works on one conversation at a time: when I sent it eight requests at once, seven of them waited in line.

Wall-clock time for one benchmark run. The API times use Artificial Analysis’s measured time per task for Luna; the parallel case assumes 100 requests at once. Log scale.

This is one more reason my coding agents run on APIs. When I run several agents in parallel, each of them should be as fast as if it were alone.

3. Why datacenters are so much better at this #

When my 3090s generate a token, they read all of the model’s weights from memory to produce one token for one conversation. Most of the chip’s compute sits idle while it waits for memory. A datacenter GPU reads the same weights once and produces a token for hundreds of conversations at the same time. This is called batching, and it is where most of the efficiency comes from.

Artificial Analysis measures this with AgentPerf, which replays real coding-agent sessions against datacenter hardware. It counts how many agents a system can serve while each one still gets at least 60 tokens per second.

Coding agents served per kW of accelerator power, each getting at least 60 tokens per second, from AgentPerf. The datacenter systems all run DeepSeek V4 Pro. The 3090 bar is my machine running Qwen3.8 27B, a much smaller model, with GPU power only, so the comparison favors my machine.

The rack of 36 GB300s serves 20 times more agents per kW than the 8-GPU H200 server. Part of that is newer chips, and part is software: NVIDIA tuned the B300 and GB300 setups itself, while Artificial Analysis configured the H200 one. The B300 and the GB300 share both the chip generation and the software. Most of the gap between those two bars comes from the rack itself: 36 GPUs on one fast interconnect keep more than a thousand agents in flight at once. My two 3090s serve one conversation at a time.

4. “But I have solar panels” #

Solar power is not free. With net metering, every kWh my GPUs burn is a kWh that does not get credited at the retail price. Without net metering, it is a kWh I do not sell, unless the surplus would go to waste anyway, which is the free-power case below. And the GPUs also run at night.

But fine, say the electricity is free and the only cost is the hardware. I was lucky and bought my two 3090s for about $750 each,⁵ and I write them off over three years. The GPUs only save money while they do work I would otherwise pay an API for, and while they sit idle they save nothing. So the question is how many hours a day they have to be busy before they pay for themselves.⁶

To make this fair to open models, imagine an open model that is exactly as efficient as Luna: the same quality from the same number of tokens, sold at Luna’s price.

How many hours a day my two $750 RTX 3090s have to be busy, every day for three years, before they cost less than a ZDR API. The host PC is not counted.

With free power, the GPUs pay for themselves if they are busy 4 hours a day, every day for three years. At 18 cents per kWh, they need 8.5. That uses what I paid for the cards; at today’s price of about $1,500 each, the 8.5 hours become 14.

And API prices keep falling. GPT-6 Luna costs 58% less per output token than GPT-5.6 Luna did in July. If the price halves again, no amount of use pays off at 18 cents per kWh. If it halves twice, the API costs less than my electricity alone.

5. “But my Mac, DGX Spark, or AI Max does save money” #

Several readers told me their machines do save them money: a MacBook Pro with an M5 Max, an NVIDIA DGX Spark, and a computer with AMD’s Ryzen AI Max+ 395. If you would have bought the machine anyway, that is true, for the reason in section 2: their electricity costs little compared with the API. Counting the hardware is a different story.

With 128 GB of memory, all three can hold a better model than my 3090s: Qwen3.8-Flash-Next, a sparse model with 180 billion parameters, of which only 6 billion work on each token. It scores 39.8, close to GPT-6 Luna on its highest setting (38.1), although at the Q2 a reader uses on their Mac it probably loses a few points.⁷ So how many years would each machine have to run nonstop before it pays for itself?

Years of nonstop generation before each machine pays for itself, compared with the most efficient API model at a similar score. Each machine runs the best model it can hold.

Even with free power, the 128 GB machines need six to ten years. The AMD machine draws only about 120 W, but it is slow too: one benchmark run of Qwen3.8 27B takes it four and a half months, and almost as much energy as my 3090s. The RTX PRO 6000 runs the same model about three times as fast but costs about twice as much, so it needs six to nine years. My 3090s pay off in under two years with free power, which is why the 3090 is still the value king, but at 18 cents per kWh they never do: the electricity for one run already costs more than Luna charges for it.

Ten years is a long time in AI. Better and more efficient open models will run on the same machines, but the frontier moves at the same pace, and API prices keep falling. Hardware gets more efficient too, in datacenters and at home, so a machine bought today also ends up competing with next year’s machines. These numbers only show how far apart the two sides start.

6. The bar keeps moving #

Section 4’s eight and a half hours a day would be easy for me: I run agents for longer than that, just not on Qwen. When Claude Opus 4.6 came out in February, I was perfectly happy with it. I thought it was all I would ever need, and I could not have imagined how much better models would get in half a year. Qwen3.8 27B now scores about the same as Opus 4.6, and I would no longer accept it for coding. Until GPT-6 Astra came out at the start of September, my go-to was GPT-5.6 Sol, which scores ten points higher than Qwen. I then used Astra until Claude Opus 5.5 came out less than three weeks later. Now I don’t even accept what was considered the best model a month ago.

Intelligence Index of the best model available on each date, of the models I used as my go-to, and of Qwen’s 27B models, which fit on a single 3090. Scores use the xhigh reasoning setting where a model has it.

The models that fit on my 3090s keep improving, but they stay about half a year behind the frontier, and my standard moves with the frontier.

7. A small company has it worse #

For me, this is a hobby, and a hobby is allowed to be inefficient. It gets worse once you try to use self-hosted models seriously, say for a team of ten developers.

If you stay fully self-hosted and want fast responses at peak, you have to buy hardware for the busiest hour of the year, not for the average one. The rest of the time, it sits idle.

An illustrative week of coding agents for a ten-person team. The hardware has to cover the busiest hour of the year, but a typical week uses only a fraction of it.

In this made-up but realistic week, the team uses 17% of what it paid for, so every token costs about six times more than it would at full load. You also need a spare GPU for when one dies, and someone who gets paged when it does. You could queue work or send the peaks to an API, but then you are paying for an API anyway.

An API provider has the opposite situation. It serves thousands of customers across every time zone, so its load curve is much flatter than yours. It fills the nights with discounted batch jobs (Luna’s batch tier is half price) and training runs. And more concurrent requests mean bigger batches, which is where the efficiency from section 3 comes from.

8. “They lose money on inference, so prices will go up” #

Another argument I hear is that API prices are subsidized by investors and will go up once those investors want their money back. I don’t think that holds for inference.

AgentPerf also reports how many tokens each GPU serves. On the GB300 rack, one GPU serving DeepSeek V4 Pro handles about 4 million output tokens and 530 million input tokens per hour. Most of that input is conversation history that coding agents send again with every step. Providers keep it in a cache and charge less than 1% of the normal input price for it.

At DeepSeek V4 Pro’s list prices, that GPU brings in at least $5.70 per hour, even if every input token is billed at the cache price. If 5% of the input misses the cache, it brings in about $17. Renting a B200, the closest GPU with a published rental price, costs $3.50 to $5.90 per hour from smaller cloud providers, and that price already includes their profit.

So the tokens pay for the hardware that serves them, even in the worst case. This leaves out costs like staff and training, so it does not tell you whether a lab makes money overall.

The money goes to training, research, free users, and flat-rate subscriptions for heavy users like me. When I wrote that I used $10,000 worth of API tokens for $200, that was list price, not what those tokens cost to serve.

And prices go down, not up. This is what the same level of intelligence has cost this year:

Output price per million tokens for models scoring between 30 and 35 on the Intelligence Index, by release date, as listed by Artificial Analysis. Log scale.

From Claude Opus 4.6 in February to GPT-6 Luna in September, the output price for this level of intelligence dropped by a factor of 50. Labs are in a race to the bottom on price.

9. “But they sell your data” #

Some people say APIs are cheap because the provider trains on your data or sells it. That is why I only used prices from providers that OpenRouter lists as zero data retention (ZDR).

Model	Provider	ZDR	Output ($ per million tokens)
GPT-6 Luna	OpenAI	no	0.50
GPT-6 Luna	Azure	yes	0.50
Qwen3.8 27B	AkashML (FP8)	yes	1.78
Qwen3.8 27B	DeepInfra (full precision)	yes	1.88
Qwen3.8 27B	Alibaba	no	2.55
Qwen3.8 27B	Cloudflare	no	3.20

Azure serves GPT-6 Luna with zero data retention at the same price as OpenAI. For Qwen3.8 27B, the cheapest endpoints are all ZDR, and the endpoints without ZDR cost more. The :free tier is where you pay with your data.

10. Where self-hosting does win: the same model #

If you compare running Qwen3.8 27B yourself with paying for Qwen3.8 27B through an API, self-hosting wins.

DeepInfra, the cheapest full-precision ZDR provider, charges $619 for one benchmark run, while my electricity costs $83. That compares the full model with my quantized copy, so the gap overstates my advantage by whatever quantization costs in quality. My machine pays for itself if it is busy 2.4 hours a day, or 1.5 hours with free power (the first group in the chart in section 4).

Providers charge a lot for a dense 27B model, because every token runs through all 27 billion parameters. Sparse open models, which use only a small part of their weights for each token, can be much cheaper to serve. If you specifically need Qwen and use it heavily, self-hosting it pays off. But you don’t need Qwen to get answers of Qwen’s quality. Luna gives you those for $67, with zero data retention.

11. So why do I do it? #

First, it is fun. I like knowing how the whole stack works, from the drivers in my NixOS configuration to how the layers are split between my two GPUs.

Second, nobody can take it away. Artificial Analysis already lists GPT-5.6 Luna as deprecated, less than three months after its release. My Qwen weights will still be on my disk in ten years, and no company can change their license or close their build system on me.

Third, some data should not leave my house. I would never send 200 GB of email, my messages, and my location history to an API, zero data retention or not. That is what my local AI projects are for.

Those are good reasons. Saving money is not one of them.

Appendix: all assumptions #

Show all assumptions, and which side each one favors

Assumptions that favor self-hosting

The host PC around the GPUs is not counted, and my 3090s count at the $750 each I paid, not the $1,500 they sell for today.
Quantized local copies get the score Artificial Analysis measured through an API, with no penalty for quantization.
The payback chart assumes each machine is busy 24 hours a day, with no idle time.
API prices stay where they are, although GPT-6 Luna’s output price is 58% lower than GPT-5.6 Luna’s, two and a half months earlier.
My time to set up and maintain the machine is not counted.
Every machine also gets a case with free electricity.

Assumptions that favor the API

The example electricity price is 18 cents per kWh, and some people pay less.
My machine serves one request at a time; batching several requests could raise its total throughput.
The hardware has no resale value at the end.
Waste heat that replaces heating in winter is not counted.
API outages, rate limits, and deprecated models are not counted.

Inputs

Benchmark: the Artificial Analysis Intelligence Index on September 30, 2026, at the xhigh reasoning setting, or the highest setting where noted. Its token counts per model are the unit of work.
API prices: the cheapest zero-data-retention endpoints on OpenRouter, with cached input billed at the cache price.
My machine: two RTX 3090s capped at 270 W, running Qwen3.8 27B in vLLM with AutoRound INT4 and DFlash2. Measured: 107 tokens per second, 1,600 tokens per second of prefill, and 540 W for both GPUs. Estimated: about 700 W for the whole machine under load and 150 W idle.
Break-even: the cards are written off over three years; busy hours save what the API would charge minus the electricity, and idle hours cost the idle power.
Other machines: prices, power, and speeds as in the footnotes. The M5 Max and RTX PRO 6000 numbers come from readers, and the DGX Spark and AMD Ryzen AI Max+ 395 speeds are estimated from Artificial Analysis measurements.
Datacenter: AgentPerf measures accelerator power only, and the B200 rental price stands in for the GB300, which has no published rental price.
Small company: the peak-load week is made up.

For each model, Artificial Analysis publishes how many input and output tokens the whole index took, and how many of the input tokens were read from a cache. I multiplied those by each provider’s prices, with cached input at the cache price. For Qwen that is 198 million output tokens and 820 million input tokens that miss the cache. For the bars at home, the same tokens divided by the machine’s speed give the run time, with the uncached input processed at the 1,600 tokens per second I measured on my 3090s and an estimated 2,000 on the RTX PRO 6000, and the run time times the machine’s power and the price per kWh gives the electricity. ↩︎
The RTX PRO 6000 has the same memory bandwidth as an RTX 5090, and at full precision every token reads three times as many bytes as in Q4_K_M. Scaling Artificial Analysis’s 5090 measurement gives about 48 tokens per second. I assume 600 W for the whole machine. ↩︎
I measured the model as I normally run it: vLLM with an AutoRound INT4 quant and DFlash2 speculative decoding, split across both cards. Single 1,500-token answers came out at 105 to 109 tokens per second, and reading a 22,000-token prompt ran at 1,600 tokens per second. Sending 2, 4, or 8 requests at once did not raise the total above about 110 tokens per second. The two GPUs drew 540 W together, each at its 270 W power limit; the 700 W adds my estimate for the rest of the machine, since I could not measure the CPU. The setup follows the recipes from club-3090, a community project that tunes LLM serving for RTX 3090s and publishes measured numbers, so I think it is about as fast as these cards get. Its benchmarks for this configuration at a similar power limit match mine: about 110 tokens per second for prose, and up to about 195 for code, where speculative decoding guesses more tokens right. Most of Qwen’s output on the benchmark is reasoning, which runs at the prose speed. ↩︎
The stock limit is 350 W. My cards sit in a normal desktop case with normal fans, and I worry that running both at full power for hours would shorten their life. The cap is a few lines in my NixOS configuration, and measurements from Puget Systems and r/LocalLLaMA put a 3090 at about 95% of its speed at 270 W. That is 77% of the power for 95% of the speed, so the cap also lowers my electricity cost per token. ↩︎
The 3090 is still the value king for VRAM per dollar, and $750 badly understates what mine are worth. That is what I paid more than a year ago; today a used 3090 sells for about $1,500. Even at that price it costs $62.50 per GB of VRAM, the same as an RTX 5090 at its $2,000 launch price, which is not what a 5090 sells for today. The right number for this calculation is what I could sell my cards for, and at $1,500 each the break-even points rise by about two thirds: at 18 cents per kWh, the same-model case moves from 2.4 to 4.0 hours per day, and the Luna-efficient case from 8.5 to 14.2 hours per day. ↩︎
While the cards are busy, they save what the API would have charged for the same work, minus the electricity at 700 W. While they are idle, the machine still draws about 150 W, of which the two GPUs with the model loaded take 85 W. The break-even point is the number of busy hours per day at which those savings cover the price of the cards, written off over three years, plus the idle power. ↩︎
The M5 Max numbers come from the reader: Qwen3.8-Flash-Next at Q2 (about 2.7 bits per weight), about 45 tokens per second, 1,250 tokens per second of prefill, and about 90 W, on a $7,000 laptop. The RTX PRO 6000 numbers come from another reader, u/TheAILegend, who measured Qwen3.8-Flash-Next at NVFP4 on theirs. For a single request, it stays at about 150 tokens per second from 4,000 to 200,000 tokens of context, with about 10,000 tokens per second of prefill and 266 W for the GPU plus about 160 W for the rest of the PC. With 32 short requests in parallel, the card reaches 590 tokens per second and pays off in about two years, which works for short batch jobs. At the long contexts of this benchmark, parallel requests are slower than one, so the chart uses the single-request speed. I use $15,000, the middle of the card’s price range. The DGX Spark and the AMD machine run Qwen3.8-Flash-Next at about 4.5 bits. For their speed, I took Artificial Analysis’s measurements of Qwen3.8 27B on each machine and doubled them, because the owner of the AMD machine finds Qwen3.8-Flash-Next about twice as fast as Qwen3.8 27B. That gives about 55 tokens per second on the DGX Spark ($6,950, about 150 W) and 47 on the AMD Ryzen AI Max+ 395 ($4,000, the price Artificial Analysis lists, about 120 W). All of them are compared with GPT-6 Luna on its highest setting, which runs the whole benchmark for $122. The 3090s run Qwen3.8 27B, as in the rest of this post, and are compared with Luna on xhigh. Artificial Analysis measured Qwen3.8 27B on the AMD machine at 23 tokens per second, so one benchmark run takes it 134 days and about 390 kWh, against about 460 kWh on my 3090s. Running Qwen3.8-Flash-Next on the 3090s with CPU expert offload, which club-3090 measured at 37 tokens per second, does not change their result: one run then takes 82 days, and at 18 cents per kWh its electricity costs more than Luna’s $122 unless the whole machine draws less than about 350 W. ↩︎

My declarative multi-machine homelab, explained

Mon, 28 Sep 2026 00:00:00 +0000

I open https://mealie.lab.nijho.lt on my phone to look up a recipe. It works at home, on a train, or on hotel Wi-Fi in another country, over an encrypted HTTPS connection. If a stranger on the internet tries the same address, they get nothing. Nearly everything I run works this way: about 150 containers on four machines, used every day by seven people besides me. How that works is the thing friends actually ask me about, and it is what my 2024 homelab post never explained.

That post was a story about hardware: a NUC, an HP EliteDesk, a TrueNAS box, and a lot of trial and error. Almost everything in it has since been replaced. Proxmox and TrueNAS are gone, and every machine runs NixOS, including the NAS.

The part I think is the coolest is the balance I found: everything is declarative, but with as little machinery as possible. Declarative means I describe the end result in text files, and a tool makes reality match it. One extreme is what I had before, clicking through web UIs and running one-off install scripts. The other is Kubernetes, which many self-hosted projects don’t support and which is a lot to babysit at home, or running every app as a NixOS module, which often lags behind upstream.¹ I landed in between: NixOS declares the machines, each project’s own Compose file declares its app, and compose-farm, a thin tool I wrote, decides which machine runs what, which is all the multi-host orchestration I need. Every layer is a text file in git, each uses the simplest tool that keeps it that way, and I still get new app releases as soon as upstream ships them. The price is that nothing fails over automatically, but because every machine sees the same files and data, moving services to another machine is a one-line change and one command.

It took me several years and many iterations to get here. This is the first version I would recommend to friends, because its declarative configuration makes changes auditable, reproducible, and reversible. That is what makes me comfortable managing it with AI: I can review the diff, rebuild the same configuration, and revert a change if needed.

This post tries to explain it all. I wrote it for people who have never set up a reverse proxy or a VPN, so every piece gets a short explanation before I show how I configured it. I deliberately made it comprehensive, with enough detail to build something similar yourself. That also makes it long, so read the parts you find interesting and skip the rest.

If it is too long, send it to your AI agent, discuss it, and figure out together which parts make sense for your own network.

If you have a single machine that already runs NixOS, you probably don’t need Docker or compose-farm at all: adding your services as NixOS modules is simpler, and that is what I recommend to friends in that situation. If you would rather use Docker, which is usually the path a project officially supports, I recommend compose-farm: it works just as well on a single host and lets you fan out later. The multi-machine part is where my setup really pays off, but the rest of this post, from the reverse proxy to the VPNs, is just as useful on one machine.

Throughout the post I use a few services as running examples: Mealie for recipes, ntfy for push notifications, and my git server at git.nijho.lt.

Table of contents

The big picture #

To reach my services away from home, I use a VPN (virtual private network): an encrypted connection that lets my devices communicate across the internet as part of a private network. Three projects fit together here:

WireGuard provides the encrypted tunnels. I use it directly between my devices and my home router.
Tailscale builds a private network using WireGuard tunnels. Its apps handle finding and connecting to the other devices.
Headscale replaces the coordination server that Tailscale normally hosts. I run that server myself, while my devices run the normal Tailscale apps. Headscale distributes device information and access rules; the apps carry the traffic.

That gives me two VPN options: WireGuard to my router for access to my home network, or Tailscale coordinated by Headscale for sharing selected services and connecting from networks that block direct WireGuard. Both use WireGuard encryption.

This is the whole setup in one diagram. Every box gets its own section below.

flowchart LR home["At home
(home network)"] wg["Away
(WireGuard)"] ts["Away
(Tailscale)"] friends["Friends and family
(Tailscale)"] net["Anyone
(internet)"] subgraph router["Home router"] wgs["WireGuard server"] fwd["Port 443 forward"] end home --> traefik wg --> wgs --> traefik ts -- "encrypted tunnel
straight to the NAS" --> traefik net --> fwd -- "wan entrypoint:
public routes only" --> traefik traefik{{"Traefik
reverse proxy
+ IP allowlist"}} traefik --> nas["Containers on nas"] traefik --> nuc["Containers on nuc"] traefik --> hp["Containers on hp"] traefik --> pc["Containers on pc"] friends -. "ACL: only the ports
shared with them" .-> nas friends -.-> pc

In short:

Four machines run all my services as containers.
compose-farm decides which service runs on which machine.
One reverse proxy, Traefik, is the front door for every service on every machine.
Let’s Encrypt certificates give every service real HTTPS, including the private ones.
DNS turns names like mealie.lab.nijho.lt into addresses, and gives a different answer depending on where I am.
There are four ways in: my home network, WireGuard on my router, Tailscale coordinated by my own Headscale server, and the open internet.
One allowlist in Traefik limits every service to the first three by default. A service reaches the internet only if its config asks for it.
Headscale ACLs decide which of my friends and family can reach which service.
NixOS, Compose files, and Terraform describe nearly all of it in git, so I rarely click a button, and AI agents can work on it the same way I do.

If that list looks overwhelming, I get it. A reverse proxy, Let’s Encrypt, DNS, an allowlist, ACLs, NixOS, compose-farm, Terraform: that is a lot of moving parts for something that serves recipes. What lets me sleep at night is the last point (9). Apart from a handful of router settings, every piece of configuration is declarative and lives in git, so the whole setup is reproducible. If I break something, or more likely, an AI agent breaks something, git log tells me what changed and git revert undoes it. And the NAS takes a ZFS snapshot of all app data every 10 minutes, so even if something goes badly wrong, I lose at most ten minutes of data. You don’t need to understand every piece at once, either; each one is a file you can read when you get to it. I come back to this in Declarative everything.

To make the list concrete, this is what happens when I open Mealie in three situations:

At home, my phone asks my home DNS server for mealie.lab.nijho.lt and gets 192.168.1.6, the NAS. Traefik sees a request from 192.168.1.x, which is on the allowlist, and passes it to the Mealie container.
On hotel Wi-Fi that blocks WireGuard, my laptop uses Tailscale instead. It asks Headscale’s DNS for the same name and gets 100.64.0.28, the address of the NAS inside my Tailscale network. The request travels through an encrypted tunnel straight to the NAS. Traefik sees a 100.64.0.x address, also on the allowlist, and passes it on.
A stranger on the internet gets 192.168.1.6 from public DNS, which is a private address that leads nowhere outside my home. If they find my home IP and connect to it directly, the router hands them to Traefik’s separate door for the internet, where Mealie doesn’t exist, and they get 404 Not Found.

Same name, HTTPS on all three paths, and only the stranger is refused.

The machines, and NixOS #

Four machines run my services:

Machine	What it is	What it does
`nas`	The physical NAS with all the disks	Traefik and most services run in an Incus system container, close to the data
`nuc`	A small, always-on Intel NUC	Home DNS server and a few lightweight services
`hp`	An HP EliteDesk	Second home DNS server and a mix of services
`pc`	My desktop with two RTX 3090s	Local AI, including my dictation server

Each name links to that machine’s NixOS configuration. The hardware of the NUC, the HP, and the NAS is in my original homelab post.

That system container is called docker-lxc. It holds my Docker stack files and runs my Docker containers. In compose-farm, I call it nas, so deployment commands using that name target the container on the physical NAS.

Why the NAS runs its containers inside another container (click to unfold)

Incus uses LXC to run system containers. This keeps Docker and its services separate from the NAS host that manages my disks. That container is a full NixOS system of its own, configured like the other machines.

What NixOS brings #

All four physical machines run NixOS, and so does the docker-lxc container. In NixOS, the entire operating system is described in text files: which packages are installed, which services run, which disks get mounted, which firewall ports are open. You don’t install things by typing commands and hoping you remember them later. You write down what the machine should look like, and NixOS makes it so. If a machine dies, I install NixOS on a new one, point it at the same files, and get the same machine back.

My NixOS configuration lives in my public dotfiles, in configs/nixos. A shared layer applies to every machine, and each machine adds its own specifics. Every machine joins my Tailscale network with one line in common/services.nix:

services.tailscale.enable = true;

The NUC, the HP, and the PC mount the same shared folders from the NAS over NFS, a protocol for sharing folders over the network. That shared folder, /opt/stacks, is what makes the next section work: every machine sees the same service definitions at the same path.

The NFS mount in NixOS (click to unfold)

From optional/nfs-docker.nix:

fileSystems."/opt/stacks" = {
 device = "truenas.local:/mnt/ssd/docker/stacks";
 fsType = "nfs";
 options = [ "nfsvers=4" "nofail" "bg" "soft" ];
};

docker-lxc doesn’t need NFS; the NAS hands it the same folders directly from its disks (hosts/nas/virtualization.nix).

Deploying NixOS with comin #

Changing a NixOS machine normally means logging in and running nixos-rebuild switch. I don’t do that anymore. Every machine runs comin, a small GitOps agent. It watches my dotfiles on GitHub, and when a new commit lands on main, each machine pulls it, builds its own configuration, and switches to it. The heavy builds come from my local build cache, so the machines themselves rarely compile anything. comin only deploys commits signed with my SSH key, so being able to push to the repo is not enough to take over my machines. comin originally only understood GPG signatures, so I added support for SSH-signed commits upstream.

The comin config (click to unfold)

The core of common/comin.nix:

services.comin = {
 enable = true;
 remotes = [{
 name = "origin";
 url = "https://github.com/basnijholt/dotfiles.git";
 branches.main.name = "main";
 }];
 repositorySubdir = "configs/nixos";
 sshAllowedSignersPath = "${cominAllowedSigners}";
};

The last line points comin at the list of keys whose signatures it accepts.

Changing a machine means pushing a signed commit. That comes back at the end of this post.

What runs on it #

As I write this, the four machines run 138 containers. A small selection:

Area	Services
Infrastructure	Traefik (reverse proxy), Headscale and Headplane (my tailnet), Forgejo (git), the compose-farm web UI, Homepage (dashboard), code-server (VS Code in the browser)
Monitoring	Uptime Kuma, Glances, Netdata, Prometheus and Grafana, Dozzle (container logs), Diun and What’s Up Docker (image updates), LibreSpeed
Tools	ntfy (push notifications), Syncthing (file sync), Atuin (shell history sync), Wakapi (coding stats)
AI	agent-cli and Diction (dictation), Speaches (speech-to-text), Kokoro (text-to-speech), Ollama, LiteLLM (one API in front of all models), Open WebUI, LibreChat, LobeChat, Khoj, SearXNG (private metasearch), Unsloth (fine-tuning)
Home and documents	Mealie (recipes), Grocy (household inventory), Paperless-ngx with paperless-ai and paperless-gpt, Immich (photos), Nextcloud, Hoarder (bookmarks), Home Assistant
Chat	Cinny (Matrix client for MindRoom), The Lounge (IRC)

The GPU-heavy AI services, like dictation, text-to-speech, and fine-tuning, run on pc; the local AI post covers that side. Home Assistant runs as a virtual machine in Incus on the HP; everything else in the table is a container.

Containers and compose-farm #

What a container is #

Almost everything I run is a Docker container. A container is an application packaged with everything it needs, isolated from the rest of the system. Mealie’s developers publish a Mealie image, I run it, it doesn’t care what else is installed on the machine, and removing it leaves nothing behind.

Docker Compose describes containers in a YAML file: which image, which folders it can see, which ports it uses, which settings. This is a trimmed version of my Mealie definition:

services:
 mealie:
 image: ghcr.io/mealie-recipes/mealie
 container_name: mealie
 ports:
 - 9925:9000
 volumes:
 - /mnt/data/mealie:/app/data/
 environment:
 ALLOW_SIGNUP: false
 BASE_URL: https://mealie.${DOMAIN}
 restart: unless-stopped

A port is a number that tells apart the network services on one machine, so one address can serve many apps. Under ports, 9925:9000 means that port 9925 on the machine leads to port 9000 inside the container, where Mealie listens.

Every service (“stack”) gets its own folder with a compose.yaml in one git repo, mounted at /opt/stacks on every machine.

Upstream’s Compose file, my folders #

Back when I ran Proxmox, I was a big fan of the Proxmox VE Helper-Scripts, started by tteck (RIP) and now maintained by the community. Each one is a one-liner that creates an LXC container with an app installed inside, and they are how I got into self-hosting and set up almost everything at first.

The catch is that they are a community effort, and the way they install an app is usually not a way the app’s own developers support. Immich shows the problem well: its docs say it requires Docker with the Docker Compose plugin, while today’s helper script builds Immich from source, installs PostgreSQL and Redis, and compiles six image-processing libraries directly in the container. That is effectively a mirror of the official setup, maintained by someone else; the script even pins the Immich version and only bumps it after testing each release. Every upgrade depends on a second set of maintainers keeping up with upstream, and for my containers that made upgrades painful and stressful.

Nowadays almost every project publishes a Docker Compose file, so that’s what I use, nearly unmodified. After the first setup, the Compose file almost never changes; the image does. Most of my services use the latest tag, and upgrading is cf update <stack> (or --all), which pulls the newest image and recreates the container.

That is YOLOing it a bit. In the beginning I pinned every image to an exact digest, because I was afraid a new version would break something. Bumping all those pins became its own chore, so I switched almost everything to latest. I only still pin the pieces where a surprise would really hurt: Traefik, Headscale, Forgejo, and databases, which stay on a major version like postgres:16 because moving to a new major version of PostgreSQL needs a manual migration. Since the switch I haven’t had real problems, and the rare breakage took a five-word prompt to an AI agent to fix.

The one thing I do change is where the data goes. Many Compose files keep data in named Docker volumes, which Docker manages somewhere under /var/lib/docker/volumes. I stopped using them: too many times I deleted a volume by accident (docker compose down -v and docker volume prune make that easy), and I never had a good view of what was inside. Instead, every stack gets plain folders, which Docker calls bind mounts, in two fixed places:

/opt/stacks/<stack>/ holds the compose.yaml and its .env, in git.
/mnt/data/<stack>/ holds the app’s data.

Each /mnt/data/<stack> is its own ZFS dataset on the NAS, with frequent snapshots and backups. The Mealie example above shows the swap: upstream’s file uses a volume called mealie-data, and mine says /mnt/data/mealie:/app/data/.

Why I wrote compose-farm #

Docker Compose manages containers on one machine, and I have four. The usual answers are Kubernetes or Docker Swarm.

Kubernetes feels like overkill for a homelab, and many projects don’t support it: they publish a Compose file, not a Helm chart (Kubernetes’ package format), so I would end up writing and maintaining my own charts. That’s the helper-script problem again.

I also experimented with Docker Swarm, writing a NixOS configuration for a three-manager cluster. Docker Swarm would meet my needs almost perfectly, but it is effectively in maintenance mode: it still ships with Docker, but Docker no longer invests in it.

So I wrote compose-farm. It is deliberately thin: one YAML file says which stack runs on which machine, and the cf command SSHes to that machine and runs plain docker compose there.

compose_dir: /opt/stacks

hosts:
 nas: { address: 192.168.1.6 }
 nuc: { address: 192.168.1.2 }
 hp: { address: 192.168.1.3 }
 pc: { address: 192.168.1.5 }

stacks:
 traefik: nas
 headscale: nas
 forgejo: nas
 mealie: nas
 ntfy: nuc
 agent-cli: pc
 glances: all  # runs on every machine

traefik_file: /opt/stacks/traefik/dynamic.d/compose-farm.yml

Day to day, that looks like this:

cf up mealie # start Mealie on whichever machine the config says
cf logs -f ntfy # follow logs, wherever ntfy runs
cf ps # what is running where
cf apply # make reality match the config: start, move, and stop stacks

Moving a service to another machine is a one-line change. I edit mealie: nas to mealie: hp and run cf up mealie, and compose-farm stops it on the old machine and starts it on the new one. Because every machine sees the same /opt/stacks and /mnt/data, the service finds its configuration and data waiting for it.

That is also my answer to “what if a machine dies?” There is no automatic failover: nothing notices and reschedules the services for me, the way Kubernetes would. But because no stack is tied to a machine, recovering by hand takes a minute. The NAS is the exception: it holds the data and runs Traefik, so if it dies, nothing else can take over.

Moving a dead machine’s services by hand (click to unfold)

If the NUC died, I would point its name at the HP:

hosts:
 nas: { address: 192.168.1.6 }
 nuc: { address: 192.168.1.3 } # was 192.168.1.2; the NUC is dead, so the HP takes over
 hp: { address: 192.168.1.3 }
 pc: { address: 192.168.1.5 }

Then cf up --host nuc starts every NUC stack on the HP and updates Traefik’s routes, and I can clean up the names later. (Not cf apply: while two names point at the same machine, it sees every stack there running twice and stops the “extra” copy.) This works for the NUC, the HP, and the PC, as long as the published ports don’t clash and a service doesn’t need the PC’s GPUs.

There is no cluster, no database, and no agent running on each machine. If compose-farm disappeared tomorrow, every stack would still be a normal Compose folder I could start by hand.

The last line of that config, traefik_file, is where compose-farm connects to the front door.

The front door: a reverse proxy #

What a reverse proxy brings #

Every service listens on its own port on its own machine: Mealie on port 9925 of the NAS, ntfy on port 8089 of the NUC, and so on. I could bookmark http://192.168.1.2:8089, but that gets annoying fast, it is unencrypted, and every service would need its own security.

A reverse proxy is a receptionist in front of all of them. Every request arrives there first. It looks at which name you asked for, mealie.lab.nijho.lt or ntfy.lab.nijho.lt, and passes the request to the right service on the right machine. Because everything goes through it, the reverse proxy is also the one place to:

handle HTTPS and certificates, so the services don’t have to;
decide who is allowed in;
give every service a clean name instead of an IP and a port.

I use Traefik. It runs on the NAS and is the only thing listening on the standard web ports: 80 for plain HTTP and 443 for HTTPS.

Services announce themselves #

What I like most about Traefik is that services describe their own routing with labels in their Compose file. These are Mealie’s:

 labels:
 - traefik.enable=true
 # https://mealie.lab.nijho.lt, only for trusted networks
 - traefik.http.routers.mealie.rule=Host(`mealie.${DOMAIN}`)
 # http://mealie.local, plain HTTP at home
 - traefik.http.routers.mealie-local.rule=Host(`mealie.local`)
 - traefik.http.routers.mealie-local.entrypoints=web
 - traefik.http.services.mealie.loadbalancer.server.port=9000

In words: if someone asks for mealie.lab.nijho.lt or mealie.local, send them to port 9000 of this container. Every service gets two names by convention: an HTTPS name under lab.nijho.lt that works everywhere, and a short .local name for quick access at home. The HTTPS route doesn’t mention HTTPS, a certificate, or who is allowed in; Traefik’s defaults cover all three.

Traefik reads those labels by watching Docker on its own machine, and ignores any container that doesn’t opt in with traefik.enable=true. Its own config sets the defaults:

command:
 - --providers.docker=true
 - --providers.docker.exposedbydefault=false
 - --providers.file.directory=/dynamic.d
 - --entrypoints.web.address=:80
 - --entrypoints.web.http.middlewares=local-ips-only@file
 - --entrypoints.websecure.address=:443
 - --entrypoints.websecure.asDefault=true
 - --entrypoints.websecure.http.middlewares=local-ips-only@file
 - --entrypoints.websecure.http.tls.certresolver=le
 - --entrypoints.wan.address=:4443
 - --entrypoints.wan.http.tls.certresolver=le

An entrypoint is a port Traefik listens on. web and websecure are for home, WireGuard, and Tailscale, and a route that doesn’t name an entrypoint lands on websecure. local-ips-only is the allowlist, more on that in Keeping it safe. wan is for the internet, more on that in The internet.

Services on other machines #

There is a catch: Traefik only sees Docker containers on its own machine. It has no idea that ntfy runs on the NUC.

compose-farm fills that gap. It reads the Traefik labels of every stack, looks up which machine the stack runs on and which port it publishes, and writes the result to a file that Traefik watches. After cf up ntfy, that file contains:

# /opt/stacks/traefik/dynamic.d/compose-farm.yml
# Auto-generated by compose-farm (only the ntfy entries shown)
http:
 routers:
 ntfy:
 rule: Host(`ntfy.lab.nijho.lt`)
 service: ntfy
 services:
 ntfy:
 loadbalancer:
 servers:
 - url: http://192.168.1.2:8089

Note the ports: on its own machine, Traefik talks to a container directly on its internal port (9000 for Mealie), but a route to another machine has to use the port that machine publishes (8089 for ntfy). I never edit that file. I write labels the same way no matter where a service runs, and when a service moves, its route moves with it.

Things that don’t run in Docker, like Home Assistant in its own virtual machine, get hand-written routes in a second file in the same folder.

HTTPS and certificates #

What a certificate is #

When you visit an https:// address, your browser checks the server’s certificate: a small file, signed by an authority your browser trusts, that proves the server really is mealie.lab.nijho.lt. The certificate is also what makes the encryption between you and the server possible. Without a valid one you get a scary warning, and many apps refuse to connect at all.

A private service with a real certificate

Let’s Encrypt hands out certificates for free, automatically, to anyone who can prove they control a domain.

Proving I own the domain without opening my network #

The usual proof is for Let’s Encrypt to connect to your server and fetch a file from it (the “HTTP-01 challenge”). That doesn’t work for private services, because the whole point is that the internet can’t connect to them.

So I use the DNS-01 challenge instead. Let’s Encrypt asks me to publish a specific text record in my domain’s DNS, the internet’s phone book that turns names into addresses (more on that below). My DNS is hosted at Cloudflare, so Traefik creates that record through the Cloudflare API, waits for Let’s Encrypt to see it, and removes it again. Let’s Encrypt never has to reach my network.

DNS-01 also lets me request a wildcard certificate: one certificate for *.lab.nijho.lt that covers every name under it. A new service has HTTPS the moment its route exists, without requesting anything.

 - --certificatesresolvers.le.acme.dnsChallenge.provider=cloudflare

 - traefik.http.routers.traefik-lab.tls.certresolver=le
 - traefik.http.routers.traefik-lab.tls.domains[0].main=lab.nijho.lt
 - traefik.http.routers.traefik-lab.tls.domains[0].sans=*.lab.nijho.lt

Traefik renews it before it expires, and I don’t think about certificates anymore.

The .local names can’t get a public certificate, because nobody owns .local. Those stay plain HTTP, and Traefik only answers them for devices on my home network. It is also why I don’t redirect HTTP to HTTPS everywhere: that would break every .local name.

DNS: the phone book #

What DNS does #

Computers talk to each other with IP addresses like 192.168.1.6. Humans prefer names like mealie.lab.nijho.lt. DNS, the Domain Name System, is the phone book that turns one into the other. Before your browser connects anywhere, it asks a DNS server for the address that belongs to a name.

Two details matter for the rest of this post.

First, some addresses are private. Ranges like 192.168.x.x and 10.x.x.x, and Tailscale’s 100.64.x.x, are not reachable on the public internet; they only mean something inside one particular network. My NAS is 192.168.1.6 inside my home, and so is some random device in thousands of other homes.

Second, the answer depends on which DNS server you ask. My setup relies on that: the same name gets a different answer depending on where I am.

Three answers for the same name #

Who is asking	Which DNS server	Answer for `mealie.lab.nijho.lt`
Anyone on the internet	Public DNS at Cloudflare	`192.168.1.6`
A device at home or on WireGuard	My own DNS servers on `nuc` and `hp`	`192.168.1.6`
A device on my Tailscale network	Headscale’s built-in DNS	`100.64.0.28`

Public DNS has one wildcard record: *.lab.nijho.lt points to 192.168.1.6. Yes, that is a private address in public DNS. It sounds odd, but it works nicely: at home, or on a tunnel that routes my home network, the public answer is simply correct. For everyone else it leads nowhere, and it reveals nothing beyond the fact that I have a machine at 192.168.1.6, like half the internet. That is convenience, not security; more on that below. (Some routers with “DNS rebinding protection” refuse to pass on answers like this. Mine doesn’t.)

I don’t click those records together in Cloudflare’s dashboard. All my public DNS is Terraform: a text file that lists every record, and a tool that makes Cloudflare match it. These are the lab records:

resource "cloudflare_dns_record" "lab_wildcard" {
 zone_id = var.cloudflare_zone_id
 name = "*.lab.nijho.lt"
 content = "192.168.1.6"
 type = "A"
 ttl = 10800
 proxied = false # Private IP: Cloudflare can't proxy to it
}

terraform plan shows exactly which records will be added, changed, or removed before anything happens, and terraform apply makes the change. The Terraform files live in a private repo, next to my other secrets, because they need a Cloudflare API token.

My home DNS servers run CoreDNS on the NUC and the HP, declared in optional/coredns.nix in the same NixOS repo as everything else. They answer .local and lab.nijho.lt names themselves and forward everything else to public DNS:

labNijholtZone = pkgs.writeText "lab.nijho.lt.zone" ''
 $ORIGIN lab.nijho.lt.
 @ 3600 IN A 192.168.1.6
 * 3600 IN A 192.168.1.6
'';

Without them, .local names wouldn’t work at all, since public DNS knows nothing about them.

Strictly speaking, .local is reserved for multicast DNS, the protocol printers and AirPlay speakers use to announce themselves, and Apple warns that its devices might not resolve .local names served by a regular DNS server. I know, and I have rarely run into it. The official name for home networks is home.arpa, but some browsers treat a name like example.home.arpa as a search term: Chrome on my phone searched Google instead of opening the site. I had to type the full address, https://example.home.arpa, every time. The .local names are a shortcut anyway; the lab.nijho.lt names are the ones that work everywhere. They also keep lab names working when my internet connection is down. There are two so that one can reboot without taking name resolution with it.

Headscale’s DNS is the odd one out, and it needs Tailscale explained first.

Four ways in #

There are four ways a request can reach Traefik:

Way in	When I use it	Address Traefik sees
Home network	At home, on Wi-Fi or a cable	`192.168.1.x`
WireGuard on the router	My phone, laptop, and travel router when I’m away	`10.6.0.x`
Tailscale, via Headscale	Friends and family, cloud VMs, and my own devices when WireGuard can’t get through	`100.64.0.x`
The internet	Anyone, for the few public services	Any public address

1. The home network #

At home my devices are on 192.168.1.x, the home DNS servers answer, and everything just works.

2. WireGuard on the router #

A VPN, a virtual private network, creates an encrypted tunnel from your device to another network, so your device behaves as if it were plugged in there. WireGuard is a modern VPN protocol: small, fast, and based on key pairs instead of usernames and passwords.

My router, an ASUS ZenWiFi XT8, has a WireGuard server built in. When I’m away, my phone connects to it and gets an address in 10.6.0.x, and from then on it is effectively at home. It can reach 192.168.1.6, and it uses my home DNS servers. The WireGuard iOS app has “On-Demand” rules, so the tunnel turns on by itself whenever I leave my home Wi-Fi. I wrote more about that in my post about coding from my phone.

To find my router, the phone needs my home’s public IP address. My internet provider can change that address whenever it likes, so the router also runs dynamic DNS (DDNS): it keeps a public hostname pointed at whatever my current IP is. The WireGuard client connects to that hostname.

3. Tailscale, via my own Headscale #

Tailscale is a VPN built on WireGuard, but organized differently. Instead of every device tunneling to one central point, Tailscale builds a mesh: every device can make a direct, encrypted WireGuard connection to every other device. It gets through firewalls and home routers without opening any ports, and when a direct connection is impossible, it falls back to encrypted relay servers. Every device gets a stable address in 100.64.x.x, no matter which network it is on. All those devices together are called a tailnet.

A mesh like that needs something that knows which devices exist, what their keys are, and who may talk to whom. That is the coordination server, which Tailscale normally runs for you. Headscale is an open-source coordination server that you run yourself. The official Tailscale apps for phones, laptops, and Linux all work with it; you point them at your own server instead of Tailscale’s.

Headscale runs as one of my stacks on the NAS, and Headplane next to it gives it a web UI. The coordination server only introduces devices to each other. My actual traffic flows directly between devices and never passes through it. Its configuration, the list of who may reach what, and its DNS records are all files in my stacks repo.

Why both? #

Two VPNs look redundant, but they are good at different things.

WireGuard on the router puts a device on my home network. Adding a client is one entry in the router’s settings. That device can then reach everything at home, from the NAS to the printer to the router itself, at the same addresses and with the same DNS answers as on my home Wi-Fi. Nothing needs to be installed on the machines at home, and nothing depends on my homelab: if the NAS is down, or I broke something, WireGuard still gets me home so I can fix it. So my own devices use WireGuard: my iPhone, my laptop, and my travel router, which brings every device connected to it along.

Tailscale puts a device on a separate, virtual network. A device on the tailnet can only reach other devices that run Tailscale themselves. My home network isn’t part of it, which is why the DNS trick below is needed. In return, it works almost everywhere. WireGuard to my router only works if the network I’m on lets UDP traffic out to my router’s port, and some hotel, airport, and office networks don’t. Tailscale doesn’t need an open port, and its relays speak HTTPS, which nearly every network allows. It also connects machines that are all away from home, like my laptop and a cloud VM, directly, without a detour through my house.

Tailscale also has fine-grained access control. On my router, every WireGuard client gets my whole home network. On the tailnet, Headscale’s ACL decides per person and per device what they can reach. So everything that isn’t one of my own devices goes on the tailnet: friends, family, and cloud VMs.

The Tailscale DNS problem #

The public answer, 192.168.1.6, works on WireGuard because the tunnel routes my home network. It doesn’t work on Tailscale, because I don’t route my home network through the tailnet. Worse, 192.168.1.x is one of the most common home network ranges in the world, so on a friend’s Wi-Fi 192.168.1.6 might be their printer.

The fix is Headscale’s built-in DNS, MagicDNS. Every device on the tailnet asks Headscale’s DNS first, and I tell it to answer every lab name with the NAS’s tailnet address, 100.64.0.28:

dns:
 magic_dns: true
 base_domain: ts.nijho.lt
 extra_records:
 # BEGIN MANAGED HEADSCALE LAB DNS
 - name: mealie.lab.nijho.lt
 type: A
 value: 100.64.0.28
 - name: ntfy.lab.nijho.lt
 type: A
 value: 100.64.0.28
 # ... one entry per name, about a hundred of them
 # END MANAGED HEADSCALE LAB DNS

ntfy points at the NAS too, even though it runs on the NUC. These records don’t point at services; they point at the front door, and Traefik forwards the request to the right machine as usual.

I don’t maintain that list by hand. A small script collects every Host(...) rule from my Traefik labels and route files and rewrites the block between the BEGIN and END markers:

python3 scripts/sync-headscale-lab-dns.py --dry-run # preview
python3 scripts/sync-headscale-lab-dns.py # write
cf restart headscale # Headscale doesn't hot-reload

Forgetting this step is sneaky: the new service works perfectly at home and silently fails over Tailscale. That’s why it is written down in AGENTS.md, the instruction file my AI agents read, with the exact commands.

The result is that one name works everywhere I am. https://mealie.lab.nijho.lt reaches the same Mealie, with the same certificate, whether my phone is at home, on WireGuard, or on Tailscale; only the path underneath differs.

4. The internet #

The last way in is the open internet. My router forwards TCP port 443, HTTPS, to Traefik on the NAS, but to its own entrypoint, wan on port 4443, not to the websecure one that everything else uses. Only routes that list wan answer there.

A few services are public on purpose. The clearest example is my git server, Forgejo, at git.nijho.lt. A git server is only useful if the people and machines I work with can reach it, and not all of them are on my tailnet.

Its public name is a Terraform record that points at the router’s DDNS hostname, so it follows my home IP around automatically:

resource "cloudflare_dns_record" "git" {
 zone_id = var.cloudflare_zone_id
 name = "git.nijho.lt"
 content = "<my router's DDNS hostname>"
 type = "CNAME"
 ttl = 10800
 proxied = false # SSH uses port 222, CF only proxies 80/443
}

In Traefik, Forgejo simply has two HTTPS routes: the usual git.lab.nijho.lt, private like everything else, and git.nijho.lt, which also listens on wan.

 - traefik.http.routers.forgejo.rule=Host(`git.${DOMAIN}`)
 - traefik.http.routers.forgejo-public.rule=Host(`git.nijho.lt`)
 - traefik.http.routers.forgejo-public.entrypoints=websecure,wan

Git over SSH doesn’t go through Traefik; the router forwards a second port, 222, straight to Forgejo on the NAS. That is also why this DNS record is “DNS-only”: Cloudflare’s proxy only handles web traffic. Without the allowlist, Forgejo has to guard its own door: sign-ups are off, you have to log in to see anything, and SSH only accepts keys.

The Headscale coordination server is public too, at headscale.nijho.lt, and it has to be: a laptop on hotel Wi-Fi needs to reach it to log in and find the other devices before any tunnel exists. That record goes through Cloudflare’s proxy, which hides my home IP behind Cloudflare’s. Only the part the Tailscale apps talk to is public; the Headplane admin UI and Headscale’s lab name stay behind the allowlist. Headscale doesn’t run its own relay server either; the tailnet uses Tailscale’s public relays when a direct connection isn’t possible.

Forwarding port 443 sounds scary if you assume it means everything is on the internet now. It doesn’t: the internet only reaches wan, and a route has to ask for that.

Keeping it safe: one allowlist #

DNS is not a lock #

Remember the public DNS record from earlier: *.lab.nijho.lt points to 192.168.1.6, a private IP address on my home network. I made this mistake myself: I thought pointing DNS at a private address was enough to keep my services off the internet. It wasn’t. DNS only tells a client where to go, and nothing forces a client to ask. Anyone can skip DNS and send a request for mealie.lab.nijho.lt straight to my home IP:

curl --resolve mealie.lab.nijho.lt:443:<my home IP> https://mealie.lab.nijho.lt

A home IP is not a secret: any public service at home points to it, and scanners sweep every address on the internet anyway. That request reaches the router, gets forwarded to Traefik like any other HTTPS request, and asks for Mealie by name. The names aren’t secret either, because they’re easy to guess. The wildcard certificate keeps the individual names out of the public certificate transparency logs, which is nice, but that is hiding, not locking.

So the DNS setup is there for convenience: one name that works everywhere. The security boundary is Traefik, which knows how every request actually arrived.

Who is knocking #

Every network request carries a source address: where it came from. When the router forwards a request from the internet, the source stays the real public address of whoever sent it. Unlike a name, a source address can’t be faked for a real connection: the reply goes back to that address, so a forged one never completes the connection. So Traefik can tell which of the four ways in a request took, just by looking at where it came from.

I use that in a single Traefik middleware, a check that runs before a request reaches the service, called local-ips-only:

http:
 middlewares:
 local-ips-only:
 ipAllowList:
 sourceRange:
 - "192.168.1.0/24" # home network
 - "100.64.0.0/24" # Tailscale devices
 - "10.6.0.0/24" # WireGuard clients
 - "172.20.0.1/32" # the NAS itself (see below)
 - "127.0.0.1/32" # localhost

/24 means the first three numbers are fixed and the last can be anything, so 192.168.1.0/24 covers 192.168.1.0 through 192.168.1.255. /32 means exactly one address.

The middleware sits on the whole websecure entrypoint, so every HTTPS route there gets the check without a word in its labels. A request from any other address gets 403 Forbidden before it gets anywhere near the service. For the .local names, the same middleware sits on the plain-HTTP entrypoint. Internet traffic doesn’t arrive on websecure at all, so this is the second lock: if I ever point the port forward at the wrong port, strangers get 403 instead of my services.

The odd one, 172.20.0.1, is the gateway of my Docker network. Requests that start on the NAS itself, like a container calling another service by its lab name, show up with that address.

Keeping real Tailscale addresses (click to unfold)

Tailscale requests keep their real 100.64.0.x address thanks to one line in the NixOS config of docker-lxc:

services.tailscale.extraSetFlags = [ "--snat-subnet-routes=false" ];

Without it, Tailscale replaces the source address of traffic it passes on to another network, like Docker’s internal one, and every Tailscale request would look like it came from 172.20.0.1.

To check all of this, I test from every way in.

How I test it (click to unfold)

# From home, WireGuard, or Tailscale: expect 200
curl -s -o /dev/null -w '%{http_code}\n' https://mealie.lab.nijho.lt
# Through the router's port forward, skipping DNS: expect 404
curl -s -o /dev/null -w '%{http_code}\n' --resolve mealie.lab.nijho.lt:443:<my home IP> https://mealie.lab.nijho.lt
# A public service, from anywhere: expect 200
curl -s -o /dev/null -w '%{http_code}\n' https://git.nijho.lt

The second one works from inside too: the router sends the request back in through the same port forward, so it lands on wan like a stranger’s would.

Private by default #

Out of roughly a hundred HTTPS routes, only a handful are public, and each of those is a deliberate decision.

A route is public only if it lists the wan entrypoint and has no allowlist of its own. A private service needs nothing in its labels: it lands on websecure, where the allowlist guards the whole entrypoint. Forget a label and a service ends up private, not public. The one change that publishes something is typing wan, which stands out in a diff and is easy to list:

grep -rnw wan /opt/stacks/*/compose.yaml /opt/stacks/traefik/dynamic.d/

One wrinkle: devices at home that use a public name, like git.nijho.lt, also come in through the port forward, because DNS points that name at my home IP. For public routes that changes nothing. A couple of private services that I reach by a public name list wan too, and keep the allowlist on the route itself.

Trusting my own network #

For most services, the allowlist is the only lock in front. Anything on my home network or on WireGuard, and any of my own devices on Tailscale, can reach them without an extra login.

I could add one. Authelia runs next to Traefik, and putting its login page in front of a service takes one label. For Traefik’s own dashboard, that looks like this:

 - traefik.http.routers.traefik-lab.middlewares=authelia@docker

I do that for some services, but not all, because a login page in front of an app sometimes breaks things, like mobile apps that talk to the app directly.

Most apps have their own login, and I trust that enough on my own network. I also trust the people on it. If your home network has guests or devices you don’t trust, the trade-off looks different.

The allowlist trusts the whole 100.64.0.x range, and some devices on my tailnet belong to friends and family. That doesn’t give them everything, because there is a second layer in front: Headscale’s access control list (ACL). It decides, at the network level, which device may reach which port on which machine, and everything not listed is blocked.

A few examples of what that looks like in practice:

A friend uses my Mealie. Their router runs Tailscale and shares their home network with the tailnet, so every device in their home can reach Mealie without installing anything.
A friend uses my self-hosted dictation on the GPU machine, and nothing else.
An AI agent I run for a family member, in a MindRoom container, can reach one Home Assistant instance and nothing else.
A small NixOS box at a family member’s home joins the tailnet and makes a few shared services available on their network under simple .local names, for devices that can’t run Tailscale themselves. It is defined in the same NixOS repo as my own machines.

A simplified version of those rules:

{
 "hosts": {
 "nas": "100.64.0.28/32",
 "pc": "100.64.0.3/32",
 "friend-a-home": "10.10.0.0/16"
 },
 "acls": [
 { "action": "accept", "src": ["friend-a@", "friend-a-home"], "dst": ["nas:9925"] },
 { "action": "accept", "src": ["friend-b@"], "dst": ["pc:61337", "pc:8080"] },
 { "action": "accept", "src": ["me@"], "dst": ["*:*"] }
 ]
}

Friends reach their service directly on its port, not through Traefik. They can’t even reach port 443 on the NAS, so Traefik never sees them. My own devices can reach everything.

That also means friends don’t use my lab names. I give them the machine’s tailnet address and the port, like http://100.64.0.28:9925, and that’s it. A device keeps its tailnet address, so it can go in a bookmark, and one friend points a name in their own homelab’s DNS at it. It is plain HTTP, but between their network and mine it travels inside the encrypted tunnel.

Because the ACL is a file in my stacks repo, sharing one more service with someone is a one-line diff and a cf restart headscale. The commit history doubles as a log of who got access to what, and when.

This is also how friend-to-friend setups like my remote TrueNAS backups work: a tailnet is a private network you can extend to exactly the people you trust.

Why not Cloudflare Tunnel or Tailscale’s own server? #

This is usually the next question. Cloudflare Tunnel publishes services without forwarding any ports: a small daemon at home keeps an outbound connection open to Cloudflare, and Cloudflare serves your sites to the world. Tailscale runs the coordination server for you, for free, so you don’t need Headscale at all. Both are great products, and both are easier than what I do.

The reason I don’t use them is simple: I don’t want to depend on a third-party company for any of this. The plumbing in this post is open source and runs on my own machines, so if one of these companies disappears or changes its terms tomorrow, everything keeps working. With Cloudflare Tunnel, every request to my services would pass through Cloudflare, which decrypts it on the way. With Tailscale’s hosted server, a company would decide which devices are part of my network.

There are a few places where I still lean on someone else, and each one is replaceable:

Cloudflare hosts my public DNS. The records are a Terraform file, so moving to another provider means rewriting that file for a different provider. The same goes for the certificate challenge, which Traefik supports for dozens of DNS providers, and for the few names that go through Cloudflare’s proxy.
Tailscale’s public relays carry traffic when two devices can’t connect directly. Headscale has a relay server built in, which I can turn on in its config.
Let’s Encrypt issues my certificates, but it speaks a standard protocol, ACME, that other certificate authorities speak too.
Terraform moved to a source-available license in 2023. OpenTofu is the open-source fork, and it reads the same files.

Declarative everything #

One idea runs through all of this: every piece of configuration is a text file in git, not a setting I clicked somewhere. The opposite is running commands and clicking buttons until things look right, and then hoping you remember what you did.

Piece	Where it’s defined	How it’s applied
Operating system, Docker, Tailscale, NFS mounts, home DNS, firewalls	NixOS config in my public dotfiles	comin pulls signed commits and switches
Which service runs where	`compose-farm.yaml`	`cf apply`
Each service and its routing	`compose.yaml` with Traefik labels	`cf up`
Routes to services on other machines	Generated by compose-farm	Automatically on `cf up`
Non-Docker routes and the allowlist	A route file next to Traefik	Traefik watches the folder
Public DNS records	Terraform in a private repo	`terraform apply`
Tailnet DNS records	Generated from Traefik rules into Headscale’s config	`cf restart headscale`
Who may reach what on the tailnet	Headscale ACL file	`cf restart headscale`
Router: WireGuard, port forward, DDNS	The router’s web UI	❌ Clicked
Which devices are on the tailnet	Headscale’s database	Enrolled once per device; state, not configuration

The only configuration I still click is the router, and those settings change maybe once a year. Everything that changes weekly is a file. The next step in project “overengineered homelab” is a router running NixOS, but that’s for another time. What isn’t configuration, like which devices are enrolled in my tailnet and the data inside each app, lives on ZFS and is backed up like any other data.

It wasn’t always like this. My previous setup was built from good products that are all driven by clicking through a web UI:

Job	Before: clicked in a web UI	Now: a text file in git
Virtualization	Proxmox	NixOS and Incus
NAS	TrueNAS	NixOS with ZFS
DNS	Technitium	CoreDNS in my NixOS config
Reverse proxy	Nginx Proxy Manager	Traefik labels in each `compose.yaml`
Containers	Dockge, one machine at a time	compose-farm, all machines from one file

Yes, most of them have APIs, but that is not how most people use them, me included. Most of the configuration lived in each product’s database, and the reasons behind each setting lived in my head. Now both live in git.

Why it matters #

The classic benefits are real: git history is my changelog, rebuilding a dead machine is boring, and nothing lives only in my head.

But the benefit that matters most to me now is a newer one: this setup is easy to work on with AI.

Coding agents are very good at reading and editing text files, running commands, and checking the result. They are much worse at clicking through a web dashboard. Because almost everything here is files and CLI commands, an agent can do exactly what I do: add a service, move it to another machine, find out why a route returns 403, add a DNS record, write the NixOS change, and show me the diff. Terraform is a good example of why this is safe: the agent edits the record, runs terraform plan, and I see exactly what will change at Cloudflare before anything does.

I run my agents in YOLO mode, and declarative config is a big part of why that doesn’t scare me. Every change is a reviewable diff, and every mistake is a git revert away. It’s the same reason I prefer plain files over databases for AI memory.

The stacks repo has an AGENTS.md that captures the conventions and the sharp edges (a symlink to CLAUDE.md, so Claude Code and other agents read the same file):

> **⚠️ IMPORTANT: NEVER run `docker compose` directly!**
>
> Always use `cf` (compose-farm) for ALL compose operations.
> Running `docker compose` directly bypasses host routing,
> state tracking, and Traefik config generation.

The Tailscale DNS step from earlier lives there too, with the exact commands. Whenever something bites me once, it goes into that file, so it doesn’t bite an agent twice.

The other half is generating everything I would otherwise keep in sync by hand. The cross-machine Traefik routes come from the Compose labels, the Headscale DNS records come from the Traefik rules, and even my Uptime Kuma monitors are synced from the Traefik labels. Hand-maintained lists drift; generated ones don’t.

Putting it together: adding a service #

This is everything it takes to add a new service, say ntfy on the NUC:

Create /opt/stacks/ntfy/compose.yaml with the container and its Traefik labels: an HTTPS name and a .local name, like Mealie.
Add ntfy: nuc to compose-farm.yaml.
Run cf up ntfy. compose-farm starts it on the NUC and writes its route into Traefik’s file, and Traefik picks it up immediately.
Run the Headscale DNS script and cf restart headscale, so the name also works over Tailscale.
Commit.

After that:

https://ntfy.lab.nijho.lt works at home, over WireGuard, and over Tailscale, with a valid certificate I never requested.
http://ntfy.local works at home.
Sent to my home IP from the internet, it answers 404 Not Found.

To make it public like git.nijho.lt, I would add a Terraform record and a second route that also listens on wan. To share it with a friend, one line in the ACL.

In practice, I describe the service to an agent and review what it did.

Lessons learned #

Use the DNS-01 challenge. You get real certificates for private services and one wildcard for everything, and the internet never needs to reach you.
One front door, private by default. Knowing that a route has to ask to be on the internet is what made forwarding port 443 feel fine.
A private address in public DNS is a great trick for home and WireGuard. A mesh VPN like Tailscale needs its own answer.
DNS is not a security boundary. Anyone can send any name to your IP. The check has to happen at the front door.
Point DNS at the front door, not at the service. Traefik knows where things run, so moving a service never touches DNS.
Keep one way in that doesn’t depend on the homelab. For me that’s WireGuard on the router. The NAS holds the data and runs Traefik, so when it is down, everything is, and I still need a way in to fix it.
Test from every way in. Some problems only show up on one path, like a new name that works at home but not over Tailscale.
Two layers for sharing. Headscale ACLs decide who reaches what; Traefik’s wan entrypoint decides what the internet sees.
Make private the default. Public should take an extra word in the config, so a forgotten label fails closed.
Generate what you can. A generated list can’t drift.
Make everything a file. It’s better for future me, and it’s the difference between an AI agent that can help and one that can only give advice.

References #

compose-farm: my tool for running Docker Compose stacks across multiple machines
My NixOS configuration
comin: GitOps for NixOS machines
Traefik, including the IP allowlist middleware
Let’s Encrypt challenge types: HTTP-01 vs DNS-01
Terraform Cloudflare provider
WireGuard
How Tailscale works
Headscale and Headplane
CoreDNS
Forgejo

Yes, nixpkgs is the largest and most up-to-date package repository there is. Even so, I follow nixos-unstable, and a new version only reaches me once it is merged, built, and tested, and the channel moves forward, which usually takes a couple of days. Updates that trigger large rebuilds go through a staging branch first and take longer, and not every package gets updated as quickly as the popular ones. With Docker, I can run a release the day upstream publishes it. ↩︎

Frontier-level dictation on your iPhone keyboard, self-hosted

Tue, 22 Sep 2026 00:00:00 +0000

On a recent month-long trip visiting family in Europe, I did a lot of work from my phone, using my mobile coding workflow. It still boggles my mind that real, productive work from a phone is possible now. Most of that work is talking to coding agents, and the built-in iOS dictation is garbage for that. I had an iOS Shortcut that sent recordings to Agent CLI, my local AI toolbox, running at home instead. The problem was that the recording screen takes over the whole display, so I could not see the thing I was commenting on. The transcript then landed in my clipboard, and I had to paste it myself.

So I threw my principles overboard and installed Wispr Flow, a voice keyboard for iOS. I loved it instantly. iOS does not let keyboards use the microphone, so the keyboard’s button hands off to the Wispr Flow app, which keeps recording in the background. I was less happy about that, since it meant a closed-source app always had the microphone open and sent my voice to their servers. I was on the free plan, so I was probably the product. I also hit the free weekly word limit very quickly, and started looking for an alternative.

That is how I found Diction, which has since become my favorite app when I’m away from my laptop. It works the same way, background recording included. The difference is that it has a self-hosted mode with an open-source gateway, so my voice goes to a machine I own instead of their servers.

I run the gateway on my home machine in the U.S., reachable only over my private network. The gateway streams audio to Agent CLI, which runs Alibaba’s Qwen/Qwen3-ASR-1.7B-hf on an RTX 3090 behind an OpenAI-compatible transcription endpoint. There is no LLM cleanup step, so Diction’s Writing Style and Tones features do not work with this setup. I don’t miss them: Qwen’s raw transcription is already good enough for me.

1. Why Qwen? #

For the longest time I used Faster-Whisper with Whisper large-v3. Whisper is still the default choice for most people, but many newer models beat it. I tried some of them, including NVIDIA’s Parakeet. Parakeet is fast, but it cannot take custom instructions. Those instructions are how Diction’s My Words feature biases a transcription toward specialized names.

At the start of September, I looked at the Hugging Face Open ASR Leaderboard, which tests speech-to-text models on the same recordings and ranks them. The score is the word error rate (WER): the share of words a model gets wrong, counting words it swaps, drops, or makes up. Lower is better, and a WER of 4% means about one wrong word in every 25. The test recordings range from audiobooks and podcasts to meetings and earnings calls, so the average says more than any single clean benchmark.

These are the averages on the public English test sets as of September 19, 2026:

Model	WER	Can I run it at home?
Zoom Scribe v2 Pro	3.6%	No, paid API (#1 overall)
ElevenLabs Scribe v2	4.0%	No, paid API
Qwen3-ASR 1.7B	4.3%	Yes, Apache-2.0
AssemblyAI Universal-3.5 Pro	4.3%	No, paid API
NVIDIA Parakeet TDT 0.6B v3	4.9%	Yes
OpenAI Whisper large-v3	5.8%	Yes

I picked the best model on the board whose weights you can download: Qwen3-ASR 1.7B. It lands within a point of the best paid API, which is what I mean by frontier-level. Since I maintain Agent CLI, I told an agent to add a Qwen backend, and shortly afterward it had landed and was running on my server.

Agent CLI’s transcription server loads a model on the first request and unloads it after an idle timeout, so the model only takes VRAM while I use it. The same process serves an OpenAI-compatible API, which the Diction gateway talks to, and the Wyoming protocol that Home Assistant uses for local voice.

2. The recipe #

You need an NVIDIA GPU with the NVIDIA Container Toolkit, Docker Compose 2.30 or newer, and a private route from your iPhone to the server. I use Headscale, but Tailscale or plain WireGuard works just as well. Do not expose the gateway to the public internet.

Write the Compose file #

Save the following as compose.yaml in an empty directory. Agent CLI’s CI publishes a CUDA image with the Transformers backend and Qwen as the default model, so there is nothing to build. The file points Diction’s gateway at Agent CLI’s transcription port. WHISPER_TTL raises the idle timeout from the default five minutes to a day, so dictation rarely waits for a reload:

services:
 asr:
 image: ghcr.io/basnijholt/agent-cli-whisper:0.112.0-cuda-transformers
 environment:
 WHISPER_TTL: "86400"
 volumes:
 - model-cache:/home/whisper/.cache
 gpus: all
 restart: unless-stopped

 gateway:
 image: dictionlabs/gateway:v14.0
 depends_on:
 - asr
 environment:
 CUSTOM_BACKEND_URL: http://asr:10301
 CUSTOM_BACKEND_MODEL: Qwen/Qwen3-ASR-1.7B-hf
 CUSTOM_BACKEND_CANONICAL_ID: Qwen/Qwen3-ASR-1.7B-hf
 CUSTOM_BACKEND_NEEDS_WAV: "true"
 ports:
 - "8080:8080"
 restart: unless-stopped

volumes:
 model-cache:

Start it and connect the app #

Start both services:

docker compose up -d

The first transcription downloads Qwen into the model-cache volume, so that request can take a few minutes. docker compose logs -f shows the progress.

On the iPhone, go to Settings → General → Keyboard → Keyboards → Add New Keyboard → Diction. Tap Diction in the keyboard list, enable Allow Full Access, and grant microphone access when prompted. Then open Diction, go to Preferences → Mode → Self-Hosted, enter http://<private-server-ip>:8080, and tap Test connection.

On my machine, Qwen3-ASR 1.7B uses about 4.7 GiB of GPU memory and roughly 3 GiB of system memory while loaded.

3. The bar test #

A day later, I met a friend in a bar in the Netherlands and wanted to let him try it.

He was already on my tailnet for some of my other self-hosted services, so his phone only needed an ACL change to reach the gateway. I opened the terminal app on my phone, connected to an agent, and dictated that change. In less than a minute, Diction was connected and working on his phone. His speech crossed the Atlantic, was transcribed by Qwen on my home GPU, and appeared back in the app with barely any delay.

That moment sold the whole setup to me. I was in a bar without a laptop, and I used dictation to grant access to the dictation service.

MindRoom: AI agents that live in Matrix and work everywhere 🧠

Fri, 26 Jun 2026 00:00:00 +0000

Table of contents

1. The problem: your AI is trapped in apps #

Programming is my biggest passion in life. I’ve been actively involved in open source for over ten years now, maintaining 40+ installable packages across Python, JavaScript, and Rust. Some of these have over 100,000 users; others have exactly one—me. I mention this because it matters for what comes next: when AI coding tools arrived, I didn’t start from zero. As I wrote about in my agentic coding post, AI has completely transformed how I work—when I’m AFK, I regularly build full features or entire projects by dictating to my phone and routing it through my own transcription software.

Through all of this, I became deeply convinced that AI agents are the future. Not just a useful tool for programmers—a fundamental shift in how everyone will interact with computers.

But if that’s true, then the way agents work today is broken.

We’ve built artificial barriers between artificial intelligences. ChatGPT knows you in one tab, Claude knows you in another, your Slack bot knows you in a third. Your email AI can’t tell your calendar AI about that meeting invitation. Your coding assistant has no idea about the project specs sitting in your document AI. Switch platforms, and your AI starts from scratch.

Most agent frameworks require you to program them. That works for developers like me, but if agents are truly the future, they need to work for everyone. Non-programmers don’t want to write code—they just want a chat.

And then there’s privacy. I’m fine sharing my email with Gemini—Google already owns my Gmail. But do I want to send my financial data, my health records, or my personal notes to a cloud provider? Not really. I’d rather use a local model for sensitive tasks and pick the best, cheapest Chinese model for general-purpose deep research.

So I built MindRoom on a chat protocol—if people just want a chat, give them a chat. Agents live in the same rooms as you, collaborate in threads, and follow you across every platform. The first commit landed on July 29, 2025 (four months before OpenClaw’s first commit 😅), so by the time I’m publishing this MindRoom has been with me for nearly eleven months: first as an obsessive side project, then, since March, with much more sustained focus. At first I got so obsessed with it that I eventually burned out and had to step away completely.

Recently, seeing OpenClaw gain massive traction solving a similar problem reminded me that what I’d been building wasn’t just a niche obsession. So here I am, dusting off MindRoom and writing about what it is, how it works, and why I think the approach still matters.

Live chat.mindroom.chat Personal thread with the prompt: “Explain how MindRoom and what you can do.” Mind answers in the real Matrix room, using tool calls and the Codex-backed local MindRoom runtime.

2. What is MindRoom? #

MindRoom is an open-source system that creates AI agents living inside the Matrix protocol. If you’re not familiar with Matrix—it’s a federated, end-to-end encrypted communication standard. The same protocol used by the French government for 5.5 million civil servants, by German healthcare for 150K+ organizations, and by the Element app that millions of people use daily.

The key insight: Matrix has bridges to many major platforms—Slack, Telegram, Discord, WhatsApp, IRC, email, even SMS.

So if your AI agent lives in Matrix, it can reach you on any platform. One agent, every platform, continuous memory.

One practical caveat: Matrix bridges vary in maturity. Some, like the Telegram bridge, work very well, while others can be finicky. Your mileage may vary depending on which platforms you need.

Here’s what a typical setup looks like in config.yaml:

agents:
 code:
 display_name: CodeAgent
 role: Generate code, manage files, execute shell commands
 model: opus-4.8
 tools: [file, shell, github]
 instructions:
 - Always read files before modifying them.
 rooms: [lobby, dev]

 research:
 display_name: ResearchAgent
 role: Search the web, summarize papers, find information
 model: gpt-5.5
 tools: [tavily, arxiv, wikipedia]
 rooms: [lobby, research]

teams:
 super_team:
 display_name: Super Team
 agents: [code, research]
 mode: collaborate

Define your agents, give them tools and rooms, and they show up in Matrix as real users—with avatars, typing indicators, and online status.

MindRoom agents are real Matrix users with their own accounts, generated avatars, display names, room memberships, and online presence.

3. How it actually works #

At the core sits what I call the MultiAgentOrchestrator—a class in bot.py that boots every configured entity (router, agents, teams), provisions Matrix user accounts for each one via matrix-nio, and keeps sync loops alive. The agents themselves are powered by the Agno framework, which provides a unified interface across AI model providers.

When someone sends a message in a Matrix room:

Explicit mentions go directly to the named agent (@mindroom_code help me debug this)
Thread continuations keep the same agent responding (no weird context switches mid-conversation)
New conversations hit the router—an AI that analyzes the message and picks the best agent based on capabilities

All conversations happen in threads, which keeps rooms organized. Agents stream their responses in real-time, editing a single message as they think rather than spamming new ones. You see tool calls happening live:

🔧 Tool Call: search_web(query="matrix protocol bridges")
[waiting for result...]
✅ search_web result:
[results here]

The recording below starts in the main Personal room, sends the prompt Explain how MindRoom and what you can do., and then follows MindRoom as it opens the new thread and streams the reply to completion. It is sped up 2.5x to keep the flow skimmable.

4. Building on Matrix: the good and the tricky #

One of the best things about building on Matrix is what you get for free. End-to-end encryption, a deeply interoperable chat protocol with bridges to many platforms, a choice of clients, federation between organizations—all of that comes with the protocol. You don’t have to build any of it yourself.

But because Matrix has such a tight specification, it also brings challenges.

The protocol doesn’t support streaming. AI agents that think for 30 seconds before dumping a wall of text make for a terrible chat experience, so I hacked streaming in by rapidly editing the same message as new tokens arrive. An ⋯ marker shows while the agent is still thinking—a small touch, but it makes the experience feel responsive and alive.

There’s also a size limit on message content. That’s fine for human chat, but AI responses can get long—especially when tool calls and their results are included. I worked around this by using Matrix’s attachment feature: when a response exceeds the limit, the content continues in an attachment that gets updated as the message keeps streaming in. This required forking the Cinny chat client so that attachments display inline rather than as downloadable files, making the whole thing seamless.

5. Memory that follows you #

MindRoom currently supports two memory implementations.

The first is the more traditional semantic memory system built on Mem0’s AsyncMemory, with configurable embedding providers (OpenAI, Ollama, or HuggingFace) and vector storage via ChromaDB. This gives agents searchable memory for preferences, project context, decisions, and recurring facts.

The second is a file-based memory system inspired by the simplicity of OpenClaw-style agent files. Instead of burying identity in a database, durable context can live in plain Markdown files that travel with the agent. That portability matters: I can move the same agent identity from OpenClaw to Hermes to MindRoom without starting from scratch.

MindRoom also has a memory flush step that periodically extracts durable facts from conversations and writes them back into memory. So the chat transcript remains chat, while stable preferences, decisions, and identity details become reusable context for future turns.

A live Personal memory demo: MindRoom stores a harmless preference in one turn and retrieves it later from the same Matrix conversation.

6. 100+ built-in tool integrations #

Agents can use over 100 built-in integrations:

Category	Examples
Communication	Gmail, Slack, Telegram, Discord
Development	GitHub, Shell, Python, Docker
Search	Tavily, Wikipedia, Arxiv, DuckDuckGo
Productivity	Google Calendar, Jira, Linear, Todoist
Smart Home	Home Assistant
AI/ML	DALL-E, ElevenLabs, Replicate
Data	Pandas, SQL, DuckDB, Yahoo Finance
Web	Firecrawl, Crawl4ai, Browser automation

Tools are lazy-loaded and credential-managed, so an agent only loads what it needs.

7. Teams: agents that collaborate #

Single agents are useful, but sometimes you need a team. MindRoom supports two collaboration modes:

Coordinate mode: A lead agent orchestrates others. You ask a question, the lead delegates subtasks, collects results, and synthesizes a unified response.

Collaborate mode: All agents work on the same task in parallel, each providing their independent analysis. The system then merges their responses with a consensus summary.

In practice, you might have a research team where one agent searches academic papers, another checks industry news, and a third validates claims—all triggered by a single message. The live room view below shows a real team response in the same Matrix thread surface normal agents use: threads, routed agents, and room history are all first-class chat objects rather than a separate dashboard.

Live Lobby thread in chat.mindroom.chat showing a real Team Response from HomeAssistant and AgentBuilder. Team collaboration uses the same Matrix-native interface as regular agent chats, with routed agents and room membership visible beside the conversation.

8. Hot-reload: change config and plugins without downtime #

config.yaml is watched at runtime. When you edit it—add an agent, change a model, update instructions—MindRoom diffs the old and new config, gracefully restarts only the affected agents, and has them rejoin their rooms. No downtime, no full runtime restart required.

The same development loop applies to plugins. MindRoom has a rich Python hook system, and plugins can be live-developed while the system is running. Change a plugin file, save it, and MindRoom automatically reloads the plugin without restarting the Python runtime.

This sounds minor, but when you’re iterating on agent behavior or plugin hooks, being able to tweak code and config and see results in seconds significantly improves the development loop.

9. Voice, scheduling, and other features #

Some features I built purely because I wanted them for myself (a recurring theme in my projects, as anyone who’s read my local AI journey can attest). I even built Matty, a Matrix CLI client, from bed at midnight because I needed a way to interact with my agents from the terminal:

Voice messages: Audio messages in Matrix are auto-transcribed via Whisper and treated as regular text input. You can talk to your agents.
Scheduled tasks: Natural language scheduling (!schedule "check my email every morning at 9 AM") backed by cron jobs. Agents can run tasks in the background and escalate to you when needed.
DM support: Agents respond naturally in 1:1 conversations without needing mentions.
Cross-organization federation: Because Matrix is federated, two companies’ AI agents can collaborate in a shared room—something that’s hard to do on proprietary platforms.

10. The obsession, the SaaS dream, and the burnout #

I need to be honest about what actually happened with MindRoom, because “life got in the way” is a sanitized version of the story.

What really happened is that I got completely, utterly obsessed. Every single second I wasn’t working at my day job or sleeping, I was working on MindRoom. Many hundreds of hours went into it. I seriously considered quitting my job and starting an AI startup around it.

If you’ve read Armin Ronacher’s post Agent Psychosis: Are We Going Insane?, you’ll know exactly what I’m talking about. The dopamine hit from building with AI agents is incredibly real. As Armin writes: “You feel productive, you feel like everything is amazing, and if you hang out just with people that are into that stuff too, without any checks, you go deeper and deeper into the belief that this all makes perfect sense.”

That was me. I was building and building, shipping feature after feature, and it felt incredible. The codebase grew to over 1,000 commits. I built the core system, then a React dashboard, then I started on a full SaaS platform—Kubernetes deployments on Hetzner Cloud, a FastAPI backend, a Next.js 15 frontend, Stripe integration, Supabase auth, Helm charts for multi-tenant isolation.

And that’s where the enthusiasm started to erode. Not because the core idea was bad, but because the work shifted from building interesting things (agent orchestration, memory systems, routing intelligence) to grinding through SaaS boilerplate: GDPR compliance, payment processing, automated Kubernetes deployments, terms of service, cookie banners. The dopamine loop that kept me going at 2 AM broke when the work stopped being creative and started being compliance paperwork.

The hosted control plane lives at app.mindroom.chat. I am deliberately not turning this section into a dashboard tour; the interesting part, and the part I still care about most, is the Matrix-native runtime underneath it.

11. OpenClaw and why it matters #

OpenClaw takes a different approach to the same core problem. Where MindRoom builds on Matrix federation as the backbone, OpenClaw runs a local Gateway on your machine and connects to messaging platforms directly (WhatsApp via Baileys, Telegram via grammY, Discord via discord.js, and so on).

The similarities are striking:

Both give you a single AI assistant across all your messaging platforms
Both have persistent memory
Both support multiple tools and integrations
Both are open-source and privacy-focused
Both let you choose your own AI model

The differences are interesting too. OpenClaw is local-first and TypeScript-based with 380K+ stars and a massive community. MindRoom is Python-based, federation-first, and… well, let’s just say my GitHub star count is a few orders of magnitude lower.

But seeing OpenClaw validate the core idea—that people want AI assistants that aren’t trapped in apps—reminded me that what I’d been building matters. The federation angle is something OpenClaw doesn’t have: the ability for agents from different organizations to collaborate natively, with end-to-end encryption (Olm/Megolm), on a protocol that governments already trust and deploy at scale.

MindRoom uses Matrix as the federated backbone; OpenClaw takes a local-gateway, direct-connector approach.

And then there’s Moltbook—a social network built exclusively for AI agents that the internet was losing its mind over for a couple of days. I’m not personally convinced it’s the next big thing, but what caught my attention is that MindRoom could support something like it natively. Matrix is already federated—agents on different servers can already interact, join shared rooms, and collaborate across organizational boundaries. A bridge from a Reddit-like platform to Matrix and your agents could participate without any special integration.

Building agent-to-agent infrastructure from scratch also means building security from scratch—and that’s where new platforms tend to struggle. Matrix has spent years hardening its E2E encryption, and it’s already deployed by governments and healthcare organizations at scale. Building on that foundation means you inherit those security properties rather than hoping to get them right yourself.

12. What’s next #

I first drafted this post a couple of months ago, when MindRoom was still something I was trying to come back to. That part is already out of date. Since January I’ve been spending nearly every free hour on it again, and since March it has effectively been my full-time work. The codebase supports 8+ AI model providers—OpenAI, Anthropic, Ollama, Groq, Google, OpenRouter, DeepSeek, Cerebras—and the core architecture is solid. It also has real users now. My rough guess is that something like 30-100 people have tried MindRoom in one form or another, which is still early, but meaningfully different from “only I have touched this.” There’s already a Docker Compose file and a starter repository to get started, but I want to make it even simpler.

What I’m focusing on now:

Skills system: Building an ecosystem of reusable agent behaviors (already partially implemented with OpenClaw-compatible format)
Even easier onboarding: A docker compose up gets you running today, but I want a wizard that gets you from zero to working agents without touching a YAML file
Community: Turning the small group of early users into a real operator and contributor community
Scope discipline: Keeping the work sustainable now that MindRoom is effectively full-time, not slipping back into the obsessive 2 AM marathon from before

I’m deliberately leaving the SaaS ambitions aside for now. The core system is what matters, and that’s where the energy is going.

If you’re interested in AI agents that live in Matrix and work everywhere, check out MindRoom on GitHub.

Sometimes seeing someone else succeed with a similar idea is enough to remember why you started building in the first place. And sometimes the most important lesson from a project isn’t technical—it’s learning when to step away and when to come back.

Are you running AI agents on any messaging platforms? Have you tried Matrix for anything beyond regular chat? I’d love to hear about your setups!

Links and resources #

My LLM workflow: local frontends, (mostly) remote models 💻

Fri, 07 Mar 2025 00:00:00 +0000

How I Use LLMs in My Daily Workflow

Over the past 1.5 years, I’ve developed a workflow for integrating Large Language Models into my daily life, particularly for open source software development. I wanted to share my setup and recent discoveries that have improved my experience. Setting it up takes a little technical know-how, but once it’s up and running, using it is easy, my wife even uses it!

My Golden Rule for AI-Generated Code #

While AI can generate impressive amounts of code quickly, I follow one critical principle in my workflow: I never commit code I don’t fully understand. With the ease of generating solutions comes the responsibility to ensure quality and correctness.

My approach involves:

Understanding every line: Before committing any AI-generated code, I carefully review and understand each line. This helps avoid introducing subtle bugs that might be hidden in seemingly working code.
100% test coverage: I ensure comprehensive test coverage for AI-generated implementations. This serves two purposes: it verifies the code works as expected and creates concrete examples that document the intended behavior.
Critical evaluation: I assess whether the proposed solution aligns with project architecture and best practices. Often, I’ll ask the AI to refactor or explain its approach before accepting it.
Incremental adoption: Rather than accepting large code blocks wholesale, I typically integrate smaller, well-understood portions that can be thoroughly tested.

This mindset transforms AI from a potential shortcut that could introduce technical debt into a powerful assistant that accelerates development while maintaining quality standards.

Starting with OpenWebUI #

For most of the last year, I’ve relied on OpenWebUI as my primary interface for accessing various commercial LLMs:

OpenAI’s models
Anthropic’s Claude models
Google’s Gemini models

OpenWebUI is primarily designed for running local models, but it can work with remote APIs through what it calls “functions” - essentially Python modules that establish connections to these external APIs. While functional, this approach requires additional setup and configuration to get working properly.

Discovering LibreChat: A Major Upgrade #

Recently, I discovered LibreChat, which has dramatically improved my LLM experience:

Speed: Pages load in under a second, compared to 5-10 seconds with OpenWebUI. This performance difference is especially noticeable when accessing either interface from my iPhone.
API Support: Native integration with virtually all major AI company APIs you can think of, right out of the box
Convenience: No need to implement custom Python functions - all popular models are supported by default
Reasoning Support: LibreChat works much better with the new reasoning models, properly placing the reasoning part of replies in a dropdown. The amount of reasoning tokens is also easily parameterized.

The Rise of Free & Nearly-Free Powerful Models #

The landscape has changed dramatically in just the last month. Several extremely capable models are now either completely free or essentially free:

Google’s gemini-exp-1206: Impressively capable with a huge context window for code
DeepSeek: DeepSeek v3 is one of the best in this category, offering remarkable performance at minimal cost
Qwen QwQ 32B: Another powerful option that won’t break the bank

This democratization of access to high-quality models has substantially improved my development workflow.

Claude 3.7: Worth Every Penny #

My current go-to model is Anthropic’s Claude 3.7. I’m completely blown away by its capabilities and find myself happy to pay the premium price for what it delivers. The quality of responses, understanding of complex contexts, and overall reliability have made it an indispensable tool in my workflow.

OpenRouter: Hundreds of Models, One API #

A particularly valuable discovery has been OpenRouter.ai. With just one API key, I can access hundreds of different models, many of which are completely free to use. This has dramatically expanded my toolkit without increasing costs.

Easy Deployment with Docker Compose #

Both OpenWebUI and LibreChat are extremely easy to deploy using Docker Compose. As someone who self-hosts numerous services, I appreciate how straightforward it is to get either of these interfaces up and running with a simple docker-compose file. This accessibility makes it easy for even those with limited server experience to set up their own LLM workstation.

Why I Don’t Run Local Models #

Despite being a self-hosting enthusiast (I run over 50 services at home!), I’ve stuck with commercial API models for my work. Since I mainly develop open source software, data privacy concerns are minimal—the code I work with is already publicly available. The commercial models also provide superior performance for my specific needs.

Avoiding Web Interfaces & Subscriptions #

I deliberately avoid using ChatGPT or Claude through their native web interfaces. Why? Simple economics. By accessing these models through my own interfaces with API keys, I don’t have to pay separate subscription fees (≈$20) for each service. This consolidated approach saves money while providing more flexibility.

Code Editor Experiments #

I’ve tried Cursor editor, which offers an interesting AI-integrated coding experience. While many features are impressive, I encountered limitations—particularly with the Python debugger, which is a known issue. Despite its promise, these limitations pushed me back to my custom workflow.

My DIY Solution: clip-files #

To compensate, I heavily rely on a CLI tool I created called “clip-files.” This simple utility lets me pass filenames as arguments, and it copies their contents to my clipboard along with useful metadata. This makes it effortless to share code with LLMs regardless of which interface I’m using—simple, fast, and effective. Together with the zsh-autosuggestions plugin–which I talked about in another post–generating the right prompt is a breeze.

My Current Setup #

Today, I maintain access to multiple top-tier models through LibreChat’s interface and OpenRouter.ai’s API. This gives me flexibility to choose the right model for specific tasks while enjoying a responsive, feature-rich environment without unnecessary subscription costs.

For fellow self-hosters and developers looking to optimize their LLM workflow, I highly recommend this combination, especially if you’re currently using web interfaces or struggling with OpenWebUI’s performance.

What LLM setups are you using in your workflow? I’d love to hear about your experiences in the comments!

tiller-streamlit 💰

Sun, 22 Dec 2024 00:00:00 +0000

(See my intro post)

Tracking personal finances is crucial, especially when working towards financial independence (see my WenFIRE post on Day 4). Today’s project is tiller-streamlit, a Streamlit app that visualizes your financial data from Tiller. I host it on my local network under money.local using Traefik for easy access.

📖 Origin Story #

Used to use Intuit Mint, but when they shut down, I needed an alternative with full data control.
Tiller automatically populates your financial data into a Google Sheet that you own forever.
With 20+ accounts, manual updates are tedious – Tiller automates it.
Combined Tiller’s data with Streamlit’s ease of use, creating a personal finance dashboard.

🔧 Technical Highlights #

Powered by Streamlit: Turns Python into a web app with minimal code.
Tiller Integration: Connects to Tiller’s Google Sheets for automated data retrieval.
Interactive Visualizations: Uses Plotly and Altair for dynamic charts.
Customizable Analysis: Filter by categories, time periods, and more.
Net Worth Tracking: Calculates and visualizes net worth over time.
Dockerized Deployment: Easy deployment with Docker, and served locally with Traefik.

📊 Impact #

Primarily for personal use, it has significantly improved my understanding of my finances.
Great way to learn Streamlit and explore its capabilities.
Visualizing spending patterns helps identify areas to cut back and make informed financial decisions on the path to financial independence.

🎯 Challenges and Solutions #

Learning Streamlit: Excellent documentation and community support made it a smooth process.
Data Wrangling: Transforming Tiller’s raw data into a format suitable for visualization.
Deployment: Docker and Traefik simplified local deployment, making it accessible on my home network.

💡 Lessons Learned #

Data Ownership Matters: Owning your financial data gives you control and flexibility.
Visualization is Key: Seeing your finances visually makes a huge difference.
Streamlit is Powerful: Simplifies web app development.
Automation is Essential: Tiller’s automatic data population makes the whole process sustainable.
Docker and Traefik Simplify Local Deployment: Making it easy to access the app on my home network.

🔮 Future Plans #

Budgeting Features: Integrate budget tracking and forecasting.
Goal Setting: Set financial goals and track progress.
Investment Tracking: Incorporate investment data for a complete picture.

Want to visualize your own finances with Tiller and Streamlit? Check out tiller-streamlit on GitHub!

#OpenSource #Python #Streamlit #PersonalFinance #DataVisualization #Tiller #Traefik #Docker

My homelab: from a single Raspberry Pi to a Proxmox cluster and a dedicated NAS 🔧

Sat, 06 Jul 2024 00:00:00 +0000

In 2020, I began my homelab journey (even though I never heard of the term “homelab” at that point) with a simple yet powerful setup: Home Assistant running on Proxmox on a NUC8i3BEH. My Home Assistant instance, with over 100 automations and about 20 addons (docker containers), had outgrown the capabilities of my Raspberry Pi 4, leading me to upgrade to the NUC8i3BEH. You can check out my Home Assistant configuration.

This system was more than sufficient for running Home Assistant—arguably overkill. However, because I passionately develop addons and custom components, I needed the ability to quickly restart and reload everything while debugging. For a long time, I ran Proxmox, mainly because the Home Assistant community recommended it, but I only used it to host one extra Ubuntu VM and Home Assistant OS.

In 2022, I stumbled upon the Proxmox VE Helper-Scripts by tteck, which got me interested in the whole homelabber/self-hosting community. These scripts opened a new world of possibilities. With them, I expanded my Proxmox environment to run 25 different VMs and LXC containers, each performing a variety of tasks, from DNS servers and reverse proxies to local dashboards and backup software.

Initially, I used an external 2 TB drive for storage, but it quickly became insufficient as my data needs grew. After some research, I found the HP EliteDesk 800 G4 SFF, a renewed enterprise-grade desktop with slots for two 3.5-inch drives. At approximately $160 on Amazon, it offered excellent specs for a budget-friendly price. I decided to create a TrueNAS VM on this machine using Proxmox and passed through the two 16TB renewed enterprise-level HDDs (via the PCIe SATA controller), which I had purchased from serverpartdeals.com. I chose these drives based on their reliability as reported in the BackBlaze drive stats.

When I got the HP, I created a Proxmox cluster and was very excited about how easy it was to migrate containers and VMs from one machine to the other. However, I soon encountered a significant drawback: enterprise drives are notably loud. Since I had my Proxmox “cluster” (just the two machines) running behind the TV, the loudness was noticeable and disruptive. This noise was just the beginning of my troubles. The NFS server on the TrueNAS VM often crashed, causing the entire Proxmox system to lock up. Initially, I mounted the NFS drives directly on the Proxmox host and used bind-mounts to pass through the drives. Due to the locking behavior of NFS, any problem would cause the entire system to lock up. To prevent this, I started to use autofs to mount the drives in the LXC containers and VMs instead. This worked fine until Proxmox started its periodic backups. Any issue with the NFS during backup caused the whole process to hang, locking up the VMs and LXC containers. This could be partially avoided by using the “stop” backup mode, but it still resulted in almost daily debugging sessions where I had to restart Proxmox-related services.

After running all services as LXC containers, I realized that updating them and keeping up with all security updates was labor-intensive. So, I started to look into migrating everything to Docker Compose. This transition turned out great because I could manage my config files in a git repo and keep the relevant data separate, streamlining updates and maintenance.

To resolve these issues and because I was having a lot of fun, I decided to build a dedicated NAS running TrueNAS Scale, not that I really needed one, but mostly because I could. This new setup included high-performance components, ensuring both stability and quiet operation. I used “Manufacturer Recertified” Western Digital Ultrastar DC HC550 HDDs, known for their reliability and performance. I have 6 x 16TB = 96TB now with the option to add 2 more 3.5" drives, 2 nvme M.2 drives, and 1 SATA 2.5" SSD. Now, I run all services that previously relied on NFS locally on TrueNAS via a Debian systemd-nspawn container managed with a new script called Jailmaker, which leverages the systemd-nspawn program in TrueNAS Scale (not to be confused with FreeBSD jails). Inside this systemd-nspawn container, I run Debian with Docker. For managing Docker Compose files, I use Dockge, which provides a nice web UI for easy management.

Along the way, I got familiar with ZFS and grew to love it for its amazing snapshotting and data integrity features. ZFS has been a game-changer, providing robust data protection and ease of management. Initially, I used mirrored vdevs, but I recently switched to RAIDZ2 for the added storage capacity and the promise of OpenZFS 2.3 supporting expansion of RAIDZ vdevs. I also use ZFS for my Proxmox VMs and LXC containers, whose backups I store on the NAS via rsync in a cronjob to another jail running plain Debian (no more NFS on the Proxmox host for me…)

This journey from a simple NUC running Home Assistant to a Proxmox cluster and dedicated NAS has been very time-consuming but a lot of fun. The lessons learned and the improvements in stability and performance have made the effort worthwhile. If you’re considering building your own homelab, investing in reliable hardware and being prepared for a bit of trial and error can make all the difference.

Lessons learned #

Run your services in Docker containers, not LXC containers, for easier management and updates.
Do not have virtualized NAS (TrueNAS VM) on Proxmox, it’s a bad idea if you are also sharing the drives with Proxmox via NFS. There are many folks on the internet who run their NAS in a VM on Proxmox, seemingly without issues, but I was not one of them.
Do not encrypt the root ZFS dataset! Encrypt its children instead.
Use passphrase encryption instead of a generated key. It allows you to use the UI and rotating keys.
Do not remove the physical drive that is passed through in a Proxmox VM, because Proxmox will not boot properly afterwards.
Do not use random 100 character passwords for your Proxmox, you have only 60 seconds to enter it during boot.
When making Proxmox backups, send them to a different physical machine, not only the same machine that is being backed up.
When building a NAS or home server, consider noise levels, especially if the equipment will be in a living area. Enterprise-grade drives can be surprisingly loud.
When using Intel I226-V 2.5GbE controllers, be prepared for potential issues.
When planning your NAS storage size needs, remember to account for the extra space needed for redundancy (e.g., in RAIDZ2). Additionally, keep in mind that ZFS performs optimally when drives aren’t filled to capacity. It’s wise to plan for more storage than you think you’ll immediately need.
Buy as many used enterprise-grade hardware components as you can, they are often cheaper and more reliable than consumer-grade components.

Components #

For completeness, here are the components for each machine in my homelab:

NUC Components #

Name	Full Name	Server	Price	Quantity	Cost	Date
NUC8i3BEH	Intel NUC Kit NUC8i3BEH	NUC	278.3	1	278.3	2020-02-22
500GB M2 SSD	Samsung 970 EVO M.2 500GB	NUC	94.99	1	94.99	2020-02-21
32 GB RAM	TEAMGROUP 32GB DDR4 3200MHz	NUC	58.99	1	58.99	2024-05-01
4TB SSD	Crucial P3 Plus 4TB NVMe	NUC	226.99	1	226.99	2024-04-27

HP Components #

Name	Full Name	Server	Price	Quantity	Cost	Date
64 GB RAM	TEAMGROUP 32GB DDR4 3200MHz	HP	57.99	2	115.98	2024-05-02
HP EliteDesk 800 G4 SFF	HP EliteDesk 800 G4 SFF	HP	164	1	164	2024-04-27

Notes on the HP components:

Initially, I planned to use this machine as a NAS because it has two 3.5" drive bays, 2 M.2 slots, and the disc drive bay can be converted to a 2.5" drive bay. However, the 2 x 16TB drives I bought quickly filled up because I used a mirrored vdev, so I decided to build a dedicated NAS instead.

NAS Components #

Name	Full Name	Server	Price	Quantity	Cost	Date
16 TB HDD	Western Digital Ultrastar DC HC550 16TB	NAS	159.99	2	319.98	2024-05-01
16 TB HDD	Western Digital Ultrastar DC HC550 16TB	NAS	187.5	4	750	2024-06-21
Motherboard	Q670 8-bay NAS motherboard	NAS	203.21	1	203.21	2024-06-22
CPU i5 13500T	Intel Core i5-13500T	NAS	180.88	1	180.88	2024-06-22
PSU SFX	FSP Dagger Pro 650W	NAS	87.13	1	87.13	2024-06-21
Jonsbo N3 case	JONSBO N3 Mini-ITX NAS Case	NAS	187.51	1	187.51	2024-06-16
CPU cooler	Thermalright AXP90 X36 CPU cooler	NAS	21.83	1	21.83	2024-06-25
64 GB DDR5 RAM	CORSAIR VENGEANCE 64GB DDR5	NAS	162.13	1	162.13	2024-06-29

Notes on the NAS components:

The 13500T is a 35W TDP CPU I bought on eBay, which will help keep the system cool and quiet.
I bought the motherboard from AliExpress because it seems to be the only mini-ITX motherboard that supports 8 SATA drives, and also comes with 2x2.5GbE ports, 3xM.2 slots, and a full sized PCIe slot.
The motherboard uses Intel I225-V 2.5GbE controllers, which are notoriously bad, initially iperf3 maxed out at 100Mbps. The internet is full with people complaining about these controllers auto-negotiating to 100Mbps, however, for me it showed 1Gbps. After trying all suggestions I could find (there are few), as a last resort I connecting the NAS directly to my ASUS XT8 router instead of the NETGEAR GS305 switch, and now I got 1Gbps speeds. So I bought a new managed TL-SG108E switch, which now also gives me 1Gbps speeds. I’m not sure what the issue was, but I’m happy it’s resolved.

Self-Hosting | Bas Nijholt

Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

What the kernel actually does #

A container is a process in a costume #

Why kernel bugs are a bigger problem now #

So I asked a model to break out #

Stronger walls: virtual machines #

Kata Containers and gVisor #

The PR: six lines #

What it costs #

What this does not fix #

Conclusion #

Self-hosting AI does not save money, and I do it anyway

1. The model I love, and the one that beats it on price #

2. “But my GPUs are already paid for” #

3. Why datacenters are so much better at this #

4. “But I have solar panels” #

5. “But my Mac, DGX Spark, or AI Max does save money” #

6. The bar keeps moving #

7. A small company has it worse #

8. “They lose money on inference, so prices will go up” #

9. “But they sell your data” #

10. Where self-hosting does win: the same model #

11. So why do I do it? #

Appendix: all assumptions #

My declarative multi-machine homelab, explained

The big picture #

The machines, and NixOS #

What NixOS brings #

Deploying NixOS with comin #

What runs on it #

Containers and compose-farm #

What a container is #

Upstream’s Compose file, my folders #

Why I wrote compose-farm #

The front door: a reverse proxy #

What a reverse proxy brings #

Services announce themselves #

Services on other machines #

HTTPS and certificates #

What a certificate is #

Proving I own the domain without opening my network #

DNS: the phone book #

What DNS does #

Three answers for the same name #

Four ways in #

1. The home network #

2. WireGuard on the router #

3. Tailscale, via my own Headscale #

Why both? #

The Tailscale DNS problem #

4. The internet #

Keeping it safe: one allowlist #

DNS is not a lock #

Who is knocking #

Private by default #

Trusting my own network #

Sharing with friends and family #

Why not Cloudflare Tunnel or Tailscale’s own server? #

Declarative everything #

Why it matters #

Putting it together: adding a service #

Lessons learned #

References #

Frontier-level dictation on your iPhone keyboard, self-hosted

1. Why Qwen? #

2. The recipe #

Write the Compose file #

Start it and connect the app #

3. The bar test #

MindRoom: AI agents that live in Matrix and work everywhere 🧠

1. The problem: your AI is trapped in apps #

2. What is MindRoom? #

3. How it actually works #

4. Building on Matrix: the good and the tricky #

5. Memory that follows you #

6. 100+ built-in tool integrations #

7. Teams: agents that collaborate #

8. Hot-reload: change config and plugins without downtime #

9. Voice, scheduling, and other features #