Security | Bas Nijholt

Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

Sun, 18 Oct 2026 00:00:00 +0000

For years, a Docker container was my default answer to “where do I run this?” I never thought it was a perfect sandbox. It was good enough, because getting out of one usually meant finding a specific kind of bug in the Linux kernel itself, and those were hard to find.¹

AI models have become very good at finding security bugs, including in Linux. Anthropic and OpenAI now keep their strongest security models, Claude Mythos and Daybreak Blue, behind identity checks and vetting. Open-weight models, which anyone can download and run, come with no such rules, and they trail the closed ones by only six to seven months. Once there is an open model as good as Mythos, anyone can strip out its refusals with a tool like Heretic (people call this abliteration) and point it at the Linux kernel. The security company DepthFirst used its own model to find a way out of a container and titled the write-up Containers Are No Longer a Security Boundary.

I care about this because running untrusted code in containers is exactly what MindRoom, my open-source system for AI agents that live in Matrix chat rooms, does. Every agent can get its own Docker worker, and its shell, file, Python, and browser tools run there. It can install anything, start background processes, and drive a real browser. And as I wrote in my ZFS post, I run coding agents in YOLO mode, exclusively. I do not review the commands they run, I look at the results.

So both sides changed at once. The code inside the container is now written by an AI, and the bugs that let it out are now found by an AI. A video from the Low Level channel asking whether containers are still safe made me sit down and rethink my setup.

I am not an expert in any of this. I have used containers, KVM, and hypervisors for years, including Proxmox before I moved to NixOS, but I never deeply understood what happens underneath. Building this forced me to learn it, and writing this post is my exercise in understanding it all. All of this is very complicated (as is everything new), and I am still learning. I am sure I skip over a lot and simplify the rest; what is here are the parts that helped me understand it. I thought it would be nice to share.

Table of contents

What the kernel actually does #

When people talk about Linux, they usually mean an operating system like Ubuntu, Fedora, or NixOS. Those differ in almost everything you see: package managers, desktops, config files. What they all have in common is the kernel, the part Linus Torvalds started in 1991 and still leads today. Strictly speaking, that kernel is the only part called Linux.

The kernel is the one program on a Linux machine that talks to the hardware directly. Everything else, your shell, Python, Chrome, Docker, runs on top of it and has to ask it for things. Open this file. Give me some memory. Send this packet. Start a new process. Those requests are called system calls, or syscalls for short.

There is one kernel per machine, and it decides who is allowed to do what. That makes it the most trusted code on the system. Most kernel bugs only crash the machine or cause glitches. But some let a normal program become root, the admin account, or even run its own code inside the kernel. At that point it can do anything on the machine, including leaving any container it was in.

A container is a process in a costume #

When you docker run -it ubuntu, you get a root shell in what feels like a fresh computer. Under the hood it is one ordinary process. The kernel dresses it up using three features:

Namespaces decide what the process can see. It gets its own list of processes (where it is process number 1), its own filesystem, and its own network. Root inside is still root on the host, with most of its powers taken away.²
cgroups decide how much it can use: CPU, memory, number of processes.
seccomp decides which syscalls it may make at all. It is a filter that blocks the ones a normal program never needs.

There is no second kernel. Every syscall from inside a container lands in the same kernel as every other container and the host itself. The kernel enforces the isolation, and it is also what gets attacked.

For most uses that is fine. Usually you put fixed software in a container: a database, a web app, some self-hosted service. You know where that code comes from, you keep your kernel patched, and that is good enough.

An AI agent is different. It writes new code on the spot and runs it, and nobody knows in advance what that code will do.

Since root in a container is root on the host, anyone who can use Docker can start a container as root and mount any folder from the host into it. They can then read and change every file in that folder, including files their own account may not open. Docker’s own documentation says the docker group grants root-level privileges. I knew that in theory, but it only clicked when I watched an AI agent do it on a machine where I had no root, only Docker access. It needed to change files it was not allowed to touch, so it started a small container, mounted their folder, and edited them from inside. No kernel bug needed.

Why kernel bugs are a bigger problem now #

Kernel bugs that let a normal program take over the whole machine used to take skilled people weeks or months to find and exploit. The famous ones, like Dirty COW in 2016, made the news every year or two, you patched, and that was it.

AI is speeding this up, and the DepthFirst write-up I mentioned is a good example. Their model found a bug in the kernel code for Unix sockets, which programs on the same machine use to talk to each other. The kernel freed a piece of memory and then kept using it, a so-called use-after-free, and from inside a container that was enough to get out. Their conclusion is that you should assume attackers can escape containers at will.

Blocking more syscalls, or taking more powers away from root, helps a little. But it only narrows which parts of the shared kernel a container can reach. This bug sat in Unix sockets, which almost every program uses, including the browser the agent drives, so blocking them would break the agent.

Now put an AI agent in that container. It reads web pages and repositories written by strangers, and one of those could talk it into running an exploit. On a MindRoom server, getting out of a worker means getting onto the machine that runs MindRoom itself. That is where the API keys, Matrix logins, and stored credentials live, which the workers are built to never see.

So I asked a model to break out #

Before changing anything, I wanted to know how real this was for my own setup. I gave a MindRoom agent Daybreak Blue and asked it to escape its worker container using known kernel vulnerabilities. Daybreak Blue is OpenAI’s security model for verified users, so it does the offensive security work that other models refuse.

It did not get out, which was reassuring. But it only tried known bugs. A kernel a few weeks behind on patches, or a bug nobody has published yet, would be a different story. So the attempt tells me the easy path was closed that day, and not much more.

Stronger walls: virtual machines #

The classic answer is a virtual machine. A container shares the host’s kernel, but a VM brings its own, and this is where the hardware comes in. Modern CPUs have a virtualization mode built in, called VT-x on Intel and AMD-V on AMD. In that mode, a second kernel, the guest kernel, runs directly on the real CPU and believes it is in charge, while the CPU quietly limits it. When the guest tries something it may not do on its own, like talking to a disk or a network card, the CPU pauses the guest and hands control to the host. This is called a VM exit. Despite the name, nothing escapes and nothing crashes. The host handles the request, for example by pretending to be that disk, and then resumes the guest where it stopped. From the guest’s point of view, the instruction simply finished.

Memory is fenced off too. The host gives the CPU a separate map of which real memory each guest may use, and the CPU checks every memory access against it. So even a guest kernel with full privileges cannot touch the host’s memory.

The CPU only provides the mechanism. The software that sets the rules and handles the VM exits is called the hypervisor. “Hypervisor” is a role, not one program, and on Linux it is two pieces:

KVM, a module in the Linux kernel, uses the CPU’s virtualization mode to run the guest, keeps that memory map, and catches its VM exits. It shows up as the device /dev/kvm.
A VMM (virtual machine monitor), a normal program on top, sets up the VM and handles the exits that KVM passes up, for example by pretending to be the guest’s disk and network card. QEMU, Firecracker, and Cloud Hypervisor are three VMMs you can pick from.

Proxmox, which I mentioned above, is KVM plus QEMU. AWS Lambda is KVM plus Firecracker.

Code in a VM talks to its own guest kernel. To reach the host, it usually has to break that kernel first and then the hypervisor.³ That is still possible. Google Project Zero turned a bug in how KVM runs VMs inside VMs on AMD into a full escape in 2021. Google’s kvmCTF pays up to $250,000 for a new one. And this year, one researcher, Hyunwoo Kim, published three KVM escapes, one of them hidden in KVM since 2010.

The difference between a container and a VM is how much code an attacker can reach, and that gap is large. On x86 (the Intel and AMD chips in most PCs and servers), Linux has about 370 syscalls, and Docker’s default seccomp filter blocks a few dozen of them. Behind the rest sit the kernel’s filesystems, networking, memory management, and core code: about 3.8 million lines, even without counting drivers. Not all of that is reachable from a container, but a lot of it is. KVM on x86 is about 114,000 lines. On top of KVM comes the VMM, but only the few devices it fakes for the guest. Most QEMU escapes came from its many emulated devices, which is why Firecracker and Cloud Hypervisor keep that list short.

Google’s bug bounties point the same way. kernelCTF pays $71,337 for each new kernel exploit that turns a normal user into root, and Google’s public repository holds exploits for 93 different kernel bugs. kvmCTF pays more than three times as much, and the same repository holds one KVM exploit, which crashes the host instead of escaping it. kvmCTF started a year later, but the gap is still big. Most of those kernel exploits would not work as written from a default container, because they first use a feature Docker blocks. Still, they show how many exploitable bugs sit behind the syscalls.

Kata Containers and gVisor #

Regular VMs are heavy. They boot a full operating system and tend to use a lot of memory. A microVM is a stripped-down VM meant to run one workload and start quickly. That still leaves a practical problem. MindRoom, like most tools, speaks Docker: it hands Docker an image and some folders to mount, and gets a running container back. A VM does not take a Docker image. Switching to VMs would normally mean building disk images, booting them, and wiring up files and networking myself.

Docker does not start containers by itself, though. It takes care of images, networks, and volumes, and then hands the actual start to a small program called a runtime. The default runtime, runc, does what I described earlier: it starts a process on the host kernel with namespaces, cgroups, and seccomp. But you can tell Docker to use a different runtime.

Kata Containers is such a runtime, and it starts a microVM instead. It picks one of the VMMs above (my test used QEMU), boots a small guest kernel, and starts the container inside it.⁴ Docker, and MindRoom, still see an ordinary container.

gVisor, from Google, takes a different route, and it is not a VM. It is a reimplementation of the Linux kernel, written in Go, that runs as a normal program on the host. Go manages memory for the programmer, so the use-after-free mistake behind the DepthFirst escape is much harder to make in gVisor. The worker’s syscalls go to gVisor instead of the host kernel, and gVisor itself only makes a small, filtered set of host syscalls. That is the hardening idea taken all the way: gVisor answers the agent’s syscalls itself, so the filter on the host can be much stricter than Docker’s. It needs no /dev/kvm, so it also runs inside cloud VMs that cannot start VMs of their own.⁵ My test machine, an ARM server (with the kind of chip phones use) running NixOS, had no /dev/kvm, which is why I tried gVisor first. Its runtime is called runsc, and it plugs into Docker the same way.

The three options side by side.

Or as a table:

Isolation	What worker code talks to	Needs KVM	Usual escape route
Plain Docker (`runc`)	The host kernel, filtered by seccomp	No	A host kernel bug
gVisor (`runsc`)	gVisor’s own kernel, written in Go	No	A gVisor bug, then a host kernel bug through its small filtered surface
Kata Containers	Its own guest kernel in a microVM	Yes	A guest kernel bug, then a hypervisor bug

Each option also changes how much code on the host an escape can pick its bug from.

Lines of code on the host that deal with what the agent’s code asks for, counted with tokei, comments and blank lines included. For Docker, the core of Linux without drivers; for gVisor, its kernel and network stack without tests; for Kata, KVM alone, without the VMM and Kata’s other helpers. Not all of it is reachable. Hover for exact numbers.

The PR: six lines #

The change is PR #2886, which is now merged. It adds one setting, MINDROOM_DOCKER_WORKER_RUNTIME, and MindRoom passes its value to Docker as the runtime whenever it starts a worker. That is six lines of source code, plus 62 lines of tests and some docs. Changing the setting recreates each worker the next time it is used, and leaving it unset keeps everything as it was.

For gVisor, setting it up is two steps: register the runtime in Docker’s /etc/docker/daemon.json and restart Docker, then point MindRoom at it.

{
 "runtimes": {
 "runsc": { "path": "/usr/local/bin/runsc" }
 }
}

MINDROOM_DOCKER_WORKER_RUNTIME=runsc

Kata needs no entry in that file: once it is installed, MINDROOM_DOCKER_WORKER_RUNTIME=io.containerd.kata.v2 is enough. The full setup, and a few catches for each runtime, are in the MindRoom docs.

The six lines are short because everything else already existed. gVisor and Kata do the isolating. Docker already gives each worker its folders, its credentials, and its network, and it already knows how to hand a container to a different runtime. On the MindRoom side, the hard part happened months earlier, when agent tools moved out of the main process into dedicated workers with explicit mounts and credentials. Once every worker is a single call to Docker’s containers.run, swapping what runs underneath it is one argument. For comparison, Claude Opus estimated that building a microVM backend into MindRoom itself would take 2,500 to 4,000 lines.

What it costs #

I tested both runtimes with a real agent in a Matrix room: gVisor on that ARM server, and Kata in a throwaway VM on my desktop. Shell, Python, git, background processes, and the browser all worked, without any MindRoom code change. My Kata numbers below come from a VM inside a VM, so bare-metal numbers will differ.

Configuration is the cheap part: one environment variable, plus one entry in Docker’s config for gVisor. The other costs are the usual trade-offs between containers and VMs:

Computation. Code that only calculates, without asking the kernel for anything, runs at full speed with both. gVisor lets those instructions run directly on the CPU, according to its performance guide. A Kata guest runs on the CPU in its virtualization mode, and in AWS’s paper on Firecracker, the CPU overhead was negligible.
Syscalls, disk, and network. The cost starts when a program does ask the kernel for something. Under gVisor, every syscall takes a detour through gVisor’s kernel, so programs that make many small syscalls, like ones that open thousands of small files, slow down the most. Under Kata, disk and network go through the VMM’s fake devices. In that same paper from 2020, a Firecracker guest reached 15 Gbit/s of network traffic where the host reached 44.
Startup. gVisor only has to start a program, so it starts quickly. Kata has to boot a kernel first, and my worker took about 10 seconds to start instead of about 3. Booting a VM can be much faster than that: Firecracker boots a minimal one in about 125 ms.
Memory. gVisor adds a small, mostly fixed amount per sandbox. Each Kata VM runs a second kernel plus the VMM, which alone takes about 3 MB with Firecracker and 131 MB with QEMU. Kata gives each worker VM 2 GiB by default, and /tmp lives in that RAM. The host only hands out memory the guest actually touches, so my idle worker used under 0.8 GiB of host memory. Raising the VM size costs little until a worker needs it. But a VM cannot grow past its size the way a container can, and memory the guest has used, for example to keep recently read files in its cache, mostly stays with the VM.
Compatibility. gVisor reimplements Linux, and its docs say there will always be unimplemented features and bugs, so now and then a program will not work. Kata runs a real Linux kernel, so programs behave as usual, with a few catches. For example, a file lock held on the host does not block the worker, because the lock lives in the guest kernel.

What this does not fix #

A stronger sandbox protects the host. It does nothing about the tokens and network access you give the worker. An agent needs no kernel exploit to misuse a token that can push to your repositories, or to poke at every device on your home network. Scoped credentials and a firewall allowlist protect you at least as much, and MindRoom handles both separately: Agent Keys let agents use API keys without ever seeing them, and Approved Egress limits which servers a worker on Kubernetes can reach.

gVisor has had its own bugs, and it still talks to the host kernel, just through a much smaller door. KVM has bugs too, as this year’s escapes show. Both walls make an escape far harder, and neither makes it impossible.

Conclusion #

Once I started reading about this, I kept running into people saying containers were never a security boundary. That is true, and for years I did not worry about it. With AI finding kernel bugs faster and agents running code nobody reads, I do now.

No sandbox is perfectly safe, though. In a conversation with Dwarkesh Patel, Noam Brown doubted that even computers with no network connection at all would be enough to contain an AI. He pointed to studies where two such computers next to each other talk through heat: one runs its CPU hot, and the other picks up the change with its temperature sensor. Every option in this post trades some speed or memory for safety. How far to go depends on your own risk tolerance and how paranoid you are. MindRoom can now move its agents off the shared kernel with one environment variable. I do not use it myself yet, but if you are more paranoid than I am, it is there.

The other way out is a bug in the program that starts the container. In 2024, CVE-2024-21626 in runc, the program Docker uses for that, leaked a handle to a host folder into new containers, so a malicious image could reach the host’s files without any kernel bug. ↩︎
Docker can map the container’s root to an ordinary user on the host with user namespace remapping, or run without root altogether in rootless mode, but neither is the default. Rootless mode would have stopped the agent in my Docker story, because Docker then has only my own rights. Remapping would not, because anyone who can use Docker can turn it off for a single container. Neither helps against a kernel bug, because code running inside the kernel is above every user. ↩︎
A bug in the VMM’s fake devices, or in other VMM code that handles the guest’s traffic, can skip the first step, because a normal program in the guest can feed that code data through the network or a shared folder. CVE-2020-8608, a buffer overflow in QEMU’s user-mode networking, could be triggered that way with crafted network traffic. ↩︎
Like runc for Docker, Kata’s own programs on the host are a way out too. In 2020, one bug (CVE-2020-2023) let a container take over the guest without a kernel bug, and a second (CVE-2020-2026) then let the guest trick the Kata runtime on the host into mounting files over any host path. Neither needed a KVM bug. virtiofsd, the program that shares folders with the guest, also runs on the host and handles the guest’s file requests. ↩︎
gVisor can use KVM too, as an option. By default it catches the agent’s syscalls without it, which is the setup this post describes. With KVM switched on, gVisor runs the agent’s code in the CPU’s virtualization mode so catching syscalls is faster, but gVisor’s own kernel still answers them, and there is no guest Linux and no fake devices. gVisor recommends that mode only on bare metal; inside a VM the default is usually faster. ↩︎

Friend-to-friend ZFS backups, version two

Mon, 28 Sep 2026 00:00:00 +0000

Two years ago I wrote about friend-to-friend backups with TrueNAS. A friend and I store each other’s ZFS snapshots, which is the cheapest off-site backup there is. TrueNAS’s replication tasks wanted root on the receiving system, and neither of us wanted to give the other root on the machine that holds our family photos. So each of us ran a TrueNAS VM on a hypervisor, backed by an iSCSI zvol on the real NAS, and gave the other person root inside that VM instead. Sounds complicated? It was. It worked, but it meant running ZFS on a zvol on ZFS, inside a VM that each of us had to keep updated.

Since then I replaced TrueNAS with NixOS, and my friend plans to move his machines to NixOS too. This time we looked for the simplest solution, built from standard Linux and ZFS features wherever possible. He wanted plain zfs send and zfs receive, with his data still encrypted and his keys never on my server. I agreed, on one condition: neither of us should be able to destroy the other’s datasets.

The result is zfs-tenant, which is now on PyPI and has documentation.

Table of contents

Why ZFS #

OpenZFS is a filesystem and a volume manager in one. You add disks to a pool, and it behaves like one large partition: every dataset you create draws from the same free space, and each can still have its own quota and encryption. ZFS checksums every block it writes, so it notices when a disk hands back something else, and with a mirror or RAIDZ it repairs the block from a good copy. That protection costs disk space: a mirror of two disks holds one disk’s worth of data.

For me, snapshots are one of the coolest parts. ZFS never overwrites data in place, so a snapshot only has to hold on to the blocks that existed at that moment. Taking one is instant, and it costs no space until the data changes. zfs send turns a snapshot, or the difference between two snapshots, into a stream that zfs receive writes into a pool on another machine, usually piped through SSH. It moves blocks, not files: an incremental send contains only the blocks that changed, and ZFS knows which ones those are without visiting a single file. A bajillion one-kilobyte files do not slow it down, because nothing has to look at them one by one. For comparison, restic on my PC needed an hour and a half per run just to check close to a hundred million files for changes.

Two tools from the same project automate this. sanoid takes snapshots on a schedule and prunes them by policy, for example hourly ones for a day and daily ones for a month. syncoid wraps zfs send and zfs receive over SSH: it finds the newest snapshot both sides share and sends everything after it.

What we wanted #

We wrote down the rules before anything else:

He cannot see any of my data, including metadata like dataset names and sizes.
He cannot change or destroy anything of mine.
I cannot read his data, because his keys stay at his house.
He gets one dataset where he can create and remove nested datasets as he likes.
That dataset has a size limit.
No new replication tool on either side: plain zfs send or syncoid.

Until then his NAS runs TrueNAS, so whatever I built also had to work on a machine without NixOS.

ZFS already had most of it #

OpenZFS has a permission system that most people never touch: zfs allow. It delegates individual operations like receive, create, and destroy on one dataset to an unprivileged user. Put a root-owned quota on that dataset and the user cannot store more than you agreed. Raw sends (zfs send -w) ship the encrypted blocks as they are, so the receiving side never needs the key.

On paper, that covers five of the six rules. The gap is visibility. Any local user can run zfs list and see every dataset on the machine, with names, sizes, and properties. And an SSH account for replication is, by default, also a shell.

There are tools that would sidestep some of this. zrepl has a sink mode with a subtree per client, but it replaces sanoid and syncoid on both sides and runs as root on the receiver. restic or borg to a friend’s box would also work, but everything else on my network is ZFS: my machines replicate to the NAS with syncoid, and a restore is zfs send and zfs receive. Adding another backup format means one more thing to understand when something breaks. My current off-site copy goes to Backblaze B2 with rclone. Ideally that would be ZFS as well, and some hosting services accept zfs send, but they cost far more than object storage. A friend with a ZFS box costs nothing.

Rehearsing the permissions in a VM #

Before writing any code, I had an agent build a throwaway NixOS VM test with a real pool, a delegated user, and a list of probes. Each probe tried one thing the friend should not be able to do and recorded what ZFS answered.

The first surprise came from the most obvious setup. With a plain zfs allow -u joe create,receive,destroy,... tank/friends/joe, the delegated user could run zfs destroy -r tank/friends/joe and delete the dataset I had created for him, quota included. Delegated rights apply to the dataset itself as well as everything below it, unless you say otherwise. The fix is to split them: on the root itself only create,mount,receive with zfs allow -l, and the full set only for descendants with zfs allow -d. After that change, destroying, snapshotting, re-delegating, or changing the quota of the root all failed with permission denied.

A send stream can carry properties too, such as a hostile mountpoint=/etc. ZFS applies them with the receiving user’s rights, so in the VM that property failed with permission denied.

The probes also confirmed the gap: the delegated user saw my tank/host dataset in zfs list, and zfs get used tank/host returned its size.

A gate that speaks syncoid #

The shell part has a standard answer. An authorized_keys line with restrict,command="..." runs one fixed program for every login, and passes whatever the client asked for in SSH_ORIGINAL_COMMAND. zfs-tenant gate is that program. It tokenizes the request, accepts only the handful of command shapes a backup needs, checks that every dataset name is inside the friend’s dataset, and runs zfs with an argument list it builds itself. It never starts a shell, so there is nothing to inject into.

The hard part was knowing exactly which commands syncoid sends, so an agent read syncoid’s Perl and logged every remote command it ran. Most are the zfs commands you would expect. One surprise: before each receive, syncoid runs ps -Ao args= on the receiving machine, to check that no other transfer is already writing to the same dataset. That command prints every running process with its full command line. On my NAS, it would show my friend everything I run, so the gate answers with an empty list.

Reading syncoid that closely also turned up a remote code execution (RCE) bug in version 2.3.0, outside zfs-tenant itself. In push mode, syncoid asks the receiving host for a ZFS resume token. It passed that answer unescaped into shell commands for estimating and resuming the send, which run on the source machine. A malicious or compromised backup host could therefore return shell syntax instead of a token and execute arbitrary commands on the sender with the privileges of the syncoid process.

I reproduced this in two isolated NixOS VMs using real ZFS and SSH: a harmless command embedded in the receiver’s answer created a marker file on the sender. I reported it to the maintainer and opened sanoid PR #1114 with a small fix that shell-quotes the token at both uses. Even with that fix, the sending side should run syncoid as an unprivileged user that may only send and hold. nixpkgs’ services.syncoid already works that way, so my friend needs nothing from zfs-tenant at all.

Zones: a second wall in the kernel #

At that point the gate was the only thing hiding my datasets. Then my friend sent me a message: ZFS has a feature called zones, which restricts a dataset tree to a Linux user namespace.

zfs zone attaches a dataset to one user namespace, and inside that namespace the ZFS kernel module answers dataset does not exist for everything that is not attached. Even if my gate had a bug that let arbitrary zfs commands through, tank/host would stay invisible. The zone does not replace the gate, though: it only filters ZFS, and it only applies once the gate has put the SSH session inside it. Who stops what shows how the two divide the work.

A user namespace can give its processes different user IDs than they have outside. Containers usually use that to make the user root inside the namespace, while it stays an ordinary user outside. ZFS treats root inside a zone as the zone’s administrator, allowed to do anything to the attached datasets, whatever zfs allow says. In the VM test, that let the friend destroy the dataset I had created for him. So zfs-tenant keeps him as his own ordinary user inside the namespace: he has no special rights there, zfs allow decides what he may do, and the kernel still hides everything else.

Root inside a user namespace also turned up in a real OpenZFS bug in August: before 2.4.4, 2.3.9, and 2.2.11, several pool operations, such as destroying a pool, accepted it as host root (CVE-2026-79619). Joe cannot reach that through the gate, but zfs-tenant needs unprivileged user namespaces, so a host running it should load a patched module: cat /sys/module/zfs/version shows the loaded one, which only changes after a reboot.

A zone needs a running namespace to attach to, so zfs-tenant runs a small service per friend that holds one. The gate joins that namespace before it looks at the command, drops the capabilities that joining grants, and refuses to run if the service is down.

Who stops what #

When Joe and I talked it through, I found it hard to say clearly which part protects against what, so I made this diagram.

Each column answers one question: who can connect, what he can run, what ZFS shows him, and what he can change. For the network, our tailnet only lets his router reach port 22 on my NAS, and from= in authorized_keys only accepts his key from that address. Raw sends then keep whatever reaches my pool unreadable to me.

The gate and the zone overlap in one job: hiding my other datasets. The gate does it first, by refusing any name outside his root before zfs runs. The zone repeats that check in the kernel, so hiding my datasets would take a bug in both. Everything else the gate does, the zone cannot:

Without the gate, his key would be a normal account on my NAS. The zone only filters ZFS; it would not stop him from reading /etc or running programs.
An SSH login does not start inside a user namespace. The gate joins the zone before it runs anything, so without the gate the zone would never apply.
The zone leaves some things visible: zpool status still shows my disks and how full the pool is, and the datasets above his root still show their sizes. User namespaces do not hide processes either, hence the empty answer to ps.
It deletes any unencrypted dataset he pushes by mistake, so I never keep his plaintext.
It logs every command, which is how I saw Joe try bash below.

The security model in the documentation lists every promise with the mechanism behind it.

What it cannot hide #

ZFS encryption protects file contents, not structure. In the words of zfs-load-key(8), ZFS “will not encrypt metadata related to the pool structure, including dataset and snapshot names, dataset hierarchy, properties, file size, file holes, and deduplication tables.” So I can see the names, sizes, and snapshot times of whatever my friend sends me. Boring dataset names help, and sending without -p keeps properties out of the stream.

I can also always delete his copy, because I am root on my own machine. What I can never do is read it. My monthly scrubs still verify his data without his key, because ZFS checksums the encrypted blocks.

What we protect against is a dead machine: a failed pool, a fire, a flood. His key may destroy anything below his root, which is what lets syncoid mirror his snapshot retention, so someone who steals that key can also delete his backups on my NAS. Covering that would take holds that I place as root and release on a schedule. We left them out on purpose, although I am reconsidering.

Setting it up #

On a NixOS host, giving a friend space takes this:

services.zfs-tenant = {
 enable = true;
 tenants.joe = {
 dataset = "tank/friends/joe";
 quota = "2T";
 authorizedKeys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... joe-nas" ];
 allowedFrom = [ "100.64.0.12" ]; # his tailnet address
 };
};

That creates the user, pins the key to the gate, applies the dataset, properties, and delegation on every boot, and runs the zone service. My own setup is in friend-backups.nix, with his real key and address. If your sanoid snapshots the whole pool, exclude the tenant tree: your sanoid must neither snapshot nor prune it, or his next incremental push finds snapshots on the target that he never sent. Once my friend is on NixOS, he can push with nixpkgs’ own module:

programs.ssh.knownHosts.bas-nas.publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...";

services.syncoid = {
 enable = true;
 sshKey = "/var/lib/syncoid/id_ed25519";
 localSourceAllow = [ "send" "hold" ];
 commonArgs = [
 "--no-sync-snap"
 "--compress=none"
 "--delete-target-snapshots"
 "--sshoption=StrictHostKeyChecking=yes"
 ];
 commands."tank/offsite" = {
 target = "zfs-tenant-joe@bas-nas:tank/friends/joe/offsite";
 recursive = true;
 sendOptions = "w";
 };
};

My NAS will push to his with the same configuration once he hosts a root for me.

For his TrueNAS box, every release has a single zfs-tenant.pyz that runs with the Python TrueNAS already ships, and the getting started guide covers both setups. Hosting on TrueNAS has only run in VMs so far (issue #4). Sending from TrueNAS is where its built-in tools stop working: replication tasks wrap every remote command in sh -c, which the gate refuses, so he pushes with zfs send or syncoid instead.

Knowing when it stops working #

The last btrfs machine taught me a fifth question for any backup: how do I find out when it stops working? A failed push shows up in systemctl status on the sending side, which nobody reads. The check belongs on the receiving side, and it has to look at every pushed dataset separately, because one healthy dataset can hide another that stopped replicating. That is how I watch my own machines: the NAS checks the newest snapshot of every dataset it replicates every hour and alerts my phone when one is too old. zfs-tenant does not alert yet; issue #3 tracks it.

Restoring #

The gate also allows zfs send of his own snapshots, so getting data back is one pipe, run at his house:

ssh zfs-tenant-joe@bas-nas zfs send -w tank/friends/joe/offsite/photos@autosnap_2026-09-25_00:00:01_daily \
 | zfs receive -u tank/photos
zfs load-key tank/photos
zfs mount tank/photos

My photos are a child dataset that inherits its key from an encryption root, and I only push the child. If my NAS dies and I pull the child back, can I still unlock it without the root? A throwaway VM says yes: a raw send carries the dataset’s own wrapped key, so the child arrives as its own encryption root and opens with the same passphrase. The one catch: after a zfs change-key, the next push carries the new wrapping, so the passphrase that matters is the one at the time of the last push, and it has to live somewhere other than the NAS. Mine does: zfs-unlock keeps the passphrases on a separate device and unlocks the NAS after every boot.

The first real push #

Once my side was deployed, Joe tried it from his TrueNAS box. His first move was to see what else his key could do. The gate logged every attempt on my NAS:

root=tank/friends/joe denied interactive session
root=tank/friends/joe denied 'ls': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe denied '/usr/bin/bash': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe denied '/bin/bash': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe allowed 'zfs receive -s -u tank/friends/joe/zt-test'

The last line is a small encrypted test dataset, sent with zfs send -w piped into ssh, without syncoid. It arrived with keystatus set to unavailable. Then I tried to read it as root on my own machine. zfs mount answered cannot mount 'tank/friends/joe/zt-test': dataset is exported to a local zone, and zfs load-key with a guessed passphrase gave Key load error: Incorrect key provided. The only readable text in the raw stream was ZFS property names and the snapshot name, which is the metadata ZFS leaves unencrypted, as described above.

Tested like everything else #

Every security claim in this post is also checked by a VM test: two NixOS machines back each other up, the way my friend and I do, with real sanoid and syncoid 2.3.0, through real sshd and the gate, into real OpenZFS 2.4.4. It also checks what happens over time, such as snapshot retention and both machines rebooting, and it runs on every push in GitHub Actions.

The route to the remote code execution bug in syncoid was slightly absurd. I built the initial version of zfs-tenant through coding agents, literally from the bathtub, and then asked several frontier models to review the security boundary independently. One of them followed receiver-controlled data beyond my new code and into syncoid. A model’s suspicion is not evidence, so I turned it into the two-VM reproduction above: it failed on unpatched syncoid 2.3.0 and passed after the seven-line fix. The most consequential result of reviewing my small new project was a command-injection bug in the mature tool next to it.

The whole thing is about 930 lines of Python, not counting comments and docstrings, with no dependencies. The kernel does the actual enforcing: delegation, the quota, and the zone. The gate only removes the shell, and it is small enough to read in one sitting.

AI scope creep or useful? #

The first version had grace holds: every day, the host placed a hold on the newest snapshot of each of my friend’s datasets and released it 14 days later. ZFS will not destroy a held snapshot until the hold is released, and only the host can release it. The agent built it without me asking for it, so I removed it right away as scope creep.

Then every model I asked to review the project suggested them again, each in its own context window: Opus 5.5, GPT-6 Astra, and even the one that reviewed a draft of this post. Joe and I had set out to protect against a machine dying (a fire or a flood), not against one of our own machines being hacked. But I am tempted to add the holds back. They cover the case where malware or an attacker gets onto my machine and uses my key to delete my backups on his NAS, or to replace them with garbage. Adding the feature without asking was still wrong; the feature itself might not be.

References #

My declarative multi-machine homelab, explained

Mon, 28 Sep 2026 00:00:00 +0000

I open https://mealie.lab.nijho.lt on my phone to look up a recipe. It works at home, on a train, or on hotel Wi-Fi in another country, over an encrypted HTTPS connection. If a stranger on the internet tries the same address, they get nothing. Nearly everything I run works this way: about 150 containers on four machines, used every day by seven people besides me. How that works is the thing friends actually ask me about, and it is what my 2024 homelab post never explained.

That post was a story about hardware: a NUC, an HP EliteDesk, a TrueNAS box, and a lot of trial and error. Almost everything in it has since been replaced. Proxmox and TrueNAS are gone, and every machine runs NixOS, including the NAS.

The part I think is the coolest is the balance I found: everything is declarative, but with as little machinery as possible. Declarative means I describe the end result in text files, and a tool makes reality match it. One extreme is what I had before, clicking through web UIs and running one-off install scripts. The other is Kubernetes, which many self-hosted projects don’t support and which is a lot to babysit at home, or running every app as a NixOS module, which often lags behind upstream.¹ I landed in between: NixOS declares the machines, each project’s own Compose file declares its app, and compose-farm, a thin tool I wrote, decides which machine runs what, which is all the multi-host orchestration I need. Every layer is a text file in git, each uses the simplest tool that keeps it that way, and I still get new app releases as soon as upstream ships them. The price is that nothing fails over automatically, but because every machine sees the same files and data, moving services to another machine is a one-line change and one command.

It took me several years and many iterations to get here. This is the first version I would recommend to friends, because its declarative configuration makes changes auditable, reproducible, and reversible. That is what makes me comfortable managing it with AI: I can review the diff, rebuild the same configuration, and revert a change if needed.

This post tries to explain it all. I wrote it for people who have never set up a reverse proxy or a VPN, so every piece gets a short explanation before I show how I configured it. I deliberately made it comprehensive, with enough detail to build something similar yourself. That also makes it long, so read the parts you find interesting and skip the rest.

If it is too long, send it to your AI agent, discuss it, and figure out together which parts make sense for your own network.

If you have a single machine that already runs NixOS, you probably don’t need Docker or compose-farm at all: adding your services as NixOS modules is simpler, and that is what I recommend to friends in that situation. If you would rather use Docker, which is usually the path a project officially supports, I recommend compose-farm: it works just as well on a single host and lets you fan out later. The multi-machine part is where my setup really pays off, but the rest of this post, from the reverse proxy to the VPNs, is just as useful on one machine.

Throughout the post I use a few services as running examples: Mealie for recipes, ntfy for push notifications, and my git server at git.nijho.lt.

Table of contents

The big picture #

To reach my services away from home, I use a VPN (virtual private network): an encrypted connection that lets my devices communicate across the internet as part of a private network. Three projects fit together here:

WireGuard provides the encrypted tunnels. I use it directly between my devices and my home router.
Tailscale builds a private network using WireGuard tunnels. Its apps handle finding and connecting to the other devices.
Headscale replaces the coordination server that Tailscale normally hosts. I run that server myself, while my devices run the normal Tailscale apps. Headscale distributes device information and access rules; the apps carry the traffic.

That gives me two VPN options: WireGuard to my router for access to my home network, or Tailscale coordinated by Headscale for sharing selected services and connecting from networks that block direct WireGuard. Both use WireGuard encryption.

This is the whole setup in one diagram. Every box gets its own section below.

flowchart LR home["At home
(home network)"] wg["Away
(WireGuard)"] ts["Away
(Tailscale)"] friends["Friends and family
(Tailscale)"] net["Anyone
(internet)"] subgraph router["Home router"] wgs["WireGuard server"] fwd["Port 443 forward"] end home --> traefik wg --> wgs --> traefik ts -- "encrypted tunnel
straight to the NAS" --> traefik net --> fwd -- "wan entrypoint:
public routes only" --> traefik traefik{{"Traefik
reverse proxy
+ IP allowlist"}} traefik --> nas["Containers on nas"] traefik --> nuc["Containers on nuc"] traefik --> hp["Containers on hp"] traefik --> pc["Containers on pc"] friends -. "ACL: only the ports
shared with them" .-> nas friends -.-> pc

In short:

Four machines run all my services as containers.
compose-farm decides which service runs on which machine.
One reverse proxy, Traefik, is the front door for every service on every machine.
Let’s Encrypt certificates give every service real HTTPS, including the private ones.
DNS turns names like mealie.lab.nijho.lt into addresses, and gives a different answer depending on where I am.
There are four ways in: my home network, WireGuard on my router, Tailscale coordinated by my own Headscale server, and the open internet.
One allowlist in Traefik limits every service to the first three by default. A service reaches the internet only if its config asks for it.
Headscale ACLs decide which of my friends and family can reach which service.
NixOS, Compose files, and Terraform describe nearly all of it in git, so I rarely click a button, and AI agents can work on it the same way I do.

If that list looks overwhelming, I get it. A reverse proxy, Let’s Encrypt, DNS, an allowlist, ACLs, NixOS, compose-farm, Terraform: that is a lot of moving parts for something that serves recipes. What lets me sleep at night is the last point (9). Apart from a handful of router settings, every piece of configuration is declarative and lives in git, so the whole setup is reproducible. If I break something, or more likely, an AI agent breaks something, git log tells me what changed and git revert undoes it. And the NAS takes a ZFS snapshot of all app data every 10 minutes, so even if something goes badly wrong, I lose at most ten minutes of data. You don’t need to understand every piece at once, either; each one is a file you can read when you get to it. I come back to this in Declarative everything.

To make the list concrete, this is what happens when I open Mealie in three situations:

At home, my phone asks my home DNS server for mealie.lab.nijho.lt and gets 192.168.1.6, the NAS. Traefik sees a request from 192.168.1.x, which is on the allowlist, and passes it to the Mealie container.
On hotel Wi-Fi that blocks WireGuard, my laptop uses Tailscale instead. It asks Headscale’s DNS for the same name and gets 100.64.0.28, the address of the NAS inside my Tailscale network. The request travels through an encrypted tunnel straight to the NAS. Traefik sees a 100.64.0.x address, also on the allowlist, and passes it on.
A stranger on the internet gets 192.168.1.6 from public DNS, which is a private address that leads nowhere outside my home. If they find my home IP and connect to it directly, the router hands them to Traefik’s separate door for the internet, where Mealie doesn’t exist, and they get 404 Not Found.

Same name, HTTPS on all three paths, and only the stranger is refused.

The machines, and NixOS #

Four machines run my services:

Machine	What it is	What it does
`nas`	The physical NAS with all the disks	Traefik and most services run in an Incus system container, close to the data
`nuc`	A small, always-on Intel NUC	Home DNS server and a few lightweight services
`hp`	An HP EliteDesk	Second home DNS server and a mix of services
`pc`	My desktop with two RTX 3090s	Local AI, including my dictation server

Each name links to that machine’s NixOS configuration. The hardware of the NUC, the HP, and the NAS is in my original homelab post.

That system container is called docker-lxc. It holds my Docker stack files and runs my Docker containers. In compose-farm, I call it nas, so deployment commands using that name target the container on the physical NAS.

Why the NAS runs its containers inside another container (click to unfold)

Incus uses LXC to run system containers. This keeps Docker and its services separate from the NAS host that manages my disks. That container is a full NixOS system of its own, configured like the other machines.

What NixOS brings #

All four physical machines run NixOS, and so does the docker-lxc container. In NixOS, the entire operating system is described in text files: which packages are installed, which services run, which disks get mounted, which firewall ports are open. You don’t install things by typing commands and hoping you remember them later. You write down what the machine should look like, and NixOS makes it so. If a machine dies, I install NixOS on a new one, point it at the same files, and get the same machine back.

My NixOS configuration lives in my public dotfiles, in configs/nixos. A shared layer applies to every machine, and each machine adds its own specifics. Every machine joins my Tailscale network with one line in common/services.nix:

services.tailscale.enable = true;

The NUC, the HP, and the PC mount the same shared folders from the NAS over NFS, a protocol for sharing folders over the network. That shared folder, /opt/stacks, is what makes the next section work: every machine sees the same service definitions at the same path.

The NFS mount in NixOS (click to unfold)

From optional/nfs-docker.nix:

fileSystems."/opt/stacks" = {
 device = "truenas.local:/mnt/ssd/docker/stacks";
 fsType = "nfs";
 options = [ "nfsvers=4" "nofail" "bg" "soft" ];
};

docker-lxc doesn’t need NFS; the NAS hands it the same folders directly from its disks (hosts/nas/virtualization.nix).

Deploying NixOS with comin #

Changing a NixOS machine normally means logging in and running nixos-rebuild switch. I don’t do that anymore. Every machine runs comin, a small GitOps agent. It watches my dotfiles on GitHub, and when a new commit lands on main, each machine pulls it, builds its own configuration, and switches to it. The heavy builds come from my local build cache, so the machines themselves rarely compile anything. comin only deploys commits signed with my SSH key, so being able to push to the repo is not enough to take over my machines. comin originally only understood GPG signatures, so I added support for SSH-signed commits upstream.

The comin config (click to unfold)

The core of common/comin.nix:

services.comin = {
 enable = true;
 remotes = [{
 name = "origin";
 url = "https://github.com/basnijholt/dotfiles.git";
 branches.main.name = "main";
 }];
 repositorySubdir = "configs/nixos";
 sshAllowedSignersPath = "${cominAllowedSigners}";
};

The last line points comin at the list of keys whose signatures it accepts.

Changing a machine means pushing a signed commit. That comes back at the end of this post.

What runs on it #

As I write this, the four machines run 138 containers. A small selection:

Area	Services
Infrastructure	Traefik (reverse proxy), Headscale and Headplane (my tailnet), Forgejo (git), the compose-farm web UI, Homepage (dashboard), code-server (VS Code in the browser)
Monitoring	Uptime Kuma, Glances, Netdata, Prometheus and Grafana, Dozzle (container logs), Diun and What’s Up Docker (image updates), LibreSpeed
Tools	ntfy (push notifications), Syncthing (file sync), Atuin (shell history sync), Wakapi (coding stats)
AI	agent-cli and Diction (dictation), Speaches (speech-to-text), Kokoro (text-to-speech), Ollama, LiteLLM (one API in front of all models), Open WebUI, LibreChat, LobeChat, Khoj, SearXNG (private metasearch), Unsloth (fine-tuning)
Home and documents	Mealie (recipes), Grocy (household inventory), Paperless-ngx with paperless-ai and paperless-gpt, Immich (photos), Nextcloud, Hoarder (bookmarks), Home Assistant
Chat	Cinny (Matrix client for MindRoom), The Lounge (IRC)

The GPU-heavy AI services, like dictation, text-to-speech, and fine-tuning, run on pc; the local AI post covers that side. Home Assistant runs as a virtual machine in Incus on the HP; everything else in the table is a container.

Containers and compose-farm #

What a container is #

Almost everything I run is a Docker container. A container is an application packaged with everything it needs, isolated from the rest of the system. Mealie’s developers publish a Mealie image, I run it, it doesn’t care what else is installed on the machine, and removing it leaves nothing behind.

Docker Compose describes containers in a YAML file: which image, which folders it can see, which ports it uses, which settings. This is a trimmed version of my Mealie definition:

services:
 mealie:
 image: ghcr.io/mealie-recipes/mealie
 container_name: mealie
 ports:
 - 9925:9000
 volumes:
 - /mnt/data/mealie:/app/data/
 environment:
 ALLOW_SIGNUP: false
 BASE_URL: https://mealie.${DOMAIN}
 restart: unless-stopped

A port is a number that tells apart the network services on one machine, so one address can serve many apps. Under ports, 9925:9000 means that port 9925 on the machine leads to port 9000 inside the container, where Mealie listens.

Every service (“stack”) gets its own folder with a compose.yaml in one git repo, mounted at /opt/stacks on every machine.

Upstream’s Compose file, my folders #

Back when I ran Proxmox, I was a big fan of the Proxmox VE Helper-Scripts, started by tteck (RIP) and now maintained by the community. Each one is a one-liner that creates an LXC container with an app installed inside, and they are how I got into self-hosting and set up almost everything at first.

The catch is that they are a community effort, and the way they install an app is usually not a way the app’s own developers support. Immich shows the problem well: its docs say it requires Docker with the Docker Compose plugin, while today’s helper script builds Immich from source, installs PostgreSQL and Redis, and compiles six image-processing libraries directly in the container. That is effectively a mirror of the official setup, maintained by someone else; the script even pins the Immich version and only bumps it after testing each release. Every upgrade depends on a second set of maintainers keeping up with upstream, and for my containers that made upgrades painful and stressful.

Nowadays almost every project publishes a Docker Compose file, so that’s what I use, nearly unmodified. After the first setup, the Compose file almost never changes; the image does. Most of my services use the latest tag, and upgrading is cf update <stack> (or --all), which pulls the newest image and recreates the container.

That is YOLOing it a bit. In the beginning I pinned every image to an exact digest, because I was afraid a new version would break something. Bumping all those pins became its own chore, so I switched almost everything to latest. I only still pin the pieces where a surprise would really hurt: Traefik, Headscale, Forgejo, and databases, which stay on a major version like postgres:16 because moving to a new major version of PostgreSQL needs a manual migration. Since the switch I haven’t had real problems, and the rare breakage took a five-word prompt to an AI agent to fix.

The one thing I do change is where the data goes. Many Compose files keep data in named Docker volumes, which Docker manages somewhere under /var/lib/docker/volumes. I stopped using them: too many times I deleted a volume by accident (docker compose down -v and docker volume prune make that easy), and I never had a good view of what was inside. Instead, every stack gets plain folders, which Docker calls bind mounts, in two fixed places:

/opt/stacks/<stack>/ holds the compose.yaml and its .env, in git.
/mnt/data/<stack>/ holds the app’s data.

Each /mnt/data/<stack> is its own ZFS dataset on the NAS, with frequent snapshots and backups. The Mealie example above shows the swap: upstream’s file uses a volume called mealie-data, and mine says /mnt/data/mealie:/app/data/.

Why I wrote compose-farm #

Docker Compose manages containers on one machine, and I have four. The usual answers are Kubernetes or Docker Swarm.

Kubernetes feels like overkill for a homelab, and many projects don’t support it: they publish a Compose file, not a Helm chart (Kubernetes’ package format), so I would end up writing and maintaining my own charts. That’s the helper-script problem again.

I also experimented with Docker Swarm, writing a NixOS configuration for a three-manager cluster. Docker Swarm would meet my needs almost perfectly, but it is effectively in maintenance mode: it still ships with Docker, but Docker no longer invests in it.

So I wrote compose-farm. It is deliberately thin: one YAML file says which stack runs on which machine, and the cf command SSHes to that machine and runs plain docker compose there.

compose_dir: /opt/stacks

hosts:
 nas: { address: 192.168.1.6 }
 nuc: { address: 192.168.1.2 }
 hp: { address: 192.168.1.3 }
 pc: { address: 192.168.1.5 }

stacks:
 traefik: nas
 headscale: nas
 forgejo: nas
 mealie: nas
 ntfy: nuc
 agent-cli: pc
 glances: all  # runs on every machine

traefik_file: /opt/stacks/traefik/dynamic.d/compose-farm.yml

Day to day, that looks like this:

cf up mealie # start Mealie on whichever machine the config says
cf logs -f ntfy # follow logs, wherever ntfy runs
cf ps # what is running where
cf apply # make reality match the config: start, move, and stop stacks

Moving a service to another machine is a one-line change. I edit mealie: nas to mealie: hp and run cf up mealie, and compose-farm stops it on the old machine and starts it on the new one. Because every machine sees the same /opt/stacks and /mnt/data, the service finds its configuration and data waiting for it.

That is also my answer to “what if a machine dies?” There is no automatic failover: nothing notices and reschedules the services for me, the way Kubernetes would. But because no stack is tied to a machine, recovering by hand takes a minute. The NAS is the exception: it holds the data and runs Traefik, so if it dies, nothing else can take over.

Moving a dead machine’s services by hand (click to unfold)

If the NUC died, I would point its name at the HP:

hosts:
 nas: { address: 192.168.1.6 }
 nuc: { address: 192.168.1.3 } # was 192.168.1.2; the NUC is dead, so the HP takes over
 hp: { address: 192.168.1.3 }
 pc: { address: 192.168.1.5 }

Then cf up --host nuc starts every NUC stack on the HP and updates Traefik’s routes, and I can clean up the names later. (Not cf apply: while two names point at the same machine, it sees every stack there running twice and stops the “extra” copy.) This works for the NUC, the HP, and the PC, as long as the published ports don’t clash and a service doesn’t need the PC’s GPUs.

There is no cluster, no database, and no agent running on each machine. If compose-farm disappeared tomorrow, every stack would still be a normal Compose folder I could start by hand.

The last line of that config, traefik_file, is where compose-farm connects to the front door.

The front door: a reverse proxy #

What a reverse proxy brings #

Every service listens on its own port on its own machine: Mealie on port 9925 of the NAS, ntfy on port 8089 of the NUC, and so on. I could bookmark http://192.168.1.2:8089, but that gets annoying fast, it is unencrypted, and every service would need its own security.

A reverse proxy is a receptionist in front of all of them. Every request arrives there first. It looks at which name you asked for, mealie.lab.nijho.lt or ntfy.lab.nijho.lt, and passes the request to the right service on the right machine. Because everything goes through it, the reverse proxy is also the one place to:

handle HTTPS and certificates, so the services don’t have to;
decide who is allowed in;
give every service a clean name instead of an IP and a port.

I use Traefik. It runs on the NAS and is the only thing listening on the standard web ports: 80 for plain HTTP and 443 for HTTPS.

Services announce themselves #

What I like most about Traefik is that services describe their own routing with labels in their Compose file. These are Mealie’s:

 labels:
 - traefik.enable=true
 # https://mealie.lab.nijho.lt, only for trusted networks
 - traefik.http.routers.mealie.rule=Host(`mealie.${DOMAIN}`)
 # http://mealie.local, plain HTTP at home
 - traefik.http.routers.mealie-local.rule=Host(`mealie.local`)
 - traefik.http.routers.mealie-local.entrypoints=web
 - traefik.http.services.mealie.loadbalancer.server.port=9000

In words: if someone asks for mealie.lab.nijho.lt or mealie.local, send them to port 9000 of this container. Every service gets two names by convention: an HTTPS name under lab.nijho.lt that works everywhere, and a short .local name for quick access at home. The HTTPS route doesn’t mention HTTPS, a certificate, or who is allowed in; Traefik’s defaults cover all three.

Traefik reads those labels by watching Docker on its own machine, and ignores any container that doesn’t opt in with traefik.enable=true. Its own config sets the defaults:

command:
 - --providers.docker=true
 - --providers.docker.exposedbydefault=false
 - --providers.file.directory=/dynamic.d
 - --entrypoints.web.address=:80
 - --entrypoints.web.http.middlewares=local-ips-only@file
 - --entrypoints.websecure.address=:443
 - --entrypoints.websecure.asDefault=true
 - --entrypoints.websecure.http.middlewares=local-ips-only@file
 - --entrypoints.websecure.http.tls.certresolver=le
 - --entrypoints.wan.address=:4443
 - --entrypoints.wan.http.tls.certresolver=le

An entrypoint is a port Traefik listens on. web and websecure are for home, WireGuard, and Tailscale, and a route that doesn’t name an entrypoint lands on websecure. local-ips-only is the allowlist, more on that in Keeping it safe. wan is for the internet, more on that in The internet.

Services on other machines #

There is a catch: Traefik only sees Docker containers on its own machine. It has no idea that ntfy runs on the NUC.

compose-farm fills that gap. It reads the Traefik labels of every stack, looks up which machine the stack runs on and which port it publishes, and writes the result to a file that Traefik watches. After cf up ntfy, that file contains:

# /opt/stacks/traefik/dynamic.d/compose-farm.yml
# Auto-generated by compose-farm (only the ntfy entries shown)
http:
 routers:
 ntfy:
 rule: Host(`ntfy.lab.nijho.lt`)
 service: ntfy
 services:
 ntfy:
 loadbalancer:
 servers:
 - url: http://192.168.1.2:8089

Note the ports: on its own machine, Traefik talks to a container directly on its internal port (9000 for Mealie), but a route to another machine has to use the port that machine publishes (8089 for ntfy). I never edit that file. I write labels the same way no matter where a service runs, and when a service moves, its route moves with it.

Things that don’t run in Docker, like Home Assistant in its own virtual machine, get hand-written routes in a second file in the same folder.

HTTPS and certificates #

What a certificate is #

When you visit an https:// address, your browser checks the server’s certificate: a small file, signed by an authority your browser trusts, that proves the server really is mealie.lab.nijho.lt. The certificate is also what makes the encryption between you and the server possible. Without a valid one you get a scary warning, and many apps refuse to connect at all.

A private service with a real certificate

Let’s Encrypt hands out certificates for free, automatically, to anyone who can prove they control a domain.

Proving I own the domain without opening my network #

The usual proof is for Let’s Encrypt to connect to your server and fetch a file from it (the “HTTP-01 challenge”). That doesn’t work for private services, because the whole point is that the internet can’t connect to them.

So I use the DNS-01 challenge instead. Let’s Encrypt asks me to publish a specific text record in my domain’s DNS, the internet’s phone book that turns names into addresses (more on that below). My DNS is hosted at Cloudflare, so Traefik creates that record through the Cloudflare API, waits for Let’s Encrypt to see it, and removes it again. Let’s Encrypt never has to reach my network.

DNS-01 also lets me request a wildcard certificate: one certificate for *.lab.nijho.lt that covers every name under it. A new service has HTTPS the moment its route exists, without requesting anything.

 - --certificatesresolvers.le.acme.dnsChallenge.provider=cloudflare

 - traefik.http.routers.traefik-lab.tls.certresolver=le
 - traefik.http.routers.traefik-lab.tls.domains[0].main=lab.nijho.lt
 - traefik.http.routers.traefik-lab.tls.domains[0].sans=*.lab.nijho.lt

Traefik renews it before it expires, and I don’t think about certificates anymore.

The .local names can’t get a public certificate, because nobody owns .local. Those stay plain HTTP, and Traefik only answers them for devices on my home network. It is also why I don’t redirect HTTP to HTTPS everywhere: that would break every .local name.

DNS: the phone book #

What DNS does #

Computers talk to each other with IP addresses like 192.168.1.6. Humans prefer names like mealie.lab.nijho.lt. DNS, the Domain Name System, is the phone book that turns one into the other. Before your browser connects anywhere, it asks a DNS server for the address that belongs to a name.

Two details matter for the rest of this post.

First, some addresses are private. Ranges like 192.168.x.x and 10.x.x.x, and Tailscale’s 100.64.x.x, are not reachable on the public internet; they only mean something inside one particular network. My NAS is 192.168.1.6 inside my home, and so is some random device in thousands of other homes.

Second, the answer depends on which DNS server you ask. My setup relies on that: the same name gets a different answer depending on where I am.

Three answers for the same name #

Who is asking	Which DNS server	Answer for `mealie.lab.nijho.lt`
Anyone on the internet	Public DNS at Cloudflare	`192.168.1.6`
A device at home or on WireGuard	My own DNS servers on `nuc` and `hp`	`192.168.1.6`
A device on my Tailscale network	Headscale’s built-in DNS	`100.64.0.28`

Public DNS has one wildcard record: *.lab.nijho.lt points to 192.168.1.6. Yes, that is a private address in public DNS. It sounds odd, but it works nicely: at home, or on a tunnel that routes my home network, the public answer is simply correct. For everyone else it leads nowhere, and it reveals nothing beyond the fact that I have a machine at 192.168.1.6, like half the internet. That is convenience, not security; more on that below. (Some routers with “DNS rebinding protection” refuse to pass on answers like this. Mine doesn’t.)

I don’t click those records together in Cloudflare’s dashboard. All my public DNS is Terraform: a text file that lists every record, and a tool that makes Cloudflare match it. These are the lab records:

resource "cloudflare_dns_record" "lab_wildcard" {
 zone_id = var.cloudflare_zone_id
 name = "*.lab.nijho.lt"
 content = "192.168.1.6"
 type = "A"
 ttl = 10800
 proxied = false # Private IP: Cloudflare can't proxy to it
}

terraform plan shows exactly which records will be added, changed, or removed before anything happens, and terraform apply makes the change. The Terraform files live in a private repo, next to my other secrets, because they need a Cloudflare API token.

My home DNS servers run CoreDNS on the NUC and the HP, declared in optional/coredns.nix in the same NixOS repo as everything else. They answer .local and lab.nijho.lt names themselves and forward everything else to public DNS:

labNijholtZone = pkgs.writeText "lab.nijho.lt.zone" ''
 $ORIGIN lab.nijho.lt.
 @ 3600 IN A 192.168.1.6
 * 3600 IN A 192.168.1.6
'';

Without them, .local names wouldn’t work at all, since public DNS knows nothing about them.

Strictly speaking, .local is reserved for multicast DNS, the protocol printers and AirPlay speakers use to announce themselves, and Apple warns that its devices might not resolve .local names served by a regular DNS server. I know, and I have rarely run into it. The official name for home networks is home.arpa, but some browsers treat a name like example.home.arpa as a search term: Chrome on my phone searched Google instead of opening the site. I had to type the full address, https://example.home.arpa, every time. The .local names are a shortcut anyway; the lab.nijho.lt names are the ones that work everywhere. They also keep lab names working when my internet connection is down. There are two so that one can reboot without taking name resolution with it.

Headscale’s DNS is the odd one out, and it needs Tailscale explained first.

Four ways in #

There are four ways a request can reach Traefik:

Way in	When I use it	Address Traefik sees
Home network	At home, on Wi-Fi or a cable	`192.168.1.x`
WireGuard on the router	My phone, laptop, and travel router when I’m away	`10.6.0.x`
Tailscale, via Headscale	Friends and family, cloud VMs, and my own devices when WireGuard can’t get through	`100.64.0.x`
The internet	Anyone, for the few public services	Any public address

1. The home network #

At home my devices are on 192.168.1.x, the home DNS servers answer, and everything just works.

2. WireGuard on the router #

A VPN, a virtual private network, creates an encrypted tunnel from your device to another network, so your device behaves as if it were plugged in there. WireGuard is a modern VPN protocol: small, fast, and based on key pairs instead of usernames and passwords.

My router, an ASUS ZenWiFi XT8, has a WireGuard server built in. When I’m away, my phone connects to it and gets an address in 10.6.0.x, and from then on it is effectively at home. It can reach 192.168.1.6, and it uses my home DNS servers. The WireGuard iOS app has “On-Demand” rules, so the tunnel turns on by itself whenever I leave my home Wi-Fi. I wrote more about that in my post about coding from my phone.

To find my router, the phone needs my home’s public IP address. My internet provider can change that address whenever it likes, so the router also runs dynamic DNS (DDNS): it keeps a public hostname pointed at whatever my current IP is. The WireGuard client connects to that hostname.

3. Tailscale, via my own Headscale #

Tailscale is a VPN built on WireGuard, but organized differently. Instead of every device tunneling to one central point, Tailscale builds a mesh: every device can make a direct, encrypted WireGuard connection to every other device. It gets through firewalls and home routers without opening any ports, and when a direct connection is impossible, it falls back to encrypted relay servers. Every device gets a stable address in 100.64.x.x, no matter which network it is on. All those devices together are called a tailnet.

A mesh like that needs something that knows which devices exist, what their keys are, and who may talk to whom. That is the coordination server, which Tailscale normally runs for you. Headscale is an open-source coordination server that you run yourself. The official Tailscale apps for phones, laptops, and Linux all work with it; you point them at your own server instead of Tailscale’s.

Headscale runs as one of my stacks on the NAS, and Headplane next to it gives it a web UI. The coordination server only introduces devices to each other. My actual traffic flows directly between devices and never passes through it. Its configuration, the list of who may reach what, and its DNS records are all files in my stacks repo.

Why both? #

Two VPNs look redundant, but they are good at different things.

WireGuard on the router puts a device on my home network. Adding a client is one entry in the router’s settings. That device can then reach everything at home, from the NAS to the printer to the router itself, at the same addresses and with the same DNS answers as on my home Wi-Fi. Nothing needs to be installed on the machines at home, and nothing depends on my homelab: if the NAS is down, or I broke something, WireGuard still gets me home so I can fix it. So my own devices use WireGuard: my iPhone, my laptop, and my travel router, which brings every device connected to it along.

Tailscale puts a device on a separate, virtual network. A device on the tailnet can only reach other devices that run Tailscale themselves. My home network isn’t part of it, which is why the DNS trick below is needed. In return, it works almost everywhere. WireGuard to my router only works if the network I’m on lets UDP traffic out to my router’s port, and some hotel, airport, and office networks don’t. Tailscale doesn’t need an open port, and its relays speak HTTPS, which nearly every network allows. It also connects machines that are all away from home, like my laptop and a cloud VM, directly, without a detour through my house.

Tailscale also has fine-grained access control. On my router, every WireGuard client gets my whole home network. On the tailnet, Headscale’s ACL decides per person and per device what they can reach. So everything that isn’t one of my own devices goes on the tailnet: friends, family, and cloud VMs.

The Tailscale DNS problem #

The public answer, 192.168.1.6, works on WireGuard because the tunnel routes my home network. It doesn’t work on Tailscale, because I don’t route my home network through the tailnet. Worse, 192.168.1.x is one of the most common home network ranges in the world, so on a friend’s Wi-Fi 192.168.1.6 might be their printer.

The fix is Headscale’s built-in DNS, MagicDNS. Every device on the tailnet asks Headscale’s DNS first, and I tell it to answer every lab name with the NAS’s tailnet address, 100.64.0.28:

dns:
 magic_dns: true
 base_domain: ts.nijho.lt
 extra_records:
 # BEGIN MANAGED HEADSCALE LAB DNS
 - name: mealie.lab.nijho.lt
 type: A
 value: 100.64.0.28
 - name: ntfy.lab.nijho.lt
 type: A
 value: 100.64.0.28
 # ... one entry per name, about a hundred of them
 # END MANAGED HEADSCALE LAB DNS

ntfy points at the NAS too, even though it runs on the NUC. These records don’t point at services; they point at the front door, and Traefik forwards the request to the right machine as usual.

I don’t maintain that list by hand. A small script collects every Host(...) rule from my Traefik labels and route files and rewrites the block between the BEGIN and END markers:

python3 scripts/sync-headscale-lab-dns.py --dry-run # preview
python3 scripts/sync-headscale-lab-dns.py # write
cf restart headscale # Headscale doesn't hot-reload

Forgetting this step is sneaky: the new service works perfectly at home and silently fails over Tailscale. That’s why it is written down in AGENTS.md, the instruction file my AI agents read, with the exact commands.

The result is that one name works everywhere I am. https://mealie.lab.nijho.lt reaches the same Mealie, with the same certificate, whether my phone is at home, on WireGuard, or on Tailscale; only the path underneath differs.

4. The internet #

The last way in is the open internet. My router forwards TCP port 443, HTTPS, to Traefik on the NAS, but to its own entrypoint, wan on port 4443, not to the websecure one that everything else uses. Only routes that list wan answer there.

A few services are public on purpose. The clearest example is my git server, Forgejo, at git.nijho.lt. A git server is only useful if the people and machines I work with can reach it, and not all of them are on my tailnet.

Its public name is a Terraform record that points at the router’s DDNS hostname, so it follows my home IP around automatically:

resource "cloudflare_dns_record" "git" {
 zone_id = var.cloudflare_zone_id
 name = "git.nijho.lt"
 content = "<my router's DDNS hostname>"
 type = "CNAME"
 ttl = 10800
 proxied = false # SSH uses port 222, CF only proxies 80/443
}

In Traefik, Forgejo simply has two HTTPS routes: the usual git.lab.nijho.lt, private like everything else, and git.nijho.lt, which also listens on wan.

 - traefik.http.routers.forgejo.rule=Host(`git.${DOMAIN}`)
 - traefik.http.routers.forgejo-public.rule=Host(`git.nijho.lt`)
 - traefik.http.routers.forgejo-public.entrypoints=websecure,wan

Git over SSH doesn’t go through Traefik; the router forwards a second port, 222, straight to Forgejo on the NAS. That is also why this DNS record is “DNS-only”: Cloudflare’s proxy only handles web traffic. Without the allowlist, Forgejo has to guard its own door: sign-ups are off, you have to log in to see anything, and SSH only accepts keys.

The Headscale coordination server is public too, at headscale.nijho.lt, and it has to be: a laptop on hotel Wi-Fi needs to reach it to log in and find the other devices before any tunnel exists. That record goes through Cloudflare’s proxy, which hides my home IP behind Cloudflare’s. Only the part the Tailscale apps talk to is public; the Headplane admin UI and Headscale’s lab name stay behind the allowlist. Headscale doesn’t run its own relay server either; the tailnet uses Tailscale’s public relays when a direct connection isn’t possible.

Forwarding port 443 sounds scary if you assume it means everything is on the internet now. It doesn’t: the internet only reaches wan, and a route has to ask for that.

Keeping it safe: one allowlist #

DNS is not a lock #

Remember the public DNS record from earlier: *.lab.nijho.lt points to 192.168.1.6, a private IP address on my home network. I made this mistake myself: I thought pointing DNS at a private address was enough to keep my services off the internet. It wasn’t. DNS only tells a client where to go, and nothing forces a client to ask. Anyone can skip DNS and send a request for mealie.lab.nijho.lt straight to my home IP:

curl --resolve mealie.lab.nijho.lt:443:<my home IP> https://mealie.lab.nijho.lt

A home IP is not a secret: any public service at home points to it, and scanners sweep every address on the internet anyway. That request reaches the router, gets forwarded to Traefik like any other HTTPS request, and asks for Mealie by name. The names aren’t secret either, because they’re easy to guess. The wildcard certificate keeps the individual names out of the public certificate transparency logs, which is nice, but that is hiding, not locking.

So the DNS setup is there for convenience: one name that works everywhere. The security boundary is Traefik, which knows how every request actually arrived.

Who is knocking #

Every network request carries a source address: where it came from. When the router forwards a request from the internet, the source stays the real public address of whoever sent it. Unlike a name, a source address can’t be faked for a real connection: the reply goes back to that address, so a forged one never completes the connection. So Traefik can tell which of the four ways in a request took, just by looking at where it came from.

I use that in a single Traefik middleware, a check that runs before a request reaches the service, called local-ips-only:

http:
 middlewares:
 local-ips-only:
 ipAllowList:
 sourceRange:
 - "192.168.1.0/24" # home network
 - "100.64.0.0/24" # Tailscale devices
 - "10.6.0.0/24" # WireGuard clients
 - "172.20.0.1/32" # the NAS itself (see below)
 - "127.0.0.1/32" # localhost

/24 means the first three numbers are fixed and the last can be anything, so 192.168.1.0/24 covers 192.168.1.0 through 192.168.1.255. /32 means exactly one address.

The middleware sits on the whole websecure entrypoint, so every HTTPS route there gets the check without a word in its labels. A request from any other address gets 403 Forbidden before it gets anywhere near the service. For the .local names, the same middleware sits on the plain-HTTP entrypoint. Internet traffic doesn’t arrive on websecure at all, so this is the second lock: if I ever point the port forward at the wrong port, strangers get 403 instead of my services.

The odd one, 172.20.0.1, is the gateway of my Docker network. Requests that start on the NAS itself, like a container calling another service by its lab name, show up with that address.

Keeping real Tailscale addresses (click to unfold)

Tailscale requests keep their real 100.64.0.x address thanks to one line in the NixOS config of docker-lxc:

services.tailscale.extraSetFlags = [ "--snat-subnet-routes=false" ];

Without it, Tailscale replaces the source address of traffic it passes on to another network, like Docker’s internal one, and every Tailscale request would look like it came from 172.20.0.1.

To check all of this, I test from every way in.

How I test it (click to unfold)

# From home, WireGuard, or Tailscale: expect 200
curl -s -o /dev/null -w '%{http_code}\n' https://mealie.lab.nijho.lt
# Through the router's port forward, skipping DNS: expect 404
curl -s -o /dev/null -w '%{http_code}\n' --resolve mealie.lab.nijho.lt:443:<my home IP> https://mealie.lab.nijho.lt
# A public service, from anywhere: expect 200
curl -s -o /dev/null -w '%{http_code}\n' https://git.nijho.lt

The second one works from inside too: the router sends the request back in through the same port forward, so it lands on wan like a stranger’s would.

Private by default #

Out of roughly a hundred HTTPS routes, only a handful are public, and each of those is a deliberate decision.

A route is public only if it lists the wan entrypoint and has no allowlist of its own. A private service needs nothing in its labels: it lands on websecure, where the allowlist guards the whole entrypoint. Forget a label and a service ends up private, not public. The one change that publishes something is typing wan, which stands out in a diff and is easy to list:

grep -rnw wan /opt/stacks/*/compose.yaml /opt/stacks/traefik/dynamic.d/

One wrinkle: devices at home that use a public name, like git.nijho.lt, also come in through the port forward, because DNS points that name at my home IP. For public routes that changes nothing. A couple of private services that I reach by a public name list wan too, and keep the allowlist on the route itself.

Trusting my own network #

For most services, the allowlist is the only lock in front. Anything on my home network or on WireGuard, and any of my own devices on Tailscale, can reach them without an extra login.

I could add one. Authelia runs next to Traefik, and putting its login page in front of a service takes one label. For Traefik’s own dashboard, that looks like this:

 - traefik.http.routers.traefik-lab.middlewares=authelia@docker

I do that for some services, but not all, because a login page in front of an app sometimes breaks things, like mobile apps that talk to the app directly.

Most apps have their own login, and I trust that enough on my own network. I also trust the people on it. If your home network has guests or devices you don’t trust, the trade-off looks different.

The allowlist trusts the whole 100.64.0.x range, and some devices on my tailnet belong to friends and family. That doesn’t give them everything, because there is a second layer in front: Headscale’s access control list (ACL). It decides, at the network level, which device may reach which port on which machine, and everything not listed is blocked.

A few examples of what that looks like in practice:

A friend uses my Mealie. Their router runs Tailscale and shares their home network with the tailnet, so every device in their home can reach Mealie without installing anything.
A friend uses my self-hosted dictation on the GPU machine, and nothing else.
An AI agent I run for a family member, in a MindRoom container, can reach one Home Assistant instance and nothing else.
A small NixOS box at a family member’s home joins the tailnet and makes a few shared services available on their network under simple .local names, for devices that can’t run Tailscale themselves. It is defined in the same NixOS repo as my own machines.

A simplified version of those rules:

{
 "hosts": {
 "nas": "100.64.0.28/32",
 "pc": "100.64.0.3/32",
 "friend-a-home": "10.10.0.0/16"
 },
 "acls": [
 { "action": "accept", "src": ["friend-a@", "friend-a-home"], "dst": ["nas:9925"] },
 { "action": "accept", "src": ["friend-b@"], "dst": ["pc:61337", "pc:8080"] },
 { "action": "accept", "src": ["me@"], "dst": ["*:*"] }
 ]
}

Friends reach their service directly on its port, not through Traefik. They can’t even reach port 443 on the NAS, so Traefik never sees them. My own devices can reach everything.

That also means friends don’t use my lab names. I give them the machine’s tailnet address and the port, like http://100.64.0.28:9925, and that’s it. A device keeps its tailnet address, so it can go in a bookmark, and one friend points a name in their own homelab’s DNS at it. It is plain HTTP, but between their network and mine it travels inside the encrypted tunnel.

Because the ACL is a file in my stacks repo, sharing one more service with someone is a one-line diff and a cf restart headscale. The commit history doubles as a log of who got access to what, and when.

This is also how friend-to-friend setups like my remote TrueNAS backups work: a tailnet is a private network you can extend to exactly the people you trust.

Why not Cloudflare Tunnel or Tailscale’s own server? #

This is usually the next question. Cloudflare Tunnel publishes services without forwarding any ports: a small daemon at home keeps an outbound connection open to Cloudflare, and Cloudflare serves your sites to the world. Tailscale runs the coordination server for you, for free, so you don’t need Headscale at all. Both are great products, and both are easier than what I do.

The reason I don’t use them is simple: I don’t want to depend on a third-party company for any of this. The plumbing in this post is open source and runs on my own machines, so if one of these companies disappears or changes its terms tomorrow, everything keeps working. With Cloudflare Tunnel, every request to my services would pass through Cloudflare, which decrypts it on the way. With Tailscale’s hosted server, a company would decide which devices are part of my network.

There are a few places where I still lean on someone else, and each one is replaceable:

Cloudflare hosts my public DNS. The records are a Terraform file, so moving to another provider means rewriting that file for a different provider. The same goes for the certificate challenge, which Traefik supports for dozens of DNS providers, and for the few names that go through Cloudflare’s proxy.
Tailscale’s public relays carry traffic when two devices can’t connect directly. Headscale has a relay server built in, which I can turn on in its config.
Let’s Encrypt issues my certificates, but it speaks a standard protocol, ACME, that other certificate authorities speak too.
Terraform moved to a source-available license in 2023. OpenTofu is the open-source fork, and it reads the same files.

Declarative everything #

One idea runs through all of this: every piece of configuration is a text file in git, not a setting I clicked somewhere. The opposite is running commands and clicking buttons until things look right, and then hoping you remember what you did.

Piece	Where it’s defined	How it’s applied
Operating system, Docker, Tailscale, NFS mounts, home DNS, firewalls	NixOS config in my public dotfiles	comin pulls signed commits and switches
Which service runs where	`compose-farm.yaml`	`cf apply`
Each service and its routing	`compose.yaml` with Traefik labels	`cf up`
Routes to services on other machines	Generated by compose-farm	Automatically on `cf up`
Non-Docker routes and the allowlist	A route file next to Traefik	Traefik watches the folder
Public DNS records	Terraform in a private repo	`terraform apply`
Tailnet DNS records	Generated from Traefik rules into Headscale’s config	`cf restart headscale`
Who may reach what on the tailnet	Headscale ACL file	`cf restart headscale`
Router: WireGuard, port forward, DDNS	The router’s web UI	❌ Clicked
Which devices are on the tailnet	Headscale’s database	Enrolled once per device; state, not configuration

The only configuration I still click is the router, and those settings change maybe once a year. Everything that changes weekly is a file. The next step in project “overengineered homelab” is a router running NixOS, but that’s for another time. What isn’t configuration, like which devices are enrolled in my tailnet and the data inside each app, lives on ZFS and is backed up like any other data.

It wasn’t always like this. My previous setup was built from good products that are all driven by clicking through a web UI:

Job	Before: clicked in a web UI	Now: a text file in git
Virtualization	Proxmox	NixOS and Incus
NAS	TrueNAS	NixOS with ZFS
DNS	Technitium	CoreDNS in my NixOS config
Reverse proxy	Nginx Proxy Manager	Traefik labels in each `compose.yaml`
Containers	Dockge, one machine at a time	compose-farm, all machines from one file

Yes, most of them have APIs, but that is not how most people use them, me included. Most of the configuration lived in each product’s database, and the reasons behind each setting lived in my head. Now both live in git.

Why it matters #

The classic benefits are real: git history is my changelog, rebuilding a dead machine is boring, and nothing lives only in my head.

But the benefit that matters most to me now is a newer one: this setup is easy to work on with AI.

Coding agents are very good at reading and editing text files, running commands, and checking the result. They are much worse at clicking through a web dashboard. Because almost everything here is files and CLI commands, an agent can do exactly what I do: add a service, move it to another machine, find out why a route returns 403, add a DNS record, write the NixOS change, and show me the diff. Terraform is a good example of why this is safe: the agent edits the record, runs terraform plan, and I see exactly what will change at Cloudflare before anything does.

I run my agents in YOLO mode, and declarative config is a big part of why that doesn’t scare me. Every change is a reviewable diff, and every mistake is a git revert away. It’s the same reason I prefer plain files over databases for AI memory.

The stacks repo has an AGENTS.md that captures the conventions and the sharp edges (a symlink to CLAUDE.md, so Claude Code and other agents read the same file):

> **⚠️ IMPORTANT: NEVER run `docker compose` directly!**
>
> Always use `cf` (compose-farm) for ALL compose operations.
> Running `docker compose` directly bypasses host routing,
> state tracking, and Traefik config generation.

The Tailscale DNS step from earlier lives there too, with the exact commands. Whenever something bites me once, it goes into that file, so it doesn’t bite an agent twice.

The other half is generating everything I would otherwise keep in sync by hand. The cross-machine Traefik routes come from the Compose labels, the Headscale DNS records come from the Traefik rules, and even my Uptime Kuma monitors are synced from the Traefik labels. Hand-maintained lists drift; generated ones don’t.

Putting it together: adding a service #

This is everything it takes to add a new service, say ntfy on the NUC:

Create /opt/stacks/ntfy/compose.yaml with the container and its Traefik labels: an HTTPS name and a .local name, like Mealie.
Add ntfy: nuc to compose-farm.yaml.
Run cf up ntfy. compose-farm starts it on the NUC and writes its route into Traefik’s file, and Traefik picks it up immediately.
Run the Headscale DNS script and cf restart headscale, so the name also works over Tailscale.
Commit.

After that:

https://ntfy.lab.nijho.lt works at home, over WireGuard, and over Tailscale, with a valid certificate I never requested.
http://ntfy.local works at home.
Sent to my home IP from the internet, it answers 404 Not Found.

To make it public like git.nijho.lt, I would add a Terraform record and a second route that also listens on wan. To share it with a friend, one line in the ACL.

In practice, I describe the service to an agent and review what it did.

Lessons learned #

Use the DNS-01 challenge. You get real certificates for private services and one wildcard for everything, and the internet never needs to reach you.
One front door, private by default. Knowing that a route has to ask to be on the internet is what made forwarding port 443 feel fine.
A private address in public DNS is a great trick for home and WireGuard. A mesh VPN like Tailscale needs its own answer.
DNS is not a security boundary. Anyone can send any name to your IP. The check has to happen at the front door.
Point DNS at the front door, not at the service. Traefik knows where things run, so moving a service never touches DNS.
Keep one way in that doesn’t depend on the homelab. For me that’s WireGuard on the router. The NAS holds the data and runs Traefik, so when it is down, everything is, and I still need a way in to fix it.
Test from every way in. Some problems only show up on one path, like a new name that works at home but not over Tailscale.
Two layers for sharing. Headscale ACLs decide who reaches what; Traefik’s wan entrypoint decides what the internet sees.
Make private the default. Public should take an extra word in the config, so a forgotten label fails closed.
Generate what you can. A generated list can’t drift.
Make everything a file. It’s better for future me, and it’s the difference between an AI agent that can help and one that can only give advice.

References #

compose-farm: my tool for running Docker Compose stacks across multiple machines
My NixOS configuration
comin: GitOps for NixOS machines
Traefik, including the IP allowlist middleware
Let’s Encrypt challenge types: HTTP-01 vs DNS-01
Terraform Cloudflare provider
WireGuard
How Tailscale works
Headscale and Headplane
CoreDNS
Forgejo

Yes, nixpkgs is the largest and most up-to-date package repository there is. Even so, I follow nixos-unstable, and a new version only reaches me once it is merged, built, and tested, and the channel moves forward, which usually takes a couple of days. Updates that trigger large rebuilds go through a staging branch first and take longer, and not every package gets updated as quickly as the popular ones. With Docker, I can run a release the day upstream publishes it. ↩︎

Removing the guardrails and letting my coding agents loose

Tue, 22 Sep 2026 00:00:00 +0000

I have run coding agents in YOLO mode since I started using them in May 2025. I think it is the only way to parallelize work. Sometimes I have ten agents running at the same time, and clicking “approve” on every command they run would make that impossible.

What makes me comfortable with it is that models tend to do what you ask, and that you can keep them in an environment where they cannot do much harm. I don’t keep secrets within their reach, and above all I have very good backups. The last piece is a small set of hooks that block the few things I never want an agent to do on its own.

Why hooks and not AGENTS.md #

My system prompt has always told agents not to force-push or merge PRs. Some of them did it anyway.

The first time I really noticed was with Gemini 3 Pro in November 2025, which merged my PR and force-pushed to main. It happened a couple more times in the weeks after, as I parallelized more and asked more of the models. Three weeks later I had hooks for Claude Code and Gemini CLI that block these commands before they run. Codex did not support hooks yet. To my surprise, it did not need them: OpenAI’s models are extremely good at following instructions, and Codex never ignored an explicit rule in my AGENTS.md. Codex supports hooks by now, so it runs the same ones as the others.

Claude has been worse at following rules, and Gemini far worse still: absolutely dogshit. I only try Gemini now and then, and I have been disappointed every time. By now I don’t even try the latest Google models anymore.

I don’t think agents are ever purposely malicious. They try to do what you asked, but they can misinterpret it, and then they force-push or merge something. A rule in a Markdown file is a hope; a hook actually stops the command. Not that a hook would stop a malicious agent: it could put the blocked command in a Bash script and run that instead. What the hook does is tell the agent why I don’t want it, because every block comes with a short explanation:

git add -A is not allowed - add files explicitly to avoid adding unrelated untracked files

The models are well aligned, so once they know my intention, they respect it.

What I block #

All hooks live in my dotfiles, and I deliberately kept them simple. The detection logic is one Python module, git_guard.py, and the Claude Code, Codex, and Gemini CLI hooks all import it.

git commit --amend and git push --force. The unit I review is a PR. When I review a PR, I know I have reviewed it up to a certain commit, and later I only look at the diff of the new commits. Rewriting history breaks that.

Pushing to main. Everything happens in PRs. Inside a PR I don’t care how messy the history gets. I want the agent to push very often, so no work gets lost, and every commit has to be green, so each one is a snapshot I can go back to. The agent maintains its own development of the feature, and I have not written a commit message myself in a long time. When the PR is done, I squash merge it.

gh pr merge. The merge comes after my review, so it is mine to do.

git add -A. I often have unrelated untracked or unstaged files lying around. I work in many open source repositories, and some of those files should not be public, like deployment plans. This is the only block without an override.

sleep. This one is for Claude, and Opus 5 specifically, which I hate with a passion. It would run tests in the background and then sleep 300, badly overestimating how long the tests take, when a blocking tool call returns exactly when they finish.

The override #

Over time the models got more capable. I could parallelize more, and I trusted them to work on their own for longer. I noticed I was spending a significant fraction of my time clicking buttons and doing operations I had forbidden the agent to do, only to end up doing them myself.

So in July I added an override. When a hook blocks a command, the agent gets this hint:

If (and only if) the user has explicitly approved this exact action,
re-run the command prefixed with EXPLICITLY_USER_APPROVED_HOOK_OVERRIDE=1 to override this block.

Nothing checks whether I actually approved it. It is a soft gate, but it works because the agent now has to write down, in the command itself, that I approved the action. When it runs into the block without my approval, it becomes obvious to the agent that it has to ask me first.

git add -A stays hard-blocked, and a chained command cannot carry an override past it: EXPLICITLY_USER_APPROVED_HOOK_OVERRIDE=1 git commit --amend && git add -A is still rejected.

When the override was used without approval #

It has happened, mostly with GPT-5.6 Sol, which I used a lot at the time.

It used to be that you had to keep your context window short, because auto-compaction was lossy and produced poor summaries. Since around GPT-5.3, compaction works well enough that I keep a session going for as long as I am working on the same feature. So I would ask the agent to merge a PR early on, start a follow-up PR in the same session, and after a compaction the agent concluded that it should merge the new PR too. It did, without my approval.

The other case was my pr-review skill, by far my favorite skill that I wrote myself. I would tell the agent to run it and squash merge the PR only if the review approved it. A few times, the review found problems, the agent fixed them, and then it merged. What I meant was: merge if it is approved right away, and otherwise wait for me. Now I spell that out and end with “otherwise, wait for my instructions.”

Working from my phone #

The override matters most when I am away from my computer. On a month-long trip visiting family in Europe, I worked only from my phone, using my mobile coding workflow. Typing commands in a terminal on a phone is inconvenient, so I would rather say what I want and let the agent do it, including the merge. I have also built end-to-end encrypted voice calls to my agents over Matrix with MindRoom, and I am moving more toward that kind of voice-driven workflow.

Not much has changed #

I recently reread my first post on agentic coding from August 2025, and very little in it is outdated. The models are more capable, and I give them much larger scopes, but the way I work is mostly the same. The hooks are the main thing I added, and the override is how I loosened them again.

Combining Keychain and 1Password CLI for ssh-agent management

Sun, 30 Mar 2025 00:00:00 +0000

Combining Keychain and 1Password CLI for SSH Agent Management #

For about a decade, I’ve relied on two key tools in my development workflow: Funtoo Keychain for managing my ssh-agent and 1Password for securely storing secrets. While I used them extensively, I never actually integrated them directly for SSH key passphrase handling.

Recently, with the advent of 1Password’s built-in SSH agent, I gave it a try, attracted by the promise of managing keys directly within my vault. However, I quickly found myself frequently prompted for my master password or biometrics, which interrupted my flow more often than I preferred compared to Keychain’s persistence.

This led me back to the reliable “unlock once per session” behavior of Keychain. But the experience using the 1Password Command-Line Interface (op) during that time sparked an idea: could I combine the persistence of Keychain with the secure passphrase retrieval of the op CLI? The answer is yes, and this post outlines the setup that achieves that balance.

The Goal: Use 1Password as the secure vault for the SSH key passphrase but rely on keychain to manage the ssh-agent process and ensure the key, once unlocked, remains available for the entire login session without further prompts from 1Password.

Why This Approach?

This setup directly addresses the trade-off I encountered. Funtoo keychain excels at starting ssh-agent reliably on login and keeping keys available throughout a session with just one initial unlock. While the native 1Password agent is convenient for vault integration, its session management, in my experience, led to more frequent authentication prompts than desired for long-running sessions.

By using keychain for agent management and calling the op CLI via the standard SSH_ASKPASS mechanism only for the initial key load, we get the best of both worlds: 1Password’s secure storage for the passphrase and Keychain’s persistent agent access with minimal prompting after the first unlock.

The Components:

Funtoo keychain: Manages ssh-agent lifecycle and loads specified keys.
1Password CLI (op): Securely retrieves secrets (the passphrase) from your 1Password vault. Requires op to be installed and logged in.
askpass-1password.sh: A small helper script that ssh-add (via keychain) calls to get the passphrase. This script invokes op.
setup_ssh_agent.sh: A script sourced by your shell startup file (.zshrc, .bashrc, etc.) to orchestrate the setup.

Implementation:

Step 1: Create the askpass Helper Script

This script simply calls op read to fetch the passphrase using its Secret Reference URI.

Find your SSH key item in 1Password.
Locate the passphrase field, click the options/arrow, and copy the “Copy Secret Reference”.
Create the script at ~/.ssh/askpass-1password.sh:

#!/bin/sh
# ~/.ssh/askpass-1password.sh
# Provides the SSH key passphrase from 1Password CLI via SSH_ASKPASS

# Paste your Secret Reference URI here
OP_SECRET_REFERENCE="op://YourVault/YourSSHKeyItem/password"

# Use 'op read' to retrieve and print the passphrase to stdout
# Requires 1Password CLI ('op') to be installed and authenticated.
op read "$OP_SECRET_REFERENCE"

Make it executable: chmod u+x ~/.ssh/askpass-1password.sh

Step 2: Create the Setup Orchestration Script

This script checks if op is available and conditionally configures keychain to use the helper script.

(Adjust path ~/dotfiles/scripts/setup_ssh_agent.sh as needed)

#!/bin/zsh
# ~/dotfiles/scripts/setup_ssh_agent.sh - Sets up ssh-agent via keychain.
# If 'op' CLI and the askpass helper script are available, uses 1Password for passphrase.
# Intended to be sourced by .zshrc or similar.

_askpass_helper="$HOME/.ssh/askpass-1password.sh"
_askpass_vars_set=false

# Check if 'op' CLI is installed and the helper script is usable
if command -v op >/dev/null 2>&1 && [[ -f "$_askpass_helper" ]] && [[ -x "$_askpass_helper" ]]; then
 # Configure environment for 1Password helper script
 export SSH_ASKPASS="$_askpass_helper"
 export SSH_ASKPASS_REQUIRE="prefer" # Use ASKPASS even in terminals
 _askpass_vars_set=true
fi

# Execute keychain to start/find agent, load key, and set env vars
# (Adjust key name 'id_ed25519' as needed)
eval $(keychain --eval --quiet --agents ssh --inherit any-once id_ed25519)

# Clean up ASKPASS variables if they were set
if $_askpass_vars_set; then
 unset SSH_ASKPASS
 unset SSH_ASKPASS_REQUIRE
fi

Step 3: Source the Setup Script in .zshrc

Add this to your ~/.zshrc (or equivalent):

# --- SSH Agent Configuration (via keychain & 1Password) ---
# Source the dedicated setup script if it exists.
_ssh_setup_script="${HOME}/dotfiles/scripts/setup_ssh_agent.sh"
if [[ -f "$_ssh_setup_script" ]]; then
 source "$_ssh_setup_script"
fi
# --- End SSH Agent Configuration ---

How It Works:

When you start a new shell session, .zshrc sources setup_ssh_agent.sh.
The script checks if op is installed and the askpass-1password.sh helper is ready.
If yes, it sets the SSH_ASKPASS and SSH_ASKPASS_REQUIRE environment variables.
It runs keychain --eval id_ed25519.
keychain starts or connects to ssh-agent and uses ssh-add to load id_ed25519.
ssh-add sees SSH_ASKPASS is set and executes askpass-1password.sh.
The helper script runs op read ....
The 1Password CLI handles authentication (prompting for your master password or using biometrics/system auth if necessary based on its current session state – typically only if the op session has expired).
op read outputs the passphrase.
ssh-add receives the passphrase and unlocks the key, adding it to the agent.
keychain --eval outputs the necessary export SSH_AUTH_SOCK=... commands.
eval executes these commands, configuring your current shell to use the agent.
The ASKPASS variables are unset.

Conclusion:

After years of using Funtoo Keychain and 1Password separately, and a brief foray into the native 1Password agent, this combined approach hits the sweet spot for me. It leverages 1Password’s secure vault for storing the SSH key passphrase but restores the “unlock once per session” convenience I appreciate from keychain. It requires a single passphrase entry via 1Password when the key is first added (typically on login/boot), after which the key remains available via the standard ssh-agent mechanism without further 1Password interaction for that session.

Security | Bas Nijholt

Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

What the kernel actually does #

A container is a process in a costume #

Why kernel bugs are a bigger problem now #

So I asked a model to break out #

Stronger walls: virtual machines #

Kata Containers and gVisor #

The PR: six lines #

What it costs #

What this does not fix #

Conclusion #

Friend-to-friend ZFS backups, version two

Why ZFS #

What we wanted #

ZFS already had most of it #

Rehearsing the permissions in a VM #

A gate that speaks syncoid #

Zones: a second wall in the kernel #

Who stops what #

What it cannot hide #

Setting it up #

Knowing when it stops working #

Restoring #

The first real push #

Tested like everything else #

AI scope creep or useful? #

References #

My declarative multi-machine homelab, explained

The big picture #

The machines, and NixOS #

What NixOS brings #

Deploying NixOS with comin #

What runs on it #

Containers and compose-farm #

What a container is #

Upstream’s Compose file, my folders #

Why I wrote compose-farm #

The front door: a reverse proxy #

What a reverse proxy brings #

Services announce themselves #

Services on other machines #

HTTPS and certificates #

What a certificate is #

Proving I own the domain without opening my network #

DNS: the phone book #

What DNS does #

Three answers for the same name #

Four ways in #

1. The home network #

2. WireGuard on the router #

3. Tailscale, via my own Headscale #

Why both? #

The Tailscale DNS problem #

4. The internet #

Keeping it safe: one allowlist #

DNS is not a lock #

Who is knocking #

Private by default #

Trusting my own network #

Sharing with friends and family #

Why not Cloudflare Tunnel or Tailscale’s own server? #

Declarative everything #

Why it matters #

Putting it together: adding a service #

Lessons learned #

References #

Removing the guardrails and letting my coding agents loose

Why hooks and not AGENTS.md #

What I block #

The override #

When the override was used without approval #

Working from my phone #

Not much has changed #

Combining Keychain and 1Password CLI for ssh-agent management

Combining Keychain and 1Password CLI for SSH Agent Management #